Class: AppEngine::Exec

Inherits:
Object
  • Object
show all
Defined in:
lib/appengine/exec.rb

Overview

App Engine remote execution

This class provides a client for App Engine remote execution, allowing App Engine applications to perform on-demand tasks in the App Engine environment. This may be used for safe running of ops and maintenance tasks, such as database migrations, that access production cloud resources.

About App Engine execution

App Engine execution spins up a one-off copy of an App Engine app, and runs a command against it. For example, if your app runs on Ruby on Rails, then you might use App Engine execution to run a command such as bundle exec bin/rails db:migrate in production infrastructure (to avoid having to connect directly to your production database from a local workstation).

App Engine execution provides two strategies for generating that "one-off copy":

  • A deployment strategy, which deploys a temporary version of your app to a single backend instance and runs the command there.
  • A cloud_build strategy, which deploys your application image to Google Cloud Build and runs the command there.

Both strategies are generally designed to emulate the App Engine runtime environment on cloud VMs similar to those used by actual deployments of your app. Both provide your application code and environment variables, and both provide access to Cloud SQL connections used by your app. However, they differ in what version of your app code they run against, and in certain other constraints and performance characteristics. More detailed information on using the two strategies is provided in the sections below.

Apps deployed to the App Engine flexible environment will use the cloud_build strategy by default. However, you can force an app to use the deployment strategy instead. (You might do so if you need to connect to a Cloud SQL database on a VPC using a private IP, because the cloud_build strategy does not support private IPs.) To force use of deployment, set the strategy parameter in the Exec constructor (or the corresponding GAE_EXEC_STRATEGY parameter in the Rake task). Note that the deployment strategy is usually significantly slower than cloud_build for apps in the flexible environment.

Apps deployed to the App Engine standard environment will always use the deployment strategy. You cannot force use of the cloud_build strategy.

Prerequisites

To use App Engine remote execution, you will need:

  • An app deployed to Google App Engine, of course!
  • The gcloud SDK installed and configured.
  • The appengine gem.

You may use the AppEngine::Exec class to run commands directly. However, in most cases, it will be easier to run commands via the provided rake tasks (see Tasks).

Using the "deployment" strategy

The deployment strategy deploys a temporary version of your app to a single backend App Engine instance, runs the command there, and then deletes the temporary version when it is finished.

This is the default strategy (and indeed the only option) for apps running on the App Engine standard environment. It can also be used for flexible environment apps, but this is not commonly done because deployment of flexible environment apps can take a long time.

Because the deployment strategy deploys a temporary version of your app, it runs against the current application code present where the command was initiated (i.e. the code currently on your workstation if you run the rake task from your workstation, or the current code on the branch if you are running from a CI/CD system.) This may be different from the code actually running in production, so it is important that you run from a compatible code branch.

Specifying the host application

The deployment strategy works by deploying a temporary version of your app, so that it has access to your app's project and settings in App Engine. In most cases, it can determine this automatically, but depending on how your app or environment is structured, you may need to give it some help.

By default, your Google Cloud project is taken from the current gcloud project. If you need to override this, set the :project parameter in the Exec constructor (or the corresponding GAE_PROJECT parameter in the Rake task).

By default, the service name is taken from the App Engine config file. App Engine execution will assume this file is called app.yaml in the current directory. To use a different config file, set the config_path parameter in the Exec constructor (or the corresponding GAE_CONFIG parameter in the Rake task). You may also set the service name directly, using the service parameter (or GAE_SERVICE in Rake).

Providing credentials

Your command will effectively be a deployment of your App Engine app itself, and will have access to the same credentials. For example, App Engine provides a service account by default for your app, or your app may be making use of its own service account key. In either case, make sure the service account has sufficient access for the command you want to run (such as database admin credentials).

Other options

You may also provide a timeout, which is the length of time that App Engine execution will allow your command to run before it is considered to have stalled and is terminated. The timeout should be a string of the form 2h15m10s. The default is 10m.

The timeout is set via the timeout parameter to the Exec constructor, or by setting the GAE_TIMEOUT environment variable when invoking using Rake.

Resource usage and billing

The deployment strategy deploys to a temporary instance of your app in order to run the command. You may be billed for that usage. However, the cost should be minimal, because it will then immediately delete that instance in order to minimize usage.

If you interrupt the execution (or it crashes), it is possible that the temporary instance may not get deleted properly. If you suspect this may have happened, go to the App Engine tab in the cloud console, under "versions" of your service, and delete the temporary version manually. It will have a name matching the pattern appengine-exec-<timestamp>.

Using the "cloud_build" strategy

The cloud_build strategy takes the application image that App Engine is actually using to run your app, and uses it to spin up a copy of your app in Google Cloud Build (along with an emulation layer that emulates certain App Engine services such as Cloud SQL connection sockets). The command then gets run in the Cloud Build environment.

This is the default strategy for apps running on the App Engine flexible environment. (It is not available for standard environment apps.) Note that the cloud_build strategy cannot be used if your command needs to connect to a database over a VPC private IP address. This is because it runs on virtual machines provided by the Cloud Build service, which are not part of your VPC. If your database can be accessed only over a private IP, you should use the deployment strategy instead.

Specifying the host application

The cloud_build strategy needs to know exactly which app, service, and version of your app, to identify the application image to use.

By default, your Google Cloud project is taken from the current gcloud project. If you need to override this, set the :project parameter in the Exec constructor (or the corresponding GAE_PROJECT parameter in the Rake task).

By default, the service name is taken from the App Engine config file. App Engine execution will assume this file is called app.yaml in the current directory. To use a different config file, set the config_path parameter in the Exec constructor (or the corresponding GAE_CONFIG parameter in the Rake task). You may also set the service name directly, using the service parameter (or GAE_SERVICE in Rake).

By default, the image of the most recently deployed version of your app is used. (Note that this most recently deployed version may not be the same version that is currently receiving traffic: for example, if you deployed with --no-promote.) To use a different version, set the version parameter in the Exec constructor (or the corresponding GAE_VERSION parameter in the Rake task).

Providing credentials

By default, the cloud_build strategy uses your project's Cloud Build service account for its credentials. Unless your command provides its own service account key, you may need to grant the Cloud Build service account any permissions needed to execute your command (such as access to your database). For most tasks, it is sufficient to grant Project Editor permissions to the service account. You can find the service account configuration in the IAM tab in the Cloud Console under the name [your-project-number]@cloudbuild.gserviceaccount.com.

Other options

You may also provide a timeout, which is the length of time that App Engine execution will allow your command to run before it is considered to have stalled and is terminated. The timeout should be a string of the form 2h15m10s. The default is 10m.

The timeout is set via the timeout parameter to the Exec constructor, or by setting the GAE_TIMEOUT environment variable when invoking using Rake.

You can also set the wrapper image used to emulate the App Engine runtime environment, by setting the wrapper_image parameter to the constructor, or by setting the GAE_EXEC_WRAPPER_IMAGE environment variable. Generally, you will not need to do this unless you are testing a new wrapper image.

Resource usage and billing

The cloud_build strategy uses virtual machine resources provided by Google Cloud Build. Generally, a certain number of usage minutes per day is covered under a free tier, but additional compute usage beyond that time is billed to your Google Cloud account. For more details, see https://cloud.google.com/cloud-build/pricing

If your command makes API calls or utilizes other cloud resources, you may also be billed for that usage. However, the cloud_build strategy (unlike the deployment strategy) does not use actual App Engine instances, and you will not be billed for additional App Engine instance usage.

Defined Under Namespace

Classes: BadConfigFileFormat, BadParameter, ConfigFileNotFound, NoDefaultProject, NoSuchVersion, UnsupportedStrategy, UsageError

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(command, project: nil, service: nil, config_path: nil, version: nil, timeout: nil, wrapper_image: nil, strategy: nil) {|_self| ... } ⇒ Exec

Create an execution for the given command.

Parameters:

  • command (Array<String>) —

    The command in array form.

  • project (String, nil) (defaults to: nil) —

    ID of the project. If omitted, obtains the project from gcloud.

  • service (String, nil) (defaults to: nil) —

    Name of the service. If omitted, obtains the service name from the config file.

  • config_path (String, nil) (defaults to: nil) —

    App Engine config file to get the service name from if the service name is not provided directly. If omitted, defaults to the value returned by default_config_path.

  • version (String, nil) (defaults to: nil) —

    Version string. If omitted, defaults to the most recently created version of the given service (which may not be the one currently receiving traffic).

  • timeout (String, nil) (defaults to: nil) —

    Timeout string. If omitted, defaults to the value returned by default_timeout.

  • wrapper_image (String, nil) (defaults to: nil) —

    The fully qualified name of the wrapper image to use. (Applies only to the "cloud_build" strategy.)

  • strategy (String, nil) (defaults to: nil) —

    The execution strategy to use, or nil to choose a default based on the App Engine environment (flexible or standard). Allowed values are nil, "deployment" (which is the default for Standard), and "cloud_build" (which is the default for Flexible).

Yields:

  • (_self)

Yield Parameters:



414
415
416
417
418
419
420
421
422
423
424
425
426
427
 
# File 'lib/appengine/exec.rb', line 414

def initialize command,
               project: nil, service: nil, config_path: nil, version: nil,
               timeout: nil, wrapper_image: nil, strategy: nil
  @command = command
  @service = service
  @config_path = config_path
  @version = version
  @timeout = timeout
  @project = project
  @wrapper_image = wrapper_image
  @strategy = strategy

  yield self if block_given?
end

Class Attribute Details

.default_config_path ⇒ String

Returns Path to default config file.

Returns:

  • (String) —

    Path to default config file.



338
339
340
 
# File 'lib/appengine/exec.rb', line 338

def default_config_path
  @default_config_path
end

.default_service ⇒ String

Returns Default service name if the config doesn't specify.

Returns:

  • (String) —

    Default service name if the config doesn't specify.



335
336
337
 
# File 'lib/appengine/exec.rb', line 335

def default_service
  @default_service
end

.default_timeout ⇒ String

Returns Default command timeout.

Returns:

  • (String) —

    Default command timeout.



332
333
334
 
# File 'lib/appengine/exec.rb', line 332

def default_timeout
  @default_timeout
end

.default_wrapper_image ⇒ String

Returns Docker image that implements the app engine wrapper.

Returns:

  • (String) —

    Docker image that implements the app engine wrapper.



341
342
343
 
# File 'lib/appengine/exec.rb', line 341

def default_wrapper_image
  @default_wrapper_image
end

Instance Attribute Details

#command ⇒ String+

The command to run.

Returns:

  • (String) —

    if the command is a script to be run in a shell.

  • (Array<String>) —

    if the command is a posix command to be run directly without a shell.



466
467
468
 
# File 'lib/appengine/exec.rb', line 466

def command
  @command
end

#config_path ⇒ String?

Returns:

  • (String) —

    Path to the config file.

  • (nil) —

    if the default of ./app.yaml should be used.



445
446
447
 
# File 'lib/appengine/exec.rb', line 445

def config_path
  @config_path
end

#project ⇒ String?

Returns:

  • (String) —

    The project ID.

  • (nil) —

    if the default gcloud project should be used.



433
434
435
 
# File 'lib/appengine/exec.rb', line 433

def project
  @project
end

#service ⇒ String?

Returns:

  • (String) —

    The service name.

  • (nil) —

    if the service should be obtained from the app config.



439
440
441
 
# File 'lib/appengine/exec.rb', line 439

def service
  @service
end

#strategy ⇒ String?

Returns:

  • (String) —

    The execution strategy to use. Allowed values are "deployment" and "cloud_build".

  • (nil) —

    to choose a default based on the App Engine environment (flexible or standard).



480
481
482
 
# File 'lib/appengine/exec.rb', line 480

def strategy
  @strategy
end

#timeout ⇒ String?

Returns:

  • (String) —

    The command timeout, in 1h23m45s format.

  • (nil) —

    if the default of 10m should be used.



457
458
459
 
# File 'lib/appengine/exec.rb', line 457

def timeout
  @timeout
end

#version ⇒ String?

Returns:

  • (String) —

    Service version of the image to use.

  • (nil) —

    if the most recent should be used.



451
452
453
 
# File 'lib/appengine/exec.rb', line 451

def version
  @version
end

#wrapper_image ⇒ String?

Returns:

  • (String) —

    Custom wrapper image to use.

  • (nil) —

    if the default should be used.



472
473
474
 
# File 'lib/appengine/exec.rb', line 472

def wrapper_image
  @wrapper_image
end

Class Method Details

.new_rake_task(name, args: [], env_args: [], service: nil, config_path: nil, version: nil, timeout: nil, project: nil, wrapper_image: nil, strategy: nil) ⇒ Object

Create an execution for a rake task.

Parameters:

  • name (String) —

    Name of the task

  • args (Array<String>) (defaults to: []) —

    Args to pass to the task

  • env_args (Array<String>) (defaults to: []) —

    Environment variable settings, each of the form NAME=value.

  • service (String, nil) (defaults to: nil) —

    Name of the service. If omitted, obtains the service name from the config file.

  • config_path (String, nil) (defaults to: nil) —

    App Engine config file to get the service name from if the service name is not provided directly. If omitted, defaults to the value returned by default_config_path.

  • version (String, nil) (defaults to: nil) —

    Version string. If omitted, defaults to the most recently created version of the given service (which may not be the one currently receiving traffic).

  • timeout (String, nil) (defaults to: nil) —

    Timeout string. If omitted, defaults to the value returned by default_timeout.

  • wrapper_image (String, nil) (defaults to: nil) —

    The fully qualified name of the wrapper image to use. (Applies only to the "cloud_build" strategy.)

  • strategy (String, nil) (defaults to: nil) —

    The execution strategy to use, or nil to choose a default based on the App Engine environment (flexible or standard). Allowed values are nil, "deployment" (which is the default for Standard), and "cloud_build" (which is the default for Flexible).



369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
 
# File 'lib/appengine/exec.rb', line 369

def new_rake_task name, args: [], env_args: [],
                  service: nil, config_path: nil, version: nil,
                  timeout: nil, project: nil, wrapper_image: nil,
                  strategy: nil
  escaped_args = args.map do |arg|
    arg.gsub(/[,\[\]]/) { |m| "\\#{m}" }
  end
  name_with_args =
    if escaped_args.empty?
      name
    else
      "#{name}[#{escaped_args.join ','}]"
    end
  new ["bundle", "exec", "rake", name_with_args] + env_args,
      service: service, config_path: config_path, version: version,
      timeout: timeout, project: project, wrapper_image: wrapper_image,
      strategy: strategy
end

Instance Method Details

#start ⇒ Object

Executes the command synchronously. Streams the logs back to standard out and does not return until the command has completed or timed out.



486
487
488
489
490
491
492
493
494
495
 
# File 'lib/appengine/exec.rb', line 486

def start
  resolve_parameters
  app_info = version_info @service, @version
  resolve_strategy app_info["env"]
  if @strategy == "cloud_build"
    start_build_strategy app_info
  else
    start_deployment_strategy app_info
  end
end