conan

Conan da Deployer

This project provides automation for a deployment pipeline (ala 'Continous Delivery') which provisions, configures and deploys applications (Java and Ruby) to "ship-clouds". Configuration is modeled through a DSL as a list of logical target environments, each with a number of ship-clouds. Endpoints for web applications and Stackato PaaS APIs are created based on conventions, including support for Blue/Green deployments.

Applications are deployed from a Sonotype Nexus artifact repository, and not from the source repository. Java applications must be executable jars, and must the provide deployment resources needed in the jar. Ruby apps (Rails or otherwise) must be vendored and packaged as a tar ball, with a compatible structrue to provide the deployment resources.

Application artifacts are idenitifed and provisioned by Maven coordinates - group id, artifact id, packaging and version. The most recent version available on the repository will be used, or a promotion workflow along the pipeline(s) can be implemented by pinning the versions. Artifacts are pinned by keeping the build meta-data in an environments directory structure that mirrors the configuration model, which should be commited to version control after successful deployment and testing. At this time, the environments directory and the environments.rb are located from the execution directory, which is assumed to be the nexus-apps-manifest project.

Conan is also designed to be idempotent. It can be executed against a target deployment over and over, and it should not change an application unless it out of alignment with the configuration model. Applications are also assumed to support standard set of managments APIs which provide build metadata, and health to support this.

Configuration

Configuration for the deployment pipline is loaded through a DSL which models a target environment(s). The DSL is loaded from a environments.rb file that should be provided in the execution directory.

  pipeline(:'team1') {
    app(:foo, 'com.mtnsat:foo', :jvm)
    app(:bar,   'com.mtnsat:bar', :rails_zip)
  }
  pipeline(:'team2')
    app(:baz,  'com.mtnsat:baz-web', :rails_zip)
    app(:bang,  'com.mtnsat:bang', :rails_zip)
  }

  environment(:integ) {
    org('mtn') {
      deploy('aws') {
        apps :foo
      }
      deploy('sea1') {
        apps :foo, :bar
        facility '999'
      }
      deploy('mia1') {
        apps :foo, :bar, :baz
        facility '888'
      }
    }
  }  

  environment(:prod) {
    promotes_from :integ
    org('mtn') {
      deploy('aws') {
        apps :foo
      }
    }
    org('acme') {
      deploy('minnow1') {
        enabled false
        apps :foo, :bar, :baz
        facility '42'
      }
      deploy('minnow1') {
        apps :bang
        option 'mapping', 'acme-stuff.com'
        option 'do-dad', 'flim-flam'
      }
      deploy('bounty1') {
        apps :foo, :bar, :baz
        facility '100'
      }
      deploy('bounty1') {
        apps :bang
        option 'mapping', 'acme-stuff.com'
        option 'do-dad', 'woo-woo'
      }
    }
  }

pipeline
any number of application pipelines. Takes a symbol for the pipeline, and defines the apps that are delivered by that pipeline

app
any number of applications. Takes a symbol for the app, a maven group_id:artifact_id and the type of platform the app uses (:jvm or :rails_zip)

environment
any number of environments. takes a symbol for the environment, and any number of deploys. NOTE: the order is not significant.

promotes_from
a declaration to control what environment the provisioning uses to check for new versions to promote. The absence of this declartation means that the artifact repository will be checked for the latest version

org
scopes the deployment to the organization served by those deployments.

deploy
describes a deployment to the given ship-cloud. It takes a key for the ship, and some attributes

endabled
deployments will be avoided if not 'true', default value is 'true' if absent

apps
a list of symbols for applications (defined in pipeline/app above) which should be deployed to the environment

facility
the facitility id

option
a way to define a key/value pair for deploy/app specific configurations (can be used multiple times to define N number key/value pairs)

Usage

$ conan [options] pipeline environment"

   pipeline - name of the pipeline to use
   environment - name of a the target environment: integ,stage,prod

   Options:
    -d, --output-dir [DIR]           Generate manifest data to this directory
    -f, --output-format [FORMAT]     The format used to write output. Either "properties" for Jenkins build properties
                                      or "stackato" (default) to provision binaries and write stackato yml files
                                      from project templates.
    -a, --action [ACTION]            The action(s) to perform. Either "provision" (default), "configure","deploy" or "all".
                                      Support for blue/green deployments use actions: "provision", "bg_configure", "bg_deploy",
                                      "bg_switch", "bg_clean" or "bg_all".
    -u, --paas-user [USER]           The stackato user name for deployments
    -p, --paas-password [PASSWORD]   The stackato password name for deployments
    -F, --force-deploy               Force deployment to bypass version checks.
    -N, --new-relic-api-key          New Relic API key for deployment alerts
    -s [ORG-SHIP],                   Only include deployments for the given shipcloud (org-ship)
        --deploy-shipcloud
    -D, --dry-run                    Deployment dry-run. Prints Stackato commands
    -V, --verbose                    Verbose console output
    -v, --version                    Show the software version
    -h, --help                       Display this screen


There are three phases... provision, configure, deploy, The current pipeline only preforms the provision phase.

  1. provision: Locate the binaries to deploy, download them from the artifact repository, and persist version information used to pin artifact versions and provides a workflow/promotion process. Version information (now really just the artifact meta-data from the artifact repository, instead of a properties file) is persisted in the environments directory tree. Artifacts that are downloaded are staged for configuration and deployment in tmp/apps. Artifacts which are tar-balls are extracted in the staging directory. Projects should provide configuration templates in a deploy-templates directory (assuming that get's approval from the Rails crowd). For jar artifacts, it attempts to extract configuration templates from the jar, at META-INF/deploy/templates and puts them in the staging directory at the same deploy-templates location.

  2. configure: Generate deployment-time configuration files from the model. It uses any ERB templates found at deploy-templates in the staging directory. The templates generate files in the staging root directory, with the same name as the template, but the extension is removed. e.g. stackato.yml.erb -> stackato.yml

  3. deploy: Run Stackato deployment of the application staging directory. It will avoid doing the deployment unless no service is found at the target hostname, the version check shows an older version running at the target hostname, or the -F flag forces deployment.

TBD: B/G actions

Templates

Templates for deloyment artifacts are .erb files (Erubis user guide: http://www.kuwata-lab.com/erubis/users-guide.html).

The templates are given a Context when it is evaluated. The Context provides the template with a set of key/values. The expected set of these is still maturing but here is an example of what is currently provided for a seanetsevice deployment, which has options defined:

app.artifact_id = "seanetservice"
app.extension = 
app.group_id = "com.mtnsat"
app.id = foo
app.platform_type = rails_zip
app.sha1 = 
app.version = 
deploy.app_ids = [:foo]
deploy.apps = []
deploy.enabled? = true
deploy.environment.deployments = [mtn-central, mtn-minnow1, mtn-minnow1, mtn-minnow1]
deploy.environment.id = integ
deploy.environment.upstream_env = 
deploy.facility_id = "10007"
deploy.shipcloud = "mtn-minnow1"
deploy.options = {"mapping"=>["integ.example.com"], "network"=>["guest=1.1.1.10", "crew=1.1.1.91", "cloud=1.1.1.91"]}
deploy.org = "mtn"
deploy.ship = "minnow1"
deploy_name = "foo-dev-minnow1"

App tokens for oauth are a special case, and assigned to special keys that will look something like:

oauth_id = application-gs8f8ae6-64a9-c987-c470-9c18b6fb918d
oauth_secret = "4z3r6e779455d89db8896b3f4d8c2369ae71e9ca662abb366eea905a41de1dcf"

The are also a number of shortcuts provided for shorter keys:

app_id => app.id
env => deploy.environment.id
org => deploy.org
ship => deploy.ship
shipcloud => deploy.shipcloud
infra => deploy.shipcloud
facility => deploy.facility_id
options => deploy.options

Warning: infra will be removed at some point

Here is an example Stackato template:

name: <%= @deploy_name %>
instances: 1
mem: 1G
framework:
  name: standalone
  runtime: java7
command: "java -Xms768m -Xmx768m -Dlog.logstash.source=\"${name}\" -Dlogback.configurationFile=logback-logstash.xml -jar poseidon.jar <%= @env %>"
url:
  - <%= @env %>.poseidon.<%= @ship %>.<%= @org %>.mtnsatcloud.com
  - <%= @env %>.poseidon.mtnsatcloud.com
min_version:
  client: 1.4.3
services:
  ${name}-db: mysql

Test

$ bundle install $ bundle exec rspec

To continously run specs when files are saved (or you hit return at the prompt):

$ bundle exec guard