concourse (the rubygem)

The concourse gem provides rake tasks to help you manage Concourse CI pipelines, jobs, and workers, to assist in running tasks with fly execute, and even run a local ephemeral deployment of Concourse on your development machine.

If you're not familiar with Concourse CI, you can read up on it at https://concourse-ci.org

Here's an example pipeline maintained by this gem:

Contents

• Basic Usage
  • Install
  • Add to your Rakefile
  • Set up your Concourse pipeline
• Real-world Examples
• Concourse pipeline configuration
  • ERB Templating
  • RUBIES
  • Secrets
  • Multiple pipelines
• Configuration
  • directory: Concourse subdirectory name
  • fly_target: Concourse fly target name
  • format: Emit the final pipelines in fly format-pipeline canonical format
  • pipeline_erb_filename: Pipeline filename
  • secrets_filename: Secrets filename
  • fly_args_: fly command arguments
• Rake Tasks
  • Manage your Concourse pipeline
  • List pipeline tasks
  • Run a Concourse task with fly execute
  • Abort running builds
  • Prune stalled concourse workers
• Running a local ephemeral Concourse
  • Spinning up a local deployment
  • Doing things with the local deployment
  • Shutting down the local deployment
• Contributing
• License

Basic Usage

Install

Add this line to your application's Gemfile:

gem 'concourse'

and then run bundle.

You can also install it via gem install concourse.

Add to your Rakefile

The primary functionality of the Concourse gem is to provide rake tasks. So, in your Rakefile:

require 'concourse'

Concourse.new("myproject").create_tasks!

Set up your Concourse pipeline

rake concourse:init

The concourse:init task will do a few different things for you:

1. create a subdirectory named concourse (or whatever you've configured with the :directory parameter)
2. create an empty Concourse pipeline file in that subdirectory named <myproject>.yml (or whatever you've configured with the :pipeline_erb_filename parameter)
3. ensure git will ignore your secrets file named private.yml (or whatever you've configured with the :secrets_filename parameter)

Real-world Examples

It might be helpful to look at how other projects are using this gem. Here are a sample, ordered (approximately) from simplest to most complex:

• jbarnette/hoe-debugging has one pipeline that test multiple versions of MRI. (see it live)
• flavorjones/hoe-gemspec has one pipeline that test multiple versions of MRI and JRuby. (see it live)
• flavorjones/calendar-assistant has one pipeline that runs tests against master and pull-requests, uses secrets, and integrates with Slack. (see it live)
• flavorjones/mini_portile has one pipeline that run tests on both Linux and Windows. (see it live)
• flavorjones/chromedriver-helper has two pipelines that test master and pull requests. (see Rakefile config) (see master and PR pipelines live)
• sparklemotion/nokogiri has multiple pipelines that use require and erbify_file to share common elements, and integrate with IRC. (see master and PR pipelines live)

Concourse pipeline configuration

ERB Templating

Your Concourse pipeline configuration file, <myproject>.yml (or whatever you've configured with the :pipeline_erb_filename parameter), will be treated like an ERB template.

The concourse directory is added the $LOAD_PATH and so local ruby files may be required if you so desire. Also note that you can include other yaml erb snippets with the method erbify_file.

An example using both of these features:

% require "common_prelude" #  will find "common_prelude.rb" in the concourse directory

resources:
<%= erbify_file "common_resources.yml" -%> #  will find this file in the concourse directory and recursively erbify it
  - name: resource_unique_to_this_pipeline
    type: git
    source:
      uri: <%= $common_prelude_defined_uri_prefix %>/username/projectname.git
      branch: master

(If you're unfamiliar with ERB and how you can mix Ruby into the document, you can read about it here.)

RUBIES

The ruby variable RUBIES is defined in the ERB binding during pipeline file generation. This variable looks like:

  # these numbers/names align with public docker image names
  RUBIES = {
    mri:     %w[2.3 2.4 2.5 2.6], # docker repository: "ruby"
    jruby:   %w[9.1 9.2],     # docker repository: "jruby"
    rbx:     %w[latest],      # docker repository: "rubinius/docker"
    windows: %w[2.3 2.4 2.5 2.6]  # windows-ruby-dev-tools-release
  }

and allows you to write a pipeline like this to get coverage on all the supported rubies:

# myproject.yml
jobs:
% RUBIES[:mri].each do |ruby_version|
  - name: "ruby-<%= ruby_version %>"
    plan:
      ...
      - task: rake-test
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: {repository: "ruby", tag: "<%= ruby_version %>"}
    ...
% end

Note that the windows rubies are not Docker images, since Concourse's Houdini backend doesn't use Docker. Instead, these are implicitly referring to the supported ruby versions installed by the BOSH release at https://github.com/flavorjones/windows-ruby-dev-tools-release

Secrets

You can use a separate file to keep your pipeline variables secret. By default, concourse/private.yml will be used. This filename can also be configured (see below)

If the secrets file exists, it will be passed to the fly commandline with the -l option to fill in template values.

For example, I might have a concourse config that looks like this:

  - name: nokogiri-pr
    type: pull-request
    source:
      repo: sparklemotion/nokogiri
      access_token: {{github-repo-status-access-token}}
      ignore_paths:
        - concourse/**

I can put my access token in private.yml like this:

github-repo-status-access-token: "your-token-here"

and the final generate template will substitute your credentials into the appropriate place.

Multiple pipelines

Setting up multiple pipelines might be useful for you, for example one pipeline for running tests against master and another for pull requests; or a pipeline for tests and a pipeline for deployments.

If you'd like to set up multiple pipelines, invoke Concourse in your Rakefile like this:

require 'concourse'

Concourse.new("myproject") do |c|
  c.add_pipeline "master", "myproject.yml"
  c.add_pipeline "pull-requests", "pr.yml"
  c.add_pipeline "deploy", "big-red-button.yml"
end

Note that when you use the block form:

• it's not necessary to explicitly call #create_tasks!
• only the pipelines declared via #add_pipeline will be managed

Configuration

directory: Concourse subdirectory name

You can choose a directory name other than the default concourse:

Concourse.new("myproject", directory: "ci").create_tasks!

fly_target: Concourse fly target name

If the initializer is given no additional parameters, your fly target is assumed to be named "default". But this is likely an inappropriate default, and so you can specify your target name:

Concourse.new("myproject", fly_target: "myci").create_tasks! # `fly -t myci <command>`

format: Emit the final pipelines in fly format-pipeline canonical format

If you'd prefer to have your final pipeline files in fly's "canonical format" (via format-pipeline), then set this to true!

Concourse.new("myproject", format: true).create_tasks!

This might be useful if you're heavily refactoring your template, and want to make sure there aren't unexpected changes to the pipeline.

pipeline_erb_filename: Pipeline filename

By default the pipeline file will be named <myproject>.yml, but can be set to something else:

Concourse.new("myproject", pipeline_erb_filename: "pipeline.yml").create_tasks!

Note that the generated, final pipeline file is always named <pipeline_erb_filename>.generated.

secrets_filename: Secrets filename

You can use a separate file to keep your pipeline variables secret. By default, concourse/private.yml will be used. You can specify a different filename:

Concourse.new("myproject", secrets_filename: "secrets.yml").create_tasks!

fly_args_<fly_command>: fly command arguments

Rarely, you may need to inject additional commandline arguments into fly to get the behavior you want. For example, I wanted to inject --enable-across-step into my validate-pipeline commands when I started exploring matrix builds via the across step.

You can pass in additional keys named fly_args_<command-name-with-underscores> like this:

Concourse.new("myproject", fly_args_validate_pipeline: "--enable-across-step --another-flag")

With the above initializer call, whenever the concourse gem invokes validate-pipeline, it will inject --enable-across-step --another-flag.

Rake Tasks

Manage your Concourse pipeline

Tasks to manage a local pipeline file, generated from an ERB template:

rake concourse:clean     # remove generated pipeline files
rake concourse:generate  # generate and validate all pipeline files

A task to update your pipeline configuration:

rake concourse:set       # upload all pipeline files

Tasks to publicly expose or hide your pipeline:

rake concourse:expose    # expose all pipelines
rake concourse:hide      # hide all pipelines

Tasks to pause and unpause your pipeline:

rake concourse:pause     # pause all pipelines
rake concourse:unpause   # unpause all pipelines

And, should you ever need to nuke the site from orbit, a task to destroy your pipeline:

rake concourse:destroy   # destroy all pipelines

List pipeline tasks

You can see a list of all the tasks defined in your pipelines:

rake concourse:tasks  # list all available tasks from all pipelines

You'll see output suitable for the concourse:task rake task, formatted as job-name/task-name, looking something like:

RUNNING: (in /home/flavorjones/code/oss/nokogiri) fly validate-pipeline -c concourse/nokogiri.yml.generated
looks good
NOTE: Available Concourse tasks for nokogiri are:
 * jruby-9.1/rake-test
 * jruby-9.2-pr/rake-test
 * jruby-9.2/rake-test
 * ruby-2.5-system/rake-test
 * ruby-2.5-vendored/rake-test
 * ruby-2.6-system-pr/rake-test
 * ruby-2.6-system/rake-test
 * ruby-2.6-vendored-pr/rake-test
 * ruby-2.6-vendored/rake-test

Run a Concourse task with fly execute

You can fly execute a task defined in your pipelines:

rake concourse:task[job_task,fly_execute_args]  # fly execute the specified task

where:

• required: job_task is formatted as job-name/task-name, for example, ruby-2.4/rake-test. (See concourse:tasks for a list of all available task names.)
• optional: fly_execute_args will default to map the project directory to a resource with the project name, e.g. --input=myproject=., so your pipeline must name the input resource appropriately in order to use the default.

Abort running builds

rake concourse:abort-builds  # abort all running builds for this pipeline

Prune stalled concourse workers

Especially useful if you're deploying via BOSH, which often results in stalled workers;

rake concourse:prune-stalled-workers  # prune any stalled workers

Running a local ephemeral Concourse

You can run a local Concourse deployment on your development system, if you have docker and docker-compose installed.

Spinning up a local deployment

To spin up the local cluster:

rake concourse:local:up

You can view that Concourse deployment at http://127.0.0.1:8080 using the credentials test/test.

Doing things with the local deployment

To target that local cluster, simply prepend the concourse:local task on the command line. For example, to fly execute a task on the local cluster:

rake concourse:local concourse:task[ruby-2.4/rake-task]

Or to push your pipelines to that local cluster:

rake concourse:local concourse:set

Shutting down the local deployment

Finally, to spin down the cluster (when you're done with it):

rake concourse:local:down

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/flavorjones/concourse-gem. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct. See CODE_OF_CONDUCT.md.

License

The gem is available as open source under the terms of the MIT License. See LICENSE.txt.