concourse (the rubygem)

The concourse gem provides rake tasks to help you manage Concourse CI pipelines, jobs, and workers, to assist in running tasks with fly execute, and even run a local ephemeral deployment of Concourse on your development machine.

If you're not familiar with Concourse CI, you can read up on it at

Here's an example pipeline maintained by this gem:



Basic Usage


Add this line to your application's Gemfile:

gem 'concourse'

and then run bundle.

You can also install it via gem install concourse.

Add to your Rakefile

The primary functionality of the Concourse gem is to provide rake tasks. So, in your Rakefile:

require 'concourse'"myproject").create_tasks!

Set up your Concourse pipeline

rake concourse:init

The concourse:init task will do a few different things for you:

  1. create a subdirectory named concourse (or whatever you've configured with the :directory parameter)
  2. create an empty Concourse pipeline file in that subdirectory named <myproject>.yml (or whatever you've configured with the :pipeline_erb_filename parameter)
  3. ensure git will ignore your secrets file named private.yml (or whatever you've configured with the :secrets_filename parameter)

Real-world Examples

It might be helpful to look at how other projects are using this gem. Here are a sample, ordered (approximately) from simplest to most complex:

Concourse pipeline configuration

ERB Templating

Your Concourse pipeline configuration file, <myproject>.yml (or whatever you've configured with the :pipeline_erb_filename parameter), will be treated like an ERB template.

The concourse directory is added the $LOAD_PATH and so local ruby files may be required if you so desire. Also note that you can include other yaml erb snippets with the method erbify_file.

An example using both of these features:

% require "common_prelude" #  will find "common_prelude.rb" in the concourse directory

<%= erbify_file "common_resources.yml" -%> #  will find this file in the concourse directory and recursively erbify it
  - name: resource_unique_to_this_pipeline
    type: git
      uri: <%= $common_prelude_defined_uri_prefix %>/username/projectname.git
      branch: master

(If you're unfamiliar with ERB and how you can mix Ruby into the document, you can read about it here.)


The ruby variable RUBIES is defined in the ERB binding during pipeline file generation. This variable looks like:

  # these numbers/names align with public docker image names
  RUBIES = {
    mri:     %w[2.3 2.4 2.5 2.6], # docker repository: "ruby"
    jruby:   %w[9.1 9.2],     # docker repository: "jruby"
    rbx:     %w[latest],      # docker repository: "rubinius/docker"
    windows: %w[2.3 2.4 2.5 2.6]  # windows-ruby-dev-tools-release

and allows you to write a pipeline like this to get coverage on all the supported rubies:

# myproject.yml
% RUBIES[:mri].each do |ruby_version|
  - name: "ruby-<%= ruby_version %>"
      - task: rake-test
          platform: linux
            type: docker-image
            source: {repository: "ruby", tag: "<%= ruby_version %>"}
% end

Note that the windows rubies are not Docker images, since Concourse's Houdini backend doesn't use Docker. Instead, these are implicitly referring to the supported ruby versions installed by the BOSH release at


You can use a separate file to keep your pipeline variables secret. By default, concourse/private.yml will be used. This filename can also be configured (see below)

If the secrets file exists, it will be passed to the fly commandline with the -l option to fill in template values.

For example, I might have a concourse config that looks like this:

  - name: nokogiri-pr
    type: pull-request
      repo: sparklemotion/nokogiri
      access_token: {{github-repo-status-access-token}}
        - concourse/**

I can put my access token in private.yml like this:

github-repo-status-access-token: "your-token-here"

and the final generate template will substitute your credentials into the appropriate place.

Multiple pipelines

Setting up multiple pipelines might be useful for you, for example one pipeline for running tests against master and another for pull requests; or a pipeline for tests and a pipeline for deployments.

If you'd like to set up multiple pipelines, invoke Concourse in your Rakefile like this:

require 'concourse'"myproject") do |c|
  c.add_pipeline "master", "myproject.yml"
  c.add_pipeline "pull-requests", "pr.yml"
  c.add_pipeline "deploy", "big-red-button.yml"

Note that when you use the block form:

  • it's not necessary to explicitly call #create_tasks!
  • only the pipelines declared via #add_pipeline will be managed


directory: Concourse subdirectory name

You can choose a directory name other than the default concourse:"myproject", directory: "ci").create_tasks!

fly_target: Concourse fly target name

If the initializer is given no additional parameters, your fly target is assumed to be named "default". But this is likely an inappropriate default, and so you can specify your target name:"myproject", fly_target: "myci").create_tasks! # `fly -t myci <command>`

format: Emit the final pipelines in fly format-pipeline canonical format

If you'd prefer to have your final pipeline files in fly's "canonical format" (via format-pipeline), then set this to true!"myproject", format: true).create_tasks!

This might be useful if you're heavily refactoring your template, and want to make sure there aren't unexpected changes to the pipeline.

pipeline_erb_filename: Pipeline filename

By default the pipeline file will be named <myproject>.yml, but can be set to something else:"myproject", pipeline_erb_filename: "pipeline.yml").create_tasks!

Note that the generated, final pipeline file is always named <pipeline_erb_filename>.generated.

secrets_filename: Secrets filename

You can use a separate file to keep your pipeline variables secret. By default, concourse/private.yml will be used. You can specify a different filename:"myproject", secrets_filename: "secrets.yml").create_tasks!

Rake Tasks

Manage your Concourse pipeline

Tasks to manage a local pipeline file, generated from an ERB template:

rake concourse:clean     # remove generated pipeline files
rake concourse:generate  # generate and validate all pipeline files

A task to update your pipeline configuration:

rake concourse:set       # upload all pipeline files

Tasks to publicly expose or hide your pipeline:

rake concourse:expose    # expose all pipelines
rake concourse:hide      # hide all pipelines

Tasks to pause and unpause your pipeline:

rake concourse:pause     # pause all pipelines
rake concourse:unpause   # unpause all pipelines

And, should you ever need to nuke the site from orbit, a task to destroy your pipeline:

rake concourse:destroy   # destroy all pipelines

List pipeline tasks

You can see a list of all the tasks defined in your pipelines:

rake concourse:tasks  # list all available tasks from all pipelines

You'll see output suitable for the concourse:task rake task, formatted as job-name/task-name, looking something like:

RUNNING: (in /home/flavorjones/code/oss/nokogiri) fly validate-pipeline -c concourse/nokogiri.yml.generated
looks good
NOTE: Available Concourse tasks for nokogiri are:
 * jruby-9.1/rake-test
 * jruby-9.2-pr/rake-test
 * jruby-9.2/rake-test
 * ruby-2.5-system/rake-test
 * ruby-2.5-vendored/rake-test
 * ruby-2.6-system-pr/rake-test
 * ruby-2.6-system/rake-test
 * ruby-2.6-vendored-pr/rake-test
 * ruby-2.6-vendored/rake-test

Run a Concourse task with fly execute

You can fly execute a task defined in your pipelines:

rake concourse:task[job_task,fly_execute_args]  # fly execute the specified task


  • required: job_task is formatted as job-name/task-name, for example, ruby-2.4/rake-test. (See concourse:tasks for a list of all available task names.)
  • optional: fly_execute_args will default to map the project directory to a resource with the project name, e.g. --input=myproject=., so your pipeline must name the input resource appropriately in order to use the default.

Abort running builds

rake concourse:abort-builds  # abort all running builds for this pipeline

Prune stalled concourse workers

Especially useful if you're deploying via BOSH, which often results in stalled workers;

rake concourse:prune-stalled-workers  # prune any stalled workers

Running a local ephemeral Concourse

You can run a local Concourse deployment on your development system, if you have docker and docker-compose installed.

Spinning up a local deployment

To spin up the local cluster:

rake concourse:local:up

You can view that Concourse deployment at using the credentials test/test.

Doing things with the local deployment

To target that local cluster, simply prepend the concourse:local task on the command line. For example, to fly execute a task on the local cluster:

rake concourse:local concourse:task[ruby-2.4/rake-task]

Or to push your pipelines to that local cluster:

rake concourse:local concourse:set

Shutting down the local deployment

Finally, to spin down the cluster (when you're done with it):

rake concourse:local:down


Bug reports and pull requests are welcome on GitHub at This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct. See


The gem is available as open source under the terms of the MIT License. See LICENSE.txt.