BBC::Cosmos::Tools

• Introduction
• Example commands
• Project directory structure
• File structure explained
• Using YAML or JSON
• Example repositories

Introduction

The BBC Cosmos Tool is a Ruby gem that runs on the command line and helps make generating CloudFormation specifically for the BBC Cosmos infrastructure easier.

The gem supports the following features:

• Pushing component config to the cosmos API
• Create/update and delete stacks for a component in cosmos
• Do release and snapshot deployments to any environment
• List instances running for a component and filter by tag

The gem supports the following formats:

• CFNDSL
• YAML
• JSON

Installation

Add this line to your application's Gemfile:

gem 'bbc-cosmos-tools'

And then execute:

$ bundle

Or install it yourself as:

$ gem install bbc-cosmos-tools

Project directory structure

The cli tool needs to be used inside a project that has a specific directory structure. This structure looks something like the following (if using CFNDSL):

.
├── configs
│   ├── app.yaml
│   └── {component}.yaml.erb
├── resources
│   ├── int
│   │   └── {component}.yaml
│   ├── live
│   │   └── {component}.yaml
│   └── test
│   │   └── {component}.yaml
└── stacks
    └── {component}
        └── {stack}
            ├── aws
            │   ├── {resource}
            │       ├── {type}.rb
            │   └── {resource}
            │       ├── {type}.rb
            └── template.rb

Where by the {x} items equate to the following:

• {component}
  The name of your Cosmos component (e.g. responsive-jmeter)

• {stack}
  Typically this will be main (as there must always be a main stack)
  But be aware you can have multiple "stacks" for a component (it's just a structural convenience)

• {resource}
  This could be (for example) a Scaling Group, IAM policy, SQS, or DynamoDB (any AWS service)

• {type}
  This is a subsection of the resource e.g. if the resource was AWS::SQS::Queue then this file would be named queue.rb

Note: if you're using the YAML or JSON formats then the following structure is used (notice it's almost identical)

.
├── configs
│   ├── app.yaml
│   └── {component}.yaml.erb
├── resources
│   ├── int
│   │   └── {component}.yaml
│   ├── live
│   │   └── {component}.yaml
│   └── test
│   │   └── {component}.yaml
└── stacks
    └── {component}
        └── {stack}.{yaml|json}

Quick directory creation

A quick way to create the YAML/JSON directory structure is shown below:

function setup() {
  local component_name=$1
  local type=$2

  mkdir {configs,resources,stacks}
  mkdir resources/{int,test,live}
  mkdir stacks/$component_name

  touch configs/$component_name.yaml.erb \
        resources/{int,test,live}/$component_name.yaml \
        stacks/$component_name/main.$type
}

setup foobar yaml
setup bazqux json

File structure explained

The file structure required by the BBC Cosmos CLI Tool can be confusing so let’s break it down into sections so we can better understand the purpose of each directory:

Config

The files in this folder let you define custom configuration that will be baked into your EC2 instances.

You need to use the bbc-cosmos-tools config push command to upload the updated configuration data (run the help command bbc-cosmos-tools help config push to see more options and details).

The content of this folder is usually:

• app.yaml (defaults)
• {component}.yaml.erb (custom)

So the contents of app.yaml can look like the following:

bbc:
  cosmos:
    api: 'https://api.live.bbc.co.uk'
    config_endpoint: '/cosmos/env/%s/component/%s/configuration'
    stack_create_endpoint: '/cosmos/env/%s/component/%s/stacks/create'
    stack_update_endpoint: '/cosmos/env/%s/component/%s/stack/%s/update'
    deployments: '/cosmos/deployments/component/%s/env/%s'
    release: '/cosmos/component/%s/releases'
    deploy: '/cosmos/env/%s/component/%s/deploy_release'
    snapshot_deploy: '/cosmos/env/%s/component/%s/deploy_snapshot'
    stack_events: '/cosmos/env/%s/component/%s/stack/%s/events'
    stacks: '/cosmos/env/%s/component/%s/stacks'
    stack_delete: '/cosmos/env/%s/component/%s/stack/%s/delete'
    stack_details: '/cosmos/env/%s/component/%s/stack/%s'
    stack_template: '/cosmos/env/%s/component/%s/stack/%s/template'

Note: stack_details is needed for when a stack update happens. As we don't have all the config values that Cosmos should automatically add in for us when it's created, we need to get them from Cosmos and add them to the parameter list when doing the update

Steven Jack [12:38 PM] otherwise they'll be blank

Where as the contents of {component}.yaml.erb can look like the following:

components:
  '{component}':
    s3_bucket_id: "foo"
    s3_bucket_path: "bar"

In YAML you can get fancy and make certain properties more re-usable by using & to create an "anchor" which you can then reference with *, along with <<: to inject content. See the following file for an example (although it’s not a very good example as there would be no need to use any of those features for such a small file, but hopefully it gives you an idea on how they are used)…

base: &shared
  s3_bucket_id: "foo"
  s3_object_path: "bar"

components:
  'responsive-jmeter':
    <<: *shared

Secure values

The cosmos API allows you to mark certain values as "secure" when adding them to the API. This means that when hitting the config REST endpoint, anything marked as secure won't show. It'll only be available when it's baked onto the instance. This is useful for stuff like API keys etc where you don't want everyone with a dev cert being able to see/use it.

To use this you need to have the following structure for a key in your config:

components:
  my_component:
    api_key:
      secure: true
      value: "SUPER_SECRET_KEY"
    foo: bar
    bar: qux

Using CFNDSL

The project config is an ERB template that is renderer into a YAML file. it gets passed all resources from the resource config that can be used in the template. Below is an example of this:

sequencer: &shared_sequencer
  sequencer_table_name: '<%= config['sequencer']['name'] %>'

storage: &shared_storage
  storage_path: '<%= config['storage']['name'] %>'

base: &shared
  <<: *shared_sequencer
  logger_path: 'some.loggingserver.com'
  logger_port: '12201'
  storage_path: '<% config['storage']['name]' %>
  keep_all_messages: 'true'

components:
  'test_component':
    <<: *shared
    some_specific_value: 'foo, bar'

The config has a number of components under a project, and the shared key allows you to have shared config for the components. The config is built up from the shared data with the component config merged in afterwards.

Application config

There's also an application config that stores data used by the app, at the moment this is only the API endpoints.

bbc:
  cosmos:
    api: "https://www.my.api.com"
    create_endpoint: "/some/restful/endpoint"
    update_endpoint: "/some/restful/endpoint"

Resources

The resources directory is used to provide custom values to the Parameters defined within your stacks (see the Stacks directory below). There is usually three folders inside this directory that reflect the Cosmos environments:

• int
• test
• live

Each of those folders will have a {component}.yaml file where you define the values for your Parameters (if you were using AWS directly, rather than via the Cosmos abstraction layer, then this is typically where you would specify the custom values either using the AWS CLI tool or by entering the values into form fields when using the AWS Console GUI).

The format of this file looks something like the following:

cloudformation:
  components:
    '{component_folder_name}':
        {stack}:
          variants:
            default:
              {parameter_a}: value
              {parameter_b}: value
              {parameter_c}: value

Using CFNDSL

The project resources yaml file is used to describe the resources used within the project. The reason for having a single file is that you can define resources that are used in the cloudformation templates and config, so when for example an ARN changes this will be picked up across the CF templates and the config.

The file is split up into two section:

• The standard resource list, this is generally the ARNS for all the resources your component uses.
• Cloudformation parameters (These are usually referenced from the aforementioned)

Below is an annotated example of a resource config:

sequencer:
  name: 'example_name'
  arn: 'example_arn'

roles:
  broker:
    name: &roles_broker_name 'some_iam_role_name'

cloudformation:
  shared: &component_shared #<- This node is ignored, it's used as a base for other config templates
    ACFParameter: 'Value'
    AnotherCFParameter: 'Another value'

  components:
    'my-example-component': #<- The name of the component in cosmos
      main: <- This is the name of the stack in cosmos
        variants: <- This is so you can have a different variant of the same config, i.e for int/test one for during the day and one for evening/weekends.
          default: &my-example-component_default #<- We set this anchor so we can use this defaults in other variants
            <<: *component_shared #<- We import the shared component params into these so we're not duplicating
            ARoleParamForDefault: *roles_broker_name #<- We use the anchor for the roles set above to add them here, again stopping repetition
          scheduled:
            <<: *my-example-component_default
            MinSize: 0
            MaxSize: 0

Stacks

The stacks directory is the most important part of the configuration as it defines your infrastructure requirements (i.e. creates all the specified AWS resources as you’ve defined them using CloudFormation).

The folder structure resembles the following:

.
├── stacks
    └── {component}
        └── {stack}
            ├── aws
            │   ├── {resource}
            │   │   ├── {type}

An example of this could be:

.
├── stacks
    └── responsive-jmeter
        └── main
            ├── aws
            │   ├── sqs
            │   │   ├── policy.rb
            │   │   ├── queue.rb
            │   ├── iam
            │   │   ├── instance_profile.rb

Note: when opening up the files policy.rb and queue.rb we’ll see the Type syntax and that should correspond to the folder structure described above. So aws/sqs/policy.rb will have a Type "AWS::SQS::QueuePolicy”, and aws/iam/instance_profile.rb will have a Type "AWS::IAM::InstanceProfile”

Example commands

When you want to push up some new configuration into your instance (for your application to use) then the following command is what you need (this isn't done very often):

bbc-cosmos-tools config push {cosmos_component_name} \
  --project={resources yaml filename} \
  --key-path=/custom/path/to/your/pem/certificate

To update your stack:

bbc-cosmos-tools stack update {cosmos_component_name} \
  --project={resources yaml filename} \
  --key-path=/custom/path/to/your/pem/certificate

To generate CloudFormation that is sent to stdout:

bbc-cosmos-tools stack generate {cosmos_component_name} \
  --project={resources yaml filename} \
  --key-path=/custom/path/to/your/pem/certificate

Note: in case you're not a CLI wizard, instead of specifying the --key-path option you can set a $DEV_CERT_PATH environment variable that the tool uses by default if it detects it. Inside your shell's configuration file (either ~/.bashrc or ~/.zshrc) you'll want to add export DEV_CERT_PATH=/custom/path/to/your/pem/certificate

Deploy component to a specific environment:

bbc-cosmos-tools release deploy <component> --project=<project> --from=X --env=Y

Using YAML or JSON

Use the same commands as above but make sure to include a --type={yaml|json}. If it's not specified then cfndsl is assumed as the default (almost as if you had written --type=cfndsl, but it's clearer to just remove the flag in that instance).

.
├── configs
│   ├── app.yaml
│   └── foo.yaml.erb
├── resources
│   ├── int
│   │   └── bar.yaml
│   ├── live
│   │   └── bar.yaml
│   └── test
│       └── bar.yaml
└── stacks
    └── foo
        ├── dns.yaml
        ├── main.yaml

To update the DNS stack in the live environment, you would execute:

bbc-cosmos-tools stack update foo --project=bar --env=live --type=yaml --stack=dns

Example repositories

There are a few example projects using this tool now:

• Election Data Config
• Jmeter Config
• Kaleidoscope Config

Note: it's important to realise that you don't need to have your configuration in a separate repository. If anything it probably would be better maintained as part of your application repository (placed inside a /config/ directory)

The reason some of the projects choose to locate their configuration in a separate repo is because they're made up of a number of different components.

Help?

You can use the built-in help command to see a list of parameters and options that need to be passed into the command you wish to execute. For example, if we were unsure of the options for the stack command then we would execute bbc-cosmos-tools stack help and this would display the following output:

Commands:
  bbc-cosmos-tools stack help [COMMAND]                          # Describe subcommands or one specific subco...
  bbc-cosmos-tools stack create [COMPONENT, MAIN_STACK = true]  # Generates and create the cloudformation te...
  bbc-cosmos-tools stack delete [COMPONENT]                      # Deletes a stack
  bbc-cosmos-tools stack events [COMPONENT = nil]                # Shows the stack events for a component
  bbc-cosmos-tools stack generate [COMPONENT]                    # Generates a cloudformation template based ...
  bbc-cosmos-tools stack list [COMPONENT = nil]                  # Lists stacks for a component
  bbc-cosmos-tools stack update [COMPONENT = nil]                # Generates and updates the cloudformation t...

SSH

You can SSH into an instance quickly by calling the component ssh command such as:

bbc-cosmos-tools component ssh news-javelin --env=live

Your IP for i-38c96895 is 10.0.233.46

If there are multiple instances for a component, a list will be provided to let you chose an instance ID:

bbc-cosmos-tools component ssh news-jenkins-agent --env=live

Multiple instances detected, please select one
1. i-67ad08ca
2. i-be8e9814
?  1 ### Selection

Creating Session
Creating Session
Creating Session
Creating Session

Your IP for i-67ad08ca is 10.0.234.217

Other Commands

List Instances

bbc-cosmos-tools component instances news-javelin --env=live

Instance ID
i-38c96895

List Releases

bbc-cosmos-tools release list news-javelin

Component: news-javelin
Version      Created at
0.1.6-1.el6  2015-06-30T16:10:54+01:00

List Deployed Releases

bbc-cosmos-tools release deployed news-javelin --env=live

Component: news-javelin
Release       Deployed at                Deployment id  Deployed by                  Status
0.1.83-1.el6  2015-07-16T15:39:59+01:00  282759         PlatformToolsTeam@bbc.co.uk  done

Redeploy a Component

bbc-cosmos-tools release redeploy news-release --env=int

news-release successfully redeployed
View deployment of news-release here https://admin.live.bbc.co.uk/cosmos/env/int/deployment/286431

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request