Humidifier::Reservoir

Build Status Coverage Status Gem

Reservoir is a plugin for the humidifier gem that allows you to specify CloudFormation resources in yaml syntax, while still allowing you to take advantage of the flexibility of humidifier.

Installation

Add this line to your application's Gemfile:

gem 'humidifier-reservoir'

And then execute:

$ bundle

Or install it yourself as:

$ gem install humidifier-reservoir

Usage

Getting started

Reservoir is meant to be run as a CLI tool. First, build a ruby script (for example bin/reservoir) that executes the Humidifier::Reservoir::CLI class, like so:

#!/usr/bin/env ruby
require 'humidifier/reservoir'

Humidifier::Reservoir.configure do |config|
  config.stack_path = 'stacks'
  config.stack_prefix = 'reservoir-'

  config.map :users, to: 'AWS::IAM::User'
end

Humidifier::Reservoir::CLI.start(ARGV)

This configuration specifies that the directories containing the specifications for each stack are contained under a "stacks" directory, the stack names should be prefixed with a "reservoir-" signifier, and that users.yml files should map to IAM users.

Resource files

Inside of the above-configured stacks directory, create a subdirectory for each CloudFormation stack that you wish to manage in code. With the above configuration, we can create YAML files in the form of users.yml for each stack, which will specify IAM users to create. The file format looks like the below:

EngUser:
  path: /reservoir/
  user_name: EngUser
  groups:
  - Engineering
  - Testing
  - Deployment

AdminUser:
  path: /reservoir/
  user_name: AdminUser
  groups:
  - Management
  - Administration

The top-level keys are the logical resource names that will be displayed in the CloudFormation screen. They point to a map of key/value pairs that will be passed on to humidifier. Any humidifier (and therefore any CloudFormation) attribute may be specified. For more information on CloudFormation templates and which attributes may be specified, see both the humidifier docs and the CloudFormation docs.

Mappers

Oftentimes, specifying these attributes can become repetitive, e.g., each user should automatically receive the same "path" attribute. Other times, you may want custom logic to execute depending on which AWS environment you're running in. Finally, you may want to reference resources in the same or other stacks.

Reservoir's solution for this is to allow customized "mapper" classes to take the user-provided attributes and transform them into the attributes that CloudFormation expects. Consider the following example for mapping a user:

class UserMapper < Humidifier::Reservoir::BaseMapper
  GROUPS = {
    'eng' => %w[Engineering Testing Deployment],
    'admin' => %w[Management Administration]
  }

  defaults do |logical_name|
    { path: '/reservoir/', user_name: logical_name }
  end

  attribute :group do |group|
    groups = GROUPS[group]
    groups.any? ? { groups: GROUPS[group] } : {}
  end
end

Humidifier::Reservoir.configure do |config|
  config.map :users, to: 'AWS::IAM::User', using: UserMapper
end

This means that by default, all entries in the users.yml files will get a /reservoir/ path, the user_name attribute will be set based on the logical name that was provided for the resource, and you can additionally specify a group attribute, even though it is not native to CloudFormation. With this group attribute, it will actually map to the groups attribute that CloudFormation expects.

With this new mapper in place, we can simplify our YAML file to:

EngUser:
  group: eng

AdminUser:
  group: admin

Using the CLI

Now that you've configured your CLI, your resources, and your mappers, you can use the CLI to display, validate, and deploy your infrastructure to CloudFormation. Run your script without any arguments to get the help message and explanations for each command.

Each command has an --aws-profile (or -p) option for specifying which profile to authenticate against when querying AWS. You should ensure that this profile has the correct permissions for creating whatever resources are going to part of your stack. You can also rely on the AWS_* environment variables, or the EC2 instance profile if you're deploying from an instance. For more information, see the AWS docs under the "Configuration" section.

Below are the list of commands and some of their options.

change [?stack]

Creates a change set for either the specified stack or all stacks in the repo. The change set represents the changes between what is currently deployed versus the resources represented by the configuration.

deploy [?stack] [*parameters]

Creates or updates (depending on if the stack already exists) one or all stacks in the repo.

--prefix

The deploy command also allows a --prefix command line argument that will override the default prefix (if one is configured) for the stack that is being deployed. This is especially useful when you're deploying multiple copies of the same stack (for instance, multiple autoscaling groups) that have different purposes or semantically mean newer versions of resources.

display [stack] [?pattern]

Displays the specified stack in JSON format on the command line. If you optionally pass a pattern argument, it will filter the resources down to just ones whose names match the given pattern.

upload [?stack]

Upload one or all stacks in the repo to S3 for reference later. Note that this must be combined with the humidifier s3_bucket configuration option.

validate [?stack]

Validate that one or all stacks in the repo are properly configured and using values that CloudFormation understands.

Parameters

CloudFormation template parameters can be specified by having a special parameters.yml file in your stack directory. This file should contain a YAML-encoded object whose keys are the names of the parameters and whose values are the parameter configuration (using the same underscore paradigm as humidifier resources for specifying configuration).

You can pass values to the CLI deploy command after the stack name on the command line as in:

bin/reservoir deploy foobar Param1=Foo Param2=Bar

Those parameters will get passed in as values when the stack is deployed.

Shortcuts

A couple of convenient shortcuts are built into humidifier-reservoir so that writing templates and mappers both can be more concise.

Automatic id properties

There are a lot of properties in the AWS CloudFormation resource specification that are simply pointers to other entities within the AWS ecosystem. For example, an AWS::EC2::VPCGatewayAttachment entity has a VpcId property that represents the ID of the associated AWS::EC2::VPC.

Because this pattern is so common, humidifier-reservoir detects all properties ending in Id and allows you to specify them without the suffix. If you choose to use this format, humidifier-reservoir will automatically turn that value into a CloudFormation resource reference.

Anonymous mappers

A lot of the time, mappers that you create will not be overly complicated, especially if you're using the aforementioned automatic id properties. Therefore, the config.map method takes a block, and allows you to specify the mapper inline. This is recommended for mappers that aren't too complicates as to warrant their own class (for instance, for testing purposes). An example of this using the UserMapper from above is below:

Humidifier::Reservoir.configure do |config|
  config.map :users, to: 'AWS::IAM::User' do
    GROUPS = {
      'eng' => %w[Engineering Testing Deployment],
      'admin' => %w[Management Administration]
    }

    defaults do |logical_name|
      { path: '/reservoir/', user_name: logical_name }
    end

    attribute :group do |group|
      groups = GROUPS[group]
      groups.any? ? { groups: GROUPS[group] } : {}
    end
  end
end

Cross-stack references

AWS allows cross-stack references through the intrinsic Fn::ImportValue function. You can take advantage of this with humidifier-reservoir by using the export: true option on resources in your stacks. For instance, if in one stack you have a subnet that you need to reference in another, you could (stacks/vpc/subnets.yml):

ProductionPrivateSubnet2a:
  vpc: ProductionVPC
  cidr_block: 10.0.0.0/19
  availability_zone: us-west-2a
  export: true

ProductionPrivateSubnet2b:
  vpc: ProductionVPC
  cidr_block: 10.0.64.0/19
  availability_zone: us-west-2b
  export: true

ProductionPrivateSubnet2c:
  vpc: ProductionVPC
  cidr_block: 10.0.128.0/19
  availability_zone: us-west-2c
  export: true

And then in another stack, you could reference those values (stacks/rds/db_subnets_groups.yml):

ProductionDBSubnetGroup:
  db_subnet_group_description: Production DB private subnet group
  subnets:
  - ProductionPrivateSubnet2a
  - ProductionPrivateSubnet2b
  - ProductionPrivateSubnet2c

Within the configuration, you would specify to use the Fn::ImportValue function like so:

Humidifier::Reservoir.configure do |config|
  config.stack_path = 'stacks'

  config.map :subnets, to: 'EC2::Subnet'

  config.map :db_subnet_groups, to: 'RDS::DBSubnetGroup' do
    attribute :subnets do |subnet_names|
      subnet_ids =
        subnet_names.map do |subnet_name|
          Humidifier.fn.import_value(subnet_name)
        end

      { subnet_ids: subnet_ids }
    end
  end
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/localytics/humidifier-reservoir.

License

The gem is available as open source under the terms of the MIT License.