Humidifier::Reservoir
Reservoir
is a plugin for the humidifier
gem that allows you to specify CloudFormation resources in yaml syntax, while still allowing you to take advantage of the flexibility of humidifier
.
Installation
Add this line to your application's Gemfile:
gem 'humidifier-reservoir'
And then execute:
$ bundle
Or install it yourself as:
$ gem install humidifier-reservoir
Usage
Getting started
Reservoir
is meant to be run as a CLI tool. First, build a ruby script (for example bin/reservoir
) that executes the Humidifier::Reservoir::CLI
class, like so:
#!/usr/bin/env ruby
require 'humidifier/reservoir'
Humidifier::Reservoir.configure do |config|
config.stack_path = 'stacks'
config.stack_prefix = 'reservoir-'
config.map :users, to: 'AWS::IAM::User'
end
Humidifier::Reservoir::CLI.start(ARGV)
This configuration specifies that the directories containing the specifications for each stack are contained under a "stacks" directory, the stack names should be prefixed with a "reservoir-" signifier, and that users.yml
files should map to IAM users.
Resource files
Inside of the above-configured stacks
directory, create a subdirectory for each CloudFormation stack that you wish to manage in code. With the above configuration, we can create YAML files in the form of users.yml
for each stack, which will specify IAM users to create. The file format looks like the below:
EngUser:
path: /reservoir/
user_name: EngUser
groups:
- Engineering
- Testing
- Deployment
AdminUser:
path: /reservoir/
user_name: AdminUser
groups:
- Management
- Administration
The top-level keys are the logical resource names that will be displayed in the CloudFormation screen. They point to a map of key/value pairs that will be passed on to humidifier
. Any humidifier
(and therefore any CloudFormation) attribute may be specified. For more information on CloudFormation templates and which attributes may be specified, see both the humidifier
docs and the CloudFormation docs.
Mappers
Oftentimes, specifying these attributes can become repetitive, e.g., each user should automatically receive the same "path" attribute. Other times, you may want custom logic to execute depending on which AWS environment you're running in. Finally, you may want to reference resources in the same or other stacks.
Reservoir
's solution for this is to allow customized "mapper" classes to take the user-provided attributes and transform them into the attributes that CloudFormation expects. Consider the following example for mapping a user:
class UserMapper < Humidifier::Reservoir::BaseMapper
GROUPS = {
'eng' => %w[Engineering Testing Deployment],
'admin' => %w[Management Administration]
}
defaults do |logical_name|
{ path: '/reservoir/', user_name: logical_name }
end
attribute :group do |group|
groups = GROUPS[group]
groups.any? ? { groups: GROUPS[group] } : {}
end
end
Humidifier::Reservoir.configure do |config|
config.map :users, to: 'AWS::IAM::User', using: UserMapper
end
This means that by default, all entries in the users.yml
files will get a /reservoir/
path, the user_name
attribute will be set based on the logical name that was provided for the resource, and you can additionally specify a group
attribute, even though it is not native to CloudFormation. With this group
attribute, it will actually map to the groups
attribute that CloudFormation expects.
With this new mapper in place, we can simplify our YAML file to:
EngUser:
group: eng
AdminUser:
group: admin
Using the CLI
Now that you've configured your CLI, your resources, and your mappers, you can use the CLI to display, validate, and deploy your infrastructure to CloudFormation. Run your script without any arguments to get the help message and explanations for each command.
Each command has an --aws-profile
(or -p
) option for specifying which profile to authenticate against when querying AWS. You should ensure that this profile has the correct permissions for creating whatever resources are going to part of your stack. You can also rely on the AWS_*
environment variables, or the EC2 instance profile if you're deploying from an instance. For more information, see the AWS docs under the "Configuration" section.
Below are the list of commands and some of their options.
change [?stack]
Creates a change set for either the specified stack or all stacks in the repo. The change set represents the changes between what is currently deployed versus the resources represented by the configuration.
deploy [?stack] [*parameters]
Creates or updates (depending on if the stack already exists) one or all stacks in the repo.
--prefix
The deploy
command also allows a --prefix
command line argument that will override the default prefix (if one is configured) for the stack that is being deployed. This is especially useful when you're deploying multiple copies of the same stack (for instance, multiple autoscaling groups) that have different purposes or semantically mean newer versions of resources.
display [stack] [?pattern]
Displays the specified stack in JSON format on the command line. If you optionally pass a pattern argument, it will filter the resources down to just ones whose names match the given pattern.
upload [?stack]
Upload one or all stacks in the repo to S3 for reference later. Note that this must be combined with the humidifier
s3_bucket
configuration option.
validate [?stack]
Validate that one or all stacks in the repo are properly configured and using values that CloudFormation understands.
Parameters
CloudFormation template parameters can be specified by having a special parameters.yml
file in your stack directory. This file should contain a YAML-encoded object whose keys are the names of the parameters and whose values are the parameter configuration (using the same underscore paradigm as humidifier
resources for specifying configuration).
You can pass values to the CLI deploy command after the stack name on the command line as in:
bin/reservoir deploy foobar Param1=Foo Param2=Bar
Those parameters will get passed in as values when the stack is deployed.
Shortcuts
A couple of convenient shortcuts are built into humidifier-reservoir
so that writing templates and mappers both can be more concise.
Automatic id properties
There are a lot of properties in the AWS CloudFormation resource specification that are simply pointers to other entities within the AWS ecosystem. For example, an AWS::EC2::VPCGatewayAttachment
entity has a VpcId
property that represents the ID of the associated AWS::EC2::VPC
.
Because this pattern is so common, humidifier-reservoir
detects all properties ending in Id
and allows you to specify them without the suffix. If you choose to use this format, humidifier-reservoir
will automatically turn that value into a CloudFormation resource reference.
Anonymous mappers
A lot of the time, mappers that you create will not be overly complicated, especially if you're using the aforementioned automatic id properties. Therefore, the config.map
method takes a block, and allows you to specify the mapper inline. This is recommended for mappers that aren't too complicates as to warrant their own class (for instance, for testing purposes). An example of this using the UserMapper
from above is below:
Humidifier::Reservoir.configure do |config|
config.map :users, to: 'AWS::IAM::User' do
GROUPS = {
'eng' => %w[Engineering Testing Deployment],
'admin' => %w[Management Administration]
}
defaults do |logical_name|
{ path: '/reservoir/', user_name: logical_name }
end
attribute :group do |group|
groups = GROUPS[group]
groups.any? ? { groups: GROUPS[group] } : {}
end
end
end
Cross-stack references
AWS allows cross-stack references through the intrinsic Fn::ImportValue
function. You can take advantage of this with humidifier-reservoir
by using the export: true
option on resources in your stacks. For instance, if in one stack you have a subnet that you need to reference in another, you could (stacks/vpc/subnets.yml
):
ProductionPrivateSubnet2a:
vpc: ProductionVPC
cidr_block: 10.0.0.0/19
availability_zone: us-west-2a
export: true
ProductionPrivateSubnet2b:
vpc: ProductionVPC
cidr_block: 10.0.64.0/19
availability_zone: us-west-2b
export: true
ProductionPrivateSubnet2c:
vpc: ProductionVPC
cidr_block: 10.0.128.0/19
availability_zone: us-west-2c
export: true
And then in another stack, you could reference those values (stacks/rds/db_subnets_groups.yml
):
ProductionDBSubnetGroup:
db_subnet_group_description: Production DB private subnet group
subnets:
- ProductionPrivateSubnet2a
- ProductionPrivateSubnet2b
- ProductionPrivateSubnet2c
Within the configuration, you would specify to use the Fn::ImportValue
function like so:
Humidifier::Reservoir.configure do |config|
config.stack_path = 'stacks'
config.map :subnets, to: 'EC2::Subnet'
config.map :db_subnet_groups, to: 'RDS::DBSubnetGroup' do
attribute :subnets do |subnet_names|
subnet_ids =
subnet_names.map do |subnet_name|
Humidifier.fn.import_value(subnet_name)
end
{ subnet_ids: subnet_ids }
end
end
end
Development
After checking out the repo, run bin/setup
to install dependencies. Then, run rake test
to run the tests. You can also run bin/console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
. To release a new version, update the version number in version.rb
, and then run bundle exec rake release
, which will create a git tag for the version, push git commits and tags, and push the .gem
file to rubygems.org.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/localytics/humidifier-reservoir.
License
The gem is available as open source under the terms of the MIT License.