Muchkeys

─────────▄──────────────▄
────────▌▒█───────────▄▀▒▌
────────▌▒▒▀▄───────▄▀▒▒▒▐
───────▐▄▀▒▒▀▀▀▀▄▄▄▀▒▒▒▒▒▐
─────▄▄▀▒▒▒▒▒▒▒▒▒▒▒█▒▒▄█▒▐
───▄▀▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▀██▀▒▌
──▐▒▒▒▄▄▄▒▒▒▒▒▒▒▒▒▒▒▒▒▀▄▒▒▌
──▌▒▒▐▄█▀▒▒▒▒▄▀█▄▒▒▒▒▒▒▒█▒▐
─▐▒▒▒▒▒▒▒▒▒▒▒▌██▀▒▒▒▒▒▒▒▒▀▄▌
─▌▒▀▄██▄▒▒▒▒▒▒▒▒▒▒▒░░░░▒▒▒▒▌
─▌▀▐▄█▄█▌▄▒▀▒▒▒▒▒▒░░░░░░▒▒▒▐
▐▒▀▐▀▐▀▒▒▄▄▒▄▒▒▒▒▒░░░░░░▒▒▒▒▌
▐▒▒▒▀▀▄▄▒▒▒▄▒▒▒▒▒▒░░░░░░▒▒▒▐
─▌▒▒▒▒▒▒▀▀▀▒▒▒▒▒▒▒▒░░░░▒▒▒▒▌
─▐▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▐
──▀▄▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▄▒▒▒▒▌
────▀▄▒▒▒▒▒▒▒▒▒▒▄▄▄▀▒▒▒▒▄▀
───▐▀▒▀▄▄▄▄▄▄▀▀▀▒▒▒▒▒▄▄▀
──▐▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▀▀

Muchkeys lets you store your application keys in consul and then leverages many conventions to create a pleasant API. Keys in this context can mean application settings, knobs, dials, passwords, api keys or anything. It's primary use is in production where it will check to see if a ENV variable exists first and then search consul for a key based on an opinionated hierarchy convention.

You will want to read below about the convention and assumptions first to see if this works for you.

Muchkeys also has a way of using encrypted secrets stored in consul. Muchkeys has a command line interface to help you encrypt or read secrets stored in consul.

Installation

Add this line to your application's Gemfile:

gem 'muchkeys'

And then execute:

$ bundle

Or install it yourself as:

$ gem install muchkeys

Usage

Use a snippet like this below in your YAML config files and anywhere else to use a central configuration store while retaining control of your local development environment.

<%= MUCHKEYS['widget_api_key'] %>

Now the widget_api_key is coming from a central location. Devs can override widget_api_key with an ENV setting. export WIDGET_API_KEY="development_key76" Muchkeys defers to ENV when it sees one set.

Muchkeys will look in consul (a key/value store) for keys (settings/secrets/knobs to turn). It searches in an order of convention. It is also assuming you want to use git2consul to enable your developers to add their own keys. Git2consul syncs a git repo to consul.

So let's walk through what happens when a developer is adding a new feature to an app. The developer is integrating an API from reddit into an app called feed_reader. So now they have a reddit API key and this is something new to the app.

The developer adds twitter_api_key to .env from the dotenv gem to control their own development environment. Staging and production have no idea this has happened.
The developer adds/commits a file to <git2consul repo>/feed_reader/twitter_api_key
git2consul is sync'd to consul and their new key is there. (This might happen through cron or some other way)
They use their key in a configuration file or in the app: <%= MUCHKEYS['redis_api_key'] %>
When they deploy to staging/production the key is there and the feed_reader app doesn't explode.
Ops wasn't involved. Developers are empowered.

Muchkeys does this magic through a search order. It searches consul for the key in certain order: (notice that the key isn't prefixed when you look up twitter_api_key with MUCHKEYS['twitter_api_key'])

It will search a consul hierarchy looking for twitter_api_key in this search order:

git/feed_reader/secrets
git/feed_reader/config
git/shared/secrets
git/shared/config

The above feed_reader is detected from Rails.application. If you don't have a Rails app, then you can set the application name in the configure block:

Muchkeys.configure do |config|
  config.application_name = "rack_app"
end

The order and paths that it searches is configurable:

Muchkeys.configure do |config|
  config.search_paths = %W(
    app_name/keys app_name/secrets shared/keys shared/secrets
  )
end

This would look for keys and settings in consul under these paths instead:

app_name/keys
app_name/secrets
shared/keys
shared/secrets

Anything that has /secrets/ in it is going to be assumed to be a secret. There is an assumption that you have your keys and secrets organized like this (one deep nesting). This is configurable.

# if you don't put your secrets in something like shared/secrets/
# but shared/passwords/
Muchkeys.configure do |config|
  config.secrets_path_hint = "passwords/"
end

Using in a Rails App

Add to your Gemfile.

gem 'muchkeys'

Run bundle install.

If you don't need to change your hostname (production you do not need to) or any other settings, then you are done. If you need to override defaults, create an initializer.

# config/initializers/muchkeys.rb
Muchkeys.configure do |config|
  # your settings if desired
end

Then, use in YAML files, in the app or in other initializers.

# example of centralizing a database password in database.yml
# blog/config/database.yml
production:
  <<: *default
  username: MUCHKEYS['database_user']      # set in consul at git/blog/config/database_user
  password: MUCHKEYS['database_password']  # set in consul at git/blog/secrets/database_password

In the above example, database_password needs to be encrypted with an SSL certificate and then set in consul (paste the PEM text). The private and public key are needed to decrypt and these keys should be stored in <deploy_user_home>/.keys/blog.pem. With these conventions set once, you won't have to do anything else.

Encrypted Secrets

Generate an SSL cert or use an existing one. You will need the private and public key to encrypt and the public key to decrypt.

openssl req -new -newkey rsa:4096 -nodes -x509 -keyout /tmp/self_signed_test.pem -out /tmp/self_signed_test.pem

Muchkeys.configure do |config|
  config.public_key  = "/tmp/self_signed_test.pem"
  config.private_key = "/tmp/self_signed_test.pem"
end

puts Muchkeys::Secret.encrypt_string("bacon is just ok").to_s
# => -----BEGIN PKCS7-----
# => MIICuwYJKoZIhvcNAQcDoIICrDCCAqgCAQAxggJuMIICagIBADBSMEUxCzAJ ...
# => ...

# Copy and paste this into consul under a key: ie: secrets/fake

# Now you can fetch the secret with the same public key.
Muchkeys::Secret.get_consul_secret('secrets/fake')
# => bacon is just ok

Inside your app:

# inside a rails initializer or other setup file you have
Muchkeys.configure do |config|
  config.consul_url = "http://myrealhost"             # Default is "http://localhost:8500"
  config.public_key  = "path to public .pem"          # this is only required if you are encrypting secrets
  config.private_key = "path to private .pem"         # this is only required if you are encrypting secrets
end

# then inside your YAML or app:
Muchkeys.fetch_key('number_of_threads')
# goes to git/config/myapp/number_of_threads in consul

Automagic Certificates

Muchkeys can find certs automatically in ~/.keys. Muchkeys.fetch_key("git/waffles/secrets/a_password") will try to decrypt a_password with a cert called ~/.keys/waffles.pem. The private key can be in the same file but should be protected with file permissions 0600.

CLI

Example of decrypted an encrypted key:

$ muchkeys -d --public_key=~/.keys/staging.pem --private_key=~/.keys/staging.pem --consul_url=http://consul.domain:8500 --consul_key git/pants/secrets/some_password
<unencrypted secret is displayed>

Other Usage

You can fetch individual keys if you want.

ruby -e 'require "muchkeys"; puts Muchkeys.fetch_key("mail_server")'
# => smtp.example.com (from consul)

mail_server=muffin.ninja.local ruby -e 'require "muchkeys"; puts Muchkeys.fetch_key("mail_server")'
# => muffin.ninja.local (from ENV)

Keys to Store

You will probably just want to store things that change between environments in consul. If database_pool_size never changes between development and production then don't store it in consul. If sidekiq_number_of_workers is 1 in development, 2 in staging and 16 in production, then this is a good thing to store in consul because you'll have a central place that all app servers can read from.

You could store things in rails' environments/ path but then you'll need to do a deploy to change a setting and some things might be shared between apps like URLs, api keys, secrets and scaling settings.

Limits and Caveats

Searching multiple paths in consul is because consul is hierarchical (or can be) and ENV is not. Making an easy to use API that looks and behaves like ENV was one of the goals to make it more familiar to developers. So worst case, by default, Muchkeys will search consul 4 times to find a key.

Encrypted secrets can be slightly more clunky. Developers may not have signing keys, in that case, an admin or someone from ops maybe has to manage the secrets.

Compared to Other Tools

It's sort of the opposite of envconsul or similar to dotenv designed for use in production.

Dotenv doesn't centralize keys.
envconsul has "an auto restart the launched process on change" feature, this project does not.

Future

It'd be nice if this project wasn't so tied to consul. I don't think it's impossible to decouple it. Etcd basically behaves the same afaik so we could add an etcd adapter.

License

The gem is available as open source under the terms of the MIT License.