Chef Metal

This library solves the problem of repeatably creating machines and infrastructures in Chef. It has a plugin model that lets you write bootstrappers for your favorite infrastructures, including VirtualBox, EC2, LXC, bare metal, and many more!

Currently, chef-metal supports vagrant, Unixes/ssh, and Windows/winrm with real chef-servers or with automagical chef-zero tunneling. Fog and Docker support (to cover EC2 and LXC) are next up. Further out, we'd like to extend support to image factories (using the machine resource to produce images) and PXE support.

Try It Out

To give it a spin, get chef 11.8 or greater try this:

git clone https://github.com/opscode/cheffish.git
cd cheffish
rake install
cd ..

git clone https://github.com/opscode/chef-metal.git
cd chef-metal
rake install

cd chef-metal
chef-client -z -o myapp::vagrant,myapp::linux,myapp::small

This will create two vagrant precise64 linux boxes, "mario" and "luigi1", in ~/machinetest, bootstrapped to an empty runlist. For Windows, you can replace myapp::linux with myapp::windows, but you'll need your own Windows vagrant box to do that (licensing!).

What Is Chef Metal?

Chef Metal has two major abstractions: the machine resource, and provisioners.

The machine resource

You declare what your machines do (recipes, tags, etc.) with the machine resource, the fundamental unit of Chef Metal. You will typically declare machine resources in a separate, OS/provisioning-independent file that declares the topology of your app--your machines and the recipes that will run on them.

The machine resources from the example myapp::small are pretty straightforward. Here's a copy/paste:

machine 'mario' do
  recipe 'postgresql'
  recipe 'mydb'
  tag 'mydb_master'
end

num_webservers = 1

1.upto(num_webservers) do |i|
  machine "luigi#{i}" do
    recipe 'apache'
    recipe 'mywebapp'
  end
end

You will notice the dynamic nature of the number of web servers. It's all code, your imagination is the limit :)

Kitchen

Chef Metal also works with Test Kitchen, allowing you to test entire clusters, not just machines! The repository for the kitchen-metal gem is https://github.com/doubt72/kitchen-metal.

Provisioners

Provisioners handle the real work of getting those abstract definitions into real, physical form. They handle the following tasks, idempotently (you can run the resource again and again and it will only create the machine once--though it may notice things are wrong and fix them!):

  • Acquiring machines from the cloud, creating containers or VMs, or grabbing bare metal
  • Connecting to those machines via ssh, winrm, or other transports
  • Bootstrapping chef onto the machines and converging the recipes you suggested

The provisioner API is separated out so that new provisioners can be made with minimal effort (without having to rewrite ssh, tunneling, bootstrapping, and OS support). But to the user, they appear as a single thing, so that the machine acquisition can use its smarts to autodetect the other bits (transports, OS's, etc.).

Provisioners save their data in the Chef node itself, so that they will be accessible to everyone who is using the Chef server to manage the nodes.

Provisioners each have their own repository. Current provisioners:

Cloud:

Virtualization:

Containers:

Bare Metal:

Vagrant

chef-zero comes with a provisioner for Vagrant, an abstraction that covers VirtualBox, VMWare and other Virtual Machine providers. To run it, you can check out the sample recipes with:

chef-client -z -o myapp::vagrant,myapp::linux,myapp::small

The provisioner specification is in myapp::vagrant and myapp::linux sample recipes, copy/pasted here for your convenience:

vagrant_cluster "#{ENV['HOME']}/machinetest"

directory "#{ENV['HOME']}/machinetest/repo"
with_chef_local_server :chef_repo_path => "#{ENV['HOME']}/machinetest/repo"

vagrant_box 'precise64' do
  url 'http://files.vagrantup.com/precise32.box' 
end

vagrant_cluster declares a directory where all the vagrant definitions will be stored, and uses with_provisioner internally to tell Chef Metal that this is the provisioner we want to use for machines.

vagrant_box makes sure a particular vagrant box exists, and lets you specify provisioner_options for things like port forwarding, OS definitions, and any other vagrant-isms. A more complicated vagrant box, with provisioner options, can be found in myapp::windows.

with_chef_local_server is a generic directive that creates a chef-zero server pointed at the given repository. nodes, clients, data bags, and all data will be stored here on your provisioner machine if you do this. You can use with_chef_server instead if you want to point at OSS, Hosted or Enterprise Chef, and if you don't specify a Chef server at all, it will use the one you are running chef-client against. Keep in mind when using with_chef_server and running chef-client -z on your workstation that you will also need to set the client name and signing key for the chef server. If you've already got knife.rb set up, then something like this will correctly create a client for the chef server on instance using your knife.rb configuration:

require 'chef/config'

with_chef_server "https://chef-server.example.org", {
  :client_name => Chef::Config[:node_name],
  :signing_key_filename => Chef::Config[:client_key]
}

Typically, you declare these in separate files from your machine resources. Chef Metal picks up the provisioners you have declared, and uses them to instantiate the machines you request. The actual machine definitions, in this case, are in myapp::small, and are generic--you could use them against EC2 as well:

machine 'mario' do
  recipe 'postgresql'
  recipe 'mydb'
  tag 'mydb_master'
end

num_webservers = 1

1.upto(num_webservers) do |i|
  machine "luigi#{i}" do
    recipe 'apache'
    recipe 'mywebapp'
  end
end

Fog (EC2 and friends)

chef-metal also comes with a Fog provisioner that handles provisioning to Amazon's EC2 and other cloud providers. (Only EC2 has been tested so far.) Before you begin, you will need to put your AWS credentials in ~/.aws/config in the format mentioned in Option 1 here.

Once your credentials are in, basic usage looks like this:

chef-client -z -o myapp::ec2,myapp::small

The provisioner definition in myapp::ec2 looks like this:

ec2testdir = File.expand_path('~/ec2test')

directory ec2testdir

with_fog_ec2_provisioner # :provider => 'AWS'

with_provisioner_options :image_id => 'ami-5ee70037'

fog_key_pair 'me' do
  private_key_path "#{ec2testdir}/me"
  public_key_path "#{ec2testdir}/me.pub"
end

with_fog_ec2_provisioner tells chef-metal to use the Fog provisioner against EC2. If you specify your credentials in ~/.aws/config, you don't have to specify anything else; it will use the Fog defaults. You may pass a hash of parameters to with_fog_ec2_provisioner that is described here.

fog_key_pair creates a new key pair (if the files do not already exist) and automatically tells the Provisioner to use it to bootstrap subsequent machines. The private/public key pair will be automatically authorized to log on to the instance on first boot.

To pass options like ami, you can say something like this:

with_provisioner_options :image_id => 'ami-5ee70037'

If you need to pass bootstrapping options on a per-machine basis, you can do that as well by doing something like the following:

machine "Ubuntu_64bit" do
  action :create
  provisioner_options 'bootstrap_options' => {
    'image_id' => 'ami-59a4a230',
    'flavor_id' => 't1.micro'
  }
end

You will notice that we are still using myapp::small here. Machine definitions are generally provisioner-independent. This is an important feature that allows you to spin up your clusters in different places to create staging, test or miniature dev environments.

Bugs and The Plan

It's early days. Please submit bugs at https://github.com/opscode/chef-metal/issues, contact jkeiser on Twitter at @jkeiser2, email at [email protected]

If you are interested in the Plan for Chef Metal, you can peruse our Trello board! Please add suggestions there, vote or comment on issues that are important to you, and feel free to contribute by picking up a card. Chat with me ([email protected]) if you would like some context on how to go about implementing a card, or just go hog wild and submit a PR :)