Deplomat - a stack agnostic deployment system

Deplomat is a stack agnostic deployment system that uses bash and ssh commands. The purpose of Deplomat is to be a suitable deployment system for all and easily scale from a one-man operation to big teams and multiple servers.

How does it work?

It uses SSH to send commands to remote servers and can also execute scripts on a local machine.
The SSH connection is opened in such a way that it is persistent, so it's fast to execute multiple commands over it;
The deployment script is a simple ruby script. You just create your own methods and then call them.

Let's take a look at an example script, step by step. For the purposes of this tutorial, we'll simplify things. The process will resemble a typical web-app deployment, but won't be too specific or complicated.

#!/usr/bin/env ruby
require 'rubygems'
require 'deplomat'

$env          = ARGV[0] || "staging"
$local        = Deplomat::LocalNode.new(path: "/home/user/myproject")
$server       = Deplomat::RemoteNode.new host: "myproject.com", port: 22, user: "deploy"
$branch       = ARGV[1] || $env
$app_name     = "myproject"
$project_dir  = "/var/www/#{$app_name}"
$release_dir  = Time.now.to_i

We've defined a bunch of global variables here. They're global to easily distinguish them from unimportant things, but they might as well have been regular local variables. We have also created a LocalNode object - it is used to run commands on the local machine; and a RemoteNode object, which, you guessed it, is used to run commands on a remote machine. You can theoretically have as many different remote nodes as you want, but in our example we'll just have one.

Ok, now let's write the actual deployment code.

# Create the release dir
$server.create_dir("#{$project_dir}/#$env/releases/#{$release_dir}")

# Upload to the remote dir
$server.upload("#{$local.current_path}/*", "#{$project_dir}/#$env/releases/#{$release_dir}/")

# cd to the release dir we've just created and uploaded, we'll do things inside it.
$server.cd("#{$project_dir}/#$env/releases/#{$release_dir}")

Here, we used standard Deplomat::RemoteNode methods. First, we used Deplomat::RemoteNode#create_dir which ran a "mkdir -p /home/user/myproject/staging/releases/[timestamp]" command on the server for us. Then we used the Deplomat::RemoteNode#upload which uploaded all the files from our project directory to the server. And, finally, we've changed current directory on the server to the one we've just created. So far so good, but we now need to do a few things on the server before we can restart our webapp:

$server.create_symlink("#{$project_dir}/#$env/shared/config/database.yml", "config/")
$server.create_symlink("#{$project_dir}/#$env/shared/config/secrets.yml", "config/")
$server.create_symlink("#{$project_dir}/#$env/shared/config/cable.yml", "config/")
$server.create_symlink("#{$project_dir}/#$env/shared/config/redis.yml", "config/")
$server.create_symlink("#{$project_dir}/#$env/shared/log", "./")
$server.create_symlink("#{$project_dir}/#$env/shared/public/uploads", "public/")

Here, we created symlinks to the files and directories that persist across deployments. For example, the files users upload should not evaporate with each new release and so we put them in the /var/www/myproject/staging/shared/public/uploads directory and then symlink them to the release directory.

For the final steps, we need to migrate the database, instruct the server to restart and symlink the release directory:

# Migrate DB. Our migration script requires a login shell to work properly,
# so we instruct deplomat to run it using a login shell.
$server.execute("bin/migrate_db #{$env}", login_shell: true)

# Restart the server
$server.execute("mkdir -p tmp")
$server.touch("tmp/restart.txt")

if $server.file_exists?("#{$project_dir}/#$env/current")
  $server.mv("#{$project_dir}/#$env/current", "#{$project_dir}/#$env/previous")
end
$server.create_symlink("#{$project_dir}/#$env/releases/#{$release_dir}", "#{$project_dir}/#$env/current")

You can see how we used #file_exists?, #touch, #mv methods here. Those are also standard methods of Deplomat::Node. Notice how we checked if "#{$project_dir}/#$env/current" exists first, because it might not exist on our first deployment. However if it does exist, it's wise to rename this symlink into previous so we can later undo the deployment easily by renaming that symlink back to current.

Our script is ready, now you can run ./deploy (assuming it's in the root dir of your project and you've made it executable).

Adding more structure with methods

Our script above is ok, however as your project grows you'll discover you'd want to add more structure to it. It makes sense to group some actions into methods, so you can have something like this:

#!/usr/bin/env ruby
require 'rubygems'
require 'deplomat'
require 'deployment_steps' # << THIS IS WHERE WE PUT OUR METHODS THAT WE USE BELOW

$env          = ARGV[0] || "staging"
$local        = Deplomat::LocalNode.new(path: "/home/user/myproject")
$server       = Deplomat::RemoteNode.new host: "myproject.com", port: 22, user: "deploy"
$branch       = ARGV[1] || $env
$app_name     = "myproject"
$project_dir  = "/var/www/#{$app_name}"
$release_dir  = Time.now.to_i

create_release_dir_and_upload!
create_symlinks!
migrate_db!
restart_server!

This script looks much nicer. We moved deployment code into separate methods into a file we called deployment_steps.rb and it looks like this:

def create_release_dir_and_upload!
  # Create the release dir
  $server.create_dir("#{$project_dir}/#$env/releases/#{$release_dir}")
  # Upload to the remote dir
  $server.upload("#{$local.current_path}/*", "#{$project_dir}/#$env/releases/#{$release_dir}/")
  # cd to the release dir we've just created and uploaded, we'll do things inside it.
  $server.cd("#{$project_dir}/#$env/releases/#{$release_dir}")
end

def create_symlinks!
  $server.create_symlink("#{$project_dir}/#$env/shared/config/database.yml", "config/")
  $server.create_symlink("#{$project_dir}/#$env/shared/config/secrets.yml", "config/")
  $server.create_symlink("#{$project_dir}/#$env/shared/config/cable.yml", "config/")
  $server.create_symlink("#{$project_dir}/#$env/shared/config/redis.yml", "config/")
  $server.create_symlink("#{$project_dir}/#$env/shared/log", "./")
  $server.create_symlink("#{$project_dir}/#$env/shared/public/uploads", "public/")
end

def migrate_db!
  # Migrate DB. Our migration script requires a login shell to work properly,
  # so we instruct deplomat to run it using a login shell.
  $server.execute("bin/migrate_db #{$env}", login_shell: true)
end

def restart_server!
  # Restart the server
  $server.execute("mkdir -p tmp")
  $server.touch("tmp/restart.txt")

  if $server.file_exists?("#{$project_dir}/#$env/current")
    $server.mv("#{$project_dir}/#$env/current", "#{$project_dir}/#$env/previous")
  end
  $server.create_symlink("#{$project_dir}/#$env/releases/#{$release_dir}", "#{$project_dir}/#$env/current")
end

Notice, we were able to use the same $server, $release_dir, $env and some other variables inside that file precisely because we made them global. While global vars are not great for large systems, deployment scripts such as this one can take advantage of them without complicating things too much.

Deployment requisites

Sometimes you need to run a piece of code while deploying the project, but you only need to run it once - that is, on one deployment after which it will never be run again. Much like Ruby On Rails runs migrations once and then only runs new migrations when necessary.

An example would be when you need to add something to the config file. While you can do it manually by logging into your server and editing the config file, it's much more desirable to automate that process, because then you won't forget to do it.

Deplomat has a special feature called Deployment requisites, which allow you to

Write special ruby scripts called "requisites", which have access to all the variables and features your script has access to;
Enumarate the tasks that are being run on each deployment and keep track of them by using a special requisite counter (a file created on the server);
Assign each task to be run before or after a particular deployment method in your script;

Let's see how we do that. The first step would be to replace method calls with a call to #add_task in your deployment script:

#!/usr/bin/env ruby
require 'rubygems'
require 'deplomat'
require 'deployment_steps' # << THIS IS WHERE WE PUT OUR METHODS THAT WE USE BELOW

$env          = ARGV[0] || "staging"
$local        = Deplomat::LocalNode.new(path: "/home/user/myproject")
$server       = Deplomat::RemoteNode.new host: "myproject.com", port: 22, user: "deploy"
$branch       = ARGV[1] || $env
$app_name     = "myproject"
$project_dir  = "/var/www/#{$app_name}"
$release_dir  = Time.now.to_i

add_task :create_release_dir_and_upload!
add_task :create_symlinks!
add_task :migrate_db!
add_task :restart_server!

execute_tasks!

This script's behavior is equivalent to the one we had previously. We can make it shorter though by writing:

# --- top part with requires and global var settings ommited ---

add_task :create_release_dir_and_upload!, :create_symlinks!, :migrate_db!, :restart_server!
execute_tasks!

We'll add two lines of code that will read the current requisite number from the server and then load the requisites from a local directory called ./deployment_requisites/. You can change the defaults by passing an additional path: argument to the load_requisites! method, but we're not going to do it here.

# --- top part with requires and global var settings ommited ---

# This method is kind of a callback, it's called automatically after every #before_task
# or #after_task call. We need to define it manually here,
# otherwise requisite number will be stuck on the same number
# and never updated.
def update_requisite_number!(n)
  $server.update_requisite_number!(n)
end

# read current requisite number from the server
req_n = $server.current_requisite_number

# load requisites
load_requisites!(req_n)

add_task :create_release_dir_and_upload!, :create_symlinks!, :migrate_db!, :restart_server!
execute_tasks!

The #execute_tasks! method will now not only run your methods, but also run the requisites associated with each task. Now you might ask, where's the actual code for the requisites? Let's create two files in the ./deployment_requisites/ dir on your local machine:

# ./deployment_requisites/1_add_var1_to_config_file.rb
before_task(:migrate_db, 1) do
  $server.execute("cat 'var1: true' >> config/secrets.yml")
end

# ./deployment_requisites/2_add_var2_to_config_file.rb
after_task(:create_release_dir_and_upload!, 2) do
  $server.execute("cat 'var2: true' >> config/secrets.yml")
end

Notice two things here:

Filenames start with a number. That's very important: each new requisite file should get a number that's larger than the previous number;
When calling before_task or after_task the second argument should always be the number that equals that consecutive number in the file name;
You can have only one call to before_task or after_task per requisite file.

When you deploy, that's what's going to happen:

Deplomat will check for the requisite number in the counter file at "#{@current_path}/.deployment_requisites_counter" (that's the default location, can be changed by passing an additional argument to Deplomat::Node#current_requisite_number and Deplomat::Node#update_requisite_number, see sources);
Use that fetched counter number to run only requisites with the numbers that are higher;
Update that number upon each requisite script completion (so if there's an error somehwere, we're still left at the exact requisite number that ran successfully). The update is done by calling the #update_requisite_number! we defined in our deployment script.

Where can I find the list of all methods?

For now, because this is a relatively small library, you're better off browsing the sources, specifically:

TODO

Deplomat::Node methods to read and update .yml files easily (usecase: update config files in requisite scripts).
Include git-archive-all script
Include scripts for default Ruby On Rails deployments (perhaps as a separate repo)