Blender-Serf

Provides Serf based host discovery and job dispatch mechanism for Blender

Installation

  gem install blender-serf

Usage

Config

Blender-serf uses Serf's RPC interface for communication between serf and ruby. Following serf RPC connection specific configuration options are allowed

  • host (defaults to localhost)
  • port (defaults to 7373)
  • authkey (rpc authenticatioin key, default is nil)

Example

config(:serf, host: '127.0.0.1', port: 7373, authkey: 'foobar')

Using serf for host discovery

Serf provides cluster membership list via members and members_filtered RPC calls, among them Blender-Serf uses members_filtered RPC command for host discovery. Following is an example of obtaining all live members of a cluster

  require 'blender/serf'
  members(search(:serf, status: 'alive'))
  ruby_task 'test' do
    execute do |host|
      Blender::Log.info(host)
    end
  end

Member list can be obtained based on serf tags or name.

Filter by tag expect tags to be supplied as hash (key value pair), which allows grouping hosts by multiple tags

  members(search(:serf, tags: {'role'=>'db'}))
  ruby_task 'test' do
    execute do |host|
      Blender::Log.info(host)
    end
  end

Filter by name, the specification string can be a pattern as well

  members(search(:serf, name: 'foo-baar-*'))
  ruby_task 'test' do
    execute do |host|
      Blender::Log.info(host)
    end
  end

Combinations of all three filtering mechanism is valid as well

  members(search(:serf, name: 'foo-baar-*', status: 'alive', tags:{'datacenter' => 'eu-east-2a'}))
  ruby_task 'test'
    execute do |host|
      Blender::Log.info(host)
    end
  end

Using serf for job dispatch

Serf agents allow job execution via event handlers, where the execution details is captured by the handler, while trigger mechanism is controlled by serf event. Blender-Serf uses serf query event type to dispatch event and check for successfull response. The handler itself, and associated serf' configyration needs to be setup externally (bake it in your image, or use a configuration managemenet system like chef, puppet, ansible or salt for this).

Following example will trigger serf query event test against 3 nodes, one by one

  extend Blender::SerfDSL
  members(['node_1', 'node_2', 'node_3'])
  serf_task 'test'

A more elaborate example

  extend Blender::SerfDSL
  members(['node_1', 'node_2', 'node_3'])
  serf_task 'start_nginx' do
    query 'nginx'
    payload 'start'
    timeout 3
  end

Which might be accmoplished by the following handler script (needs to be present aprior)

require 'serfx/utils/handler'

include Serfx::Utils::Handler


on :query, 'nginx' do |event|
  case event.payload
  when 'start'
    %x{/etc/init.d/nginx start}
  when 'stop'
  # ...
  when 'check'
  # ...
  end
end

run

Async task DSL:

Since serf event handler execution happens synchronously (i.e. normal serf events wont be processed when an event handler is running). Hence, it is desirable that the actual event handler returns promptly. This imposes certain design challenges in orchestrating long running commands via serf.

blender-serf integration provides a simpler solution to this problem. It recommends using the serf event handler only to fork off the long running command, and maintain a state file locally that can be used to decide whether the original task is running or finished. Most of these gears are offered by Serfx::Utils::AsynJob module. On blender side serf_async_task method is used to manage this long running job, using three separate serf events (one to start the command, another one to poll and check if its finished or not, and the last one to reap the finished command).

Following is an example handler that runs apt update. It exposes the task management as 4 distinct events, start, check, kill, reap.

require 'serfx/utils/async_job'
require 'serfx/utils/handler'
require 'json'

include Serfx::Utils::Handler

job = Serfx::Utils::AsyncJob.new( name: 'update', command: 'sudo apt-get update -y', state: '/opt/serf/states/chef')

on :query, 'apt-update' do |event|
  case event.payload
  when 'start'
    status = job.start
    code = (status == 'success') ? 0 : -1
    puts JSON.dump(code: code, result: { status: status })
  when 'kill'
    status = job.kill
    code = (status == 'success') ? 0 : -1
    puts JSON.dump(code: code, result: { status: status })
  when 'reap'
    status = job.reap
    code = (status == 'success') ? 0 : -1
    puts JSON.dump(code: code, result: { status: status })
  when 'check'
    state = job.stateinfo
    puts JSON.dump(code: 0, result: state)
  else
    puts JSON.dump(code: -1, result: { status: 'failed' })
  end
end

run

This long running command can be orchestrated using the serf_async_task like this:

serf_async_task 'run apt-get update' do

  start do
    query 'apt-update'
    payload 'start'
  end

  check do
    query 'apt-update'
    payload 'check'
    process do |responses|
      responses.all? do |resp|
        status = JSON.parse(resp['Payload'])['result']['status']
        fail 'Apt-get update is running' if status == 'running'
      end
    end
  end

  reap do
    query 'apt-update'
    payload 'reap'
  end
end

Which can also be reduced to this, as job named are default query names and serf_async_task defaults to start, check and reap as payload for managing the life cycle of the command.

serf_async_task 'apt-update' do
  check do
    process do |responses|
      responses.all? do |resp|
        status = JSON.parse(resp['Payload'])['result']['status']
        fail ChefRunning if status == 'running'
      end
    end
  end
end

Supported ruby versions

Blender-serf currently support the following MRI versions:

  • Ruby 1.9.3
  • Ruby 2.1

License

Apache 2

Contributing

  1. Fork it ( https://github.com/PagerDuty/blender-serf/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request