Execache

Run commands in parallel and cache the output. Redis queues jobs and stores the result.

Requirements

gem install execache

How Your Binaries Should Behave

Execache assumes that the script or binary you are executing has multiple results and sometimes multiple groups of results.

Example output:

$ bin/some/binary preliminary_arg arg1a arg1b arg2a arg2b
$ arg1_result_1
$ arg1_result_2
$ [END]
$ arg2_result_1
$ arg2_result_2

Your binary may take zero or more preliminary arguments (e.g. preliminary_arg), followed by argument "groups" that dictate output (e.g. arg1a arg1b).

Configure

Given the above example, our execache.yml looks like this:

redis: localhost:6379/0
parallel: 3
some_binary:
  command: '/bin/some/binary'
  separators:
    result: "\n"
    group: "[END]"

Start the Server

$ execache /path/to/execache.yml

Execute Commands

require 'rubygems'
require 'execache'

client = Execache::Client.new("localhost:6379/0")

results = client.exec(
  :ttl => 60,
  :some_binary => {
    :args => 'preliminary_arg',
    :groups => [
      'arg1a arg1b',
      'arg2a arg2b'
    ]
  }
)

results == {
  :some_binary => [
    [ 'arg1_result_1', 'arg1_result_2' ],
    [ 'arg2_result_1', 'arg2_result_2' ]
  ]
}