resque-cluster

resque-cluster allows managing resque workers distributed across multiple hosts. It allows user to set the maximum number of workers to be running per host as well as the maximum number of workers running per cluster and maintains correct counts via a shared Redis server.

resque-cluster is based on resque-pool gem and will still run in a non clustered mode if, for example, you are anticipating expanding across multiple hosts at a later time.

Installation

gem install resque-cluster

Or, more likely, add it to the Gemfile used by your resque workers

gem resque-cluster

Usage

bundle exec resque-cluster_member <options>

Options

-c, --config=<s>              Path to the config(local/host maximums config in a cluster mode) 
                                YAML file. Always required
-a, --appname=<s>             Application name
-d, --daemon                  Run as a background daemon
-o, --stdout=<s>              Redirect stdout to logfile
-e, --stderr=<s>              Redirect stderr to logfile
-n, --nosync                  Don't sync logfiles on every write
-p, --pidfile=<s>             PID file location
-E, --environment=<s>         Set RAILS_ENV/RACK_ENV/RESQUE_ENV(in a cluster mode used for
                                namespacing the cluster)
-s, --spawn-delay=<i>         Delay in milliseconds between spawning missing workers
-t, --term-graceful-wait      On TERM signal, wait for workers to shut down gracefully
-r, --term-graceful           On TERM signal, shut down workers gracefully
-m, --term-immediate          On TERM signal, shut down workers immediately (default)
-i, --single-process-group    Workers remain in the same process group as the master
-C, --cluster=<s>             Name of the cluster this resque-pool belongs to. Required 
                                if you want to launch in a cluster mode
-G, --global-config=<s>       Path to the global config file(global/cluster maximums
                                config in a cluster mode)
-v, --version                 Print version and exit

Local/Host Config

When run without –cluster option the host config creates precisely the number of workers specified in the config. When run with –cluster option set host config specifies the maximum number of workers that can be spun up on the machine

Host config follows the following format:

foo: 1
bar: 9
"foo,bar,baz": 1

Global/Cluster Config

Cluster config isn’t used by a non cluster mode and the param is ignored. In a cluster mode, cluster config defines the maximum number of workers that can run across the whole cluster. It also defines the cluster rebalancing mode. When set to true, workers will try to balance the load across all cluster members for each worker.

Cluster config follows the following format:

global_maximums:
  foo: 1
  bar: 9
  "foo,bar,baz": 1
rebalance_cluster: true

Examples

When trying to launch without a cluster you could use a command like this:

bundle exec resque-cluster_member -c config.yml -E production -o /var/log/resque-cluster.stdout.log

When trying to launch with a cluster you could use a command like this:

bundle exec resque-cluster_member -c host_maximums.yml -E production -C prod-cluster -G cluster_config.yml -o /var/log/resque-cluster.stdout.log

Contributing to resque-cluster

Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet.
Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it.
Fork the project.
Start a feature/bugfix branch.
Commit and push until you are happy with your contribution.
Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.