resque-state
<img src=“https://img.shields.io/badge/license-MIT-blue.svg” /> <img src=“https://img.shields.io/gem/v/resque-state.svg” /
<img src=“https://codeclimate.com/github/nathan-v/resque-state/badges/gpa.svg” /> <img src=“https://codeclimate.com/github/nathan-v/resque-state/badges/issue_count.svg” /> <img src=“https://codeclimate.com/github/nathan-v/resque-state/badges/coverage.svg” /> <img src=“https://travis-ci.org/nathan-v/resque-state.svg?branch=master” alt=“Build Status” /> <img src=“https://gemnasium.com/badges/github.com/nathan-v/resque-state.svg” /> <img src=“https://img.shields.io/github/issues/nathan-v/resque-state.svg” />
resque-state is an extension to the resque queue system that provides simple trackable jobs.
About
resque-state provides a set of simple classes that extend resque’s default functionality (with 0% monkey patching) to give apps a way to track specific job instances and their state. It achieves this by giving job instances UUID’s and allowing the job instances to report their state from within their iterations.
Installation
Ruby 2.2.2+ and JRuby 9.1+ are supported.
resque-state requires Redis >= 1.1 (though I recommend getting the latest stable version). You can download Redis here: redis.io/ or install it using homebrew (brew install redis).
Install the resque-state gem (which will pull in the dependencies).
gem install resque-state
With newer Rails add this to your Gemfile:
# Gemfile
gem 'resque-state'
Then in an initializer:
# config/initializers/resque.rb
Resque.redis = "your/redis/socket" # default localhost:6379
Resque::Plugins::State::Hash.expire_in = (24 * 60 * 60) # 24hrs in seconds
Usage
The most direct way to use resque-state is to create your jobs using the Resque::Plugins::State module. An example job would look something like:
class SleepJob
include Resque::Plugins::State
def perform
total = (['length'] || 1000).to_i
total.times do |i|
num = i+1
at(num, total, "At #{num} of #{total}")
sleep(1)
end
end
end
One major difference is that instead of implementing perform
as a class method, we do our job implementation within instances of the job class.
In order to queue a SleepJob up, we also won’t use Resque.enqueue
, instead we’ll use the create
class method which will wrap enqueue
and creating a unique id (UUID) for us to track the job with.
job_id = SleepJob.create(length: 100)
This will create a UUID enqueue the job and pass the :length option on the SleepJob instance as options (as you can see above).
Now that we have a UUID its really easy to get the state:
state = Resque::Plugins::State::Hash.get(job_id)
This returns a Resque::Plugins::State::Hash object, which is a Hash (with benefits).
state.pct_complete #=> 0
state.status #=> 'queued'
state.queued? #=> true
state.working? #=> false
state.time #=> Time object
state. #=> "Created at ..."
Once the worker reserves the job, the instance of SleepJob updates the state at each iteration using at()
state = Resque::Plugins::State::Hash.get(job_id)
state.working? #=> true
state.num #=> 5
state.total #=> 100
state.pct_complete #=> 5
If an error occurs within the job instance, the state is set to ‘failed’ and then the error is re-raised so that Resque can capture it.
Its also possible to get a list of current/recent job statuses:
Resque::Plugins::State::Hash.statuses #=> [#<Resque::Plugins::State::Hash>, ...]
Passing back data from the job
You may want to save data from inside the job to access it from outside the job.
A common use-case is web-triggered jobs that create files, later available for download by the user.
A Status is actually just a hash, so inside a job you can do:
set_status(filename: "myfilename")
Also, all the status setting methods take any number of hash arguments. So you could do:
completed('filename' => '/myfilename')
Kill! Kill! Kill!
Because we’re tracking UUIDs per instance, and we’re checking in/updating the status on each iteration (using at
or tick
) we can kill specific jobs by UUID.
Resque::Plugins::State::Hash.kill(job_id)
The next time the job at job_id calls at
or tick
, it will raise a Killed
error and set the status to killed.
Hold up: Pausing
Since we perhaps might want to just have a job sit and wait a bit rather than have it die completely there’s pause. This tells the job to sleep for 10 seconds before checking in again.
Resque::Plugins::State::Hash.pause(job_id)
The next time the job at job_id calls at
or tick
, it will start a while loop with a 10 second sleep until Resque::Plugins::State::Hash.unpause is called.
As of 1.1 pause! can be called from the job itself with an optional string argument that allows you to set a specific status along with the paused state if you’d like.
Back it up
Perhaps you want a job to be able to undo what it did. Well; that’s what revert is all about. Jobs that support on_revert can be told to revert and will run insructions in that method to undo whatever they might need to. While you could do similar things with on_failure on_revert was added to provide a separate status for reverting and reverted cases so you can see later which jobs failed (probably due to an error) or which ones might have been intentionally reverted by a user or automation.
Resque::Plugins::State::Hash.revert(job_id)
This is all you need to tell that job to go back and undo it’s work. For jobs that do not support on_revert those jobs will pause themeslves and leave a note to the user that the job doesn’t allow that functionality. This lets the user decide if the job should be killed or let continue.
Percent Complete and setting the message
Use at
or tick
to show progress in your job’s perform
function (which is displayed on the resque-web state tab). This will also be where Killed
is raised if the job is killed.
at(steps_completed, total_steps, "${steps_completed} of #{total_steps} steps completed!")
Expiration
Since Redis is RAM based, we probably don’t want to keep these statuses around forever (at least until @antirez releases the VM feature). By setting expire_in, all statuses and their related keys will expire in expire_in seconds from the last time theyre updated:
Resque::Plugins::State::Hash.expire_in = (60 * 60) # 1 hour
Testing
Recent versions of Resque introduced Resque.inline
which changes the behavior to instead of enqueueing and performing jobs to just executing them inline. In Resque itself this removes the dependency on a Redis, however, Resque::State
uses Redis to store information about jobs, so though inline
“works”, you will still need to use or mock a redis connection. You should be able to use a library like github.com/causes/mock_redis alongside inline
if you really want to avoid Redis connections in your test.
resque-web
Though the main purpose of these trackable jobs is to allow you to surface the status of user created jobs through your apps’ own UI, I’ve added a simple example UI as a plugin to resque-web. This adds a State tab to resque-web
To use, you need to setup a resque-web config file:
# ~/resque_conf.rb
require 'resque/state_server'
Then start resque-web with your config:
resque-web ~/resque_conf.rb
If you’re using Rails you can just require ‘resque/state_server’ in your resque initializer or application.rb.
More
-
Source: github.com/nathan-v/resque-state
-
API Docs: rdoc.info/projects/nathan-v/resque-state
-
Examples: github.com/nathan-v/resque-state/tree/master/examples
-
Resque: github.com/resque/resque
Thanks
Resque is awesome, @defunkt needs a shout-out.
Note on Patches/Pull Requests
PRs are welcome. Please always include relevant tests and ensure your changes follow Ruby style conventions as much as possible.
Copyright
Copyright © 2016 Nathan V.
Copyright © 2010 Aaron Quint.
MIT licensed. See LICENSE file for details.