kthxbye

Delayed job processing with async job completion and results notificiation*

* depends on Node.js and Socket.io

Quick Links

code: http://github.com/plukevdh/kthxbye

docs: http://rdoc.info/github/plukevdh/kthxbye/

gem: https://rubygems.org/gems/kthxbye

issues: http://github.com/plukevdh/kthxbye/issues

History

Kthxbye is the answer to a fairly unique-yet-common problem: Background job processing when we care about the result.

There are a number of projects I can think of where a job that takes longer than a user ought to be waiting for a result (due to server timeout length or out of simple app responsiveness courtesy to the user) and yet I need the result of the operation to be returned to the user.

Here's a real-world example. I work with a set of legacy Oracle databases that stores much of our business logic as PLSQL procedures. Yes, this is not "The Rails Way ™" but it's the only way for the company I work for right now. Many of the procedures that I run as part of several of the applications I support can take on average one minute or more with a standard deviation of almost 2 minutes (with a forced timeout of 5 minutes). That's kinda a long time to sit and wait on a web app.

Stats

We don't really want users sitting waiting for up to 5 minutes (when it forces failure) unable to do anything or (even worse) hitting refresh or the action again. Especially bad when this can mean the HTTP server is getting backed up as more and more people run these long running processes.

Moreover, the users need to get response from the completed job before moving on. Most job processors (DJ, Resque) are setup for running jobs that do not require the result to be returned to the web app (think mass-mailers, queue population, image resizing). They just run and the output goes to a database, an inbox or a file server.

Enter Kthxbye

Kthxbye is an attempt to solve this problem. It is based heavily off of Resque and why not an addition to Resque? I needed some hack time with Redis on my own as I've never used it before...

I can learn any language or tool in a matter of days if you give me

  1. a good manual
  2. an even better project to work on.

-Prof. Shumacher

This project accomplishes both those goals. This is an attempt to learn something, using Resque and Redis docs as a manual, while at the same time creating a much needed solution to a problem.

The idea is to be able to do the following:

# dummy job class
class MyJob
  def self.perform(data)
    puts "Do something with #{data}"
    data.gsub(/hello/i, "Goodbye")
  end
end

# setup options, then connect
Kthxbye::Config.setup(:redis_server => 'localhost', :redis_port => 8080)

# each enqueued job returns a unique id to poll with
unique_id = Kthxbye.enqueue("jobs", MyJob, "Hello World")

# ... code code code ...

# polls queue every 5 seconds
computed_value = Kthxbye.poll("jobs", unique_id, 5)

and then in some other world, on some other machine, (that still has knowledge of MyJob)

# inits with queue
worker = Kthxbye::Worker.new("jobs")

# connects to queue and runs jobs found there
worker.run

Pretty... damn... simple.™

Installation

Installation isn't hard. Simply:

gem install kthxbye

or in Rails

gem "kthxbye"

and then in your project directory (if using Rails) run

rails g kthxbye

which will install all of the necessary assets, of which there are three:

  1. public/javascripts/kthxbye.js - a responder that will catch job status message from the worker
  2. public/images/kthxbye_widget.png - a notifier widget to show the status of a job
  3. public/stylesheets/kthxbye.css - styles for the widget

These can be ignored if you really don't want to have an asynchronus widget to display the job's status to the user in real-time. You can simply use the gem and get the results on your own time.

If are using the widget however, we depend on node.js to handle the Redis PUBSUB listening and updating and Socket.io to stream the results back to the clients.

In order to install Socket.io on your application side, run

git clone http://github.com/LearnBoost/Socket.IO.git socket-io --recursive

Note: due to the way Rails tries to serve these files and the seemingly conflicting way that Socket.io has hardcoded some paths into the code, you may need to create a symlink to the public/javascripts/socket.io directory in you public directory. This was the easiest solution I found.

There is a separate node.js client at http://github.com/plukevdh/kthxbye-node. Simply clone the repo or download the poll.js file and set it up to your hearts content.

NOTE: You will need to configure your ports/redis server in the poll.js file as well as installing socket.io in the same dir as the poll.js file via git clone git://github.com/LearnBoost/Socket.IO-node.git socket.io-node --recursive (from the socket.io install instructions http://github.com/LearnBoost/Socket.IO-node)

Troubleshooting

The part of the show where I try to save you some trouble

There are a few things I found along the way that were necessary to make this integration run super smoothly:

You will need to run node as root user if you want to take advantage of flashsockets.

You may need to create a link in your public/ dir to socket.io to the install in your public/javascripts/ dir.

Want to Help? Rock on.

  • Fork.
  • Add or fix stuff.
  • Prove it works with everything else.
  • Commit changes.
  • Send me a pull request!
  • Glory in the fact that your name is now in the annals of open-sourcedom.
  • Repeat.

Copyright

Copyright © 2010 Luke van der Hoeven. See LICENSE for details.