Jobby

Jobby is a small utility and library for managing running jobs in concurrent processes. It was initially developed for offloading long running tasks from the webserver in Rails applications, but it has proven to be useful in its own right and has been extracted to work in a general manner.

Download

Contact

What happens when you run Jobby?

Jobby can be thought of as a self-managing daemon and client program rolled into one. The first time you run Jobby, a daemon is started which listens for connections on a socket. The daemon will fork every time it receives a connection (up to –max-children, further requests are queued) and the forked child will run whatever is specified by –ruby or –command.

The same ‘jobby’ call also runs a client program to connect to the daemon, passing –input as a parameter which will be used by the child process. Subsequent calls to ‘jobby’ will use the existing daemon.

Usage

Jobby is a single ‘binary’ script that, can be run from the command line and is often used within scripts. You can, of course, also use the Jobby::Server and Jobby::Client classes directly in Ruby programs if you like.

This bash script attempts to illustrate some simple, if somewhat contrived, usage of processing a directory full of text files (yes, there are better tools for this particular example): 

#!/bin/bash
for i in *; do
  jobby --ruby 'File.rename("#{input}", "#{input}.jobby"' --max-children 4 --input $i
done

The above script runs the specified Ruby code on up to four processes in parallel. In this case, each file in the current directory will be renamed with ‘.jobby’ appended. Standard Ruby string interpolation is used to replace ‘#input’ with whatever is specified by –input. You may pass a filepath in place of a string as the –ruby parameter - in this case the specified file is simply loaded (using the Kernel module method ‘load’) by Ruby.

The above ‘jobby’ command is equivalent to:

jobby --command 'mv #{input} #{input}.jobby' --max-children 4 --input $i

The difference is that whatever is in the –command parameter is exec’d by the child process rather than interpretted as Ruby code. Again, Ruby string interpolation is used.

It is important to realise that although the –ruby ‘File.…’ parameter is passed every time jobby is called in this for loop, it is only actually read and used the first time. I’ll try to explain why below.

Log rotation

Issue a HUP signal to tell a Jobby process that the log file has been rotated. You need to do this for all Jobby processes.

Stopping Jobby

Sending a USR1 signal to the Jobby daemon process (jobbyd) will gracefully shutdown Jobby. The daemon process will stop accepting any more connections and close the socket. It will honour anything in the queue and wait until all the children have exited before exiting itself. In the meantime, you can start another Jobby daemon on the old socket.

Sending a TERM signal to the Jobby daemon process (jobbyd) will terminate the related Jobby processes (kill -15 the children, then exit the daemon).

Running multiple Jobby daemons

Since the –ruby and –command parameters are ignored (and the ones used for the first ‘jobby’ call are used), you may wonder how to specify different types of jobs to be run. The most straightforward way to do this is to specify a different socket for the Jobby daemon, using the –socket option. Then, the ‘jobby’ command will run whatever –ruby or –command is specified for the daemon running on that socket.

Copy-on-write ==

The Jobby::Server will enable the copy-on-write-friendly code when the interpretter supports it. You might not want this, but can disable it by passing calling GC.copy_on_write_friendly = true within a file that is –prerun.