QPush

Fast and simple job queue microservice for Ruby. Please consider it under development at the moment.

QPush provides a scalable solution to your background job processing needs. Its Redis-backed, with support for forking and threading - letting it process an enormous amount of jobs in short order.

As a microservice, QPush is meant to be independent in its operation and deployment. This means that unlike other job processors such as Sidekiq, DelayedJob, etc - QPush does not hook into a web framework. Jobs must therefore be self-sufficent in their operation. This can often lead to better application designs - but also means QPush will have a minimal memory footprint.

Installation

Add this line to your application's Gemfile:

gem 'qpush'

And then execute:

$ bundle

Or install it yourself as:

$ gem install job_que

Usage

Before starting, ensure you have a functioning Redis server available.

The Server

In order to process queued jobs, we run the QPush server. This is a separate service beyond your web application (Rails, Sinatra, etc). To start the server simply type the following in your console.

$ bundle exec qpush-server -c path/to/config.rb

By providing a path to a configuration file, we can setup QPush with plain old ruby. At a minimum, we should provide details on our Redis server and connections. There are more configuration options available - all of which can be viewed here (to come).

# QPush server configuration
QPush.configure do |config|
  # Your redis server url and number of connections to provide
  config.redis_url = ENV['REDIS_URL']
  config.redis_pool = 10
end

Once the QPush server is running, it will begin processing any queued jobs based on priority.

The Client

Before we can add jobs to our server, we must first ensure our client has the same connection to our Redis server. We can setup our configuration in the same manner as above.

require 'qpush'

# QPush client configuration
QPush.configure do |config|
  # Your redis server url and number of connections to provide
  config.redis_url = ENV['REDIS_URL']
  config.redis_pool = 10
end

With our client setup, we can now queue jobs on our QPush server. All we have to do is:

QPush.job(klass: 'Example::Job', args: { example: 'Job' })

The job above would be equivalent to running the following command on the server.

Example::Job.new(example: 'Job').call

At a minimum, we must provide the job with a 'klass'. There are many more options that we can provide to the job though - all of which can be viewed here (to come).

Building Jobs

Jobs are simply plain old ruby objects that contain a 'call' method. If you provide a hash for the 'args' of the job, the job will be initialized with them. Below is an example of a simple mailing job utilizing the 'mail' gem.

require 'mail'

class MailJob
  def initialize(options = {})
    @mail = Mail.new(options)
  end

  def call
    @mail.deliver
  end
end

From our client, we could then queue a mail job with the following:

mail_options = { to: 'person@example.com', from: 'admin@test.com', subject: 'Hello!', body: 'From MailJob' }
QPush.job(klass: 'MailJob', args: mail_options)

Failed Jobs

Jobs that raise an error will be sent to the retry queue. As a default, they are set to attempt a maximum of 10 retries. Each failed attempt creates a longer delay for subsequent attempts. The job will permanently fail once the max retries has been hit.

Cron Jobs

QPush supports cron jobs. All you have to do is include a cron expression with your job. For example, the following would perform our job everyday at 4AM UTC.

QPush.job(klass: 'Example::Job', args: { example: 'Job' }, cron: '0 4 * * *')

Relational Databases

Although QPush is designed to independent in its operation, it still provides access to relational databases via Sequel. You can read more about how to use Sequel here. Suffice to say, its quite easy. We first will need to add the required information to our configuration:

# You must remember to require the gem for whatever database you will be using.
require 'pg'

# QPush database configuration
QPush.configure do |config|
  # Redis and additional config omitted
  # ....
  # ....
  # Your database server url and number of connections to provide
  config.database_url = ENV['DATABASE_URL']
  config.database_pool = 10
end

We can then access the database from any job. For example, we could retrieve all of our users with the following:

  QPush.db[:users].all

It is recommended that you read up on Sequel before use.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/job_que. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.