Roundhouse

Roundhouse is based on Sidekiq 3.5.0 (HEAD: f8ee0896076671b73739099e3c1eb7d8814e407d). It dispatches jobs in a very specialized way for a very narrow set of problems. For example, if you don't need to manage a large number of rate-limited access to an external resource, you probably want to use Sidekiq and not Roundhouse.

The problem Roundhouse solves resembles that of a load balancing problem: only one worker should be working on a work for a given user. Any other worker can work on any other user's work, as only as no other worker is working on it. We also want to do this fairly so that every user has a chance at getting work done steadily. This requires:

  1. A queue for each user. (Which often means a lot of queues)
  2. A turntable semaphore to control access to queues in a round-robin fashion
  3. A way to track the status of each queue (active, empty, suspended) and when to push a queue back into the turntable

Other things Roundhouse should be able to do:

  1. It should be able to run side-by side with Sidekiq client.
  2. Be able to suspend, resume, and defer work
  3. Metrics to measure lead times and give estimates on when work might be completed
  4. Coordinate across a number of worker processes, across machines

Roundhouse is in pre-alpha right now, designed for a specialized need we have. You probably don't want to try this in production.

Install

gem 'roundhouse-x', require: 'roundhouse'

Workers

class APIWorker
  include Roundhouse::Worker

  def perform(id)
    # something
  end
end

APIWorker.perform_async(api_token.id, item.id)

Why the name 'Roundhouse'?

The name 'Sidekiq' is obviously a play on the word, 'sidekick', both to mean a companion that helps a hero out in the background, and the name of a martial arts kicking technique. It is a great name for a background processing manager that works well with Rails apps.

When Roundhouse was first concieved, it was meant as a play on the word 'sidekick'. Since Roundhouse controls access to a queue in a round-robin fashion, I thought of a 'roundhouse kick'. Later, while implementing the code, I found out that a 'roundhouse' is an old term in the railway industry. In the old days, you needed a turntable to move trains around, or to reverse their direction. Although in modern times, 'Roundhouse' has come to mean the general facilities to maintain trains, this idea of a turntable at the center of a roundhouse is a powerful metaphor for the way Roundhouse dispatches work to background workers.

The structure that actually controls access is neither a counting or a binary semaphore. I had originally thought to call it a "queing semaphore' but that confuses a lot of people. The term 'turntable semaphore' better describes this.

Unfortunately, someone had written a Ruby gem several years ago called RoundhousE. I have no idea what that project does, other than that it has something to do with Ruby and .NET. So for now, this gem is available as roundhouse-x. (Yes, that is quite a creative name).

Sidekiq License

Please see LICENSE for licensing details.

Sidekiq Original Author

Mike Perham, @mperham / @sidekiq, http://www.mikeperham.com / http://www.contribsys.com