Based on awesome gems like Sidekiq and Suckerpunch, DelayedJobCelluloid allows delayed job workers to be run in multiple threads within a single process using the Celluloid actor pattern. The delayed_job workers by default start a new single-threaded process for each worker. By running workers in a single multi-threaded process instead, delayed_job_celluloid is more efficient in terms of memory use and speed.
Add delayed_job_celluloid to your gem file
Run bundle install
bundle install delayed_job_celluloid
To add the startup script to your script directory, run the generator
rails generate delayed_job_celluloid
First, make sure you have your preferred delayed job backend installed, for instance delayed_job_activerecord, or delayed_job_mongoid. See delayed_job for more information.
To start working off of your delayed_job queues simply run the below from your application's root directory
bundle exec script/delayed_job_celluloid
To specify the number of worker threads to start, pass the -n parameter to the startup script. For example, the below command would start 5 worker threads. The default is 2.
bundle exec script/delayed_job_celluloid -n 5
One important thing to bear in mind with this is that you should have a database connection pool size that is at least equal to the number of worker threads you are running due to the fact that each thread will need its own connection. if there are not enough connections available you will get database errors. You can set this in the database.yml file in your app.
development: adapter: postgresql database: dev host: localhost pool: 5
The gem also does not work particularly well with Sqlite since Sqlite does not handle concurrent reads/writes. You will see locking errors, although delayed job handles these gracefully by retrying the job later.
script/delayed_job_celluloid -d start script/delayed_job_celluloid -d stop
You can also start the daemon with a monitor process that will restart the daemon if it crashes
script/delayed_job_celluloid -d -m start
If you are running your app on Heroku, this gem will allow you to run multiple workers in a single Unicorn process. An example unicorn config is as follows:
# config/unicorn.rb worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2) timeout 15 preload_app true before_fork do |server, worker| #Start the Delayed Job worker inside of a Unicorn process @delayed_job_celluloid_pid ||= spawn("bundle exec script/delayed_job_celluloid -n 5") Signal.trap 'TERM' do puts 'Unicorn master intercepting TERM and sending myself QUIT instead' Process.kill 'QUIT', Process.pid end defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect! end after_fork do |server, worker| Signal.trap 'TERM' do puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT' end defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection end
- Fork it
- Create your feature branch (
git checkout -b my-new-feature)
- Commit your changes (
git commit -am 'Add some feature')
- Push to the branch (
git push origin my-new-feature)
- Create new Pull Request