sidekiq-bulk

Give your workers more to do!

Augments Sidekiq job classes with a push_bulk method for easier bulk pushing.

Sidekiq comes with Sidekiq::Client.push_bulk which can be faster than perform_async if you have lots and lots of jobs to enqueue.

This gem provides a wrapper around Sidekiq::Client.push_bulk so that instead of

Sidekiq::Client.push_bulk("class" => FooJob, "args" => [[1], [2], [3]])

You can write

FooJob.push_bulk([1, 2, 3])

More stuff is supported!

Installing

With Bundler:

# in Gemfile
gem "sidekiq-bulk"

Either require "sidekiq-bulk" or require "sidekiq/bulk".

To use

To enqueue a job for each element of an array:

# enqueues 3 jobs for FooJob, each with 1 argument
FooJob.push_bulk([1, 2, 3])

# equivalent to:
Sidekiq::Client.push_bulk("class" => FooJob, "args" => [[1], [2], [3]])

# which is a more efficient version of
[1, 2, 3].each do |i|
  FooJob.perform_async(i)
end

To enqueue jobs with more than one argument:

FooJob.push_bulk(all_users) do |user|
  [user.id, "foobar"]
end

# enqueues one job for each element of `all_users`, where each job
# has two arguments: the user ID and the string "foobar".
#
# equivalent to, but faster than:

all_users.each do |user|
  FooJob.perform_async(user.id, "foobar")
end

Job count splitting

push_bulk will only enqueue at most 10,000 jobs at a time. That is, if items has 20,000 elements, push_bulk(items) will push the first 10,000, then the second 20,000. You can control the threshold with limit:.

# push in groups of 50,000 jobs at a time
FooJob.push_bulk(items, limit: 50_000)

# equivalent to FooJob.push_bulk(items, limit: 10_000)
FooJob.push_bulk(items)

This also works with a block.

# this results in 5 pushes

users.length # => 100_000
FooJob.push_bulk(users, limit: 20_000) do |user|
  [user.id, "some-value"]
end

And to disable push splitting, use push_bulk!.

# one single push of 500,000 jobs, no splitting

FooJob.push_bulk!(users) do |user|
  [user.id, "some-value"]
end

License

Development

Issues (bugs, questions, etc.) should be opened with the GitHub project.

To contribute changes:

Visit the GitHub repository for sidekiq-bulk.
Fork the repository.
Make new feature branch: git checkout -b master new-feature (do not add on top of master!)
Implement the feature, along with tests.
Send a pull request.

Make sure to bundle install.

Tests live in spec/. Run them with bundle exec rspec.

To run tests against various Sidekiq versions, use appraisal rspec, after appraisal bundle if necessary. (See the Appraisal project and the Appraisals file for more details.)