If you're viewing this at https://github.com/nehalamin93/dj_split, you're reading the documentation for the master branch. View documentation for the latest release (1.0.1).

Delayed Job Split Feature

Gem Version

Class "DjSplit::Split" is designed to Split Time Taking Delayed Jobs, Crons, Bulk Operations, etc into smaller size multiple Delayed Jobs. These Sub-Jobs should be mutually exclusive of each other and should be able to run concurrently.

Parallelism can be achieved across multiple servers through Delayed Jobs which can directly impact performance.

These Sub-Jobs can be picked by Delayed Job Workers within or across Multiple Servers.

Performance can improve up to n+1 times, where n = number of workers picking the jobs.

Class behaves like Delayed Job Worker, it also picks and processes delayed job entries.

Splitting will be done on one parameter only.

Purpose

To distribute the load among Multiple Workers and to improve Performance.

Installation

Add this line to your application's Gemfile:

gem 'dj_split'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install dj_split

Run the required database migrations:

$ script/rails generate dj_split
$ rake db:migrate

Usage

class A
  def self.function1(user_ids_array1, other_attr1)
    #do something
  end
end

  $ A.function1(user_ids_array1, other_attr1)

to:

  $ DjSplit::Split.new(queue_options: {queue: queue_name}, split_options: {size: 1000, by: 2}).enqueue(A, "function1", user_ids_array1, other_attr1)

Note

  • Arguments must be in exact order.
  • "enqueue" parameters are (object/class, function_name, arguments of that function)
  • split_options[:size] is splitting size

  • split_options[:by] is position of splitting attribute in the enqueue function. In above example, splitting attribute is user_ids_array1 and has position 2(>=2).

  • Here we are splitting on the basis of user_ids_array1.

  • We can also specify the: splitting_size, Otherwise it will take default Optimal Splitting Size

  • After splitting and enqueuing, instead of waiting for the sub-jobs to be complete this function will behave like delayed job worker and will pick and process the sub-jobs instead of blocking.

What DjSplit::Enqueue does

  • Split the mentioned parameter into array of chunks(arrays) of size = split_options[:size].
  • Loop through the array and insert each chunk into Delayed Job queue(array element is placed in place of splitting params)
  • Queue jobs will be picked and executed by workers
  • Instead of waiting for jobs to be picked and processed by workers this function will also pick and process those job

Contributing

  • Bug reports and pull requests are welcome on GitHub at https://github.com/nehalamin93/dj_split.
  • Fork it
  • Create your feature branch (git checkout -b my-new-feature)
  • Commit your changes (git commit -am 'Add some feature')
  • Push to the branch (git push origin my-new-feature)
  • Create new Pull Request

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the DjSplit project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.