Shoryuken
Shoryuken sho-ryu-ken is a super-efficient AWS SQS thread-based message processor.
Key features
Load balancing
Yeah, Shoryuken load balances the messages consumption!
Given this configuration:
concurrency: 25
delay: 25
queues:
- [high_priority, 6]
- [normal_priority, 2]
- [low_priority, 1]
And supposing all the queues are full of messages, the configuration above will make Shoryuken to process high_priority
3 times more than normal_priority
and 6 times more than low_priority
,
splitting the work load among all available processors concurrency: 25
.
If high_priority
gets empty, Shoryuken will keep using the 25 processors, but only to process normal_priority
and low_priority
.
If high_priority
receives a new message, Shoryuken will smoothly increase back its weight one by one until it reaches the weight of 6 again.
If a queue gets empty, Shoryuken will pause checking it for delay: 25
.
Fetch in batches
To be even more performant and cost effective, Shoryuken fetches SQS messages in batches, so a single SQS request can fetch up to 10 messages.
Requirements
Ruby 2.0 or greater.
Installation
Add this line to your application's Gemfile:
gem 'shoryuken'
Or to get the latest updates:
gem 'shoryuken', github: 'phstc/shoryuken', branch: 'master'
And then execute:
$ bundle
Or install it yourself as:
$ gem install shoryuken
Usage
Worker class
class MyWorker
include Shoryuken::Worker
queue: 'default', auto_delete: true
# shoryuken_options queue: ->{ "#{ENV['environment']}_default" }
# shoryuken_options body_parser: :json
# shoryuken_options body_parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
# shoryuken_options body_parser: JSON
def perform(sqs_msg, body)
puts body
end
end
Check the Worker options documention.
Sending a message
Check the Sending a message documentation
Middleware
class MyMiddleware
def call(worker_instance, queue, sqs_msg, body)
puts 'Before work'
yield
puts 'After work'
end
end
Check the Middleware documentation.
Shoryuken Configuration
Sample configuration file shoryuken.yml
.
concurrency: 25 # The number of allocated threads to process messages. Default 25
delay: 25 # The delay in seconds to pause a queue when it's empty. Default 0
queues:
- [high_priority, 6]
- [normal_priority, 2]
- [low_priority, 1]
AWS Configuration
Check the Configure AWS Client documentation
Rails Integration
Check the Rails Integration Active Job documention.
Start Shoryuken
bundle exec shoryuken -r worker.rb -C shoryuken.yml
For other options check bundle exec shoryuken help start
SQS commands
Check also some available SQS commands bundle exec shoryuken help sqs
, such as:
ls
list queuesmv
move messages from one queue to anotherdump
dump messages from a queue into a JSON lines filerequeue
requeue messages from a dump file
More Information
For more information on advanced topics such as signals (shutdown), ActiveJob integration, and so on please check the Shoryuken Wiki.
Credits
Mike Perham, creator of Sidekiq, and everybody who contributed to it. Shoryuken wouldn't exist as it is without those contributions.
Contributing
- Fork it ( https://github.com/phstc/shoryuken/fork )
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request