philiprehberger-retry_queue
Batch processor with per-item retry, backoff, and dead letter collection
Requirements
- Ruby >= 3.1
Installation
Add to your Gemfile:
“by gem “philiprehberger-retry_queue”
“
Or install directly:
“sh gem install philiprehberger-retry_queue
“
Usage
“by require “philiprehberger/retry_queue”
result = Philiprehberger::RetryQueue.process(items, max_retries: 3) do |item| process_item(item) end
puts result.succeeded.size # => number of successful items puts result.failed.size # => number of failed items
“
Custom Backoff
“by result = Philiprehberger::RetryQueue.process(items, max_retries: 5, backoff: ->(n) { n * 0.5 }) do |item| external_api_call(item) end
“
Jitter
Reduce thundering-herd risk by randomizing the backoff delay. Pass a fraction in 0.0..1.0; the computed delay is multiplied by 1 + rand * jitter. A value of 0.0 (default) disables jitter.
“by result = Philiprehberger::RetryQueue.process(items, max_retries: 3, jitter: 0.3) do |item| external_api_call(item) end
“
Values outside 0.0..1.0 or non-Numeric values raise ArgumentError.
Selective Retry
“by result = Philiprehberger::RetryQueue.process(items, max_retries: 3, retry_on: [Net::OpenTimeout, Timeout::Error]) do |item| api_call(item) end
Only Net::OpenTimeout and Timeout::Error trigger retries
All other errors send the item straight to failed
“
Retry Hooks
“by logger_hook = ->(item, error, attempt) { puts “Retrying #item: #errorerror.message (attempt #attempt)” } metrics_hook = ->(item, error,attempt) { increment_counter(“retry.#item”) }
result = Philiprehberger::RetryQueue.process(items, max_retries: 3, on_retry: [logger_hook, metrics_hook]) do |item| process_item(item) end
“
Dead-letter Notifications
“by on_failure = ->(item, error) { Rails.logger.error(“Dead-lettered #item: #errorerror.message”) }
result = Philiprehberger::RetryQueue.process(items, max_retries: 3, on_failure: on_failure) do |item| process_item(item) end
“
The hook fires once per item that exhausts its retries, just as the item is recorded in Result#failed. Exceptions raised inside the hook are swallowed so a faulty callback cannot break the queue.
DLQ Reprocessing
“by result = Philiprehberger::RetryQueue.process(jobs, max_retries: 2) do |job| job.execute! end
reprocessed = result.reprocess_failed do |item, error| fallback_handler(item, error) end
puts reprocessed.succeeded.size # => items recovered during reprocessing puts reprocessed.failed.size # => items that failed reprocessing too
“
Statistics
“by result = Philiprehberger::RetryQueue.process(records, max_retries: 3) do |record| save(record) end
stats = result.stats
=>
“
Success rate
“by result = Philiprehberger::RetryQueue.process(items, max_retries: 3) { |item| call(item) } result.success_rate # => 0.92 result.failure_rate # => 0.08
“
Result#failure_rate is the counterpart to #success_rate and pairs naturally with it — for any non-empty Result, success_rate + failure_rate sums to 1.0. Empty batches return 0.0 for both.
API
| max_retries: | Integer >= 0. 0 means one attempt with no retries (not zero attempts) | | jitter: | Numeric in 0.0..1.0; multiplies backoff delay by 1 + rand * jitter. Defaults to 0.0 | | on_failure: | Callable (item, error) invoked once per item that exhausts retries; hook errors are swallowed | | Result#succeeded | Array of successfully processed items | | Result#failed | Array of hashes with :item, :error, :attempts | | Result#stats | Hash with :total, :succeeded, :failed, :success_rate, :elapsed | | Result#success_rate | Float in [0.0, 1.0]; ratio of succeeded to total items (0.0 for empty batches) | | Result#failure_rate | Ratio of failed items to total (0.0..1.0); pairs with success_rate | | Result#empty? | true when stats[:total] is 0 | | Result#size | Returns stats[:total] (succeeded + failed count) | | Result#reprocess_failed { \|item, error\| } | Reprocess failed items, returns a new Result |
Development
“sh bundle install bundle exec rspec bundle exec rubocop
“
Support
If you find this project useful: