Class: Sidekiq::Middleware::Server::RetryJobs

Inherits:
Object
  • Object
show all
Includes:
Util
Defined in:
lib/sidekiq/middleware/server/retry_jobs.rb

Overview

Automatically retry jobs that fail in Sidekiq. Sidekiq’s retry support assumes a typical development lifecycle:

  1. push some code changes with a bug in it

  2. bug causes message processing to fail, sidekiq’s middleware captures the message and pushes it onto a retry queue

  3. sidekiq retries messages in the retry queue multiple times with an exponential delay, the message continues to fail

  4. after a few days, a developer deploys a fix. the message is reprocessed successfully.

  5. if 3 never happens, sidekiq will eventually give up and throw the message away. If the worker defines a method called ‘retries_exhausted’, this will be called before throwing the message away. If the ‘retries_exhausted’ method throws an exception, it’s dropped and logged.

A message looks like:

{ 'class' => 'HardWorker', 'args' => [1, 2, 'foo'] }

The ‘retry’ option also accepts a number (in place of ‘true’):

{ 'class' => 'HardWorker', 'args' => [1, 2, 'foo'], 'retry' => 5 }

The job will be retried this number of times before giving up. (If simply ‘true’, Sidekiq retries 25 times)

We’ll add a bit more data to the message to support retries:

* 'queue' - the queue to use
* 'retry_count' - number of times we've retried so far.
* 'error_message' - the message from the exception
* 'error_class' - the exception class
* 'failed_at' - the first time it failed
* 'retried_at' - the last time it was retried

We don’t store the backtrace as that can add a lot of overhead to the message and everyone is using Airbrake, right?

Constant Summary collapse

DEFAULT_MAX_RETRY_ATTEMPTS =

delayed_job uses the same basic formula

25

Constants included from Util

Util::EXPIRY

Instance Method Summary collapse

Methods included from Util

#hostname, #logger, #process_id, #redis, #watchdog

Methods included from ExceptionHandler

#handle_exception

Instance Method Details

#call(worker, msg, queue) ⇒ Object 



49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 49

def call(worker, msg, queue)
  yield
rescue Exception => e
  raise e unless msg['retry']
  max_retry_attempts = retry_attempts_from(msg['retry'], DEFAULT_MAX_RETRY_ATTEMPTS)

  msg['queue'] = if msg['retry_queue']
    msg['retry_queue']
  else
    queue
  end
  msg['error_message'] = e.message
  msg['error_class'] = e.class.name
  count = if msg['retry_count']
    msg['retried_at'] = Time.now.utc
    msg['retry_count'] += 1
  else
    msg['failed_at'] = Time.now.utc
    msg['retry_count'] = 0
  end

  if msg['backtrace'] == true
    msg['error_backtrace'] = e.backtrace
  elsif msg['backtrace'].to_i != 0
    msg['error_backtrace'] = e.backtrace[0..msg['backtrace'].to_i]
  end

  if count < max_retry_attempts
    delay = seconds_to_delay(count)
    logger.debug { "Failure! Retry #{count} in #{delay} seconds" }
    retry_at = Time.now.to_f + delay
    payload = Sidekiq.dump_json(msg)
    Sidekiq.redis do |conn|
      conn.zadd('retry', retry_at.to_s, payload)
    end
  else
    # Goodbye dear message, you (re)tried your best I'm sure.
    retries_exhausted(worker, msg)
  end

  raise e
end

#retries_exhausted(worker, msg) ⇒ Object 



92
93
94
95
96
97
98
 
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 92

def retries_exhausted(worker, msg)
  logger.debug { "Dropping message after hitting the retry maximum: #{msg}" }
  worker.retries_exhausted(*msg['args']) if worker.respond_to?(:retries_exhausted)

rescue Exception => e
  handle_exception(e, "Error calling retries_exhausted")
end

#retry_attempts_from(msg_retry, default) ⇒ Object 



100
101
102
103
104
105
106
 
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 100

def retry_attempts_from(msg_retry, default)
  if msg_retry.is_a?(Fixnum)
    msg_retry
  else
    default
  end
end

#seconds_to_delay(count) ⇒ Object 



108
109
110
 
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 108

def seconds_to_delay(count)
  (count ** 4) + 15 + (rand(30)*(count+1))
end