Module: Resque::Plugins::Retry
- Includes:
- Logging
- Included in:
- ExponentialBackoff
- Defined in:
- lib/resque/plugins/retry.rb,
lib/resque/plugins/retry/logging.rb
Overview
If you want your job to retry on failure, simply extend your module/class with this module:
class DeliverWebHook
extend Resque::Plugins::Retry # allows 1 retry by default.
@queue = :web_hooks
def self.perform(url, hook_id, hmac_key)
heavy_lifting
end
end
Easily do something custom:
class DeliverWebHook
extend Resque::Plugins::Retry
@queue = :web_hooks
@retry_limit = 8 # default: 1
@retry_delay = 60 # default: 0
# used to build redis key, for counting job attempts.
def self.retry_identifier(url, hook_id, hmac_key)
"#{url}-#{hook_id}"
end
def self.perform(url, hook_id, hmac_key)
heavy_lifting
end
end
Defined Under Namespace
Modules: Logging Classes: AmbiguousRetryStrategyException
Instance Attribute Summary collapse
-
#expire_retry_key_after ⇒ Number
readonly
abstract
The number of seconds to set the TTL to on the resque-retry key in redis.
-
#fatal_exceptions ⇒ Array?
readonly
abstract
Controls what exceptions may not be retried.
Class Method Summary collapse
-
.extended(receiver) ⇒ Object
private
Fail fast, when extended, if the “receiver” is misconfigured.
Instance Method Summary collapse
-
#after_perform_retry(*args) ⇒ Object
private
Resque after_perform hook.
-
#before_perform_retry(*args) ⇒ Object
private
Resque before_perform hook.
-
#call_symbol_or_block(method, *args) ⇒ Object
private
Helper to call functions that may be passed as Symbols or Procs.
-
#clean_retry_key(*args) ⇒ Object
private
Clean up retry state from redis once done.
-
#give_up(exception, *args) ⇒ Object
private
We failed and we’re not retrying.
-
#give_up_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a give up callback that will be called when the job fails and is not retrying.
-
#give_up_callbacks ⇒ Array<Proc>
Returns the give up callbacks.
-
#inherited(subclass) ⇒ Object
private
Copy retry criteria checks, try again callbacks, and give up callbacks on inheritance.
-
#instance_exec(*args, &block) ⇒ Object
private
Used to perform retry criteria check blocks under the job instance’s context.
-
#on_failure_retry(exception, *args) ⇒ Object
private
Resque on_failure hook.
-
#redis_retry_key(*args) ⇒ String
Builds the redis key to be used for keeping state of the job attempts.
-
#retry_args(*args) ⇒ Array
abstract
Modify the arguments used to retry the job.
-
#retry_args_for_exception(exception, *args) ⇒ Array
abstract
Modify the arguments used to retry the job based on the exception.
-
#retry_attempt ⇒ Fixnum
Number of retry attempts used to try and perform the job.
-
#retry_criteria_check(method = nil) {|exception, *args| ... } ⇒ Object
Register a retry criteria check callback to be run before retrying the job again.
-
#retry_criteria_checks ⇒ Array
Retry criteria checks.
-
#retry_criteria_checks_pass?(exception, *args) ⇒ Boolean
private
Returns true if any of the retry criteria checks pass.
-
#retry_criteria_valid?(exception, *args) ⇒ Boolean
Test if the retry criteria is valid.
-
#retry_delay(exception_class = nil) ⇒ Number
abstract
Number of seconds to delay until the job is retried.
-
#retry_exception?(exception) ⇒ Boolean
Convenience method to test whether you may retry on a given exception.
-
#retry_exceptions ⇒ Array?
abstract
Controls what exceptions may be retried.
-
#retry_identifier(*args) ⇒ String
abstract
Builds a retry identifier using the job arguments.
-
#retry_job_delegate ⇒ Object?
abstract
Specify another resque job (module or class) to delegate retry duties to upon failure.
-
#retry_limit ⇒ Fixnum
Maximum number of retrys we can attempt to successfully perform the job.
-
#retry_limit_reached? ⇒ Boolean
Test if the retry limit has been reached.
-
#run_give_up_callbacks(exception, *args) ⇒ Object
private
Runs all the give up callbacks.
-
#run_try_again_callbacks(exception, *args) ⇒ Object
private
Runs all the try again callbacks.
-
#sleep_after_requeue ⇒ Number
abstract
Number of seconds to sleep after job is requeued.
-
#try_again(exception, *args) ⇒ Object
private
Retries the job.
-
#try_again_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a try again callback that will be called when the job fails but is trying again.
-
#try_again_callbacks ⇒ Array<Proc>
Returns the try again callbacks.
Methods included from Logging
Instance Attribute Details
#expire_retry_key_after ⇒ Number (readonly)
The number of seconds to set the TTL to on the resque-retry key in redis
254 255 256 |
# File 'lib/resque/plugins/retry.rb', line 254 def expire_retry_key_after @expire_retry_key_after end |
#fatal_exceptions ⇒ Array? (readonly)
Controls what exceptions may not be retried
Default: ‘nil` - this will retry all exceptions.
230 231 232 |
# File 'lib/resque/plugins/retry.rb', line 230 def fatal_exceptions @fatal_exceptions end |
Class Method Details
.extended(receiver) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Fail fast, when extended, if the “receiver” is misconfigured
49 50 51 52 53 |
# File 'lib/resque/plugins/retry.rb', line 49 def self.extended(receiver) if receiver.instance_variable_get('@fatal_exceptions') && receiver.instance_variable_get('@retry_exceptions') raise AmbiguousRetryStrategyException.new(%{You can't define both "@fatal_exceptions" and "@retry_exceptions"}) end end |
Instance Method Details
#after_perform_retry(*args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Resque after_perform hook
Deletes retry attempt count from Redis.
430 431 432 433 |
# File 'lib/resque/plugins/retry.rb', line 430 def after_perform_retry(*args) 'after_perform_retry, clearing retry key', args clean_retry_key(*args) end |
#before_perform_retry(*args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Resque before_perform hook
Increments ‘@retry_attempt` count and updates the “retry_key” expiration time (if applicable)
408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 |
# File 'lib/resque/plugins/retry.rb', line 408 def before_perform_retry(*args) 'before_perform_retry', args @on_failure_retry_hook_already_called = false # store number of retry attempts. retry_key = redis_retry_key(*args) Resque.redis.setnx(retry_key, -1) @retry_attempt = Resque.redis.incr(retry_key) "attempt: #{@retry_attempt} set in Redis", args # set/update the "retry_key" expiration if expire_retry_key_after "updating expiration for retry key: #{retry_key}", args Resque.redis.expire(retry_key, retry_delay + expire_retry_key_after) end end |
#call_symbol_or_block(method, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Helper to call functions that may be passed as Symbols or Procs. If a symbol, it is assumed to refer to a method that is already defined on this class.
596 597 598 599 600 601 602 |
# File 'lib/resque/plugins/retry.rb', line 596 def call_symbol_or_block(method, *args) if method.is_a?(Symbol) send(method, *args) elsif method.respond_to?(:call) instance_exec(*args, &method) end end |
#clean_retry_key(*args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Clean up retry state from redis once done
484 485 486 487 |
# File 'lib/resque/plugins/retry.rb', line 484 def clean_retry_key(*args) 'clean_retry_key', args Resque.redis.del(redis_retry_key(*args)) end |
#give_up(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
We failed and we’re not retrying.
396 397 398 399 400 |
# File 'lib/resque/plugins/retry.rb', line 396 def give_up(exception, *args) 'retry criteria not sufficient for retry', args, exception run_give_up_callbacks(exception, *args) clean_retry_key(*args) end |
#give_up_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a give up callback that will be called when the job fails and is not retrying. Can be registered with a block or a symbol.
567 568 569 570 571 572 573 |
# File 'lib/resque/plugins/retry.rb', line 567 def give_up_callback(method=nil, &block) if method.is_a? Symbol give_up_callbacks << method elsif block_given? give_up_callbacks << block end end |
#give_up_callbacks ⇒ Array<Proc>
Returns the give up callbacks.
543 544 545 |
# File 'lib/resque/plugins/retry.rb', line 543 def give_up_callbacks @give_up_callbacks ||= [] end |
#inherited(subclass) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Copy retry criteria checks, try again callbacks, and give up callbacks on inheritance.
59 60 61 62 63 64 |
# File 'lib/resque/plugins/retry.rb', line 59 def inherited(subclass) super(subclass) subclass.instance_variable_set('@retry_criteria_checks', retry_criteria_checks.dup) subclass.instance_variable_set('@try_again_callbacks', try_again_callbacks.dup) subclass.instance_variable_set('@give_up_callbacks', give_up_callbacks.dup) end |
#instance_exec(*args, &block) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Used to perform retry criteria check blocks under the job instance’s context
470 471 472 473 474 475 476 477 478 479 |
# File 'lib/resque/plugins/retry.rb', line 470 def instance_exec(*args, &block) mname = "__instance_exec_#{Thread.current.object_id.abs}" class << self; self end.class_eval{ define_method(mname, &block) } begin ret = send(mname, *args) ensure class << self; self end.class_eval{ undef_method(mname) } rescue nil end ret end |
#on_failure_retry(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
This hook will only allow execution once per job perform attempt. This was added because Resque v1.20.0 calls the hook twice. IMO; this isn’t something resque-retry should have to worry about!
Resque on_failure hook
Checks if our retry criteria is valid, if it is we try again. Otherwise the retry attempt count is deleted from Redis.
445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 |
# File 'lib/resque/plugins/retry.rb', line 445 def on_failure_retry(exception, *args) 'on_failure_retry', args, exception if exception.is_a?(Resque::DirtyExit) # This hook is called from a worker processes, not the job process # that failed with a DirtyExit, so @retry_attempt wasn't set yet @retry_attempt = Resque.redis.get(redis_retry_key(*args)).to_i elsif @on_failure_retry_hook_already_called 'on_failure_retry_hook_already_called', args, exception return end if retry_criteria_valid?(exception, *args) try_again(exception, *args) else give_up(exception, *args) end @on_failure_retry_hook_already_called = true end |
#redis_retry_key(*args) ⇒ String
Builds the redis key to be used for keeping state of the job attempts.
88 89 90 |
# File 'lib/resque/plugins/retry.rb', line 88 def redis_retry_key(*args) ['resque-retry', name, retry_identifier(*args)].compact.join(':').gsub(/\s/, '') end |
#retry_args(*args) ⇒ Array
Modify the arguments used to retry the job. Use this to do something other than try the exact same job again
160 161 162 163 164 165 166 167 168 169 170 |
# File 'lib/resque/plugins/retry.rb', line 160 def retry_args(*args) # Here for backwards compatibility. If an "args_for_retry" method exists # invoke it, but warn that it is deprecated (and will be removed in a # future revision) if respond_to?(:args_for_retry) warn "`Resque::Plugins::Retry#args_for_retry` is deprecated, please use `Resque::Plugins::Retry#retry_args` instead." args_for_retry(*args) else args end end |
#retry_args_for_exception(exception, *args) ⇒ Array
Modify the arguments used to retry the job based on the exception. Use this to do something other than try the exact same job again.
179 180 181 |
# File 'lib/resque/plugins/retry.rb', line 179 def retry_args_for_exception(exception, *args) retry_args(*args) end |
#retry_attempt ⇒ Fixnum
Number of retry attempts used to try and perform the job
The real value is kept in Redis, it is accessed and incremented using a before_perform hook.
112 113 114 |
# File 'lib/resque/plugins/retry.rb', line 112 def retry_attempt @retry_attempt ||= 0 end |
#retry_criteria_check(method = nil) {|exception, *args| ... } ⇒ Object
Register a retry criteria check callback to be run before retrying the job again. Can be registered with a block or a symbol.
If any callback returns ‘true`, the job will be retried.
335 336 337 338 339 340 341 |
# File 'lib/resque/plugins/retry.rb', line 335 def retry_criteria_check(method=nil, &block) if method.is_a? Symbol retry_criteria_checks << method elsif block_given? retry_criteria_checks << block end end |
#retry_criteria_checks ⇒ Array
Retry criteria checks
289 290 291 |
# File 'lib/resque/plugins/retry.rb', line 289 def retry_criteria_checks @retry_criteria_checks ||= [] end |
#retry_criteria_checks_pass?(exception, *args) ⇒ Boolean
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Returns true if any of the retry criteria checks pass. When a retry criteria check passes, the remaining ones are not executed.
349 350 351 352 353 354 |
# File 'lib/resque/plugins/retry.rb', line 349 def retry_criteria_checks_pass?(exception, *args) retry_criteria_checks.each do |criteria_check| return true if !!call_symbol_or_block(criteria_check, exception, *args) end false end |
#retry_criteria_valid?(exception, *args) ⇒ Boolean
Test if the retry criteria is valid
263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 |
# File 'lib/resque/plugins/retry.rb', line 263 def retry_criteria_valid?(exception, *args) # if the retry limit was reached, dont bother checking anything else. if retry_limit_reached? 'retry limit reached', args, exception return false end # We always want to retry if the exception matches. retry_based_on_exception = retry_exception?(exception) "Exception is #{retry_based_on_exception ? '' : 'not '}sufficient for a retry", args, exception retry_based_on_criteria = false unless retry_based_on_exception # call user retry criteria check blocks. retry_based_on_criteria = retry_criteria_checks_pass?(exception, *args) "user retry criteria is #{retry_based_on_criteria ? '' : 'not '}sufficient for a retry", args, exception end retry_based_on_exception || retry_based_on_criteria end |
#retry_delay(exception_class = nil) ⇒ Number
Number of seconds to delay until the job is retried
122 123 124 125 126 127 128 129 130 |
# File 'lib/resque/plugins/retry.rb', line 122 def retry_delay(exception_class = nil) if @retry_exceptions.is_a?(Hash) delay = @retry_exceptions[exception_class] || 0 # allow an array of delays. delay.is_a?(Array) ? delay[retry_attempt] || delay.last : delay else @retry_delay ||= 0 end end |
#retry_exception?(exception) ⇒ Boolean
Convenience method to test whether you may retry on a given exception
also be a Class
192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 |
# File 'lib/resque/plugins/retry.rb', line 192 def retry_exception?(exception) # If both "fatal_exceptions" and "retry_exceptions" are undefined we are # done (we should retry the exception) # # It is intentional that we check "retry_exceptions" first since it is # more likely that it will be defined (over "fatal_exceptions") as it # has been part of the API for quite a while return true if retry_exceptions.nil? && fatal_exceptions.nil? # If "fatal_exceptions" is undefined interrogate "retry_exceptions" if fatal_exceptions.nil? retry_exceptions.any? do |ex| if exception.is_a?(Class) ex >= exception else ex === exception end end # It is safe to assume we need to check "fatal_exceptions" at this point else fatal_exceptions.none? do |ex| if exception.is_a?(Class) ex >= exception else ex === exception end end end end |
#retry_exceptions ⇒ Array?
Controls what exceptions may be retried
Default: ‘nil` - this will retry all exceptions.
240 241 242 243 244 245 246 |
# File 'lib/resque/plugins/retry.rb', line 240 def retry_exceptions if @retry_exceptions.is_a?(Hash) @retry_exceptions.keys else @retry_exceptions ||= nil end end |
#retry_identifier(*args) ⇒ String
You may override to implement a custom retry identifier, you should consider doing this if your job arguments are many/long or may not cleanly convert to strings.
Builds a retry identifier using the job arguments. This identifier is used as part of the redis key
77 78 79 80 |
# File 'lib/resque/plugins/retry.rb', line 77 def retry_identifier(*args) args_string = args.join('-') args_string.empty? ? nil : Digest::SHA1.hexdigest(args_string) end |
#retry_job_delegate ⇒ Object?
Specify another resque job (module or class) to delegate retry duties to upon failure
149 150 151 |
# File 'lib/resque/plugins/retry.rb', line 149 def retry_job_delegate @retry_job_delegate ||= nil end |
#retry_limit ⇒ Fixnum
Maximum number of retrys we can attempt to successfully perform the job
A retry limit of 0 will never retry. A retry limit of -1 or below will retry forever.
100 101 102 |
# File 'lib/resque/plugins/retry.rb', line 100 def retry_limit @retry_limit ||= 1 end |
#retry_limit_reached? ⇒ Boolean
Test if the retry limit has been reached
298 299 300 301 302 303 304 305 306 |
# File 'lib/resque/plugins/retry.rb', line 298 def retry_limit_reached? if retry_limit == 0 true elsif retry_limit > 0 true if retry_attempt >= retry_limit else false end end |
#run_give_up_callbacks(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Runs all the give up callbacks.
581 582 583 584 585 |
# File 'lib/resque/plugins/retry.rb', line 581 def run_give_up_callbacks(exception, *args) give_up_callbacks.each do |callback| call_symbol_or_block(callback, exception, *args) end end |
#run_try_again_callbacks(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Runs all the try again callbacks.
532 533 534 535 536 |
# File 'lib/resque/plugins/retry.rb', line 532 def run_try_again_callbacks(exception, *args) try_again_callbacks.each do |callback| call_symbol_or_block(callback, exception, *args) end end |
#sleep_after_requeue ⇒ Number
Number of seconds to sleep after job is requeued
138 139 140 |
# File 'lib/resque/plugins/retry.rb', line 138 def sleep_after_requeue @sleep_after_requeue ||= 0 end |
#try_again(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Retries the job
359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 |
# File 'lib/resque/plugins/retry.rb', line 359 def try_again(exception, *args) 'try_again', args, exception run_try_again_callbacks(exception, *args) # some plugins define retry_delay and have it take no arguments, so rather than break those, # we'll just check here to see whether it takes the additional exception class argument or not temp_retry_delay = ([-1, 1].include?(method(:retry_delay).arity) ? retry_delay(exception.class) : retry_delay) retry_in_queue = retry_job_delegate ? retry_job_delegate : self "retry delay: #{temp_retry_delay} for class: #{retry_in_queue}", args, exception # remember that this job is now being retried. before_perform_retry will increment # this so it represents the retry count, and MultipleWithRetrySuppression uses # the existence of this to determine if the job should be sent to the # parent failure backend (e.g. failed queue) or not. Removing this means # jobs that fail before ::perform will be both retried and sent to the failed queue. Resque.redis.setnx(redis_retry_key(*args), -1) retry_args = retry_args_for_exception(exception, *args) if temp_retry_delay <= 0 # If the delay is 0, no point passing it through the scheduler Resque.enqueue(retry_in_queue, *retry_args) else Resque.enqueue_in(temp_retry_delay, retry_in_queue, *retry_args) end # remove retry key from redis if we handed retry off to another queue. clean_retry_key(*args) if retry_job_delegate # sleep after requeue if enabled. sleep(sleep_after_requeue) if sleep_after_requeue > 0 end |
#try_again_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a try again callback that will be called when the job fails but is trying again. Can be registered with a block or a symbol.
518 519 520 521 522 523 524 |
# File 'lib/resque/plugins/retry.rb', line 518 def try_again_callback(method=nil, &block) if method.is_a? Symbol try_again_callbacks << method elsif block_given? try_again_callbacks << block end end |
#try_again_callbacks ⇒ Array<Proc>
Returns the try again callbacks.
494 495 496 |
# File 'lib/resque/plugins/retry.rb', line 494 def try_again_callbacks @try_again_callbacks ||= [] end |