Module: Resque::Plugins::Retry
- Includes:
- Logging
- Included in:
- ExponentialBackoff
- Defined in:
- lib/resque/plugins/retry.rb,
lib/resque/plugins/retry/logging.rb
Overview
If you want your job to retry on failure, simply extend your module/class with this module:
class DeliverWebHook
extend Resque::Plugins::Retry # allows 1 retry by default.
@queue = :web_hooks
def self.perform(url, hook_id, hmac_key)
heavy_lifting
end
end
Easily do something custom:
class DeliverWebHook
extend Resque::Plugins::Retry
@queue = :web_hooks
@retry_limit = 8 # default: 1
@retry_delay = 60 # default: 0
# used to build redis key, for counting job attempts.
def self.retry_identifier(url, hook_id, hmac_key)
"#{url}-#{hook_id}"
end
def self.perform(url, hook_id, hmac_key)
heavy_lifting
end
end
Defined Under Namespace
Modules: Logging Classes: AmbiguousRetryStrategyException, RetryConfigurationException
Instance Attribute Summary collapse
-
#expire_retry_key_after ⇒ Number
readonly
abstract
The number of seconds to set the TTL to on the resque-retry key in redis.
-
#fatal_exceptions ⇒ Array?
readonly
abstract
Controls what exceptions may not be retried.
Class Method Summary collapse
-
.extended(receiver) ⇒ Object
private
Fail fast, when extended, if the “receiver” is misconfigured.
Instance Method Summary collapse
-
#after_perform_retry(*args) ⇒ Object
private
Resque after_perform hook.
-
#before_perform_retry(*args) ⇒ Object
private
Resque before_perform hook.
-
#call_symbol_or_block(method, *args) ⇒ Object
private
Helper to call functions that may be passed as Symbols or Procs.
-
#clean_retry_key(*args) ⇒ Object
private
Clean up retry state from redis once done.
-
#give_up(exception, *args) ⇒ Object
private
We failed and we’re not retrying.
-
#give_up_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a give up callback that will be called when the job fails and is not retrying.
-
#give_up_callbacks ⇒ Array<Proc>
Returns the give up callbacks.
- #ignore_exceptions ⇒ Object
-
#inherited(subclass) ⇒ Object
private
Copy retry criteria checks, try again callbacks, and give up callbacks on inheritance.
-
#instance_exec(*args, &block) ⇒ Object
private
Used to perform retry criteria check blocks under the job instance’s context.
-
#on_failure_retry(exception, *args) ⇒ Object
private
Resque on_failure hook.
-
#redis_retry_key(*args) ⇒ String
Builds the redis key to be used for keeping state of the job attempts.
-
#retry_args(*args) ⇒ Array
abstract
Modify the arguments used to retry the job.
-
#retry_args_for_exception(exception, *args) ⇒ Array
abstract
Modify the arguments used to retry the job based on the exception.
-
#retry_attempt ⇒ Fixnum
Number of retry attempts used to try and perform the job.
-
#retry_criteria_check(method = nil) {|exception, *args| ... } ⇒ Object
Register a retry criteria check callback to be run before retrying the job again.
-
#retry_criteria_checks ⇒ Array
Retry criteria checks.
-
#retry_criteria_checks_pass?(exception, *args) ⇒ Boolean
private
Returns true if any of the retry criteria checks pass.
-
#retry_criteria_valid?(exception, *args) ⇒ Boolean
Test if the retry criteria is valid.
-
#retry_delay(exception_class = nil) ⇒ Number
abstract
Number of seconds to delay until the job is retried If @retry_exceptions is a Hash and there is no delay defined for exception_class, looks for closest superclass and assigns it’s delay to @retry_exceptions.
-
#retry_exception?(exception) ⇒ Boolean
Convenience method to test whether you may retry on a given exception.
-
#retry_exceptions ⇒ Array?
abstract
Controls what exceptions may be retried.
-
#retry_identifier(*args) ⇒ String
abstract
Builds a retry identifier using the job arguments.
-
#retry_job_delegate ⇒ Object?
abstract
Specify another resque job (module or class) to delegate retry duties to upon failure.
-
#retry_limit ⇒ Fixnum
Maximum number of retrys we can attempt to successfully perform the job.
-
#retry_limit_reached? ⇒ Boolean
Test if the retry limit has been reached.
-
#retry_queue(exception, *args) ⇒ Symbol
abstract
Specify the queue that the job should be placed in upon failure.
-
#run_give_up_callbacks(exception, *args) ⇒ Object
private
Runs all the give up callbacks.
-
#run_try_again_callbacks(exception, *args) ⇒ Object
private
Runs all the try again callbacks.
-
#sleep_after_requeue ⇒ Number
abstract
Number of seconds to sleep after job is requeued.
-
#try_again(exception, *args) ⇒ Object
private
Retries the job.
-
#try_again_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a try again callback that will be called when the job fails but is trying again.
-
#try_again_callbacks ⇒ Array<Proc>
Returns the try again callbacks.
Methods included from Logging
Instance Attribute Details
#expire_retry_key_after ⇒ Number (readonly)
The number of seconds to set the TTL to on the resque-retry key in redis
282 283 284 |
# File 'lib/resque/plugins/retry.rb', line 282 def expire_retry_key_after @expire_retry_key_after end |
#fatal_exceptions ⇒ Array? (readonly)
Controls what exceptions may not be retried
Default: ‘nil` - this will retry all exceptions.
258 259 260 |
# File 'lib/resque/plugins/retry.rb', line 258 def fatal_exceptions @fatal_exceptions end |
Class Method Details
.extended(receiver) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Fail fast, when extended, if the “receiver” is misconfigured
54 55 56 57 58 59 60 61 62 63 64 65 66 |
# File 'lib/resque/plugins/retry.rb', line 54 def self.extended(receiver) retry_exceptions = nil retry_exceptions = receiver.instance_variable_get(:@retry_exceptions) \ if receiver.instance_variable_defined?(:@retry_exceptions) fatal_exceptions = nil fatal_exceptions = receiver.instance_variable_get(:@fatal_exceptions) \ if receiver.instance_variable_defined?(:@fatal_exceptions) if fatal_exceptions && retry_exceptions raise AmbiguousRetryStrategyException.new(%{You can't define both "@fatal_exceptions" and "@retry_exceptions"}) end end |
Instance Method Details
#after_perform_retry(*args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Resque after_perform hook
Deletes retry attempt count from Redis.
472 473 474 475 476 |
# File 'lib/resque/plugins/retry.rb', line 472 def after_perform_retry(*args) return if Resque.inline? 'after_perform_retry, clearing retry key', args clean_retry_key(*args) end |
#before_perform_retry(*args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Resque before_perform hook
Increments ‘@retry_attempt` count and updates the “retry_key” expiration time (if applicable)
449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 |
# File 'lib/resque/plugins/retry.rb', line 449 def before_perform_retry(*args) return if Resque.inline? 'before_perform_retry', args @on_failure_retry_hook_already_called = false # store number of retry attempts. retry_key = redis_retry_key(*args) Resque.redis.setnx(retry_key, -1) @retry_attempt = Resque.redis.incr(retry_key) "attempt: #{@retry_attempt} set in Redis", args # set/update the "retry_key" expiration if expire_retry_key_after "updating expiration for retry key: #{retry_key}", args Resque.redis.expire(retry_key, retry_delay + expire_retry_key_after) end end |
#call_symbol_or_block(method, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Helper to call functions that may be passed as Symbols or Procs. If a symbol, it is assumed to refer to a method that is already defined on this class.
652 653 654 655 656 657 658 |
# File 'lib/resque/plugins/retry.rb', line 652 def call_symbol_or_block(method, *args) if method.is_a?(Symbol) send(method, *args) elsif method.respond_to?(:call) instance_exec(*args, &method) end end |
#clean_retry_key(*args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Clean up retry state from redis once done
536 537 538 539 |
# File 'lib/resque/plugins/retry.rb', line 536 def clean_retry_key(*args) 'clean_retry_key', args Resque.redis.del(redis_retry_key(*args)) end |
#give_up(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
We failed and we’re not retrying.
437 438 439 440 441 |
# File 'lib/resque/plugins/retry.rb', line 437 def give_up(exception, *args) 'retry criteria not sufficient for retry', args, exception run_give_up_callbacks(exception, *args) clean_retry_key(*args) end |
#give_up_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a give up callback that will be called when the job fails and is not retrying. Can be registered with a block or a symbol.
619 620 621 622 623 624 625 |
# File 'lib/resque/plugins/retry.rb', line 619 def give_up_callback(method=nil, &block) if method.is_a? Symbol give_up_callbacks << method elsif block_given? give_up_callbacks << block end end |
#give_up_callbacks ⇒ Array<Proc>
Returns the give up callbacks.
595 596 597 |
# File 'lib/resque/plugins/retry.rb', line 595 def give_up_callbacks @give_up_callbacks ||= [] end |
#ignore_exceptions ⇒ Object
639 640 641 |
# File 'lib/resque/plugins/retry.rb', line 639 def ignore_exceptions @ignore_exceptions ||= [] end |
#inherited(subclass) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Copy retry criteria checks, try again callbacks, and give up callbacks on inheritance.
72 73 74 75 76 77 |
# File 'lib/resque/plugins/retry.rb', line 72 def inherited(subclass) super(subclass) subclass.instance_variable_set('@retry_criteria_checks', retry_criteria_checks.dup) subclass.instance_variable_set('@try_again_callbacks', try_again_callbacks.dup) subclass.instance_variable_set('@give_up_callbacks', give_up_callbacks.dup) end |
#instance_exec(*args, &block) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Used to perform retry criteria check blocks under the job instance’s context
522 523 524 525 526 527 528 529 530 531 |
# File 'lib/resque/plugins/retry.rb', line 522 def instance_exec(*args, &block) mname = "__instance_exec_#{Thread.current.object_id.abs}" class << self; self end.class_eval{ define_method(mname, &block) } begin ret = send(mname, *args) ensure class << self; self end.class_eval{ undef_method(mname) } rescue nil end ret end |
#on_failure_retry(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
This hook will only allow execution once per job perform attempt. This was added because Resque v1.20.0 calls the hook twice. IMO; this isn’t something resque-retry should have to worry about!
Resque on_failure hook
Checks if our retry criteria is valid, if it is we try again. Otherwise the retry attempt count is deleted from Redis.
488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 |
# File 'lib/resque/plugins/retry.rb', line 488 def on_failure_retry(exception, *args) return if Resque.inline? 'on_failure_retry', args, exception if exception.is_a?(Resque::DirtyExit) # This hook is called from a worker processes, not the job process # that failed with a DirtyExit, so @retry_attempt wasn't set yet @retry_attempt = Resque.redis.get(redis_retry_key(*args)).to_i elsif instance_variable_defined?(:@on_failure_retry_hook_already_called) && \ @on_failure_retry_hook_already_called 'on_failure_retry_hook_already_called', args, exception return end # If we are "ignoring" the exception, then we decrement the retry # counter, so that the current attempt didn't count toward the retry # counter. if ignore_exceptions.include?(exception.class) @retry_attempt = Resque.redis.decr(redis_retry_key(*args)) end if retry_criteria_valid?(exception, *args) try_again(exception, *args) else give_up(exception, *args) end @on_failure_retry_hook_already_called = true end |
#redis_retry_key(*args) ⇒ String
Builds the redis key to be used for keeping state of the job attempts.
101 102 103 |
# File 'lib/resque/plugins/retry.rb', line 101 def redis_retry_key(*args) ['resque-retry', name, retry_identifier(*args)].compact.join(':').gsub(/\s/, '') end |
#retry_args(*args) ⇒ Array
Modify the arguments used to retry the job. Use this to do something other than try the exact same job again
188 189 190 191 192 193 194 195 196 197 198 |
# File 'lib/resque/plugins/retry.rb', line 188 def retry_args(*args) # Here for backwards compatibility. If an "args_for_retry" method exists # invoke it, but warn that it is deprecated (and will be removed in a # future revision) if respond_to?(:args_for_retry) warn "`Resque::Plugins::Retry#args_for_retry` is deprecated, please use `Resque::Plugins::Retry#retry_args` instead." args_for_retry(*args) else args end end |
#retry_args_for_exception(exception, *args) ⇒ Array
Modify the arguments used to retry the job based on the exception. Use this to do something other than try the exact same job again.
207 208 209 |
# File 'lib/resque/plugins/retry.rb', line 207 def retry_args_for_exception(exception, *args) retry_args(*args) end |
#retry_attempt ⇒ Fixnum
Number of retry attempts used to try and perform the job
The real value is kept in Redis, it is accessed and incremented using a before_perform hook.
125 126 127 |
# File 'lib/resque/plugins/retry.rb', line 125 def retry_attempt @retry_attempt ||= 0 end |
#retry_criteria_check(method = nil) {|exception, *args| ... } ⇒ Object
Register a retry criteria check callback to be run before retrying the job again. Can be registered with a block or a symbol.
If any callback returns ‘true`, the job will be retried.
363 364 365 366 367 368 369 |
# File 'lib/resque/plugins/retry.rb', line 363 def retry_criteria_check(method=nil, &block) if method.is_a? Symbol retry_criteria_checks << method elsif block_given? retry_criteria_checks << block end end |
#retry_criteria_checks ⇒ Array
Retry criteria checks
317 318 319 |
# File 'lib/resque/plugins/retry.rb', line 317 def retry_criteria_checks @retry_criteria_checks ||= [] end |
#retry_criteria_checks_pass?(exception, *args) ⇒ Boolean
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Returns true if any of the retry criteria checks pass. When a retry criteria check passes, the remaining ones are not executed.
377 378 379 380 381 382 |
# File 'lib/resque/plugins/retry.rb', line 377 def retry_criteria_checks_pass?(exception, *args) retry_criteria_checks.each do |criteria_check| return true if !!call_symbol_or_block(criteria_check, exception, *args) end false end |
#retry_criteria_valid?(exception, *args) ⇒ Boolean
Test if the retry criteria is valid
291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 |
# File 'lib/resque/plugins/retry.rb', line 291 def retry_criteria_valid?(exception, *args) # if the retry limit was reached, dont bother checking anything else. if retry_limit_reached? 'retry limit reached', args, exception return false end # We always want to retry if the exception matches. retry_based_on_exception = retry_exception?(exception) "Exception is #{retry_based_on_exception ? '' : 'not '}sufficient for a retry", args, exception retry_based_on_criteria = false unless retry_based_on_exception # call user retry criteria check blocks. retry_based_on_criteria = retry_criteria_checks_pass?(exception, *args) "user retry criteria is #{retry_based_on_criteria ? '' : 'not '}sufficient for a retry", args, exception end retry_based_on_exception || retry_based_on_criteria end |
#retry_delay(exception_class = nil) ⇒ Number
Number of seconds to delay until the job is retried If @retry_exceptions is a Hash and there is no delay defined for exception_class, looks for closest superclass and assigns it’s delay to @retry_exceptions
137 138 139 140 141 142 143 144 145 146 147 148 |
# File 'lib/resque/plugins/retry.rb', line 137 def retry_delay(exception_class = nil) if instance_variable_defined?(:@retry_exceptions) && @retry_exceptions.is_a?(Hash) delay = @retry_exceptions[exception_class] ||= begin relevant_definitions = @retry_exceptions.select { |ex| exception_class <= ex } relevant_definitions.any? ? relevant_definitions.sort.first[1] : 0 end # allow an array of delays. delay.is_a?(Array) ? delay[retry_attempt] || delay.last : delay else @retry_delay ||= 0 end end |
#retry_exception?(exception) ⇒ Boolean
Convenience method to test whether you may retry on a given exception
also be a Class
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 |
# File 'lib/resque/plugins/retry.rb', line 220 def retry_exception?(exception) # If both "fatal_exceptions" and "retry_exceptions" are undefined we are # done (we should retry the exception) # # It is intentional that we check "retry_exceptions" first since it is # more likely that it will be defined (over "fatal_exceptions") as it # has been part of the API for quite a while return true if retry_exceptions.nil? && fatal_exceptions.nil? # If "fatal_exceptions" is undefined interrogate "retry_exceptions" if fatal_exceptions.nil? retry_exceptions.any? do |ex| if exception.is_a?(Class) ex >= exception else ex === exception end end # It is safe to assume we need to check "fatal_exceptions" at this point else fatal_exceptions.none? do |ex| if exception.is_a?(Class) ex >= exception else ex === exception end end end end |
#retry_exceptions ⇒ Array?
Controls what exceptions may be retried
Default: ‘nil` - this will retry all exceptions.
268 269 270 271 272 273 274 |
# File 'lib/resque/plugins/retry.rb', line 268 def retry_exceptions if instance_variable_defined?(:@retry_exceptions) && @retry_exceptions.is_a?(Hash) @retry_exceptions.keys else @retry_exceptions ||= nil end end |
#retry_identifier(*args) ⇒ String
You may override to implement a custom retry identifier, you should consider doing this if your job arguments are many/long or may not cleanly convert to strings.
Builds a retry identifier using the job arguments. This identifier is used as part of the redis key
90 91 92 93 |
# File 'lib/resque/plugins/retry.rb', line 90 def retry_identifier(*args) args_string = args.join('-') args_string.empty? ? nil : Digest::SHA1.hexdigest(args_string) end |
#retry_job_delegate ⇒ Object?
Specify another resque job (module or class) to delegate retry duties to upon failure
167 168 169 |
# File 'lib/resque/plugins/retry.rb', line 167 def retry_job_delegate @retry_job_delegate ||= nil end |
#retry_limit ⇒ Fixnum
Maximum number of retrys we can attempt to successfully perform the job
A retry limit of 0 will never retry. A retry limit of -1 or below will retry forever.
113 114 115 |
# File 'lib/resque/plugins/retry.rb', line 113 def retry_limit @retry_limit ||= 1 end |
#retry_limit_reached? ⇒ Boolean
Test if the retry limit has been reached
326 327 328 329 330 331 332 333 334 |
# File 'lib/resque/plugins/retry.rb', line 326 def retry_limit_reached? if retry_limit == 0 true elsif retry_limit > 0 true if retry_attempt >= retry_limit else false end end |
#retry_queue(exception, *args) ⇒ Symbol
Specify the queue that the job should be placed in upon failure
177 178 179 |
# File 'lib/resque/plugins/retry.rb', line 177 def retry_queue(exception, *args) nil end |
#run_give_up_callbacks(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Runs all the give up callbacks.
633 634 635 636 637 |
# File 'lib/resque/plugins/retry.rb', line 633 def run_give_up_callbacks(exception, *args) give_up_callbacks.each do |callback| call_symbol_or_block(callback, exception, *args) end end |
#run_try_again_callbacks(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Runs all the try again callbacks.
584 585 586 587 588 |
# File 'lib/resque/plugins/retry.rb', line 584 def run_try_again_callbacks(exception, *args) try_again_callbacks.each do |callback| call_symbol_or_block(callback, exception, *args) end end |
#sleep_after_requeue ⇒ Number
Number of seconds to sleep after job is requeued
156 157 158 |
# File 'lib/resque/plugins/retry.rb', line 156 def sleep_after_requeue @sleep_after_requeue ||= 0 end |
#try_again(exception, *args) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Retries the job
387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 |
# File 'lib/resque/plugins/retry.rb', line 387 def try_again(exception, *args) 'try_again', args, exception run_try_again_callbacks(exception, *args) # some plugins define retry_delay and have it take no arguments, so rather than break those, # we'll just check here to see whether it takes the additional exception class argument or not # we also allow all job args to be passed to a custom `retry_delay` method retry_delay_arity = method(:retry_delay).arity temp_retry_delay = if [-2, 2].include?(retry_delay_arity) retry_delay(exception.class, *args) elsif [-1, 1].include?(retry_delay_arity) retry_delay(exception.class) else retry_delay end retry_job_class = retry_job_delegate ? retry_job_delegate : self retry_in_queue = retry_queue(exception, *args) retry_in_queue ||= Resque.queue_from_class(retry_job_class) "retry delay: #{temp_retry_delay} for queue: #{retry_in_queue}", args, exception # remember that this job is now being retried. before_perform_retry will increment # this so it represents the retry count, and MultipleWithRetrySuppression uses # the existence of this to determine if the job should be sent to the # parent failure backend (e.g. failed queue) or not. Removing this means # jobs that fail before ::perform will be both retried and sent to the failed queue. Resque.redis.setnx(redis_retry_key(*args), -1) retry_args = retry_args_for_exception(exception, *args) if temp_retry_delay <= 0 # If the delay is 0, no point passing it through the scheduler Resque.enqueue_to(retry_in_queue, retry_job_class, *retry_args) else Resque.enqueue_in_with_queue(retry_in_queue, temp_retry_delay, retry_job_class, *retry_args) end # remove retry key from redis if we handed retry off to another queue. clean_retry_key(*args) if retry_job_delegate # sleep after requeue if enabled. sleep(sleep_after_requeue) if sleep_after_requeue > 0 end |
#try_again_callback(method = nil) {|exception, *args| ... } ⇒ Object
Register a try again callback that will be called when the job fails but is trying again. Can be registered with a block or a symbol.
570 571 572 573 574 575 576 |
# File 'lib/resque/plugins/retry.rb', line 570 def try_again_callback(method=nil, &block) if method.is_a? Symbol try_again_callbacks << method elsif block_given? try_again_callbacks << block end end |
#try_again_callbacks ⇒ Array<Proc>
Returns the try again callbacks.
546 547 548 |
# File 'lib/resque/plugins/retry.rb', line 546 def try_again_callbacks @try_again_callbacks ||= [] end |