Class: Deimos::Utils::DeadlockRetry

Inherits:
Object
  • Object
show all
Defined in:
lib/deimos/utils/deadlock_retry.rb

Overview

Utility class to retry a given block if a a deadlock is encountered. Supports Postgres and MySQL deadlocks and lock wait timeouts.

Constant Summary collapse

RETRY_COUNT =

Maximum number of times to retry the block after encountering a deadlock

Returns:

  • (Integer) 
2
DEADLOCK_MESSAGES =

Need to match on error messages to support older Rails versions

Returns:

  • (Array<String>) 
[
  # MySQL
  'Deadlock found when trying to get lock',
  'Lock wait timeout exceeded',

  # Postgres
  'deadlock detected'
].freeze

Class Method Summary collapse

Class Method Details

.wrap(tags = []) { ... } ⇒ void

This method returns an undefined value.

Retry the given block when encountering a deadlock. For any other exceptions, they are reraised. This is used to handle cases where the database may be busy but the transaction would succeed if retried later. Note that your block should be idempotent and it will be wrapped in a transaction. Sleeps for a random number of seconds to prevent multiple transactions from retrying at the same time.

Parameters:

  • tags (Array) (defaults to: []) —

    Tags to attach when logging and reporting metrics.

Yields:

  • Yields to the block that may deadlock.



34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/deimos/utils/deadlock_retry.rb', line 34

def wrap(tags=[])
  count = RETRY_COUNT

  begin
    ActiveRecord::Base.transaction do
      yield
    end
  rescue ActiveRecord::StatementInvalid => e
    # Reraise if not a known deadlock
    raise if DEADLOCK_MESSAGES.none? { |m| e.message.include?(m) }

    # Reraise if all retries exhausted
    raise if count <= 0

    Deimos.config.logger.warn(
      message: 'Deadlock encountered when trying to execute query. '\
        "Retrying. #{count} attempt(s) remaining",
      tags: tags
    )

    Deimos.config.metrics&.increment(
      'deadlock',
      tags: tags
    )

    count -= 1

    # Sleep for a random amount so that if there are multiple
    # transactions deadlocking, they don't all retry at the same time
    sleep(Random.rand(5.0) + 0.5)

    retry
  end
end