Class: OnlineMigrations::LockRetrier

Inherits:
Object
  • Object
show all
Defined in:
lib/online_migrations/lock_retrier.rb

Overview

This class provides a way to automatically retry code that relies on acquiring a database lock in a way designed to minimize impact on a busy production database.

This class defines an interface for child classes to implement to configure timing configurations and the maximum number of attempts.

There are two predefined implementations (see OnlineMigrations::ConstantLockRetrier and OnlineMigrations::ExponentialLockRetrier). It is easy to provide more sophisticated implementations.

Examples:

Custom LockRetrier implementation

module OnlineMigrations
  class SophisticatedLockRetrier < LockRetrier
    TIMINGS = [
      [0.1.seconds, 0.05.seconds], # first - lock timeout, second - delay time
      [0.1.seconds, 0.05.seconds],
      [0.2.seconds, 0.05.seconds],
      [0.3.seconds, 0.10.seconds],
      [1.second, 5.seconds],
      [1.second, 1.minute],
      [0.1.seconds, 0.05.seconds],
      [0.2.seconds, 0.15.seconds],
      [0.5.seconds, 2.seconds],
      [0.5.seconds, 2.seconds],
      [3.seconds, 3.minutes],
      [0.1.seconds, 0.05.seconds],
      [0.5.seconds, 2.seconds],
      [5.seconds, 2.minutes],
      [7.seconds, 5.minutes],
      [0.5.seconds, 2.seconds],
    ]

    def attempts
      TIMINGS.size
    end

    def lock_timeout(attempt, command = nil, arguments = [])
      TIMINGS[attempt - 1][0]
    end

    def delay(attempt)
      TIMINGS[attempt - 1][1]
    end
  end

Instance Method Summary collapse

Instance Method Details

#attempts(_command = nil, _arguments = []) ⇒ Object

Returns the number of retrying attempts

Parameters:

  • _command (Symbol, nil) (defaults to: nil) —

    the migration method being executed (e.g., :add_index, :add_column). Will be nil when called from a transaction-wrapped migration (the default). Only populated for migrations using ‘disable_ddl_transaction!`.

  • _arguments (Array) (defaults to: []) —

    the arguments passed to the migration method

Raises:

  • (NotImplementedError) 


56
57
58
 
# File 'lib/online_migrations/lock_retrier.rb', line 56

def attempts(_command = nil, _arguments = [])
  raise NotImplementedError
end

#delay(_attempt, _command = nil, _arguments = []) ⇒ Object

Returns sleep time after unsuccessful lock attempt (in seconds)

Parameters:

  • _attempt (Integer) —

    attempt number

  • _command (Symbol, nil) (defaults to: nil) —

    the migration method being executed (e.g., :add_index, :add_column). Will be nil when called from a transaction-wrapped migration (the default). Only populated for migrations using ‘disable_ddl_transaction!`.

  • _arguments (Array) (defaults to: []) —

    the arguments passed to the migration method

Raises:

  • (NotImplementedError) 


78
79
80
 
# File 'lib/online_migrations/lock_retrier.rb', line 78

def delay(_attempt, _command = nil, _arguments = [])
  raise NotImplementedError
end

#lock_timeout(_attempt, _command = nil, _arguments = []) ⇒ Object

Returns database lock timeout value (in seconds) for specified attempt number

Parameters:

  • _attempt (Integer) —

    attempt number

  • _command (Symbol, nil) (defaults to: nil) —

    the migration method being executed (e.g., :add_index, :add_column). Will be nil when called from a transaction-wrapped migration (the default). Only populated for migrations using ‘disable_ddl_transaction!`.

  • _arguments (Array) (defaults to: []) —

    the arguments passed to the migration method



68
 
# File 'lib/online_migrations/lock_retrier.rb', line 68

def lock_timeout(_attempt, _command = nil, _arguments = []); end

#with_lock_retries(connection, command = nil, *arguments, &block) ⇒ void

This method returns an undefined value.

Executes the block with a retry mechanism that alters the ‘lock_timeout` and sleep time between attempts.

Examples:

retrier.with_lock_retries(connection) do
  add_column(:users, :name, :string)
end

Parameters:

  • connection —

    The connection on which to retry lock timeouts



93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'lib/online_migrations/lock_retrier.rb', line 93

def with_lock_retries(connection, command = nil, *arguments, &block)
  return yield if lock_retries_disabled?

  current_attempt = 0

  begin
    current_attempt += 1

    current_lock_timeout = lock_timeout(current_attempt, command, arguments)
    if current_lock_timeout
      with_lock_timeout(connection, current_lock_timeout.in_milliseconds, &block)
    else
      yield
    end
  rescue ActiveRecord::LockWaitTimeout, ActiveRecord::Deadlocked => e
    if current_attempt <= attempts(command, arguments)
      current_delay = delay(current_attempt, command, arguments)

      problem = e.is_a?(ActiveRecord::Deadlocked) ? "Deadlock detected." : "Lock timeout."
      Utils.say("#{problem} Retrying in #{current_delay} seconds...")

      sleep(current_delay)
      retry
    end
    raise
  end
end