Class: OnlineMigrations::LockRetrier

Inherits:
Object
  • Object
show all
Defined in:
lib/online_migrations/lock_retrier.rb

Overview

This class provides a way to automatically retry code that relies on acquiring a database lock in a way designed to minimize impact on a busy production database.

This class defines an interface for child classes to implement to configure timing configurations and the maximum number of attempts.

There are two predefined implementations (see OnlineMigrations::ConstantLockRetrier and OnlineMigrations::ExponentialLockRetrier). It is easy to provide more sophisticated implementations.

Examples:

Custom LockRetrier implementation

module OnlineMigrations
  class SophisticatedLockRetrier < LockRetrier
    TIMINGS = [
      [0.1.seconds, 0.05.seconds], # first - lock timeout, second - delay time
      [0.1.seconds, 0.05.seconds],
      [0.2.seconds, 0.05.seconds],
      [0.3.seconds, 0.10.seconds],
      [1.second, 5.seconds],
      [1.second, 1.minute],
      [0.1.seconds, 0.05.seconds],
      [0.2.seconds, 0.15.seconds],
      [0.5.seconds, 2.seconds],
      [0.5.seconds, 2.seconds],
      [3.seconds, 3.minutes],
      [0.1.seconds, 0.05.seconds],
      [0.5.seconds, 2.seconds],
      [5.seconds, 2.minutes],
      [7.seconds, 5.minutes],
      [0.5.seconds, 2.seconds],
    ]

    def attempts
      TIMINGS.size
    end

    def lock_timeout(attempt, command = nil, arguments = [])
      TIMINGS[attempt - 1][0]
    end

    def delay(attempt)
      TIMINGS[attempt - 1][1]
    end
  end

Instance Method Summary collapse

Instance Method Details

#attempts(_command = nil, _arguments = []) ⇒ Object

Returns the number of retrying attempts

Raises:

  • (NotImplementedError) 


56
57
58
 
# File 'lib/online_migrations/lock_retrier.rb', line 56

def attempts(_command = nil, _arguments = [])
  raise NotImplementedError
end

#delay(_attempt, _command = nil, _arguments = []) ⇒ Object

Returns sleep time after unsuccessful lock attempt (in seconds)

Raises:

  • (NotImplementedError) 


78
79
80
 
# File 'lib/online_migrations/lock_retrier.rb', line 78

def delay(_attempt, _command = nil, _arguments = [])
  raise NotImplementedError
end

#lock_timeout(_attempt, _command = nil, _arguments = []) ⇒ Object

Returns database lock timeout value (in seconds) for specified attempt number



68
 
# File 'lib/online_migrations/lock_retrier.rb', line 68

def lock_timeout(_attempt, _command = nil, _arguments = []); end

#with_lock_retries(connection, command = nil, *arguments, &block) ⇒ void

This method returns an undefined value.

Executes the block with a retry mechanism that alters the lock_timeout and sleep time between attempts.

Examples:

retrier.with_lock_retries(connection) do
  add_column(:users, :name, :string)
end



93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'lib/online_migrations/lock_retrier.rb', line 93

def with_lock_retries(connection, command = nil, *arguments, &block)
  return yield if lock_retries_disabled?

  current_attempt = 0

  begin
    current_attempt += 1

    current_lock_timeout = lock_timeout(current_attempt, command, arguments)
    if current_lock_timeout
      with_lock_timeout(connection, current_lock_timeout.in_milliseconds, &block)
    else
      yield
    end
  rescue ActiveRecord::LockWaitTimeout, ActiveRecord::Deadlocked => e
    if current_attempt <= attempts(command, arguments)
      current_delay = delay(current_attempt, command, arguments)

      problem = e.is_a?(ActiveRecord::Deadlocked) ? "Deadlock detected." : "Lock timeout."
      Utils.say("#{problem} Retrying in #{current_delay} seconds...")

      sleep(current_delay)
      retry
    end
    raise
  end
end