Class: Amigo::Autoscaler

Inherits:
Object
  • Object
show all
Defined in:
lib/amigo/autoscaler.rb,
lib/amigo/autoscaler/heroku.rb

Defined Under Namespace

Classes: Heroku, InvalidHandler

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(poll_interval: 20, latency_threshold: 5, hostname_regex: /^web\.1$/, handlers: [:log], alert_interval: 120, latency_restored_threshold: latency_threshold, latency_restored_handlers: [:log], log: ->(level, message, params={}) { Amigo.log(nil, level, message, params) }, on_unhandled_exception: nil) ⇒ Autoscaler

Returns a new instance of Autoscaler.

Raises:

  • (ArgumentError) 


103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
 
# File 'lib/amigo/autoscaler.rb', line 103

def initialize(
  poll_interval: 20,
  latency_threshold: 5,
  hostname_regex: /^web\.1$/,
  handlers: [:log],
  alert_interval: 120,
  latency_restored_threshold: latency_threshold,
  latency_restored_handlers: [:log],
  log: ->(level, message, params={}) { Amigo.log(nil, level, message, params) },
  on_unhandled_exception: nil
)
  raise ArgumentError, "latency_threshold must be > 0" if
    latency_threshold <= 0
  raise ArgumentError, "latency_restored_threshold must be >= 0" if
    latency_restored_threshold.negative?
  raise ArgumentError, "latency_restored_threshold must be <= latency_threshold" if
    latency_restored_threshold > latency_threshold
  @poll_interval = poll_interval
  @latency_threshold = latency_threshold
  @hostname_regex = hostname_regex
  @handlers = handlers.freeze
  @alert_interval = alert_interval
  @latency_restored_threshold = latency_restored_threshold
  @latency_restored_handlers = latency_restored_handlers.freeze
  @log = log
  @on_unhandled_exception = on_unhandled_exception
end

Instance Attribute Details

#alert_interval ⇒ Integer (readonly)

Only alert this often. For example, with poll_interval of 10 seconds and alert_interval of 200 seconds, we’d alert once and then 210 seconds later.

Returns:

  • (Integer) 


71
72
73
 
# File 'lib/amigo/autoscaler.rb', line 71

def alert_interval
  @alert_interval
end

#handlers ⇒ Array<String,Symbol,Proc,#call> (readonly)

Methods to call when alerting, as strings/symbols or procs. Valid string values are ‘log’ and ‘sentry’ (requires Sentry to be required already). Anything that responds to call will be invoked with:

  • Positional argument which is a Hash of ‘name => latency in seconds`

  • Keyword argument :depth: Number of alerts as part of this latency event. For example, the first alert has a depth of 1, and if latency stays high, it’ll be 2 on the next call, etc. depth can be used to incrementally provision additional processing capacity, and stop adding capacity at a certain depth to avoid problems with too many workers (like excessive DB load).

  • Keyword argument :duration: Number of seconds since this latency spike started.

  • Additional undefined keywords. Handlers should accept additional options, like via ‘**kw` or `opts={}`, for compatibility.

Returns:

  • (Array<String,Symbol,Proc,#call>) 


65
66
67
 
# File 'lib/amigo/autoscaler.rb', line 65

def handlers
  @handlers
end

#hostname_regex ⇒ Regexp (readonly)

What hosts/processes should this run on? Looks at ENV and Socket.gethostname for a match. Default to only run on ‘web.1’, which is the first Heroku web dyno. We run on the web, not worker, dyno, so we report backed up queues in case we, say, turn off all workers (broken web processes are generally easier to find).

Returns:

  • (Regexp) 


51
52
53
 
# File 'lib/amigo/autoscaler.rb', line 51

def hostname_regex
  @hostname_regex
end

#latency_restored_handlers ⇒ Array<String,Symbol,Proc,#call> (readonly)

Methods to call when a latency of latency_restored_threshold is reached (ie, when we get back to normal latency after a high latency event). Valid string values are ‘log’. Usually this handler will deprovision capacity procured as part of the alert handlers. Anything that responds to call will be invoked with:

  • Keyword :depth, the number of times an alert happened before the latency spike was resolved.

  • Keyword :duration, the number of seconds for the latency spike has been going on.

  • Additional undefined keywords. Handlers should accept additional options, like via ‘**kw`, for compatibility.

Returns:

  • (Array<String,Symbol,Proc,#call>) 


91
92
93
 
# File 'lib/amigo/autoscaler.rb', line 91

def latency_restored_handlers
  @latency_restored_handlers
end

#latency_restored_threshold ⇒ Object (readonly)

After an alert happens, what latency should be considered “back to normal” and latency_restored_handlers will be called? In most cases this should be the same as (and defaults to) latency_threshold so that we’re ‘back to normal’ once we’re below the threshold. It may also commonly be 0, so that the callback is fired when the queue is entirely clear. Note that, if latency_restored_threshold is less than latency_threshold, while the latency is between the two, no alerts will fire.



79
80
81
 
# File 'lib/amigo/autoscaler.rb', line 79

def latency_restored_threshold
  @latency_restored_threshold
end

#latency_threshold ⇒ Integer (readonly)

What latency should we alert on?

Returns:

  • (Integer) 


43
44
45
 
# File 'lib/amigo/autoscaler.rb', line 43

def latency_threshold
  @latency_threshold
end

#log ⇒ Object (readonly)

Proc/callable called with (level, message, params={}). By default, use Amigo.log (which logs to the Sidekiq logger).



94
95
96
 
# File 'lib/amigo/autoscaler.rb', line 94

def log
  @log
end

#on_unhandled_exception ⇒ Object (readonly)

Proc called with an exception that occurs while the thread is running. If the handler returns true, then the thread will keep going. All other values will kill the thread, which breaks autoscaling. Note that Amigo automatically logs unhandled exceptions at :error level. If you use an error reporter like Sentry, you can pass in something like:

-> (e) { Sentry.capture_exception(e) }



101
102
103
 
# File 'lib/amigo/autoscaler.rb', line 101

def on_unhandled_exception
  @on_unhandled_exception
end

#poll_interval ⇒ Integer (readonly)

How often should Autoscaler check for latency?

Returns:

  • (Integer) 


40
41
42
 
# File 'lib/amigo/autoscaler.rb', line 40

def poll_interval
  @poll_interval
end

Instance Method Details

#_check ⇒ Object 



208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
 
# File 'lib/amigo/autoscaler.rb', line 208

def _check
  now = Time.now
  skip_check = now < (@last_alerted + self.alert_interval)
  if skip_check
    self._log(:debug, "async_autoscaler_skip_check")
    return
  end
  self._log(:info, "async_autoscaler_check")
  high_latency_queues = Sidekiq::Queue.all.
    map { |q| [q.name, q.latency] }.
    select { |(_, latency)| latency > self.latency_threshold }.
    to_h
  if high_latency_queues.empty?
    # Whenever we are in a latency event, we have a depth > 0. So a depth of 0 means
    # we're not in a latency event, and still have no latency, so can noop.
    return if @depth.zero?
    # We WERE in a latency event, and now we're not, so report on it.
    @restored_methods.each do |m|
      m.call(depth: @depth, duration: (Time.now - @latency_event_started).to_f)
    end
    # Reset back to 0 depth so we know we're not in a latency event.
    @depth = 0
    @latency_event_started = Time.at(0)
    @last_alerted = now
    self.persist
    return
  end
  if @depth.positive?
    # We have already alerted, so increment the depth and when the latency started.
    @depth += 1
    duration = (Time.now - @latency_event_started).to_f
  else
    # Indicate we are starting a high latency event.
    @depth = 1
    @latency_event_started = Time.now
    duration = 0.0
  end
  # Alert each handler. For legacy reasons, we support handlers that accept
  # ({queues and latencies}) and ({queues and latencies}, {}keywords}).
  kw = {depth: @depth, duration: duration}
  @alert_methods.each do |m|
    if m.respond_to?(:arity) && m.arity == 1
      m.call(high_latency_queues)
    else
      m.call(high_latency_queues, **kw)
    end
  end
  @last_alerted = now
  self.persist
end

#alert_log(names_and_latencies, depth:, duration:) ⇒ Object 



267
268
269
 
# File 'lib/amigo/autoscaler.rb', line 267

def alert_log(names_and_latencies, depth:, duration:)
  self._log(:warn, "high_latency_queues", queues: names_and_latencies, depth: depth, duration: duration)
end

#alert_restored_log(depth:, duration:) ⇒ Object 



273
274
275
 
# File 'lib/amigo/autoscaler.rb', line 273

def alert_restored_log(depth:, duration:)
  self._log(:info, "high_latency_queues_restored", depth: depth, duration: duration)
end

#alert_sentry(names_and_latencies) ⇒ Object 



259
260
261
262
263
264
265
 
# File 'lib/amigo/autoscaler.rb', line 259

def alert_sentry(names_and_latencies)
  Sentry.with_scope do |scope|
    scope.set_extras(high_latency_queues: names_and_latencies)
    names = names_and_latencies.map(&:first).sort.join(", ")
    Sentry.capture_message("Some queues have a high latency: #{names}")
  end
end

#alert_test(_names_and_latencies, _opts = {}) ⇒ Object 



271
 
# File 'lib/amigo/autoscaler.rb', line 271

def alert_test(_names_and_latencies, _opts={}); end

#check ⇒ Object 



200
201
202
203
204
205
206
 
# File 'lib/amigo/autoscaler.rb', line 200

def check
  self._check
rescue StandardError => e
  self._log(:error, "async_autoscaler_unhandled_error", exception: e)
  handled = self.on_unhandled_exception&.call(e)
  raise e unless handled.eql?(true)
end

#polling_thread ⇒ Thread

Returns:

  • (Thread) 


132
133
134
 
# File 'lib/amigo/autoscaler.rb', line 132

def polling_thread
  return @polling_thread
end

#setup ⇒ Object 



136
137
138
139
140
141
142
143
144
145
146
147
 
# File 'lib/amigo/autoscaler.rb', line 136

def setup
  # Store these as strings OR procs, rather than grabbing self.method here.
  # It gets extremely hard ot test if we capture the method here.
  @alert_methods = self.handlers.map { |a| _handler_to_method("alert_", a) }
  @restored_methods = self.latency_restored_handlers.map { |a| _handler_to_method("alert_restored_", a) }
  @stop = false
  Sidekiq.redis do |r|
    @last_alerted = Time.at((r.get("#{namespace}/last_alerted") || 0).to_f)
    @depth = (r.get("#{namespace}/depth") || 0).to_i
    @latency_event_started = Time.at((r.get("#{namespace}/latency_event_started") || 0).to_f)
  end
end

#start ⇒ Object 



179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
 
# File 'lib/amigo/autoscaler.rb', line 179

def start
  raise "already started" unless @polling_thread.nil?

  hostname = ENV.fetch("DYNO") { Socket.gethostname }
  return false unless self.hostname_regex.match?(hostname)

  self._log(:info, "async_autoscaler_starting")
  self.setup
  @polling_thread = Thread.new do
    until @stop
      Kernel.sleep(self.poll_interval)
      self.check unless @stop
    end
  end
  return true
end

#stop ⇒ Object 



196
197
198
 
# File 'lib/amigo/autoscaler.rb', line 196

def stop
  @stop = true
end

#unpersist ⇒ Object

Delete all the keys that Autoscaler stores. Can be used in extreme cases where things need to be cleaned up, but should not be normally used.



160
161
162
163
164
165
166
 
# File 'lib/amigo/autoscaler.rb', line 160

def unpersist
  Sidekiq.redis do |r|
    r.del("#{namespace}/last_alerted")
    r.del("#{namespace}/depth")
    r.del("#{namespace}/latency_event_started")
  end
end