Class: Amigo::Autoscaler::Heroku

Inherits:
Object
  • Object
show all
Defined in:
lib/amigo/autoscaler/heroku.rb

Overview

Autoscaler to use on Heroku, that starts additional worker processes when there is a high latency event and scales them down after the event is finished.

When the first call of a high latency event happens (depth: 1), this class will ask Heroku how many dynos are in the formation. This is known as active_event_initial_workers.

If active_event_initial_workers is 0, no autoscaling will be done. This avoids a situation where a high latency event is triggered due to workers being deprovisioned intentionally, perhaps for maintenance.

Each time the alert fires (see Amigo::Autoscaler#alert_interval), an additional worker will be added to the formation, up to max_additional_workers. So with active_event_initial_workers of 1 and max_additional_workers of 2, the first time the alert times, the formation will be set to 2 workers. The next time, it’ll be set to 3 workers. After that, no additional workers will be provisioned.

After the high latency event resolves, the dyno formation is restored to active_event_initial_workers.

To use:

heroku = PlatformAPI.connect_oauth(heroku_oauth_token)
heroku_scaler = Amigo::Autoscaler::Heroku.new(heroku:, default_workers: 1)
Amigo::Autoscaler.new(
  handlers: [heroku_scaler.alert_callback],
  latency_restored_handlers: [heroku_scaler.restored_callback],
)

See instance attributes for additional options.

Note that this class is provided as an example, and potentially a base or implementation class. Your actual implementation may also want to alert when a max depth or duration is reached, since it can indicate a bigger problem. Autoscaling, especially of workers, is a tough problem without a one-size-fits-all approach.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(heroku:, max_additional_workers: 2, app_id_or_app_name: ENV.fetch("HEROKU_APP_NAME"), formation_id_or_formation_type: "worker") ⇒ Heroku

Returns a new instance of Heroku.



78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
 
# File 'lib/amigo/autoscaler/heroku.rb', line 78

def initialize(
  heroku:,
  max_additional_workers: 2,
  app_id_or_app_name: ENV.fetch("HEROKU_APP_NAME"),
  formation_id_or_formation_type: "worker"
)

  @heroku = heroku
  @max_additional_workers = max_additional_workers
  @app_id_or_app_name = app_id_or_app_name
  @formation_id_or_formation_type = formation_id_or_formation_type
  # Is nil outside of a latency event, set during a latency event. So if this is initialized to non-nil,
  # we're already in a latency event.
  @active_event_initial_workers = Sidekiq.redis do |r|
    v = r.get("#{namespace}/active_event_initial_workers")
    v&.to_i
  end
end

Instance Attribute Details

#active_event_initial_workers ⇒ Integer (readonly)

Captured at the start of a high latency event. Nil otherwise.

Returns:

  • (Integer) 


52
53
54
 
# File 'lib/amigo/autoscaler/heroku.rb', line 52

def active_event_initial_workers
  @active_event_initial_workers
end

#app_id_or_app_name ⇒ String (readonly)

Defaults to HEROKU_APP_NAME, which should already be set if you use Heroku dyna metadata, as per devcenter.heroku.com/articles/dyno-metadata. This must be provided if the env var is missing.

Returns:

  • (String) 


70
71
72
 
# File 'lib/amigo/autoscaler/heroku.rb', line 70

def app_id_or_app_name
  @app_id_or_app_name
end

#formation_id_or_formation_type ⇒ String (readonly)

Defaults to ‘worker’, which is what you’ll probably use if you have a simple system. If you use multiple worker processes for different queues, this class probably isn’t sufficient. You will probably need to look at the slow queue names and determine the formation name to scale up.

Returns:

  • (String) 


76
77
78
 
# File 'lib/amigo/autoscaler/heroku.rb', line 76

def formation_id_or_formation_type
  @formation_id_or_formation_type
end

#heroku ⇒ PlatformAPI::Client (readonly)

Heroku client, usually created via PlatformAPI.oauth_connect.

Returns:

  • (PlatformAPI::Client) 


47
48
49
 
# File 'lib/amigo/autoscaler/heroku.rb', line 47

def heroku
  @heroku
end

#max_additional_workers ⇒ Integer (readonly)

Maximum number of workers to add.

As the ‘depth’ of the alert is increased, workers are added to the recorded worker count until the max is reached. By default, this is 2 (so the max workers will be the recorded number, plus 2). Do not set this too high, since it can for example exhaust database connections or just end up increasing load.

See class docs for more information.

Returns:

  • (Integer) 


64
65
66
 
# File 'lib/amigo/autoscaler/heroku.rb', line 64

def max_additional_workers
  @max_additional_workers
end

Instance Method Details

#alert_callback ⇒ Object 



97
98
99
 
# File 'lib/amigo/autoscaler/heroku.rb', line 97

def alert_callback
  self.method(:scale_up)
end

#restored_callback ⇒ Object 



101
102
103
 
# File 'lib/amigo/autoscaler/heroku.rb', line 101

def restored_callback
  self.method(:scale_down)
end

#scale_down ⇒ :noscale, :scaled

Reset the formation to active_event_initial_workers.

Returns:

  • (:noscale, :scaled) —

    :noscale if active_event_initial_workers is 0, otherwise :scaled.



134
135
136
137
138
139
140
141
142
143
 
# File 'lib/amigo/autoscaler/heroku.rb', line 134

def scale_down(**)
  initial_workers = @active_event_initial_workers
  Sidekiq.redis do |r|
    r.del("#{namespace}/active_event_initial_workers")
  end
  @active_event_initial_workers = nil
  return :noscale if initial_workers.zero?
  @heroku.formation.update(@app_id_or_app_name, @formation_id_or_formation_type, {quantity: initial_workers})
  return :scaled
end

#scale_up(_queues_and_latencies, depth:) ⇒ :noscale, ...

Potentially add another worker to the formation.

Returns:

  • (:noscale, :maxscale, :scaled) —

    One of :noscale (no active_event_initial_workers), :maxscale (max_additional_workers reached), or :scaled.



112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/amigo/autoscaler/heroku.rb', line 112

def scale_up(_queues_and_latencies, depth:, **)
  # When the scaling event starts (or if this is the first time we've seen it
  # but the event is already in progress), store how many workers we have.
  # It needs to be stored in redis so it persists if
  # the latency event continues through restarts.
  if @active_event_initial_workers.nil?
    @active_event_initial_workers = @heroku.formation.info(@app_id_or_app_name, @formation_id_or_formation_type).
      fetch("quantity")
    Sidekiq.redis do |r|
      r.set("#{namespace}/active_event_initial_workers", @active_event_initial_workers.to_s)
    end
  end
  return :noscale if @active_event_initial_workers.zero?
  new_quantity = @active_event_initial_workers + depth
  max_quantity = @active_event_initial_workers + @max_additional_workers
  return :maxscale if new_quantity > max_quantity
  @heroku.formation.update(@app_id_or_app_name, @formation_id_or_formation_type, {quantity: new_quantity})
  return :scaled
end