Class: Amigo::Autoscaler::Handlers::Heroku

Inherits:
Amigo::Autoscaler::Handler show all
Defined in:
lib/amigo/autoscaler/handlers/heroku.rb

Overview

Autoscaler to use on Heroku, that starts additional worker processes when there is a high latency event and scales them down after the event is finished.

When the first call of a high latency event happens (depth: 1), this class will ask Heroku how many dynos are in the formation. This is known as active_event_initial_workers.

If active_event_initial_workers is 0, no autoscaling will be done. This avoids a situation where a high latency event is triggered due to workers being deprovisioned intentionally, perhaps for maintenance.

Each time the alert fires (see Amigo::Autoscaler#alert_interval), an additional worker will be added to the formation, up to max_additional_workers. So with active_event_initial_workers of 1 and max_additional_workers of 2, the first time the alert times, the formation will be set to 2 workers. The next time, it’ll be set to 3 workers. After that, no additional workers will be provisioned.

After the high latency event resolves, the dyno formation is restored to active_event_initial_workers.

To use:

heroku = PlatformAPI.connect_oauth(heroku_oauth_token)
heroku_scaler = Amigo::Autoscaler::Heroku.new(heroku:, default_workers: 1)
Amigo::Autoscaler.new(
  handlers: [heroku_scaler.alert_callback],
  latency_restored_handlers: [heroku_scaler.restored_callback],
)

See instance attributes for additional options.

Note that this class is provided as an example, and potentially a base or implementation class. Your actual implementation may also want to alert when a max depth or duration is reached, since it can indicate a bigger problem. Autoscaling, especially of workers, is a tough problem without a one-size-fits-all approach.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(client:, formation:, max_additional_workers: 2, app_id_or_app_name: ENV.fetch("HEROKU_APP_NAME")) ⇒ Heroku

Returns a new instance of Heroku.



80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 80

def initialize(
  client:,
  formation:,
  max_additional_workers: 2,
  app_id_or_app_name: ENV.fetch("HEROKU_APP_NAME")
)
  super()
  @client = client
  @max_additional_workers = max_additional_workers
  @app_id_or_app_name = app_id_or_app_name
  @formation = formation
  # Is nil outside a latency event, set during a latency event. So if this is initialized to non-nil,
  # we're already in a latency event.
  @active_event_initial_workers = Sidekiq.redis do |r|
    v = r.get("#{namespace}/active_event_initial_workers")
    v&.to_i
  end
end

Instance Attribute Details

#active_event_initial_workers ⇒ Integer (readonly)

Captured at the start of a high latency event. Nil otherwise.

Returns:

  • (Integer) 


53
54
55
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 53

def active_event_initial_workers
  @active_event_initial_workers
end

#app_id_or_app_name ⇒ String (readonly)

Defaults to HEROKU_APP_NAME, which should already be set if you use Heroku dyna metadata, as per devcenter.heroku.com/articles/dyno-metadata. This must be provided if the env var is missing.

Returns:

  • (String) 


71
72
73
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 71

def app_id_or_app_name
  @app_id_or_app_name
end

#formation ⇒ String (readonly)

Formation ID or name. Usually ‘worker’ to scale Sidekiq workers, or ‘web’ for the web worker. If you use multiple worker processes for different queues, this class probably isn’t sufficient. You will probably need to look at the slow queue names and determine the formation name to scale up.

Returns:

  • (String) 


78
79
80
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 78

def formation
  @formation
end

#heroku ⇒ PlatformAPI::Client (readonly)

Heroku client, usually created via PlatformAPI.oauth_connect.

Returns:

  • (PlatformAPI::Client) 


48
49
50
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 48

def heroku
  @heroku
end

#max_additional_workers ⇒ Integer (readonly)

Maximum number of workers to add.

As the ‘depth’ of the alert is increased, workers are added to the recorded worker count until the max is reached. By default, this is 2 (so the max workers will be the recorded number, plus 2). Do not set this too high, since it can for example exhaust database connections or just end up increasing load.

See class docs for more information.

Returns:

  • (Integer) 


65
66
67
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 65

def max_additional_workers
  @max_additional_workers
end

Instance Method Details

#scale_down ⇒ :noscale, :scaled

Reset the formation to active_event_initial_workers.

Returns:

  • (:noscale, :scaled) —

    :noscale if active_event_initial_workers is 0, otherwise :scaled.



128
129
130
131
132
133
134
135
136
137
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 128

def scale_down(**)
  initial_workers = @active_event_initial_workers
  Sidekiq.redis do |r|
    r.del("#{namespace}/active_event_initial_workers")
  end
  @active_event_initial_workers = nil
  return :noscale if initial_workers.zero?
  @client.formation.update(@app_id_or_app_name, @formation, {quantity: initial_workers})
  return :scaled
end

#scale_up(depth:) ⇒ :noscale, ...

Potentially add another worker to the formation.

Returns:

  • (:noscale, :maxscale, :scaled) —

    One of :noscale (no active_event_initial_workers), :maxscale (max_additional_workers reached), or :scaled.



106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/amigo/autoscaler/handlers/heroku.rb', line 106

def scale_up(depth:, **)
  # When the scaling event starts (or if this is the first time we've seen it
  # but the event is already in progress), store how many workers we have.
  # It needs to be stored in redis so it persists if
  # the latency event continues through restarts.
  if @active_event_initial_workers.nil?
    @active_event_initial_workers = @client.formation.info(@app_id_or_app_name, @formation).
      fetch("quantity")
    Sidekiq.redis do |r|
      r.set("#{namespace}/active_event_initial_workers", @active_event_initial_workers.to_s)
    end
  end
  return :noscale if @active_event_initial_workers.zero?
  new_quantity = @active_event_initial_workers + depth
  max_quantity = @active_event_initial_workers + @max_additional_workers
  return :maxscale if new_quantity > max_quantity
  @client.formation.update(@app_id_or_app_name, @formation, {quantity: new_quantity})
  return :scaled
end