Class: Hatchet::Reaper

Inherits:
Object
  • Object
show all
Defined in:
lib/hatchet/reaper.rb,
lib/hatchet/reaper/app_age.rb,
lib/hatchet/reaper/reaper_throttle.rb

Overview

This class lazilly deletes hatchet apps

When the reaper is called, it will check if the system has too many apps (Bassed off of HATCHET_APP_LIMIT), if so it will attempt to delete an app to free up capacity. The goal of lazilly deleting apps is to temporarilly keep apps around for debugging if they fail.

When App#teardown! is called on an app it is marked as being in a “finished” state by turning on maintenance mode. The reaper will delete these in order (oldest first).

If no apps are marked as being “finished” then the reaper will check to see if the oldest app has been alive for a long enough period for it’s tests to finish (configured by HATCHET_ALIVE_TTL_MINUTES env var). If the “unfinished” app has been alive that long it will be deleted. If not, the system will sleep for a period of time in an attempt to allow other apps to move to be “finished”.

This class only limits and the number of “hatchet” apps on the system. Prevously there was a maximum of 100 apps on a Heroku account. Now a user can belong to multiple orgs and the total number of apps they have access to is no longer fixed at 100. Instead of hard coding a maximum limit, this failure mode is handled by forcing deletion of an app when app creation fails. In the future we may find a better way of detecting this failure mode

Notes:

  • The class uses a file mutex so that multiple processes on the same machine do not attempt to run the reaper at the same time.

  • AlreadyDeletedError will be raised if an app has already been deleted (possibly by another test run on another machine). When this happens, the system will automatically attempt to reap another app.

Defined Under Namespace

Classes: AlreadyDeletedError, AppAge, ReaperThrottle

Constant Summary collapse

HATCHET_APP_LIMIT =

the number of apps hatchet keeps around

Integer(ENV["HATCHET_APP_LIMIT"] || 20)
DEFAULT_REGEX = 
/^#{Regexp.escape(Hatchet::APP_PREFIX)}[a-f0-9]+/

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(api_rate_limit:, regex: DEFAULT_REGEX, io: STDOUT, hatchet_app_limit: HATCHET_APP_LIMIT, initial_sleep: 10) ⇒ Reaper

Returns a new instance of Reaper.



37
38
39
40
41
42
43
44
45
46
 
# File 'lib/hatchet/reaper.rb', line 37

def initialize(api_rate_limit: , regex: DEFAULT_REGEX, io: STDOUT, hatchet_app_limit:  HATCHET_APP_LIMIT, initial_sleep: 10)
  @api_rate_limit = api_rate_limit
  @regex = regex
  @io = io
  @finished_hatchet_apps = []
  @unfinished_hatchet_apps = []
  @app_count = 0
  @hatchet_app_limit = hatchet_app_limit
  @reaper_throttle = ReaperThrottle.new(initial_sleep: initial_sleep)
end

Instance Attribute Details

#hatchet_app_limit ⇒ Object

Returns the value of attribute hatchet_app_limit.



35
36
37
 
# File 'lib/hatchet/reaper.rb', line 35

def hatchet_app_limit
  @hatchet_app_limit
end

#io ⇒ Object

Returns the value of attribute io.



35
36
37
 
# File 'lib/hatchet/reaper.rb', line 35

def io
  @io
end

Instance Method Details

#cycle(app_exception_message: false) ⇒ Object 



48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/hatchet/reaper.rb', line 48

def cycle(app_exception_message: false)
  # Protect against parallel deletion of the same app on the same system
  mutex_file = File.open("#{Dir.tmpdir()}/hatchet_reaper_mutex", File::CREAT)
  mutex_file.flock(File::LOCK_EX)

  refresh_app_list if @finished_hatchet_apps.empty?

  # To be safe try to delete an app even if we're not over the limit
  # since the exception may have been caused by going over the maximum account limit
  if app_exception_message
      io.puts <<-EOM.strip_heredoc
        WARNING: Running reaper due to exception on app
                 #{stats_string}
                 Exception: #{app_exception_message}
      EOM
    reap_once
  end

  while over_limit?
    reap_once
  end
ensure
  mutex_file.close
end

#destroy_all ⇒ Object

No guardrails, will delete all apps that match the hatchet namespace



82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/hatchet/reaper.rb', line 82

def destroy_all
  refresh_app_list

  (@finished_hatchet_apps + @unfinished_hatchet_apps).each do |app|
    begin
      destroy_with_log(name: app["name"], id: app["id"])
    rescue AlreadyDeletedError
      # Ignore, keep going
    end
  end
end

#over_limit? ⇒ Boolean

Returns:

  • (Boolean) 


77
78
79
 
# File 'lib/hatchet/reaper.rb', line 77

def over_limit?
  hatchet_app_count > hatchet_app_limit
end

#stats_string ⇒ Object 



73
74
75
 
# File 'lib/hatchet/reaper.rb', line 73

def stats_string
  "total_app_count: #{@app_count}, hatchet_app_count: #{hatchet_app_count}/#{HATCHET_APP_LIMIT}, finished: #{@finished_hatchet_apps.length}, unfinished: #{@unfinished_hatchet_apps.length}"
end