Class: Amigo::MemoryPressure

Inherits:
Object
  • Object
show all
Defined in:
lib/amigo/memory_pressure.rb

Overview

Helper class to detect when the Redis server is under memory pressure. In these cases, we want to disable queue backoff behavior. There is a significant risk that the backoff behavior will take jobs from the queue, and immediately try and reschedule them. If that happens in an OOM condition, the re-push will fail and the job can be lost.

Additionally, the backoff behavior causes delays that slow down the clearing of the queue.

In these high-memory-utilization conditions, it makes more sense to disable the backoff logic and just brute force to try to get through the queue.

Constant Summary collapse

DEFAULT_THRESHOLD =

Percentage at which the server is considered under memory pressure.

90
DEFAULT_CHECK_TTL =

Default seconds a memory check is good for. See check_ttl.

120

Class Attribute Summary collapse

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(check_ttl: DEFAULT_CHECK_TTL, threshold: DEFAULT_THRESHOLD) ⇒ MemoryPressure

Returns a new instance of MemoryPressure.



44
45
46
47
48
49
 
# File 'lib/amigo/memory_pressure.rb', line 44

def initialize(check_ttl: DEFAULT_CHECK_TTL, threshold: DEFAULT_THRESHOLD)
  @last_checked_at = nil
  @check_ttl = check_ttl
  @threshold = threshold
  @last_check_result = nil
end

Class Attribute Details

.instance ⇒ Object

Return the singleton instance, creating a cached value if needed.



23
24
25
 
# File 'lib/amigo/memory_pressure.rb', line 23

def instance
  return @instance ||= self.new
end

Instance Attribute Details

#check_ttl ⇒ Object (readonly)

See DEFAULT_CHECK_TTL.



39
40
41
 
# File 'lib/amigo/memory_pressure.rb', line 39

def check_ttl
  @check_ttl
end

#last_check_result ⇒ Object (readonly)

What was the result of the last check? true is under pressure, false if not.



36
37
38
 
# File 'lib/amigo/memory_pressure.rb', line 36

def last_check_result
  @last_check_result
end

#last_checked_at ⇒ Object (readonly)

When did we last check for pressure?



32
33
34
 
# File 'lib/amigo/memory_pressure.rb', line 32

def last_checked_at
  @last_checked_at
end

#threshold ⇒ Object (readonly)

See DEFAULT_THRESHOLD.



42
43
44
 
# File 'lib/amigo/memory_pressure.rb', line 42

def threshold
  @threshold
end

Instance Method Details

#get_memory_info ⇒ Object 



77
78
79
80
 
# File 'lib/amigo/memory_pressure.rb', line 77

def get_memory_info
  s = self.get_memory_info_string
  return self.parse_memory_string(s)
end

#under_pressure? ⇒ Boolean

Return true if the server is under memory pressure. When this is the case, we want to disable backoff, since it will delay working through the queue, and can also result in a higher likelihood of lost jobs, since returning them back to the queue will fail.

Returns:

  • (Boolean) 


56
57
58
59
60
61
 
# File 'lib/amigo/memory_pressure.rb', line 56

def under_pressure?
  return @last_check_result unless self.needs_check?
  @last_check_result = self.calculate_under_pressure
  @last_checked_at = Time.now
  return @last_check_result
end