Class: Gitlab::JobWaiter

Inherits:
Object
  • Object
show all
Defined in:
lib/gitlab/job_waiter.rb

Overview

JobWaiter can be used to wait for a number of Sidekiq jobs to complete.

Its use requires the cooperation of the sidekiq jobs themselves. Set up the waiter, then start the jobs, passing them its ‘key`. Their `perform` methods should look like:

def perform(args, notify_key)
  # do work
ensure
  ::Gitlab::JobWaiter.notify(notify_key, jid)
end

The JobWaiter blocks popping items from a Redis array. All the sidekiq jobs push to that array when done. Once the waiter has popped ‘count` items, it knows all the jobs are done.

Constant Summary collapse

KEY_PREFIX = 
"gitlab:job_waiter"
DEFAULT_TTL =

This TTL needs to be long enough to allow whichever Sidekiq job calls JobWaiter#wait to reach BLPOP.

6.hours.to_i

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(jobs_remaining = 0, key = "#{KEY_PREFIX}:#{SecureRandom.uuid}") ⇒ JobWaiter

jobs_remaining - the number of jobs left to wait for key - The key of this waiter.



53
54
55
56
57
 
# File 'lib/gitlab/job_waiter.rb', line 53

def initialize(jobs_remaining = 0, key = "#{KEY_PREFIX}:#{SecureRandom.uuid}")
  @key = key
  @jobs_remaining = jobs_remaining
  @finished = []
end

Instance Attribute Details

#finished ⇒ Object (readonly)

Returns the value of attribute finished.



48
49
50
 
# File 'lib/gitlab/job_waiter.rb', line 48

def finished
  @finished
end

#jobs_remaining ⇒ Object

Returns the value of attribute jobs_remaining.



49
50
51
 
# File 'lib/gitlab/job_waiter.rb', line 49

def jobs_remaining
  @jobs_remaining
end

#key ⇒ Object (readonly)

Returns the value of attribute key.



48
49
50
 
# File 'lib/gitlab/job_waiter.rb', line 48

def key
  @key
end

Class Method Details

.delete_key(key) ⇒ Object 



44
45
46
 
# File 'lib/gitlab/job_waiter.rb', line 44

def self.delete_key(key)
  Gitlab::Redis::SharedState.with { |redis| redis.del(key) } if key?(key)
end

.generate_key ⇒ Object 



40
41
42
 
# File 'lib/gitlab/job_waiter.rb', line 40

def self.generate_key
  "#{KEY_PREFIX}:#{SecureRandom.uuid}"
end

.key?(key) ⇒ Boolean

Returns:

  • (Boolean) 


36
37
38
 
# File 'lib/gitlab/job_waiter.rb', line 36

def self.key?(key)
  key.is_a?(String) && key =~ /\A#{KEY_PREFIX}:\h{8}-\h{4}-\h{4}-\h{4}-\h{12}\z/o
end

.notify(key, jid, ttl: DEFAULT_TTL) ⇒ Object 



26
27
28
29
30
31
32
33
34
 
# File 'lib/gitlab/job_waiter.rb', line 26

def self.notify(key, jid, ttl: DEFAULT_TTL)
  Gitlab::Redis::SharedState.with do |redis|
    # Use a Redis MULTI transaction to ensure we always set an expiry
    redis.multi do |multi|
      multi.lpush(key, jid)
      multi.expire(key, ttl)
    end
  end
end

Instance Method Details

#wait(timeout = 10) ⇒ Object

Waits for all the jobs to be completed.

timeout - The maximum amount of seconds to block the caller for. This

ensures we don't indefinitely block a caller in case a job takes
long to process, or is never processed.



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/gitlab/job_waiter.rb', line 64

def wait(timeout = 10)
  deadline = Time.now.utc + timeout

  Gitlab::Redis::SharedState.with do |redis|
    while jobs_remaining > 0
      # Redis will not take fractional seconds. Prefer waiting too long over
      # not waiting long enough
      seconds_left = (deadline - Time.now.utc).ceil

      # Redis interprets 0 as "wait forever", so skip the final `blpop` call
      break if seconds_left <= 0

      list, jid = redis.blpop(key, timeout: seconds_left)

      # timed out
      break unless list && jid

      @finished << jid
      @jobs_remaining -= 1
    end
  end

  finished
end