Class: Furnish::Scheduler

Inherits:
Object
  • Object
show all
Includes:
Logger::Mixins
Defined in:
lib/furnish/scheduler.rb

Overview

This is a scheduler for provisioners. It can run in parallel or serial mode, and is dependency-based, that is, it will only schedule items for execution which have all their dependencies satisfied and items that haven’t will wait to execute until that happens.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Logger::Mixins

#if_debug

Constructor Details

#initialize ⇒ Scheduler

Instantiate the Scheduler.



48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/furnish/scheduler.rb', line 48

def initialize
  @force_deprovision  = false
  @solved_mutex       = Mutex.new
  @serial             = false
  @solver_thread      = nil
  @working_threads    = { }
  @queue              = Queue.new
  @vm                 = Furnish::VM.new
  @recovering         = false
  @signal_handler     = true
end

Instance Attribute Details

#force_deprovision ⇒ Object

Ignore exceptions while deprovisioning. Default is false.



33
34
35
 
# File 'lib/furnish/scheduler.rb', line 33

def force_deprovision
  @force_deprovision
end

#serial ⇒ Object

Turn serial mode on (off by default). This forces the scheduler to execute every provision in order, even if it could handle multiple provisions at the same time.



27
28
29
 
# File 'lib/furnish/scheduler.rb', line 27

def serial
  @serial
end

#signal_handler ⇒ Object

When true, calling #run or #recover also installs a SIGINFO (Ctrl+T in the terminal on macs) and SIGUSR2 handler which can be used to get information on the status of what’s solved and what’s working.

Default is true.



43
44
45
 
# File 'lib/furnish/scheduler.rb', line 43

def signal_handler
  @signal_handler
end

#vm ⇒ Object (readonly)

Access the VM object.



20
21
22
 
# File 'lib/furnish/scheduler.rb', line 20

def vm
  @vm
end

Instance Method Details

#deprovision_group(group_name, clean_state = true) ⇒ Object

Performs the deprovision of a group by replaying its provision strategy backwards and applying the #shutdown method instead of the #startup method. Removes it from the various state tables if true is set as the second argument, which is the default.

While this is a part of the public API, you should probably use #teardown or #teardown_group instead of this method, as they have better error handling and semantics. This “just does it”.



334
335
336
337
 
# File 'lib/furnish/scheduler.rb', line 334

def deprovision_group(group_name, clean_state=true)
  shutdown(group_name)
  delete_group(group_name) if clean_state
end

#needs_recovery ⇒ Object

A map of group name to Furnish::ProvisionerGroup for groups that failed their #startup or #shutdown. See #recover for more information



96
97
98
 
# File 'lib/furnish/scheduler.rb', line 96

def needs_recovery
  vm.need_recovery
end

#needs_recovery? ⇒ Boolean

Is recovery necessary? See #recover.

Returns:

  • (Boolean) 


88
89
90
 
# File 'lib/furnish/scheduler.rb', line 88

def needs_recovery?
  needs_recovery.count > 0
end

#recover ⇒ Object

Initiate recovery. While running, #recovering? will be true.

Recovery will step through all the items in #needs_recovery and attempt to recover them according to Furnish::ProvisionerGroup#recover. If recovery succeeds, the items will be in the solved formula and effectively provisioned. They will also be removed from the needs_recovery information.

If recovery fails, #needs_recovery will not be touched (but the state at which recovery starts the next attempt may be different for those groups). Additionally, the return value of this method will be keyed by the group name, and an exception or false depending on what we got back during recovery. It is strongly recommended you check #needs_recovery? or the return value after calling this to locate flapping groups.

Recovery is a serial process and blocks the main thread. It also installs a signal handler if #signal_handler is set. It does not interrupt or stop the scheduler, but note that in serial mode, the scheduler will likely already be stopped by the time you are able to call recovery. In threaded mode, this means any dependencies that are able to be provisioned after a successful recovery of a group will automatically start provisioning.



206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
 
# File 'lib/furnish/scheduler.rb', line 206

def recover
  install_handler if signal_handler

  @recovering = true

  failures = { }

  needs_recovery.keys.each do |k|
    begin
      group = vm.groups[k]
      result = group.recover(force_deprovision)
      vm.groups[k] = group

      if result
        needs_recovery.delete(k)
        @queue << k
      else
        failures[k] = false
      end
    rescue => e
      failures[k] = e
    end
  end

  if @serial
    begin
      queue_loop
    rescue => e
      if_debug do
        puts "During recovery, serial mode, encountered: #{e}: #{e.message}"
      end
    end
  end

  @recovering = false

  return failures
end

#recovering? ⇒ Boolean

Is recovery running? See #recover.

Returns:

  • (Boolean) 


81
82
83
 
# File 'lib/furnish/scheduler.rb', line 81

def recovering?
  @recovering
end

#run ⇒ Object

Start the scheduler. In serial mode this call will block until the whole dependency graph is satisfied, or one of the provisions fails, at which point an exception will be raised. In parallel mode, this call completes immediately, and you should use #wait_for to control main thread flow, and #running? and #stop to control and monitor the threads this class manages.



165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
 
# File 'lib/furnish/scheduler.rb', line 165

def run
  # short circuit if we're not serial and already running
  return if running?

  install_handler if signal_handler

  if @serial
    service_resolved_waiters
    queue_loop
  else
    @solver_thread = Thread.new do
      with_timeout(false) { service_resolved_waiters }
      queue_loop
    end
  end
end

#running? ⇒ Boolean

Ask the scheduler if it’s running. Returns nil in serial mode.

If there’s an exception waiting and the scheduler has stopped, it will be raised here.

Returns:

  • (Boolean) 


66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/furnish/scheduler.rb', line 66

def running?
  return nil if @serial
  return nil unless @solver_thread
  if @solver_thread.alive?
    return true
  else
    # XXX if there's an exception to be raised, it'll happen here.
    @solver_thread.join
    return nil
  end
end

#schedule_provision(group_name, provisioners, dependencies = []) ⇒ Object Also known as: s, sched

Schedule a group of VMs for provision. This takes a group name, which is a string, an array of provisioner objects, and a list of string dependencies. If anything in the dependencies list hasn’t been pre-declared, it refuses to continue.

This method will return nil if the server group is already provisioned.



108
109
110
111
 
# File 'lib/furnish/scheduler.rb', line 108

def schedule_provision(group_name, provisioners, dependencies=[])
  group = Furnish::ProvisionerGroup.new(provisioners, group_name, dependencies)
  schedule_provisioner_group(group)
end

#schedule_provisioner_group(group) ⇒ Object Also known as: <<

Schedule a provision with a Furnish::ProvisionerGroup. Works exactly like Furnish::Scheduler#schedule_provision otherwise.



120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/furnish/scheduler.rb', line 120

def schedule_provisioner_group(group)
  return nil if vm.groups[group.name]

  vm.groups[group.name] = group

  unless group.dependencies.all? { |x| vm.groups.has_key?(x) }
    raise "One of your dependencies for #{group.name} has not been pre-declared. Cannot continue"
  end

  vm.dependencies[group.name] = group.dependencies

  vm.sync_waiters do |waiters|
    waiters.add(group.name)
  end

  return true
end

#stop ⇒ Object

Instructs the scheduler to stop. Note that this is not an interrupt, and the queue will still be exhausted before terminating.

It is a good idea to check #running? before calling this to ensure the scheduler did not halt with an exception.



252
253
254
255
256
257
258
259
260
261
262
263
264
265
 
# File 'lib/furnish/scheduler.rb', line 252

def stop
  if @serial
    @queue << nil
  else
    @working_threads.values.map { |v| v.join rescue nil }
    if @solver_thread and @solver_thread.alive?
      @queue << nil
      sleep 0.1 until @queue.empty?
      @solver_thread.kill
    end

    @solver_thread = nil
  end
end

#teardown(exceptions = []) ⇒ Object

Instruct all provisioners except ones in the exception list to tear down. Calls #stop as its first action.

This is always done serially. For sanity.

If #force_provision is true, failed shutdowns from provisioners will not halt the deprovisioning process.



316
317
318
319
320
321
322
 
# File 'lib/furnish/scheduler.rb', line 316

def teardown(exceptions=[])
  stop

  (vm.groups.keys.to_set - exceptions.to_set).each do |group_name|
    deprovision_group(group_name) # clean this after everything finishes
  end
end

#teardown_group(group_name, wait = true) ⇒ Object

Teardown a single group – modifies the solved formula. Be careful to resupply dependencies if you use this, as nothing will resolve until you resupply it.

This takes an optional argument to wait for the group to be solved before attempting to tear it down. Setting this to false effectively says, “I know what I’m doing”, and you should feel bad if you file an issue because you supplied it.

If #force_provision is true, failed shutdowns from provisioners will not halt the deprovisioning process.



280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
 
# File 'lib/furnish/scheduler.rb', line 280

def teardown_group(group_name, wait=true)
  wait_for(group_name) if wait

  dependent_items = vm.dependencies.partition { |k,v| v.include?(group_name) }.first.map(&:first)

  if_debug do
    if dependent_items.length > 0
      puts "Trying to terminate #{group_name}, found #{dependent_items.inspect} depending on it"
    end
  end

  @solved_mutex.synchronize do
    dependent_and_working = @working_threads.keys & dependent_items

    if dependent_and_working.count > 0
      if_debug do
        puts "#{dependent_and_working.inspect} are depending on #{group_name}, which you are trying to deprovision."
        puts "We can't resolve this problem for you, and future converges may fail during this run that would otherwise work."
        puts "Consider using wait_for to better control the dependencies, or turning serial provisioning on."
      end
    end

    deprovision_group(group_name)
  end

end

#wait_for(*dependencies) ⇒ Object

Sleep until this list of dependencies are resolved. In parallel mode, will raise if an exception occurred while waiting for these resources. In serial mode, wait_for just returns nil.



145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/furnish/scheduler.rb', line 145

def wait_for(*dependencies)
  return nil if @serial
  return nil if dependencies.empty?

  dep_set = Set[*dependencies]

  until dep_set & vm.solved == dep_set
    sleep 0.1
    @solver_thread.join unless @solver_thread.alive?
  end
end