Module: Xolo::Server::Helpers::Maintenance

Defined in:
lib/xolo/server/helpers/maintenance.rb

Overview

Nightly cleanup of deprecated and skipped packages.

Also, alerts will be posted, and Emails will be sent to the admins who added versions that have been in pilot for more than some period of time.

Constant Summary collapse

CLEANUP_HOUR =

At what hour should the nightly cleanup run?

2
UNRELEASED_PILOTS_NOTIFICATION_DAY =

on which day of the month should we send the unreleased pilot notifications?

1
DFT_DEPRECATED_LIFETIME_DAYS =

Once a version becomes deprecated, it will be automatically deleted this many days later. If not set in the server config, this is the default value. use 0 or less to disable cleanup of deprecated versions

30
DFT_UNRELEASED_PILOTS_NOTIFICATION_DAYS =

If a pilot has not been released in this many days, notify someone about it weekly, asking to release it or delete it. If not set in the server config, this is the default value.

180
SERVER_LAUNCHD_PLIST =

when doing a full shutdown, we need to unload the launchd plist

Pathname.new '/Library/LaunchDaemons/com.pixar.xoloserver.plist'

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.cleanup_mutex ⇒ Mutex

A mutex for the cleanup process

TODO: use Concrrent Ruby instead of Mutex

Returns:

  • (Mutex) —

    the mutex



64
65
66
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 64

def self.cleanup_mutex
  @cleanup_mutex ||= Mutex.new
end

.cleanup_timer_task ⇒ Concurrent::TimerTask

nightly cleanup is done by a Concurrent::TimerTask, which checks every hour to see if it should do anything.

It will only do the cleanup if the current time is in the 2am hour (02:00 - 02:59)

We trigger the cleanup by POSTing to /cleanup, so that it runs in the context of a request, having access to Title and Version instantiation.

Returns:

  • (Concurrent::TimerTask) —

    the timed task to do log rotation



79
80
81
82
83
84
85
86
87
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 79

def self.cleanup_timer_task
  return @cleanup_timer_task if @cleanup_timer_task

  @cleanup_timer_task =
    Concurrent::TimerTask.new(execution_interval: 3600) { post_to_start_cleanup }

  Xolo::Server.logger.info 'Created Concurrent::TimerTask for nightly cleanup.'
  @cleanup_timer_task
end

.included(includer) ⇒ Object

when this module is included



25
26
27
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 25

def self.included(includer)
  Xolo.verbose_include includer, self
end

.last_cleanup ⇒ Time

When was our last cleanup?

Returns:

  • (Time) —

    the time of the last cleanup, or the epoch if never



92
93
94
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 92

def self.last_cleanup
  @last_cleanup ||= Time.at(0)
end

.last_cleanup=(time) ⇒ Time

Set the time of the last cleanup

Parameters:

  • time (Time) —

    the time of the last cleanup

Returns:

  • (Time) —

    the time of the last cleanup



100
101
102
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 100

def self.last_cleanup=(time)
  @last_cleanup = time
end

.post_to_start_cleanup(force: false) ⇒ void

This method returns an undefined value.

post to the server to start the cleanup process This is done so that the cleanup can run in the context of a request, having access to Title and Version instantiation.

Parameters:

  • force (Boolean) (defaults to: false) —

    force the cleanup to run now



111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 111

def self.post_to_start_cleanup(force: false)
  if Xolo::Server.shutting_down?
    Xolo::Server.logger.info 'Not starting cleanup, server is shutting down'
    return
  end

  # only run the cleanup if it's between 2am and 3am
  # and the last one was more than 23 hrs ago
  last_cleanup_hrs_ago = (Time.now - last_cleanup) / 3600
  return unless force || (Time.now.hour == CLEANUP_HOUR && last_cleanup_hrs_ago > 23)

  uri = URI.parse "https://#{Xolo::Server::Helpers::Auth::IPV4_LOOPBACK}/maint/cleanup-internal"
  https = Net::HTTP.new(uri.host, uri.port)
  https.use_ssl = true
  # The server cert may be self-signed and/or doesn't
  # match the hostname, so we need to disable verification
  https.verify_mode = OpenSSL::SSL::VERIFY_NONE

  request = Net::HTTP::Post.new(uri.path)
  request['Authorization'] = Xolo::Server::Helpers::Auth.internal_auth_token_header

  response = https.request(request)
  Xolo::Server.logger.info "Cleanup request response: #{response.code} #{response.body}"
end

Instance Method Details

#accept_title_editor_eas ⇒ void

This method returns an undefined value.

look for any titles that need their Title Editor EA’s accepted, and auto accept them if we need to



167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 167

def accept_title_editor_eas
  unless Xolo::Server.config.jamf_auto_accept_xolo_eas
    log_info 'Cleanup: The xolo server is not configured to auto-accept Title Editor EAs'
    return
  end

  log_info 'Cleanup: Looking for Title Editor EAs to auto-accept'

  # TODO: Be DRY with this stuff and similar in title_jamf_access.rb
  Xolo::Server::Title.all_titles.each do |title|
    title_obj = instantiate_title title
    next unless title_obj.jamf_patch_ea_awaiting_acceptance?

    log_info "Cleanup: Auto-accepting Title Editor EA for title '#{title}'"
    title_obj.accept_jamf_patch_ea_via_api
  rescue => e
    log_error "Cleanup: Error auto-accepting Title Editor EA for title '#{title}': #{e}"
  end # Xolo::Server::Title.all_titles.each

  log_info 'Cleanup: Done with Title Editor EAs to auto-accept'
end

#cleanup_deprecated_version(version) ⇒ void

This method returns an undefined value.

Cleanup a deprecated version.

Parameters:



217
218
219
220
221
222
223
224
225
226
227
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 217

def cleanup_deprecated_version(version)
  # do nothing if the deprecated_lifetime_days is 0 or less
  return unless deprecated_lifetime_days.positive?

  # how many days has this version been deprecated?
  days_deprecated = (Time.now - version.deprecation_date) / 86_400
  return unless days_deprecated > deprecated_lifetime_days

  log_info "Cleanup: Deleting deprecated version '#{version.version}' of title '#{version.title}'"
  version.delete
end

#cleanup_skipped_version(version) ⇒ void

This method returns an undefined value.

Cleanup a skipped version.

Parameters:



233
234
235
236
237
238
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 233

def cleanup_skipped_version(version)
  return if Xolo::Server.config.keep_skipped_versions

  log_info "Cleanup: Deleting skipped version '#{version.version}' of title '#{version.title}'"
  version.delete
end

#cleanup_versions ⇒ void

This method returns an undefined value.

Cleanup versions.



192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 192

def cleanup_versions
  log_info 'Cleanup: cleaning up deprecated and skipped versions'

  Xolo::Server::Title.all_titles.each do |title|
    title_obj = instantiate_title title

    title_obj.version_objects.each do |version|
      if version.deprecated?
        cleanup_deprecated_version version
      elsif version.skipped?
        cleanup_skipped_version version
      end # case
    end # each version

    notify_admins_of_unreleased_pilots(title_obj)
  end # each title

  Xolo::Server::Helpers::Maintenance.last_cleanup = Time.now
  log_info 'Cleanup: versions cleanup complete'
end

#deprecated_lifetime_days ⇒ Integer

how many days can a version be deprecated?

Returns:

  • (Integer) —

    the number of days a version can be deprecated



273
274
275
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 273

def deprecated_lifetime_days
  @deprecated_lifetime_days ||= Xolo::Server.config.deprecated_lifetime_days || DFT_DEPRECATED_LIFETIME_DAYS
end

#notify_admins_of_unreleased_pilots(title_obj) ⇒ void

This method returns an undefined value.

Notify the admins about unreleased pilots if needed



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 243

def notify_admins_of_unreleased_pilots(title_obj)
  return unless Time.now.day == UNRELEASED_PILOTS_NOTIFICATION_DAY
  return unless unreleased_pilots_notification_days.positive?
  return unless title_obj.latest_version

  latest_vers_obj = instantiate_version title: title_obj, version: title_obj.latest_version
  return unless latest_vers_obj.pilot?

  days_in_pilot = ((Time.now - latest_vers_obj.creation_date) / 86_400).to_i

  return unless days_in_pilot > unreleased_pilots_notification_days

  alert_msg = "Cleanup: Notifying #{title_obj.contact_email} about unreleased pilot '#{latest_vers}' of title '#{title_obj.title}', in pilot for #{days_in_pilot} days"

  log_info alert_msg
  send_alert alert_msg

  email_msg = <<~MSG
    The newest version '#{latest_vers_obj.version}' of title '#{title_obj.title}' has been in pilot for #{days_in_pilot} days, which makes it seem like it's not going to be released.

    To reduce clutter, please consider releasing it, deleting it, or deleting the whole title if it's no longer needed.

    If this is intentional, you can ignore this monthly message.
  MSG
  send_email to: title_obj.contact_email, subject: 'Unreleased Pilot Notification', msg: email_msg
end

#run_cleanup ⇒ void

This method returns an undefined value.

Cleanup things that need to be cleaned up



139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 139

def run_cleanup
  if Xolo::Server.shutting_down?
    log_info 'Cleanup: Not starting cleanup, server is shutting down'
    return
  end
  # TODO: Use Concurrent ruby rather than this instance variable
  mutex = Xolo::Server::Helpers::Maintenance.cleanup_mutex

  if mutex.locked?
    log_warn 'Cleanup: already running, skipping this run'
    return
  end
  mutex.lock
  log_info 'Cleanup: starting'

  # add new cleanup tasks/methods here
  accept_title_editor_eas
  cleanup_versions

  log_info 'Cleanup: complete'
ensure
  mutex&.unlock
end

#shutdown_pkg_deletion_pool ⇒ void

This method returns an undefined value.

Shutdown the pkg deletion pool



374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 374

def shutdown_pkg_deletion_pool
  # Start the shutdown of the pkg_deletion_pool. Will finish anything
  # in the queue, but not accept any new tasks.
  pkg_pool = Xolo::Server::Version.pkg_deletion_pool
  pkg_pool.shutdown
  pkg_pool_shutdown_start = Time.now
  progress 'Shutting down pkg deletion pool', log: :info
  # returns true when shutdown is complete
  until pkg_pool.wait_for_termination(20)
    msg = "..Waiting for pkg deletion pool to finish, processing: #{pkg_pool.length}, in queue: #{pkg_pool.queue_length}"
    progress msg, log: :debug
    next unless Time.now - pkg_pool_shutdown_start > Xolo::Server::Constants::MAX_JAMF_WAIT_FOR_PKG_DELETION

    msg = 'ERROR: Timeout waiting for pkg deletion pool to finish, some pkgs may not be deleted'
    progress msg, log: :error
    pkg_pool.kill
    break
  end
  progress 'Pkg deletion queue is empty'
end

#shutdown_server(restart) ⇒ void

This method returns an undefined value.

Shutdown the server



287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 287

def shutdown_server(restart)
  # let all the routes know we are shutting down
  Xolo::Server.shutting_down = true

  progress "Server Shutdown by #{session[:admin]}", log: :info

  stop_cleanup_timer_task
  stop_log_rotation_timer_task
  shutdown_pkg_deletion_pool
  wait_for_object_locks
  wait_for_progress_streams

  # without unloading the launchd job, the server will restart automatically
  # when we tell it to quit
  if restart
    progress 'Restarting the server now', log: :info
    Xolo::Server::App.quit!
  else
    progress 'Shutting down the server now', log: :info
    unload_server_launchd
  end
end

#stop_cleanup_timer_task ⇒ void

This method returns an undefined value.

Stop the cleanup timer task



321
322
323
324
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 321

def stop_cleanup_timer_task
  progress 'Stopping the cleanup timer task', log: :info
  Xolo::Server::Helpers::Maintenance.cleanup_timer_task.shutdown
end

#stop_log_rotation_timer_task ⇒ void

This method returns an undefined value.

Stop the log rotation timer task



329
330
331
332
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 329

def stop_log_rotation_timer_task
  progress 'Stopping the log rotation timer task', log: :info
  Xolo::Server::Log.log_rotation_timer_task.shutdown
end

#unload_server_launchd ⇒ void

This method returns an undefined value.

full shutdown of the server by unloading the launchd plist



313
314
315
316
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 313

def unload_server_launchd
  log_info 'Unloading the server launchd plist'
  system "/bin/launchctl unload #{SERVER_LAUNCHD_PLIST}"
end

#unreleased_pilots_notification_days ⇒ Object

Notify the admins about unreleased pilots when the newest one is older than this many days.



279
280
281
282
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 279

def unreleased_pilots_notification_days
  @unreleased_pilots_notification_days ||=
    Xolo::Server.config.unreleased_pilots_notification_days || DFT_UNRELEASED_PILOTS_NOTIFICATION_DAYS
end

#wait_for_object_locks ⇒ void

This method returns an undefined value.

Wait for all object locks to be released



337
338
339
340
341
342
343
344
345
346
347
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 337

def wait_for_object_locks
  Xolo::Server.remove_expired_object_locks

  until Xolo::Server.object_locks.empty?
    progress 'Waiting for object locks to be released', log: :info
    log_debug "Object locks: #{Xolo::Server.object_locks.inspect}"
    sleep 5
    Xolo::Server.remove_expired_object_locks
  end
  progress 'All object locks released', log: :info
end

#wait_for_progress_streams ⇒ void

This method returns an undefined value.

Wait for all progress streams to finish



352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
 
# File 'lib/xolo/server/helpers/maintenance.rb', line 352

def wait_for_progress_streams
  prefix = Xolo::Server::Helpers::ProgressStreaming::PROGRESS_THREAD_NAME_PREFIX
  prog_threads = Thread.list.select { |th| th.name.to_s.start_with? prefix }
  # remove our own thread from the list
  prog_threads.delete Thread.current
  prog_threads.delete @streaming_thread

  until prog_threads.empty?
    progress 'Waiting for progress streams to finish', log: :info
    log_debug "Progress stream threads: #{prog_threads.map(&:name)}}"
    sleep 5
    prog_threads = Thread.list.select { |th| th.name.to_s.start_with? prefix }
    # remove our own thread from the list
    prog_threads.delete Thread.current
    prog_threads.delete @streaming_thread
  end
  progress 'All progress streams finished', log: :info
end