periodically

Redis-backed Ruby library for periodically running tasks, whose execution depends can depend on a custom lambda block (e.g. weekly syncs). The job execution won't be guaranteed to happen exactly as the condition is fulfilled, hence periodically is best for nonfrequent and noncritical jobs.

Example usecases:

• Sync a Rails model's data once per week
  • Achievable by e.g. a last_synced value in the database
• Launch a non-important sync operation depending on a specific condition (e.g. NULL value in some database column)
  • Achievable by checking the column against NULL inside the on condition

Example usage with Rails

# config/initializers/periodically.rb
require "periodically"

# Note: in development mode classes are loaded lazily, so periodical jobs only start once classes have been loaded
# In production classes are loaded eagerly, so this is no problem

Periodically.start

# app/models/item.rb

class Item < ApplicationRecord
  include Periodically::Model

  periodically :refresh_price,
                  on: -> { Item.where("last_synced < ?", 7.days.ago) }

  private

  def refresh_price
    self.price = PriceFetcher.fetch(item_id)
    save!
  end
end

Example usage with pure Ruby

# TODO

Execution model

Periodically launches a single background thread, which executes registered queries every x seconds. If a pending query is found, the registered callback method is called in the same thread. Hence, a blocking callback method will also block execution of other pending queries.

By default everything happens in the same process as the main Rails web server. To parallelize processing a bit, you can do bundle exec periodically to start a new process for jobs. (Remember to remove Periodically.start from initializer!)

API

Terminology

• Job: something enqueued to be called using the periodically method
• Instance job: a single Job execution concerning a specific instance of the class

Inside a Model

Definitions

# Add Periodically context to this class
include Periodically::Model

# Enqueue a Periodically job
periodically :update_method, # call instance method "update_method" for found instances
  on: -> { where("last_synced < ?", 7.days.ago) }, # Condition that must be true for update method to be called
  min_class_interval: 5.minutes, # (Optional) The minimum interval between calls to this specific class (TODO not implemented)
  max_retries: 25, # (Optional) Maximum number of retries. Periodically uses exponential backoff
  instance_id: -> { cache_key_with_version }, # (Optional) Returns this instance's unique identifying key. Used for e.g. deferring jobs and marking them as erroring (TODO not implemented)

Update method return values

Job method's return value or raised exception determines further executions of that specific instance job.

# As referred to by a previous `periodically` call
def update_method
  # Let's retrieve a normal value from the model instance
  status = my_column_status

  # No-op
  #   Since we don't update `last_synced`, this method will get called again without much delay!
  return if status == "pending"

  # Log error and defer execution
  #   This unique instance will be deferred for later execution (using exponential backoff) and the error logged
  raise "something went wrong" if status == "error"

  # Update checked delay
  #   Updates the property we check against, thus making this instance not pass the Periodically condition
  #   Note that this line is normal Rails code: Periodically conditions are database/anything-agnostic
  update(last_synced: Time.now)
end

The job method's return value can be used to defer further execution of the same model instance, even if it still passes the condition: return Periodically::Defer(60.minutes)

Dashboard

(When implemented :D) Dashboard contains recently succeeded executions, failed executions (with stacktrace) and deferred executions.

Why not Sidekiq?

With Sidekiq you can achieve something almost similar by combining a scheduled job that enqueues further unique jobs based on the condition.

However, there are few advantages Periodically has for the specific usecase of per-instance non-critical jobs:

• Improved backpressure handling. Due to knowing the conditions, we are able to track the number of pending jobs at all times. This enables early warnings for the developers in case of job buildup. (TODO)
• Better observability. We know exactly how many items fulfill the condition and how many don't; therefore, we can visualize success rates and current status as a percentage of the total. (TODO)
• Cleaner per-instance retrying. If we start executing a job, but suddenly want to defer execution by some time in Sidekiq, it is definitely doable with scheduled jobs. However, this may entrap you in a "scheduled unique job" hell: if some job keeps getting mistakenly deferred, it might be hard to find out about this behavior without some complex job tracking logic. In contrast, Periodically delivers this functionality for free due to more explicit control over job scheduling and rescheduling. (TODO)
• More clever polling. Since we know the exact condition for new periodic jobs, we can deduce the next execution time and sleep accordingly. (TODO)
• Easier priority escalation. Periodically selects jobs in order of the given condition and maintains no queue of its own; therefore it is trivial to prioritize certain jobs by adding a new query condition.

Importantly, Sidekiq and Periodically aim to solve different problems. Nothing prevents one from using both at the same time.