Module: Resque::Kubernetes::Job

Defined in:
lib/resque/kubernetes/job.rb

Overview

Resque hook to autoscale Kubernetes Jobs for workers.

To use with pure Resque, extend your Resque job class with this module and then define a class method ‘job_manifest` that produces the Kubernetes Job manifest.

To use with ActiveJob, include this module in your ActiveJob class and then define an instance method ‘job_manifest` that produces the Kubernetes Job manifest.

Example (pure Resque):

class ResourceIntensiveJob
  extend Resque::Kubernetes::Job
  class << self
    def perform
      # ... your existing code
    end

    def job_manifest
      YAML.safe_load(
        <<~MANIFEST
        apiVersion: batch/v1
          kind: Job
          metadata:
            name: worker-job
          spec:
            template:
              metadata:
                name: worker-job
              spec:
                containers:
                - name: worker
                  image: us.gcr.io/project-id/some-resque-worker
                  env:
                  - name: QUEUE
                    value: high-memory
        MANIFEST
      )
    end
  end
end

Example (ActiveJob backed by Resque):

class ResourceIntensiveJob < ApplicationJob
  include Resque::Kubernetes::Job
  def perform
    # ... your existing code
  end

  def job_manifest
    YAML.safe_load(
      <<~MANIFEST
      apiVersion: batch/v1
        kind: Job
         metadata:
           name: worker-job
         spec:
           template:
             metadata:
               name: worker-job
             spec:
               containers:
               - name: worker
                 image: us.gcr.io/project-id/some-resque-worker
                 env:
                 - name: QUEUE
                   value: high-memory
      MANIFEST
    )
  end
end

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.included(base) ⇒ Object 



79
80
81
82
 
# File 'lib/resque/kubernetes/job.rb', line 79

def self.included(base)
  return unless base.respond_to?(:before_enqueue)
  base.before_enqueue :before_enqueue_kubernetes_job
end

Instance Method Details

#before_enqueue_kubernetes_job(*_args) ⇒ Object

A before_enqueue hook that adds worker jobs to the cluster.



85
86
87
88
89
90
91
92
 
# File 'lib/resque/kubernetes/job.rb', line 85

def before_enqueue_kubernetes_job(*_args)
  return unless Resque::Kubernetes.enabled

  manager = JobsManager.new(self)
  manager.reap_finished_jobs
  manager.reap_finished_pods
  manager.apply_kubernetes_job
end

#max_workers ⇒ Object

The maximum number of workers to autoscale the job to.

While the number of active Kubernetes Jobs is less than this number, the gem will add new Jobs to auto-scale the workers.

By default, this returns ‘Resque::Kubernetes.max_workers` from the gem configuration. You may override this method to return any other value, either as a simple integer or with some complex logic.

Example:

def max_workers
  # A simple integer
  105
end

Example:

def max_workers
  # Scale based on time of day
  Time.now.hour < 8 ? 15 : 5
end



114
115
116
 
# File 'lib/resque/kubernetes/job.rb', line 114

def max_workers
  Resque::Kubernetes.max_workers
end