Pipeline

Description

Pipeline is a Rails plugin/gem to run asynchronous processes in a configurable pipeline.

Documentation

rdoc.info/projects/dtsato/pipeline

Features

Execution of sequential user-defined stages in an asynchronous pipeline
Persistence of pipeline instances and stages
Error recovery strategies:
- Irrecoverable errors fail the entire pipeline
- Recoverable errors are automatically retried (using dj’s exponential retry strategy)
- Recoverable errors that require user input pause the pipeline for further retry
Cancelling/Resuming of a paused pipeline
Callbacks before and after executing stages and pipeline

Installation and Use

pipeline can be installed as either a RubyGem (recommended) or as a plugin.

Gem

To install pipeline as a RubyGem, add the following lines to your config/environment.rb file:

config.gem "pipeline"
config.gem "delayed_job"

And execute:

rake gems:install
rake gems:unpack # Optional, if you want to vendor the gem

Plugin

To install it as a plugin, run:

script/plugin install git://github.com/dtsato/pipeline.git

Generating the required tables

In order to persist your pipelines and stages, execute the following:

script/generate pipeline # To generate the migration scripts that will store pipelines
script/generate delayed_job # To generate the migration scripts for delayed_job
rake db:migrate

Starting your workers

You will also need to run your Delayed Job workers that will process the pipeline jobs in the background:

rake jobs:work

Dependencies

Rails (ActiveRecord)
Delayed job (github.com/collectiveidea/delayed_job)

Usage

Check examples for more examples (including error-recovery and cancelling)

class Step1 < Pipeline::Stage::Base
  def perform
    puts("Started step 1")
    sleep 2
    puts("Finished step 1")
  end
end

class Step2 < Pipeline::Stage::Base
  def perform
    puts("Started step 2")
    sleep 3
    puts("Finished step 2")
  end
end

class TwoStepPipeline < Pipeline::Base
  define_stages Step1 >> Step2
end

Pipeline.start(TwoStepPipeline.new)