Jongleur Gem Version

Jongleur is a process scheduler and manager. It allows its users to declare a number of executable tasks as Ruby classes, define precedence between those tasks and run each task as a separate process.

Jongleur is particularly useful for implementing workflows modelled as a DAG (Directed Acyclic Graph), but can be also used to run multiple tasks in parallel or even sequential workflows where each task needs to run as a separate OS process.

Environment

This gem has been built using the POSIX/UNIX process model. It will work on Linux and Mac OS but not on Windows.

Jongleur has been tested with MRuby 2.4.3, 2.4.4, 2.5.0 and 2.5.1. I would also expect it to work with other Ruby implementations too, though it hasn't yet been tested on those.

Depndencies

This gem depends on the Graphviz package for drawing graphs. If this isn't already installed on your system please install with

$ sudo apt-get install graphviz (Linux)

or

$ brew cask install graphviz(Mac OS)

Installation

Add this line to your application's Gemfile:

gem 'jongleur'

And then execute:

$ bundle

Or install it yourself as:

$ gem install jongleur

In either case, call require jongleur before using the gem.

What does it do?

In a nutshell, Jongleur keeps track of a number of tasks and executes them as separate OS processes according to their precedence criteria. For instance, if there are 3 tasks A, B and C, and task C depends on A and B, Jongleur will start executing A and B in separate processes (i.e. in parallel) and will wait until they are both finished before it executes C in a separate process.

Jongleur is ideal for running workflows represented as DAGs, but is also useful for simply running tasks in parallel or for whenever you need some multi-processing capability.

Concepts

Directed Acyclic Graph (DAG)

A graph that is directed and without cycles connecting the other edges. DAGs are very useful for representing different kinds of information models, such as task scheduling and business process workflows.

Task Graph

To run Jongleur, you will need to define the tasks to run and their precedence. A Task Graph is a representation of the tasks to be run by Jongleur and it usually (but not exclusively) represents a DAG, as in the example below:

DAG example

A Task Graph is defined as a Hash in the following format:

{task-name => list[names-of-dependent-tasks]}

So the graph above can be defined as:

my_graph = {
  A: [:C, :D],
  B: [:D, :E],
  D: [:F, :G],
  E: [],
  C: [],
  G: [:I],
  H: [:I],
  F: [],
  I: []
}

where they Hash key is the class name of a Task and the Hash value is an Array of other Tasks that can be run only after this Task is finished. So in the above example:

  • Tasks C and D can only start after task A has finished.
  • Task I can only start after G and H have finished.
  • Tasks C, E, F and I have no dependants. No other tasks need wait for them.

N.B: Since the Task Graph is a Hash, any duplicate key entries will be overridden. For instance:

my_task_graph = { A: [:B, :C], B: [:D] }

is re-defined as

my_task_graph = { A: [:B], A: [:C], B: [:D] }

The 2nd assignment of A will override the first one so your graph will be:

{:A=>[:C], :B=>[:D]}

Always assign all dependent tasks together in a single list.

Task Matrix

It's a tabular real-time representation of the state of task execution. It can be invoked at any time with

Jongleur::API.task_matrix

After defining your Task Graph and before running Jongleur, your Task Matrix should look like this:

#<Jongleur::Task name=:A, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
#<Jongleur::Task name=:B, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
#<Jongleur::Task name=:C, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
#<Jongleur::Task name=:D, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
#<Jongleur::Task name=:E, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>

After Jongleur finishes, your Task Matrix will look something like this:

#<Jongleur::Task name=:A, pid=95117, running=false, exit_status=0, finish_time=1546279899.005019, success_status=true>
#<Jongleur::Task name=:B, pid=95118, running=false, exit_status=0, finish_time=1546279901.0053551, success_status=true>
#<Jongleur::Task name=:C, pid=95120, running=false, exit_status=0, finish_time=1546279900.0213592, success_status=true>
#<Jongleur::Task name=:D, pid=95122, running=false, exit_status=0, finish_time=1546279912.0673511, success_status=true>
#<Jongleur::Task name=:E, pid=95123, running=false, exit_status=0, finish_time=1546279909.0876418, success_status=true>

The Jongleur::Task attribute values are as follows

  • name : the Task name
  • pid : the Task process id (nil if the task hasn't yet ran)
  • running : true if task is currently running
  • exit_status : usually 0 if process finished without errors, <>0 or nil otherwise
  • finish_time : the Task's completion timestamp as a flating point number of seconds since Epoch
  • success_status : true if process finished successfully, false if it didn't or nil if process didn't exit at all

WorkerTask

This is the implementation template for a Task. For each Task in your Task Graph you must provide a class that derives from WorkerTask and implements the execute method. This method is what will be called by Jongleur when the Task is ready to run.

Usage

Using Jongleur is easy:

  1. (Optional) Add include jongleur so that you won't have to namespace every api call.

  2. Define your Task Graph

    test_graph = {
  A: [:B, :C],
  B: [:D],
  C: [:D],
  D: [:E],
  E: []
}

    Each Task corresponds to a Ruby class with an execute method

  3. Add your Task Graph to Jongleur

    API.add_task_graph test_graph

=> [#<struct Jongleur::Task name=:A, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
    #<struct Jongleur::Task name=:B, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
    #<struct Jongleur::Task name=:C, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
    #<struct Jongleur::Task name=:D, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>,
    #<struct Jongleur::Task name=:E, pid=-1, running=false, exit_status=nil, finish_time=0, success_status=nil>]

    Jongleur will show you the Task Matrix for your Task Graph with all attributes set at their initial values, obviously, since the Tasks haven't ran yet.

  4. (Optional) You may want to see a graphical representation of your Task Graph

    API.print_graph('/tmp')

=> "/tmp/jongleur_graph_08252018_194828.pdf"

    Opening the PDF file will display this:

    ETL DAG

  5. Implement your tasks. To do that you have to 

* create a new class, based on `WorkerTask` and named as in your Task Graph
* define an `#execute` method in your class. This is the method that Jongleur will call to run the Task. 

For instance task A from your Task Graph may look something like that:


       class A < Jongleur::WorkerTask
          @desc = 'this is task A'
          def execute
           sleep 1 # do something
           'A is running... '
          end
        end

You'll have to do the same for Tasks B, C, D and E, as these are the tasks declared in the Task Graph.

  1. Run the tasks and implement the completed callback. This will be called asynchronously when Jongleur has finished running all the tasks. 

       $ API.run do |on|
        on.completed { |task_matrix| puts "Done!"}
    end

  => Starting workflow...
  => starting task A
 => finished task: A, process: 2501, exit_status: 0, success: true
 => starting task B
 => starting task C
 => finished task: C, process: 2503, exit_status: 0, success: true
 => finished task: B, process: 2502, exit_status: 0, success: true
 => starting task D
 => finished task: D, process: 2505, exit_status: 0, success: true
 => starting task E
 => finished task: E, process: 2506, exit_status: 0, success: true
 => Workflow finished
 => Done!

Examples of running Jongleur can be found in the examples directory.

Use-Cases

Extract-Transform-Load

The ETL workflow is ideally suited to Jongleur. You can define many Extraction tasks -maybe separate Tasks for different data sources- and have them ran in parallel to each other. At the same time Transformation and Loading Tasks wait in turn for the previous task to finish before they start, as in this DAG illustration:

ETL DAG

Transactions

Transactional workflows can be greatly sped up by Jongleur by parallelising parts of the transaction that are usually performed sequentially, i.e.:

Transaction DAG

Development

After checking out the repo, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

F.A.Q

Does Jongleur allow me to pass messages between Tasks?

No it doesn't. Each task is run completely independently from the other Tasks. There is no Inter-Process Communication, no common data contexts, no shared memory.

How can I share data created by a predecessor Task?

This is something that I would like to build into Jongleur. For now, you can save a Task's data in a database or KV Store and using the Task's process id as part of the key. Subsequent Tasks can retrieve their predecessor's process ids with

API.get_predecessor_pids

and therefore retrieve the data created by those Tasks.

What's the difference between Jongleur::Task's success_status and exit_status attributes?

According to the official docs exit_status returns the least significant eight bits of the return code of the stat call while success_status returns true if stat is successful.

What happens when Jongleur finishes running?

When Jongleur finishes running all tasks in its Task Graph -and regardless of whether the Tasks themselves have failed or not- it will exit the parent process with an exit code of 0.

What happens if a Task fails

If a Task fails to run or to finish its run, Jongleur will simply go on running any other tasks it can. It will not run any Tasks which depend on the failed Task. The status of the failed Task will be indicated via an appropriate output message and will also be visible on the Task Matrix.

Can I quickly analyse the Task Matrix after Jongleur has finished?

Yes. When the completed callback is called, Jongleur will enable the following methods:

API::successful_tasks
API::failed_tasks
API::not_ran_tasks
API::hung_tasks

Are there any execution logs saved?

Jongleur serializes each run's Task Matrix as a time-stamped JSON file in the /tmp directory. You can either view this in an editor or load it and manipulate it in Ruby with

JSON.parse( File.read('/tmp/jongleur_task_matrix_08272018_103406.json') )

Roadmap

These are the things I'd like Jongleur to support in future releases:

  • Task storage mechanism, i.e. the ability for each Task to save data in a uniquely identifiable and safe way so that data can be shared between sequential tasks in a transparent and easy manner.
  • Rails integration. Pretty self-explanatory really.

Contributing

Any suggestions for new features or improvements are very welcome. Please raise bug reports and pull requests on GitLab.

License

The gem is available as open source under the terms of the MIT License