Distribot

Work scheduling and distribution engine.

Features

• Built on RabbitMQ and Redis.

Installation

In your Gemfile

gem 'distribot', git: 'git@github.com:jdrago999/distribot.git'

Usage

require 'distribot'

Distribot.configure do |config|
  config.redis_url = ENV['DISTRIBOT_REDIS_URL']
  config.rabbitmq_url = ENV['DISTRIBOT_RABBITMQ_URL']
end

{
  "name": "search",
  "phases": [
    {
      "name": "pending",
      "is_initial": true,
      "transitions_to": "searching"
    },
    {
      "name": "searching",
      "transitions_to": "fetching-pages",
      "handlers": [
        "GoogleSearcher",
        "BingSearcher"
      ]
    },
    {
      "name": "fetching-pages",
      "transitions_to": "finished",
      "handlers": [
        {
          "name": "PageDownloader",
          "version": "~> 1.2"
        }
      ]
    },
    {
      "name": "finished",
      "is_final": true
    }
  ]
}

TODO

Features

• ~~Ability to control running flows~~
  • ~~cancel~~
  • ~~pause~~
  • ~~resume~~
• ~~Handler versioning~~
  • ~~semver~~
  • ~~specify handler versions in flow definitions~~
    • ~~similar to gemfile~~

Organization

• Break this project into smaller parts.
  • gem code
    • ~~should not require infrastructure to run tests~~
    • ~~enough infrastructure to run the code~~
  • small running environment which uses the gem
    • engine
    • worker
    • controller
    • ~~infra~~
    • ~~redis~~
    • ~~rabbitmq~~
    • ~~elasticsearch~~
    • ~~kibana~~
• ~~status dashboard~~
  • ~~show running flows~~
  • ~~create a new flow~~

Notes

Clear out queues:

sudo rabbitmqctl list_queues | grep distribot | awk '{print $1}' | xargs -I qn rabbitmqadmin delete queue name=qn