About
Plugin pipeline infrastructure connected through RabbitMQ. The plugins are just normal Ruby gems.
Instance(s) of plugin 1 -> RabbitMQ broker -> Instance(s) of plugin 2
It uses amq.topic
exchange, so you can use globs to consume from within a plugin or a custom script. So if you want to see all the messages going through the system, simply create a queue a bind it to amq.topic
with routing key #
. Need all the events? #event#
. Easy as that.
Why?
It's been proved over and over that splitting app into services and is the way to go, because:
- It's easier to maintain such app.
- Rewriting components is easy.
- It's easy to scale by adding more consumers of given service.
- It's easy to inspect what's going on.
- It's much easier to split work on the project between multiple developers.
- You can restart services without losing any data, as they are kept in RabbitMQ.
- You don't have to introduce any code dependencies: just consume given queue and publish to another one.
Settings
Pipeline.rb uses JSON for configuration.
The cool thing is it supports global and local configuration. That means for instance if you want to configure a database, you're probably going to use the same database drivers on either development machine or on production server.
So first let's create a global configuration file config/database.json
:
{
"adapter": "mysql",
"username": "mysql",
"database": "example"
}
And now the local config/database.local.json
:
{
"password": "ae28cd87adb5c385117f89e9bd452d18"
}
Don't forget to ignore the local settings .gitignore
:
config/*.local.json
HOWTO
Pipeline.rb expects you to provide AMQP configuration, so it knows how to connect to RabbitMQ:
config/amqp.json
:
{
"vhost": "example",
"user": "example",
}
config/amqp.local.json
:
{
"password": "ae28cd87adb5c385117f89e9bd452d18"
}
Don't forget to ignore the local settings .gitignore
:
config/*.local.json
Shop for plugins and add them to your Gemfile
:
source 'https://rubygems.org'
# The app uses pipeline.rb itself.
gem 'pipeline.rb'
# Tasks.
gem 'nake'
# Existing pipeline.rb plugins.
group(:plugins) do
gem 'mail_queue'
end
#!/usr/bin/env bundle exec ruby
require 'nake/runner'
# Load all the plugins.
Bundler.require(:plugins)
# Tasks: declare.
require 'pipeline/tasks'
Nake.run
Now run ./tasks.rb declare
to declare all the required queues.
And finally create your app:
require 'mail_queue'
# This is already loaded,
# but just to make it clear
# that the app is actually
# just another plugin.
require 'pipeline/plugin'
class App < Pipeline::Plugin
def run
EM.add_timer(0.5) do
client.publish("Hello World!", 'emails.random')
end
end
end
And finally run it all:
- Start RabbitMQ.
- Start
bin/mail_queue.rb
to send your emails. - Run your app.
Deployment
TODO: Show example Upstart scripts.