Vincent =======

This a PROOF for a micro-framework for handling AMQP traffic between multiple daemons. It was written with Fibers on Ruby 1.9 to allow for synchronous action inside event machine and MAY work in 1.8 if patched with fibers but is untested. The is PRE-ALPHA and it needs a full design and implementation phase as well as full tests and examples. If anyone wants to contribute I am totally open.

Background:

Vincent gets you many of the AMQP benefits out of the box that most would miss if they do not know the protocol very well. For instance it hoes a clean shutdown on a SIGTERM and transparently handles message acknowledgements for you so that you never lose an AMQP message. Like-wise it includes a TTL option on messages that lets you kill off messages that might have stuck around beyond their usefulness.

Model:

Http App Server (thin/mongrel) are not the place to be processing long running work (100ms max). These app servers should be doing only simple database operations and displaying cached data. When work needs to be done an asynchronous message should be sent to worker processes. Vincent is broken into several libraries that allow you to enforce that your app servers are only sending data, while the workers can both send an receive.

Vincent needs to be run inside an Event Machine reactor. Thin uses EventMachine. If you are using an app server that does not use EM, you will need to start the reactor manually.

Client Example:

I use the ‘cast’ and the ‘call’ idioms from erlang to specify if you are sending a message (cast) or sending a message and awaiting a reply (call). It is important to not confuse these two.

require ‘vincent’

Vincent::Client.cast(“file.encrypt”, { :key_id => 1234, :file => “/gfs1/file.dat”, :enc => ‘blowfish’ })

## Vincent.file_encrypt( { … } ) does the same thing with method_missing?

Cast takes a hash and will encode the that via JSON before publishing the AMQP message and will decode the hash on the other side.

In line Server Example:

An in line server can be setup as such:

require ‘vincent/server’

Vincent::Server.start do |vincent|

vincent.cast("server.up", { :host => ENV['HOSTNAME'], :pid => Process.pid } )

vincent.listen4("file.encrypt", :queue => 'file.encrypt.queue') do |b|
  file = get_file(b['file'])
  key = get_key(b['enc'],b['key_id'])
  file.lock(key)
  b.cast("file.encrypted", { :file => file.name })
end

end

The code above sends a “server.up” message when it comes online and encrypts files when told to. All such processes listen on the same named queue so they will do the work round-robin. After finishing the file it sends a file.encrypted message.

Call:

Now imagine the files are stored on a file server and an AMQP message needs to be sent to fetch it. This is no easy task since AMQP is asynchronous and in eventmachine. Fibers come to the rescue.

Vincent::Server.start do |vincent|

vincent.listen4("file.encrypt", :queue => 'file.encrypt.queue') do |b|
  file = b.call("file.get", { :file => b['file'] })
  key = get_key(b['enc'],b['key_id'])
  file.lock(key)
  b.cast("file.encrypted", { :file => file.name })
end

end

Here is the file server code - Vincent reads the file and sends its bytes back over the wire

Vincent::Server.start do |vincent|

vincent.listen4("file.get") do |b|
  File.read(b['file'])
end

end

Note: if the call block were to throw an exception that exception would propagate up to the callers block.

This process is good, but it breaks down as your app size grows. Code is hard to reuse and spec, and the size of the server block will grow out of control. That’s where Vincent::Base comes in.

The Vincent Object Model:

require ‘vincent/base’

class FilePut < Vincent::Base

bind_to "file.put", :route_to => :one, :active => :disk_not_full

## only one worker will receive the message

## this class accepts requests to store files locally if and 
## only if the disk is not full

def disk_not_full
  not disk_full?
end

def handle_cast
  File.write(params['file_name'],"w") do |f|
    f.write(params['file_bytes'])
  end
end

end

class FileGet < Vincent::Base bind_to “file.get”, :route_to => :one

## this decrypts the file but rejects requests for a file that does ## not exist, since it probably is on a different server

def handle_call

reject unless File.exists?(params['file'])
  File.read(params['file'])
end

end

What’s Next?

There is a lot of work to be done:

* finalize the interface
* get it working on 1.8.7+fibers
* get full test coverage
* test the fiber code more - make sure it works in all cases
* make sure it's wire compatible with current AMQP network
* write examples
* write docs
* do a proper gem release

Why call it Vincent?

He was that scrappy little robot from the movie The Black Hole. =)