Flapjack

Flapjack is highly scalable and distributed monitoring system.

It understands the Nagios plugin format, and can easily be scaled from 1 server to 1000.

Setup dependencies (Ubuntu Hardy)

Add the following lines to /etc/apt/sources.list

deb http://ppa.launchpad.net/ubuntu-ruby-backports/ubuntu hardy main
deb http://ppa.launchpad.net/auxesis/ppa/ubuntu hardy main

Add GPG keys for the repos: 

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 288BA53BCB7DA731

Update your package list:

sudo apt-get update

Install Ruby dependencies: 

sudo apt-get install build-essential libsqlite3-dev

Install rubygems + beanstalkd:

sudo apt-get install rubygems beanstalkd

Set ENABLED=1 in /etc/default/beanstalkd.

Start beanstalkd: 

sudo /etc/init.d/beanstalkd start

Setup dependencies (everyone else)

Install the following software through your package manager or from source:

Installation

Add GitHub's RubyGems server to your Gem sources: 

sudo gem sources -a http://gems.github.com

Install the Flapjack gem: 

sudo gem install auxesis-flapjack

Then run the magic configuration script to set up init scripts: 

sudo install-flapjack-systemwide

The script will prompt you if it wants to do anything destructive.

Running

Make sure beanstalkd is running.

You'll want to set up /etc/flapjack/recipients.yaml so notifications can be sent via flapjack-notifier: 

---
- :name: Jane Doe
  :email: "[email protected]"
  :phone: "+61 444 222 111"
  :pager: "61444222111"
  :jid: "[email protected]"

You also need to set up /etc/flapjack/flapjack-notifier.yaml: 

--- 
:notifiers: 
  :mailer: 
    :from_address: [email protected]
  :xmpp: 
    :jid: [email protected]
    :password: foo
:database_uri: "sqlite3:///var/lib/flapjack/flapjack.db"

Start up a cluster of workers: 

flapjack-worker-manager start

This will spin up 5 workers in the background. You can specify how many workers to start: 

flapjack-worker-manager start --workers=10

Each of the flapjack-workers will output to syslog (check in /var/log/messages).

Start up the notifier: 

flapjack-notifier-manager start --recipients /etc/flapjack/recipients.yaml

Currently there are email and XMPP notifiers.

You'll want to get a copy of (http://github.com/auxesis/flapjack-admin/)[flapjack-admin] to set up some checks, then run its' populator to get them into Flapjack.

What things do

  • flapjack-worker => executes checks, reports results
  • flapjack-worker-manager => starts/stops a cluster of flapjack-worker
  • flapjack-notifier => gets results, notifies people if necessary
  • flapjack-stats => gets stats from beanstalkd tubes (useful for benchmarks + performance analysis)

init scripts

You can use the provided init scripts to start Flapjack on boot.

To start: 

/etc/init.d/flapjack-workers start
/etc/init.d/flapjack-notifier start

To set Flapjack to start on boot (Ubuntu): 

sudo update-rc.d flapjack-workers defaults
sudo update-rc.d flapjack-notifier defaults

Config for the init scripts can be found in /etc/defaults.

Developing

You can write your own notifiers and place them in lib/flapjack/notifiers/.

Your notifier just needs to implement the notify method, and take in a hash:

class Sms
  def initialize(opts={})
    # you may want to set from address here
  end

  def notify(opts={})
    who = opts[:who]
    result = opts[:result]
    # sms to your hearts content
  end
end

Testing

Tests are in spec/.

To run tests:

$ rake spec

Architecture

       -------------------
       | web interface / |
       | dsl / flat file |
       -------------------
      /
      |
      |
-------------                  ------------
| populator |---          -----| notifier |
-------------  |          |    ------------
               |          |
              --------------
              | beanstalkd |
              --------------
                    |
   -----------------------------------
   |                |                |
----------      ----------       ----------
| worker |      | worker |       | worker |
----------      ----------       ----------
  • populator determines checks that need to occur, and puts checks onto jobs tube
    • populator fetches jobs from a database
    • populator can be swapped out to get checks from a dsl or flat files
  • workers pop a check off the beanstalk, perform check, put result onto results tube, re-add check to jobs tube with a delay.
  • notifier pops results off results tube, notifies as necessary