Eye::Patch

Eye::Patch is a wrapper for the fantastic eye process monitoring gem, with support for parsing yaml configuration files and capistrano deploys.

The goal of this gem is to make eye process monitoring easier to set up, manage, and deploy for large applications.

Installation

Add this line to your application's Gemfile:

gem "eye-patch", require: false

And then execute:

$ bundle

Or install it yourself as:

$ gem install eye-patch

Usage

Configuration

Eye::Patch overrides eye's built-in DSL to allow your eye application to be configured in a structured YAML file.

Notifications

Eye::Patch forgoes granular process-level notification in favor of setting up application-wide notifications for different reporting levels. The following configuration will send all error notifications to the described contact via Amazon's SES service.

Note that SES support will only be provided if a necessary gem is installed on the system.

Eye::Patch supports 2 different SES-backed gems, and also the datadog gem:

aws-ses gem

Deprecated: The aws-sdk-ses gem is recommended in place of aws-ses, as aws-ses hasn't been maintained in years.

If the aws-ses gem is available on the system, use the type: ses setting.

Example configuration:

  notifications:
    - name: crash # this name must match the "notify" target of the process.
      type: ses # for aws-ses
      level: error
      contact: [email protected]
      config:
        from: [email protected]
        access_key_id: Your+AWS+Access+Key+ID
        secret_access_key: Your+AWS+Secret+Access+Key

aws-sdk/aws-sdk-ses gems

If either of the aws-sdk or aws-sdk-ses gems are available on the system, use the type: aws_sdk setting. (Note: usually one of these gems are installed as a consequence of installing aws-sdk-rails).

Note: As of version 2.x of aws-sdk-rails, it only includes aws-sdk-ses.

Note: As of version 3.x of aws-sdk, you can instead use aws-sdk-ses in order to only load the SES portion of the libraries. Before version 2.x of aws-sdk, the gem included all of the AWS libraries.

Example configuration:

  notifications:
    - name: crash # this name must match the "notify" target of the process.
      type: aws_sdk # for aws-sdk
      level: error
      contact: [email protected]
      config:
        from: [email protected]
        region: us-east-1 # optional
        # NOTE: the default region is us-east-1. It can be overriden.
        access_key_id: Your+AWS+Access+Key+ID # optional
        secret_access_key: Your+AWS+Secret+Access+Key # optional

datadog (dogapi gem)

If you want to send crash notifications through datadag, you can do that by specifying the api_key for datadog. You must include the dogapi gem in your project Gemfile, so that the DataDog Ruby bindings are available.

To retrieve your api_key

Example configuration:

notifications:
  - name: crash # this name must match the "notify" target of the process.
    type: datadog # for datadog
    level: error
    config:
      api_key: You+Datadog+API+Key

Example block for monitored processes:

In any case above, an example notification block for monitored process:

processes:
  - name: unicorn
    config:
      pid_file: tmp/pids/unicorn.pid
      start_command: bundle exec unicorn -c config/unicorn/<%= ENV["RAILS_ENV"] %>.rb -D
      ....
      monitor_children:
        stop_command: "kill -QUIT {PID}"
      notify:
        crash: error # this must match the "name" of the notification above

Triggers/Checks

Triggers and checks are set up much like eye's basic DSL. All trigger and check types available in eye are supported.

triggers:
  - name: flapping
    config:
      times: 10
      within: 1 minute

checks:
  - name: memory
    config:
      times: 3
      every: 20 seconds
      below: 1.5 gigabytes

Note that memory and duration values are written in plain english. See lib/eye/patch/value_parser.rb for the regular expressions used to parse these values into usable numbers.

Processes

Eye::Patch uses the same conventions as eye for basic process configuration.

processes:
  - name: my-process
    config:
      start_command: bundle exec my-process
      pid_file: tmp/pids/my-process.pid

Processes will inherit all configurations from the main application. All process options supported by eye are available by using the appropriate key in the config block.

Checks and Triggers

You can define per-process checks and triggers by defining a checks or triggers block within the process definition.

processes:
  - name: my-process
    checks:
      - name: memory
        config:
          times: 3
          every: 10 seconds
          below: 52 megabytes
...

Any check or trigger already defined at the application level will be overwritten with the new configuration. Application-level checks and triggers which are not redefined for the process will be persisted.

Grouping

Rather than using the eye DSL's nesting approaching to setting up processes within groups, Eye::Patch simply needs a group attribute to be added to the process definition.

processes:
  - name: my-grouped-process
    group: my-group
    config:
...

All processes sharing a group name will be grouped together when the configuration is loaded.

Copying

On occasion, it's desirable to spin up multiple copies of a process with the same configuration. Eye::Patch offers a shorthand for this setup with the count attribute.

processes:
  - name: my-copied-process
    count: 3
    config:
      start_command: bundle exec my-copied-process
      pid_file: "copied-process-{ID}.pid"

The above configuration will spin up a number of processes equal to count, resulting in the following monitored processes:

my-copied-process-0 <copied-process-0.pid>
my-copied-process-1 <copied-process-1.pid>
my-copied-process-2 <copied-process-2.pid>

This approach pairs well with the daemonize: true option, as eye will have control over the creation and management of the enumerated PID files.

Monkey-Patching

Eye::Patch can't cover all possible use cases, so instead, it supplies a few hooks to allow you to inject your own behavior into eye during initial load.

Defining a Setup File

For the application specified in your configuration file, you can specify a setup_file directive, which will tell Eye::Patch to load the specified file (relative to the working directory for the application) immediately after parsing your configuration.

This can be used to inject any desired behavior or hooks into eye. For example:

working-dir/config/eye.rb:

#!/usr/bin/env/ruby

Eye::Control.settings[:my_key] = "SOME_KEY"

Eye::Controller.class_eval do
  def before_spawn
    reader, writer = ::IO.pipe.map(&:binmode)
    reader.close_on_exec = false
    writer.close_on_exec = true

    ENV["MY_FD"] = reader.to_i.to_s

    writer << Eye::Control.settings[:my_key]
    writer.flush
    writer.close
  end
end

working-dir/config/eye.yml:

...
application:
  working_dir: working-dir
  setup_file: config/eye.rb
...

Using a `before_spawn` Hook

If you need to invoke some code before spawning a process (for example, if you want to make a file descriptor available to your new child process), you can accomplish this by patching Eye::Controller with a #before_spawn method in your setup file. This method will be invoked prior to Eye::System calling Process.spawn.

See above for more instructions on how to define this hook during setup.

Working with File Descriptors

Note that if you do need to persist file descriptors for a daemonized process, you'll want to set the preserve_fds: true option for that particular process in your configuration file. Process.spawn's default behavior closes all non-standard file descriptors; this option will ensure that they remain open.

Note also that since Ruby 2.0, file descriptors are closed when a process is invoked through bundle exec. From versions 1.5 and above, you can pass the --keep-file-descriptors flag to bundle exec to prevent this behavior.

Managing Pidfiles

By default, eye will attempt to manage the pidfile for any process with the daemonize: true flag. In certain cases, this may be undesirable. For example, unicorn attempts to manage its own pid, which is necessary for zero-downtime deploys.

To get around this problem, you can supply the smart_pid: true option in a process's config hash. This will instruct eye to trust the daemonized process to manage its own pid.

Running Locally

You can test your configurations locally by running the eye-patch binary like so:

bundle exec eye-patch l config/eye.yml

The eye-patch binary wraps the eye CLI, with some tweaks to insert Eye::Patch as the config file parser.

Deploying

Eye::Patch has built-in support for capistrano deploys. Include the following in your deploy.rb file:

require "eye/patch/capistrano"

For standard capistrano deployments with bundler, this should be enough. If you aren't using bundler, or your deploy process is more complicated, you may need to tweak the default binary locations or callback ordering.

See lib/eye/patch/capistrano.rb for a list of available configuration options.

If you're on Capistrano 3, you'll want to add the following to the Capfile:

require "eye/patch/capistrano3"

If you're specifically using Eye::Patch in conjunction with a Rails project, you may need to add the following to your deploy file:

set :eye_env, -> { {rails_env: fetch(:rails_env)} }

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request

Coming Soon

Process templates for commonly-used libraries (unicorn, sidekiq, delayed_jobs).

Eye::Patch

Installation

Usage

Configuration

Notifications

Example block for monitored processes:

Triggers/Checks

Processes

Checks and Triggers

Grouping

Copying

Monkey-Patching

Defining a Setup File

Using a before_spawn Hook

Working with File Descriptors

Managing Pidfiles

Running Locally

Deploying

Contributing

Coming Soon

Using a `before_spawn` Hook