roo_on_rails
roo_on_rails
is:
- A library that extends Rails (as a set of Railties) and auto-configures common dependencies.
- A command that checks whether an application's Github repository and Heroku instanciations are compliant.
... packaged into a gem, to make following our guidelines easy.
Table of Contents
Installation
Add this line at the top of your Rails application's Gemfile:
gem 'roo_on_rails'
Remove the following gems from your Gemfile, as they're provided and configured
by roo_on_rails
:
dotenv
newrelic_rpm
Remove the following configuration files:
newrelic.yml
orconfig/newrelic.yml
Also remove any other gem-specific configuration from your repository.
And then execute:
$ bundle
Then re-run your test suite to make sure everything is shipshape.
Library usage
New Relic configuration
We enforce configuration of New Relic.
- Your app must be loaded with a
NEW_RELIC_LICENSE_KEY
environment variable, otherwise it will abort. - No
new_relic.yml
file may be presentin your app. Overrides to New Relic settings through environment variables is permitted. - The
NEW_RELIC_APP_NAME
environment variable must be defined such that the app will be properly registered in New Relic.
No further configuration is required for production apps as the gem configures our standard settings.
Rack middleware
We'll insert the following middlewares into the rails stack:
Rack::Timeout
: sets a timeout for all requests. UseRACK_SERVICE_TIMEOUT
(default 15) andRACK_WAIT_TIMEOUT
(default 30) to customise.Rack::SslEnforcer
: enforces HTTPS.Rack::Deflater
: compresses responses from the application, can be disabled withROO_ON_RAILS_RACK_DEFLATE
(default: 'YES').- Optional middlewares for Google Oauth2 (more below).
Disabling SSL enforcement
If you're running your application on Hopper, you'll need to turn off SSL enforcement
as we do that at edge level in Cloudflare rather than the application code itself,
which must be served over HTTP to its associated ALB, which handles SSL termination.
To do this, you can set the ROO_ON_RAILS_DISABLE_SSL_ENFORCEMENT
to YES
.
Database configuration
The database statement timeout will be set to a low value by default. Use
DATABASE_STATEMENT_TIMEOUT
(milliseconds, default 200) to customise.
For database creation and migration (specifically the db:create
, db:migrate
,
db:migrate:down
and db:rollback
tasks) a much higher statement timeout is
set by default. Use MIGRATION_STATEMENT_TIMEOUT
(milliseconds, default 10000)
to customise.
Note: This configuration is not supported in Rails 3 and will be skipped. Set statement timeouts directly in the database.
Sidekiq
Deliveroo services implement Sidekiq with an urgency pattern. By only having
time-based SLA queue
names (eg. within5minutes
) we can automatically create incident alerting for
queues which take longer than the time the application needs them to be processed.
When SIDEKIQ_ENABLED
is set we'll:
- check for the existence of a worker line in your Procfile;
- add SLA style queues to your worker list;
- check for a
HIREFIRE_TOKEN
and if it's set enable SLA based autoscaling;
The following ENV are available:
SIDEKIQ_ENABLED
SIDEKIQ_QUEUES
- comma-separated custom queue names; if not specified, default queues are processed which are defined here. For accurate health reporting and scaling for your custom queue names, you can specify their permitted latency within this variable e.g.within5seconds,queue-one:10seconds,queue-two:20minutes,queue-three:3hours,queue-four:1day,default
. For non-default queue names that don't follow thewithinXunit
pattern, you will need to specify the permitted latency otherwise the initialization will error.SIDEKIQ_THREADS
(default: 25) - Sets sidekiq concurrency valueSIDEKIQ_DATABASE_REAPING_FREQUENCY
(default: 10) - For sidekiq processes the amount of time in seconds rails will wait before attempting to find and recover connections from dead threads
NB. If you are migrating to SLA-based queue names, do not set SIDEKIQ_ENABLED
to true
before your old queues have finished processing (this will prevent
Sidekiq from seeing the old queues at all).
HireFire
For Web Dynos
Web dynos can be autoscaled by HireFire only if it has been configured to use the Web.Logplex.Load
source and the Heroku runtime metrics lab feature has been enabled:
$ heroku labs:enable log-runtime-metrics -a your-service-name-here
You will also need a log drain for HireFire, but the RooOnRails helper below should configure this for you. You can check with
$ heroku drains | grep hirefire
https://logdrain.hirefire.io (d.00000000-0000-0000-0000-000000000000)
# No drain? Add with:
$ heroku drains:add -a your-service-name-here https://logdrain.hirefire.io
For Sidekiq Workers
When HIREFIRE_TOKEN
is set an endpoint will be mounted at /hirefire
that
reports the required worker count as a function of queue latency. By default we
add queue names in the style 'within1day', so if we notice an average latency in
that queue of more than an set threshold we'll request one more worker. If we
notice less than a threshold we'll request one less worker. These settings can
be customised via the following ENV variables
WORKER_INCREASE_THRESHOLD
(default 0.5)WORKER_DECREASE_THRESHOLD
(default 0.1)
When setting the manager up in the HireFire web ui, the following settings must be used:
- name: 'worker'
- type: 'Worker.HireFire.JobQueue'
- ratio: 1
- decrementable: 'true'
Logging
For clearer and machine-parseable log output, the Rails logger is replaced by an extended logger to add context to your logs, which is output as logfmt key/value pairs along with the log message.
You can use the logger as usual:
Rails.logger.info { 'hello world' }
From your console, the output will include the timestamp, severity, and message:
[2017-08-25 14:34:54.899] INFO | hello world
In production (or whenever the output isn't a TTY), the timestamp is stripped
(as it's provided by the logging pipes) and the output is fully valid logfmt
:
at=INFO msg="hello world"
One can also add context using the with
method:
logger.with(a: 1, b: 2) { logger.info 'Stuff' }
# => at=INFO msg=Stuff a=1 b=2
logger.with(a: 1) { logger.with(b: 2) { logger.info('Stuff') } }
# => at=INFO msg=Stuff a=1 b=2
logger.with(a: 1, b: 2).info('Stuff')
# => at=INFO msg=Stuff a=1 b=2
See the class documentation for further details.
Identity
If your service wants to accept JWTs for identity claims, then setting the
VALID_IDENTITY_URL_PREFIXES
environment variable (to be a comma separasted list of the URL prefixes
which valid JWTs come from) will set everything up, eg:
https://deliveroo.co.uk/identity-keys/,https://identity.deliveroo.com/jwks/
Any inbound request which has a valid JWT will have the claims made available:
class MyController
def index
customer_id = request.env['roo.identity']['cust']
request.env['roo.identity'].class
# => JSON::JWT
end
end
Be aware that maliciously crafted JWTs will raise 401s that your other middleware can present and poorly configured JWT set up will raise errors that you'll be able to catch in test.
Google OAuth authentication
When GOOGLE_AUTH_ENABLED
is set to true we inject a Omniauth
Rack middleware
with a pre-configured strategy for Google Oauth2.
Parameters:
GOOGLE_AUTH_CLIENT_ID
andGOOGLE_AUTH_CLIENT_SECRET
(mandatory)GOOGLE_AUTH_PATH_PREFIX
(optional, defaults to/auth
)GOOGLE_AUTH_CONTROLLER
(optional, defaults tosessions
)
This feature is bring-your-own-controller — it won't magically protect your application.
A simple but secure example is detailed in README.google_oauth2.md
.
Datadog Integration
Heroku metrics
To send system metrics to Datadog (CPU, memory usage, HTTP throughput per status, latency, etc), your application need to send their logs to the metrics bridge.
This is automatically configured when running roo_on_rails harness
.
Custom application metrics
Sending custom metrics to Datadog through Statsd requires an agent running in
each dyno.
You need to add the buildpack
heroku-buildpack-datadog
.
Once this is done, you can send metrics with e.g.:
RooOnRails.statsd.increment('my.metric', tags: 'tag:value')
Tags env:{name}
, source:{dyno}
, and app:{name}
will automatically be added
to all your metrics.
The following environment variables are being used. The default values are fine
except for STATSD_ENV
which should be set.
STATSD_ENV
STATSD_HOST
STATSD_PORT
These extra required variables are automatically set by Heroku:
DYNO
HEROKU_APP_NAME
Routemaster Client
When ROUTEMASTER_ENABLED
is set to true
we attempt to configure routemaster-client
on your application. In order for this to happen, set the following environment variables:
ROUTEMASTER_URL
– the full URL of your Routemaster application (mandatory)ROUTEMASTER_UUID
– the UUID of your application, e.g.logistics-dashboard
(mandatory)ROUTEMASTER_VERIFY_SSL
– set to false if your Routemaster application is not served with a valid cert. (optional)
If you then want to enable the publishing of events onto the event bus, you need to set ROUTEMASTER_PUBLISHING_ENABLED
to true
and implement publishers as needed. An example of how to do this is detailed in README.routemaster_client.md
.
API Authentication
RooOnRails provides a concern which will make adding rotatable API authentication to your service a breeze:
require 'roo_on_rails/concerns/require_api_key'
class ThingController < ActionController::Base
include RooOnRails::Concerns::RequireApiKey
require_api_key
# or
require_api_key(only: :update)
# or
require_api_key(only_services: %i(service_1 service_2))
def index
# etc
end
Keys are specified in environment variables ending with _CLIENT_KEY
, where the value is a comma separated list of keys which the specified service can authenticate with. This means that if your service has the environment variables:
SERVICE_1_CLIENT_KEY=abc123abc123,def456def456
SERVICE_2_CLIENT_KEY=I-never-could-get-the-hang-of-Thursdays
Then, for any controller where this concern has been initiated, Basic Authentication will be required and only service_1:abc123abc123
, service_1:def456def456
and service_2:I-never-could-get-the-hang-of-Thursdays
will be allowed access.
Command features
Usage
Run the following from your app's top-level directory:
roo_on_rails harness
That command will sequentially run a number of checks. For it to run successfully, you will need:
- a GitHub API token that can read your GitHub repository's settings placed in
~/.roo_on_rails/github-token
- the Heroku toolbelt installed and logged in
- admin privileges on the
roo-dd-bridge-production
(this will be addressed eventually)
The command can automatically fix most of the failing checks automatically;
simply run it with the --fix
flag:
roo_on_rails harness --fix
To run checks for only one environment, use the --env
flag:
roo_on_rails harness --env staging
Description
Running the roo_on_rails
command currently checks for:
- the presence of
PLAYBOOK.md
- compliant Heroku app naming;
- presence of the Heroku preboot flag;
- correct Github master branch protection;
- integration with the Heroku-Datadog metrics bridge (for CPU, memory, request throughput data);
- integration with Papertrail;
- correct Sidekiq configuration.
The command is designed to fix issues in many cases.
Contributing
Pull requests are welcome on GitHub at
https://github.com/deliveroo/roo_on_rails
.
License
The gem is available as open source under the terms of the MIT License.