Marlowe, Your Request Sleuth

code: github.com/KineticCafe/marlowe/
issues: github.com/KineticCafe/marlowe/issues
docs: www.rubydoc.info/github/KineticCafe/marlowe/master
continuous integration: <img src=“https://travis-ci.org/KineticCafe/marlowe.svg?branch=master” alt=“Build Status” />

Description

Marlowe is a Rack middleware that extracts or creates a request ID using a pre-defined header, permitting request correlation across multiple services.

When using Rails, Marlowe automatically adds itself to the middleware before Rails::Rack::Logger.

Upgrading from Marlowe 1.x

In Marlowe 1.0, the correlation header was called Correlation-Id; since then, Rails 5 and other tools and frameworks (such as Phoenix) have standardized on the header X-Request-Id. Marlowe 2.0 changes to this header by default.

To keep complete compatibility with Marlowe 1.0, the following should be used:

# Rails
config.marlowe_header = 'Correlation-Id'
config.marlowe_handler = :simple
config.marlowe_return = false

# Rack
use Marlowe::Middleware, header: 'Correlation-Id', handler: :simple, return: false

Configuration

Marlowe has three main configuration options: the request ID header, the request ID handler, and the request ID return. The options may be provided to the Rack use command as a keyword option, or set in a corresponding marlowe_option configuration variable in Rails.

Specifies the header to be used for the request correlation ID. Defaults to X-Request-Id.

# Rails
config.marlowe_header = 'Correlation-Id'
# OR: config.marlowe_correlation_header = 'Correlation-Id'

# Rack
use Marlowe::Middleware, header: 'Correlation-Id'
# OR: use Marlowe::Middleware, correlation_header: 'Correlation-Id'

Marlowe will convert this to an appropriate HTTP header (in the Rack env parameter, the above header would be represented as env['HTTP_CORRELATION_ID']).

Request ID Handler

Specifies the method for sanitizing or generating the request correlation ID. Values can be :clean (the default, which limits incoming correlation IDs to 255 alphanumeric-or-dash characters), :simple (does not limit incoming correlation IDs), or a proc to transform or generate a correlation ID.

In all cases, if a correlation request ID is not handled, a UUID will be generated.

# Rails
config.marlowe_handler = :simple
config.marlowe_handler = ->(req_id) {
  req_id.try(:reverse) || SecureRandom.uuid
}

# Rack
use Marlowe::Middleware, handler: :simple
use Marlowe::Middleware, handler: ->(req_id) {
  req_id.try(:reverse) || SecureRandom.uuid
}

Request ID Return

If true (the default), the request correlation ID will be returned to the client in the same header that it was provided in.

# Rails
config.marlowe_return = false

# Rack
use Marlowe::Middleware, return: false

Using Marlowe with Rails 5

Rails 5 includes the ActionDispatch::RequestId middleware, reducing the need for Marlowe. Marlowe is more configurable than the Rails 5 default, so set marlowe_replace_action_dispatch_request_id to true to have Marlowe::Middleware will replace ActionDispatch::RequestId:

# Rails only
config.marlowe_replace_action_dispatch_request_id = true

Accessing the Correlation ID

The correlation id can be accessed throughout the application by accessing the RequestStore storage.

RequestStore[:correlation_id]

Logging

For a Rails application, you simply need to change the log formatter to one of the provided ones. Correlated versions of both the SimpleFormatter and Formatter are included.

# config/environments/development.rb
Rails.application.configure do
  config.log_formatter = Marlowe::SimpleFormatter.new
end

To create your own formatter, you’ll need to access the RequestStore storage. You can use this pattern if you’ve rolled your own logger/formatter:

# lib/correlated_formatter.rb
require 'request_store'

class CorrelatedSimpleFormatter < ActiveSupport::Logger::SimpleFormatter
  def call(severity, timestamp, progname, msg)
    "[#{RequestStore.store[:correlation_id]}] #{super}"
  end
end

Lograge

As lograge supplies its own formatter, you will need to do something a little different:

# config/application.rb

class Application < Rails::Application
  config.before_initialize do
    ...
    # use lograge for all request logs
    config.lograge.enabled = true
    config.lograge.custom_options = lambda do |event|
      { correlation_id: RequestStore[:correlation_id] }
    end
  end
end

Clients

Catching and creating the correlation ID is a great all on its own, but to really take advantage of the correlation in a service based architecture you’ll need to pass the request ID to the next service in the change.

Here’s an example of a Hurley client:

# lib/correlated_client.rb

require 'hurley'
require 'request_store'

class Hurley::CorrelatedClient < Hurley::Client
  def initialize(*args, &block)
    super
    header['X-Request-Id'] = ::RequestStore.store[:correlation_id]
  end
end

If you have long-lived Hurley clients, it is also possible to use the Hurley callback machanism to add the outgoing headers:

client.before_call do |request|
  request.header['X-Request-Id'] = ::RequestStore.store[:correlation_id]
end

class Correlator
  def name
    :correlator
  end

  def call(request)
    request.header['X-Request-Id'] = ::RequestStore.store[:correlation_id]
  end
end

client.before_call(Correlator.new)

Install

Add Marlowe to your Gemfile:

gem 'marlowe', '~> 2.0'

Or manually install:

$ gem install marlowe

Marlowe Semantic Versioning

Marlowe uses a Semantic Versioning scheme with one significant change:

When PATCH is zero (0), it will be omitted from version references.

Additionally, the major version will generally be reserved for plug-in infrastructure changes.

Community and Contributing

Marlowe welcomes your contributions as described in Contributing.md. This project, like all Kinetic Cafe open source projects, is under the Kinetic Cafe Open Source Code of Conduct.