Field Test

:maple_leaf: A/B testing for Rails

  • Designed for web and email
  • Comes with a dashboard to view results and update variants
  • Uses your database for storage
  • Seamlessly handles the transition from anonymous visitor to logged in user

Uses Bayesian statistics to evaluate results so you don’t need to choose a sample size ahead of time.

Build Status

Installation

Add this line to your application’s Gemfile:

gem "field_test"

Run:

rails generate field_test:install
rails db:migrate

And mount the dashboard in your config/routes.rb:

mount FieldTest::Engine, at: "field_test"

Be sure to secure the dashboard in production.

Screenshot

Getting Started

Add an experiment to config/field_test.yml.

experiments:
  button_color:
    variants:
      - red
      - green
      - blue

Refer to it in controllers, views, and mailers.

button_color = field_test(:button_color)

To make testing easier, you can specify a variant with query parameters

http://localhost:3000/?field_test[button_color]=green

When someone converts, record it with:

field_test_converted(:button_color)

When an experiment is over, specify a winner:

experiments:
  button_color:
    winner: green

All calls to field_test will now return the winner, and metrics will stop being recorded.

You can keep returning the variant for existing participants after a winner is declared:

experiments:
  button_color:
    winner: green
    keep_variant: true

You can also close an experiment to new participants without declaring a winner while still recording metrics for existing participants:

experiments:
  button_color:
    closed: true

Calls to field_test for new participants will return the control, and they won’t be added to the experiment.

You can get the list of experiments and variants for a user with:

field_test_experiments

JavaScript and Native Apps

For JavaScript and native apps, add calls to your normal endpoints.

class CheckoutController < ActionController::API
  def start
    render json: {button_color: field_test(:button_color)}
  end

  def finish
    field_test_converted(:button_color)
    # ...
  end
end

For anonymous visitors in native apps, pass a Field-Test-Visitor header with a unique identifier.

Participants

Any model or string can be a participant in an experiment.

For web requests, it uses current_user (if it exists) and an anonymous visitor id to determine the participant. Set your own with:

class ApplicationController < ActionController::Base
  def field_test_participant
    current_company
  end
end

For mailers, it tries @user then params[:user] to determine the participant. Set your own with:

class ApplicationMailer < ActionMailer::Base
  def field_test_participant
    @company
  end
end

You can also manually pass a participant with:

field_test(:button_color, participant: company)

Config

By default, bots are returned the first variant and excluded from metrics. Change this with:

exclude:
  bots: false

Keep track of when experiments started and ended. Use any format Time.parse accepts. Variants assigned outside this window are not included in metrics.

experiments:
  button_color:
    started_at: Dec 1, 2016 8 am PST
    ended_at: Dec 8, 2016 2 pm PST

Add a friendlier name and description with:

experiments:
  button_color:
    name: Buttons!
    description: >
      Different button colors
      for the landing page.

By default, variants are given the same probability of being selected. Change this with:

experiments:
  button_color:
    variants:
      - red
      - blue
    weights:
      - 85
      - 15

To help with GDPR compliance, you can switch from cookies to anonymity sets for anonymous visitors. Visitors with the same IP mask and user agent are grouped together.

cookies: false

Dashboard Config

If the dashboard gets slow, you can make it faster with:

cache: true

This will use the Rails cache to speed up winning probability calculations.

If you need more precision, set:

precision: 1

Multiple Goals

You can set multiple goals for an experiment to track conversions at different parts of the funnel. First, run:

rails generate field_test:events
rails db:migrate

And add to your config:

experiments:
  button_color:
    goals:
      - signed_up
      - ordered

Specify a goal during conversion with:

field_test_converted(:button_color, goal: "ordered")

The results for all goals will appear on the dashboard.

Analytics Platforms

You may also want to send experiment data as properties to other analytics platforms like Segment, Amplitude, and Ahoy. Get the list of experiments and variants with:

field_test_experiments

Ahoy

You can configure Field Test to use Ahoy’s visitor token instead of creating its own:

class ApplicationController < ActionController::Base
  def field_test_participant
    [ahoy.user, ahoy.visitor_token]
  end
end

Dashboard Security

Devise

authenticate :user, ->(user) { user.admin? } do
  mount FieldTest::Engine, at: "field_test"
end

Basic Authentication

Set the following variables in your environment or an initializer.

ENV["FIELD_TEST_USERNAME"] = "moonrise"
ENV["FIELD_TEST_PASSWORD"] = "kingdom"

Reference

Assign a specific variant to a user with:

experiment = FieldTest::Experiment.find(:button_color)
experiment.variant(participant, variant: "green")

You can also change a user’s variant from the dashboard.

To associate models with field test memberships, use:

class User < ApplicationRecord
  has_many :field_test_memberships, class_name: "FieldTest::Membership", as: :participant
end

Now you can do:

user.field_test_memberships

Upgrading

0.3.0

Upgrade the gem and add to config/field_test.yml:

legacy_participants: true

Also, if you use Field Test in emails, know that the default way participants are determined has changed. Restore the previous way with:

class ApplicationMailer < ActionMailer::Base
  def field_test_participant
    message.to.first
  end
end

We also recommend upgrading participants when you have time.

Upgrading Participants

Field Test 0.3.0 splits the field_test_memberships.participant column into participant_type and participant_id.

To upgrade without downtime, create a migration:

rails generate migration upgrade_field_test_participants

with:

class UpgradeFieldTestParticipants < ActiveRecord::Migration[5.2]
  def change
    add_column :field_test_memberships, :participant_type, :string
    add_column :field_test_memberships, :participant_id, :string

    add_index :field_test_memberships, [:participant_type, :participant_id, :experiment],
      unique: true, name: "index_field_test_memberships_on_participant_and_experiment"
  end
end

After you run it, writes will go to both the old and new sets of columns.

Next, backfill data:

FieldTest::Membership.where(participant_id: nil).find_each do |membership|
  participant = membership.participant

  if participant.include?(":")
    participant_type, _, participant_id = participant.rpartition(":")
    participant_type = nil if participant_type == "cookie" # legacy
  else
    participant_id = participant
  end

  membership.update!(
    participant_type: participant_type,
    participant_id: participant_id
  )
end

Finally, remove legacy_participants: true from the config file. Once you confirm it’s working, you can drop the participant column (you can rename it first just to be extra safe).

Credits

A huge thanks to Evan Miller for deriving the Bayesian formulas.

History

View the changelog

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help: