GOV.UK A/B Testing

Gem to help with A/B testing on the GOV.UK platform.

Technical documentation

Installation

Add this line to your application's Gemfile:

gem 'govuk_ab_testing', '~> VERSION'

And then execute:

$ bundle

Usage

Before starting this, you'll need to

  • get your cookie listed on /help/cookies
  • configure the CDN like we did for the Education Navigation test. The cookie and header name in the CDN config must match the test name parameter that you pass to the Gem. The cookie name is case-sensitive.
  • configure Google Analytics (guidelines to follow)

To enable testing in the app, your Rails app needs:

  1. Some piece of logic to be A/B tested
  2. A HTML meta tag that will be used to measure the results, and which specifies the dimension to use in Google Analytics
  3. A response HTTP header that tells Fastly you're doing an A/B test

Start by defining which acceptance testing framework you will use. This gem supports both Capybara and ActiveSupport. In order to configure it, add this to your test helper file:

GovukAbTesting.configure do |config|
  config.acceptance_test_framework = :capybara # or :active_support
end

If we use capybara, the gem expects page to be defined in the scope of the test cases. If we use ActiveSupport, the gem expects @request to be defined in the scope of the test cases.

Now, let's say you have this controller:

# app/controllers/party_controller.rb
class PartyController < ApplicationController
  def show
    ab_test = GovukAbTesting::AbTest.new("your_ab_test_name", dimension: 300)
    @requested_variant = ab_test.requested_variant(request.headers)
    @requested_variant.configure_response(response)

    if @requested_variant.variant_b?
      render "new_show_template_to_be_tested"
    else
      render "show"
    end
  end
end

Add this to your layouts, so that we have a meta tag that can be picked up by the extension and analytics.

<!-- application.html.erb -->
<head>
  <%= @requested_variant.analytics_meta_tag.html_safe %>
</head>

Test helpers

Minitest

The most common usage of an A/B test is to serve two different variants of the same page. In this situation, you can test the controller using with_variant. It will configure the request and assert that the response is configured correctly:

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "show the user the B version" do
    with_variant your_ab_test_name: "B" do
      get :show

      # Optional assertions about page content of the B variant
    end
  end
end

Pass the assert_meta_tag: false option to skip assertions about the meta tag, for example because the variant returns a redirect response rather than returning an HTML page.

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "redirect the user the B version" do
    with_variant your_ab_test_name: "B", assert_meta_tag: false do
      get :show

      assert_response 302
      assert_redirected_to { controller: "other_controller", action: "show" }
    end
  end
end

To test the negative case in which a page is unaffected by the A/B test:

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "show the original" do
    setup_ab_variant("your_ab_test_name", "B") # optionally pass in a analytics dimension as the third argument

    get :show

    assert_response_not_modified_for_ab_test("your_ab_test_name")
  end
end

There are some more fine-grained assertions which you can use to test a page with A/B variants which should be cached separately, but which should be excluded from the analytics:

# test/controllers/party_controller_test.rb
class PartyControllerTest < ActionController::TestCase
  include GovukAbTesting::MinitestHelpers

  should "cache each variant but not add analytics" do
    setup_ab_variant("your_ab_test_name", "B")

    get :show

    assert_response_is_cached_by_variant("your_ab_test_name")
    assert_page_not_tracked_in_ab_test("your_ab_test_name")
  end
end
RSpec

It is also possible to use with_variant and all the individual setup and assertions steps in RSpec tests. Here is an example of a spec file:

# spec/features/ab_testing_spec.rb
feature "Viewing a page with an A/B test" do
  include GovukAbTesting::RspecHelpers

  scenario "viewing the B version of the page" do
    with_variant your_ab_test_name: 'B' do
      visit root_path

      expect(page).to have_breadcrumbs
      expect(page).to have_beta_label
    end
  end
end

As with the minitest version, you can also pass in the following options to with_variant:

  • assert_meta_tag: false
  • dimension: <number>

Current limitations

This library assumes we are only using one A/B test per page. The acceptance test classes look for only one analytics' meta tag and will fail in the presence of more than one.

Running the test suite

bundle exec rake

Testing in a browser

If you want to test this behaviour in a browser then you should use the GOV.UK Toolkit for Chrome.

This detects when you have a test running on a page and enables you to choose between variants.

Documentation

See RubyDoc for some limited documentation.

To run a Yard server locally to preview documentation, run:

$ bundle exec yard server --reload

Licence

MIT License