GitLab Experiment

Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our Experiment Guide docs if you're curious about how we use this gem internally.

This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.

When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.

• experiment is any deviation of code paths we want to run sometimes and not others.
• context is used to identify a consistent experience we'll provide in an experiment.
• control is the default, or "original" code path.
• candidate defines that there's one experimental code path.
• variant(s) is used when more than one experimental code path exists.

Candidate and variant are the same concept, but simplify how we speak about experimental paths -- this concept is also widely referred to as the "experiment group".

[[TOC]]

Installation

Add the gem to your Gemfile and then bundle install.

gem 'gitlab-experiment'

If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.

$ rails generate gitlab:experiment:install

Implementing an experiment

For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.

In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.

The behavior will be the same, but the interface will be different and may involve more or fewer steps.

Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.

We'll name our experiment notification_toggle. This name is prefixed based on configuration. If you've set config.name_prefix = 'gitlab', the experiment name would be gitlab_notification_toggle elsewhere.

When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.

A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching.

Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.

class SubscriptionsController < ApplicationController
  def show
    experiment(:notification_toggle, actor: user) do |e|
      e.use { render_toggle } # control
      e.try { render_button } # candidate
    end
  end
end

You can define the experiment using simple control/candidate paths, or provide named variants.

Handling multivariate experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.

experiment(:notification_toggle, actor: user) do |e|
  e.use { render_toggle } # control
  e.try(:variant_one) { render_button(confirmation: true) }
  e.try(:variant_two) { render_button(confirmation: false) }
end

Understanding how an experiment can change behavior is important in evaluating its performance.

To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.

experiment(:notification_toggle, actor: user).track(:clicked_button)

Custom experiments

You can craft more advanced behaviors by defining custom experiments at a higher level. To do this you can define a class that inherits from ApplicationExperiment (or Gitlab::Experiment).

Let's say you want to do more advanced segmentation, or provide default behavior for the variants on the experiment we've already outlined above -- that way if the variants aren't defined in the block at the time the experiment is run, these methods will be used.

You can generate a custom experiment by running:

$ rails generate gitlab:experiment NotificationToggle control candidate

This will generate a file in app/experiments/notification_toggle_experiment.rb, as well as a test file for you to further expand on.

Here are some examples of what you can introduce once you have a custom experiment defined.

class NotificationToggleExperiment < ApplicationExperiment
  # Exclude any users that aren't me.
  exclude :users_named_richard

  # Segment any account older than 2 weeks into the candidate, without
  # asking the variant resolver to decide which variant to provide.
  segment :old_account?, variant: :candidate

  # Define the default control behavior, which can be overridden at
  # experiment time.
  def control_behavior
    # render_toggle
  end

  # Define the default candidate behavior, which can be overridden
  # at experiment time.
  def candidate_behavior
    # render_button
  end

  private

  def users_named_richard
    context.actor.first_name == 'Richard'
  end

  def old_account?
    context.actor.created_at < 2.weeks.ago
  end
end

# The class will be looked up based on the experiment name provided. 
exp = experiment(:notification_toggle, actor: user)
exp # => instance of NotificationToggleExperiment

# Run the experiment -- returning the result.
exp.run

# Track an event on the experiment we've defined.
exp.track(:clicked_button)

You can now also do things very similar to the simple examples and override the default variant behaviors defined in the custom experiment -- keeping in mind that this should be carefully considered within the scope of your experiment.

experiment(:notification_toggle, actor: user) do |e|
  e.use { render_special_toggle } # override default control behavior
end

You can also specify the variant to use for segmentation...

Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration. Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled. ```ruby experiment(:notification_toggle, :no_interface, actor: user) do |e| e.use { render_toggle } # control e.try { render_button } # candidate e.try(:no_interface) { no_interface! } # no_interface variant end ``` Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it. ```ruby experiment(:notification_toggle, actor: user) do |e| # Variant selection must be done before calling run or track. e.variant(:no_interface) # set the variant # ... end ``` Or it can be specified in the call to run if you call it from within the block. ```ruby experiment(:notification_toggle, actor: user) do |e| # ... # Variant selection can be specified when calling run. e.run(:no_interface) end ```

Segmentation rules

This library comes with the capability to segment contexts into a specific variant, before asking the variant resolver which variant to provide.

Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.

class ExampleExperiment < ApplicationExperiment
  segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
  segment :old_account?, variant: :variant_two

  private

  def old_account?
    context.actor.created_at < 2.weeks.ago
  end
end

In the previous examples, any user named 'Richard' would always receive the experience defined in "variant_one". As well, any account older than 2 weeks old would get the alternate experience defined in "variant_two".

When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.

This means that any user named 'Richard', regardless of account age, will always be provided the experience as defined in "variant_one". If you wanted the opposite logic, you can flip the order.

Exclusion rules

Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.

class ExampleExperiment < ApplicationExperiment
  exclude :old_account?, ->{ context.actor.first_name == 'Richard' }

  private

  def old_account? 
    context.actor.created_at < 2.weeks.ago
  end
end

The previous examples will exclude all users named 'Richard' as well as any account older than 2 weeks old. Not only will they be given the control behavior, but no events will be tracked in these cases as well.

You may need to check exclusion in custom tracking logic by calling should_track?:

def expensive_tracking_logic
  return unless should_track?

  track(:my_event, value: expensive_method_call)
end

Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an enabled? method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This enabled? check should be as efficient as possible because it's the first early opt out path an experiment can implement.

Return value

By default the return value is a Gitlab::Experiment instance. In simple cases you may want only the results of the experiment though. You can call run within the block to get the return value of the assigned variant.

experiment(:notification_toggle) do |e|
  e.use { 'A' }
  e.try { 'B' }
  e.run
end # => 'A'

Including the DSL

By default, Gitlab::Experiment injects itself into the controller and view layers. This exposes the experiment method application wide in those layers.

Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.

Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.

class WelcomeMailer < ApplicationMailer
  include Gitlab::Experiment::Dsl # include the `experiment` method

  def welcome
    @user = params[:user]

    ex = experiment(:project_suggestions, actor: @user) do |e|
      e.use { 'welcome' }
      e.try { 'welcome_with_project_suggestions' }
    end

    mail(to: @user.email, subject: 'Welcome!', template: ex.run)
  end
end

Context migrations

There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.

Take for instance, that you might be using version: 1 in your context currently. To migrate this to version: 2, provide the portion of the context you wish to change using a migrated_with option.

In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.

# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
experiment(:example, actor: project, version: 2, migrated_with: { version: 1 })

You can add or remove context by providing a migrated_from option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.

If you wanted to introduce a version to your context, provide the full previous context.

# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
experiment(:example, actor: project, version: 1, migrated_from: { actor: project })

This can impact an experience if you:

1. haven't implemented the concept of migrations in your variant resolver
2. haven't enabled a reasonable caching mechanism

When there isn't an actor (cookie fallback)

When there isn't an identifying key in the context (this is actor by default), we fall back to cookies to provide a consistent experience for the client viewing them.

Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.

This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.

To read and write cookies, we provide the request from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.

You'll need to provide the request as an option to the experiment if it's outside of the controller and views.

experiment(:example, actor: user, request: request)

The cookie isn't set if the actor key isn't present at all in the context. Meaning that when no actor key is provided, the cookie will not be set.

# actor is not present, so no cookie is set
experiment(:example, project: project)

# actor is present and is nil, so the cookie is set and used
experiment(:example, actor: nil, project: project)

# actor is present and set to a value, so no cookie is set
experiment(:example, actor: user, project: project)

For edge cases, you can pass the cookie through by assigning it yourself -- e.g. actor: request.cookie_jar.signed['example_actor']. The cookie name is the full experiment name (including any configured prefix) with _actor appended -- e.g. gitlab_notification_toggle_actor for the :notification_toggle experiment key with a configured prefix of gitlab.

How it works

The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.

graph TD
    GP[General Pool/Population] --> Enabled?
    Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
    Enabled? -->|No| Excluded[Control / No Tracking]
    Cached? -->|No| Excluded?
    Cached? -->|Yes| Cached[Cached Value]
    Excluded? -->|Yes / Cached| Excluded
    Excluded? -->|No| Segmented?
    Segmented? -->|Yes / Cached| VariantA
    Segmented? -->|No| Included?[Experiment Group?]
    Included? -->|Yes| Rollout
    Included? -->|No| Control
    Rollout -->|Cached| VariantA
    Rollout -->|Cached| VariantB
    Rollout -->|Cached| VariantC

classDef included fill:#380d75,color:#ffffff,stroke:none
classDef excluded fill:#fca121,stroke:none
classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
classDef default fill:#fff,stroke:#6e49cb

class VariantA,VariantB,VariantC included
class Control,Excluded excluded
class Cached cached

Configuration

This gem needs to be configured before being used in a meaningful way.

The default configuration will always render the control, so it's important to configure your own logic for resolving variants.

Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using Unleash or Flipper for this.

Gitlab::Experiment.configure do |config|
  # The block here is evaluated within the scope of the experiment instance,
  # which is why we are able to access things like name and context.
  config.variant_resolver = lambda do |requested_variant|
    # Return the requested variant if a specific one has been provided in code.
    return requested_variant unless requested_variant.nil?

    # Ask Unleash to determine the variant, given the context we've built,
    # using the control as the fallback.
    fallback = Unleash::Variant.new(name: 'control', enabled: true)
    UNLEASH.get_variant(name, context.value, fallback)
  end
end

More examples for configuration are available in the provided rails initializer.

Client layer / JavaScript

This library doesn't attempt to provide any logic for the client layer.

Instead it allows you to do this yourself in configuration. Using Gon to publish your experiment information to the client layer is pretty simple.

Gitlab::Experiment.configure do |config|
  config.publishing_behavior = lambda do |_result|
    # Push the experiment knowledge into the front end. The signature contains
    # the context key, and the variant that has been determined.
    Gon.push({ experiment: { name => signature } }, true)
  end
end

In the client you can now access window.gon.experiment.notificationToggle.

Caching

Caching can be enabled in configuration, and is implemented towards the Rails.cache / ActiveSupport::Cache::Store interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.

It's important to understand that using caching can drastically change or override your rollout strategy logic.

Gitlab::Experiment.configure do |config|
  config.cache = Rails.cache
end

Middleware

There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases, gitlab-experiment comes with middleware that will redirect to a given URL while also tracking that the URL was visited.

In Rails this middleware is mounted automatically, with a base path of what's been configured for mount_at. If this path is empty the middleware won't be mounted at all.

Once mounted, the redirect URLs can be generated using the Rails route helpers. If not using Rails, mount the middleware and generate these URLs yourself.

Gitlab::Experiment.configure do |config|
  config.mount_at = '/experiment'
end

ex = experiment(:example, foo: :bar)

# using rails path/url helpers 
experiment_redirect_path(ex, 'https//docs.gitlab.com/') # => /experiment/example:[context_key]?https//docs.gitlab.com/

# manually
"#{Gitlab::Experiment.configure.mount_at}/#{ex.to_param}?https//docs.gitlab.com/"

URLS that match the base path will be handled by the middleware and will redirect to the provided redirect path.

Testing (rspec support)

This gem comes with some rspec helpers and custom matchers. These are in flux at the time of writing.

First, require the rspec support file:

require 'gitlab/experiment/rspec'

This mixes in some of the basics, but the matchers and other aspects need to be included. This happens automatically for files in spec/experiments, but for other files and specs you want to include it in, you can specify the :experiment type:

it "tests", :experiment do
end

Stub helpers

You can stub experiments using stub_experiments. Pass it a hash using experiment names as the keys and the variants you want each to resolve to as the values:

# Ensures the experiments named `:example` & `:example2` are both
# "enabled" and that each will resolve to the given variant
# (`:my_variant` & `:control` respectively).
stub_experiments(example: :my_variant, example2: :control)

experiment(:example) do |e|
  e.enabled? # => true
  e.variant.name # => 'my_variant'
end

experiment(:example2) do |e|
  e.enabled? # => true
  e.variant.name # => 'control'
end

Exclusion and segmentation matchers

You can also easily test the exclusion and segmentation matchers.

class ExampleExperiment < ApplicationExperiment
  exclude { context.actor.first_name == 'Richard' }
  segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
end

excluded = double(username: 'rdiggitty', first_name: 'Richard')
segmented = double(username: 'jejacks0n', first_name: 'Jeremy')

# exclude matcher
expect(experiment(:example)).to exclude(actor: excluded)
expect(experiment(:example)).not_to exclude(actor: segmented)

# segment matcher
expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
expect(experiment(:example)).not_to segment(actor: excluded)

Tracking matcher

Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.

You can do this on the instance level or at an "any instance" level. At an instance level this is pretty straight forward:

subject = experiment(:example)

expect(subject).to track(:my_event)

subject.track(:my_event)

You can use the on_any_instance chain method to specify that it could happen on any instance of the experiment. This can be useful if you're calling experiment(:example).track downstream:

expect(experiment(:example)).to track(:my_event).on_any_instance

experiment(:example).track(:my_event)

And here's a full example of the methods that can be chained onto the track matcher:

expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
  .on_any_instance
  .with_context(foo: :bar)
  .for(:variant_name)

experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')

Tracking, anonymity and GDPR

We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).

We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.

This library attempts to be non-user-centric, in that a context can contain things like a user or a project.

If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.

Each of these approaches could be desirable given the objectives of your experiment.

Development

After checking out the repo, run bundle install to install dependencies. Then, run bundle exec rake to run the tests. You can also run bundle exec pry for an interactive prompt that will allow you to experiment.

Contributing

Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

Release process

Please refer to the Release Process.

License

The gem is available as open source under the terms of the MIT License.

Code of conduct

Everyone interacting in the Gitlab::Experiment project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

Make code not war