edhd

Experiment A/B framework for Ruby. Thread-safety is left as an exercise to the user. I use globals. Shoot me. Supports nested experiments.

Overview

Experiments are defined as follows (look at the specs for more details):

  • Exactly one of the hypotheses blocks in an experiment will run.

  • A hypothesis will be chosen with probability proportional to its weight.

  • You can pass in a seed for the random number generator to ensure that, for example, a given user always sees the same hypothesis.

  • An experiment block will log which hypothesis actually runs by calling $observer.hypothesized(experiment_name, hypothesis_name, seed)

  • Calling observation(name, value, seed) will log to $observer.observed(name, value, seed).

  • $observer = IOObserver(STDOUT) will give you a sample observer, but you’ll probably want to log to a database or something. Exercise left to the user.

  • Statistical analysis is an exercise left to the user.

    # Method signatures
    # def experiment(name = "default", seed = rand, &block)
    # def hypothesis(name, weight = 1, &block)
    # def observation(name, value, seed)
    
    # Example
    experiment "hello world", current_user. do
      hypothesis "a1", 3 do
        @a = 1
      end
    
      hypothesis "a2" do
        @a = 2
      end
    end
    
    observation "value of a", @a, current_user.
    

Copyright © 2009 Kyle Maxwell. MIT Licensed