Module: Scientist::Experiment

Included in:
Default
Defined in:
lib/scientist/experiment.rb

Overview

This mixin provides shared behavior for experiments. Includers must implement ‘enabled?` and `publish(result)`.

Override Scientist::Experiment.new to set your own class which includes and implements Scientist::Experiment’s interface.

Defined Under Namespace

Modules: RaiseOnMismatch Classes: MismatchError

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#raise_on_mismatches ⇒ Object

Whether to raise when the control and candidate mismatch. If this is nil, raise_on_mismatches class attribute is used instead.



10
11
12
 
# File 'lib/scientist/experiment.rb', line 10

def raise_on_mismatches
  @raise_on_mismatches
end

Class Method Details

.included(base) ⇒ Object 



12
13
14
15
 
# File 'lib/scientist/experiment.rb', line 12

def self.included(base)
  self.set_default(base) if base.instance_of?(Class)
  base.extend RaiseOnMismatch
end

.new(name) ⇒ Object

Instantiate a new experiment (using the class given to the .set_default method).



18
19
20
 
# File 'lib/scientist/experiment.rb', line 18

def self.new(name)
  (@experiment_klass || Scientist::Default).new(name)
end

.set_default(klass) ⇒ Object

Configure Scientist to use the given class for all future experiments (must implement the Scientist::Experiment interface).

Called automatically when new experiments are defined.



26
27
28
 
# File 'lib/scientist/experiment.rb', line 26

def self.set_default(klass)
  @experiment_klass = klass
end

Instance Method Details

#before_run(&block) ⇒ Object

Define a block of code to run before an experiment begins, if the experiment is enabled.

The block takes no arguments.

Returns the configured block.



86
87
88
 
# File 'lib/scientist/experiment.rb', line 86

def before_run(&block)
  @_scientist_before_run = block
end

#behaviors ⇒ Object

A Hash of behavior blocks, keyed by String name. Register behavior blocks with the ‘try` and `use` methods.



92
93
94
 
# File 'lib/scientist/experiment.rb', line 92

def behaviors
  @_scientist_behaviors ||= {}
end

#clean(&block) ⇒ Object

A block to clean an observed value for publishing or storing.

The block takes one argument, the observed value which will be cleaned.

Returns the configured block.



101
102
103
 
# File 'lib/scientist/experiment.rb', line 101

def clean(&block)
  @_scientist_cleaner = block
end

#clean_value(value) ⇒ Object

Internal: Clean a value with the configured clean block, or return the value if no clean block is configured.

Rescues and reports exceptions in the clean block if they occur.



116
117
118
119
120
121
122
123
124
125
 
# File 'lib/scientist/experiment.rb', line 116

def clean_value(value)
  if @_scientist_cleaner
    @_scientist_cleaner.call value
  else
    value
  end
rescue StandardError => ex
  raised :clean, ex
  value
end

#cleaner ⇒ Object

Accessor for the clean block, if one is available.

Returns the configured block, or nil.



108
109
110
 
# File 'lib/scientist/experiment.rb', line 108

def cleaner
  @_scientist_cleaner
end

#compare(*args, &block) ⇒ Object

A block which compares two experimental values.

The block must take two arguments, the control value and a candidate value, and return true or false.

Returns the block.



133
134
135
 
# File 'lib/scientist/experiment.rb', line 133

def compare(*args, &block)
  @_scientist_comparator = block
end

#context(context = nil) ⇒ Object

A Symbol-keyed Hash of extra experiment data.



138
139
140
141
142
 
# File 'lib/scientist/experiment.rb', line 138

def context(context = nil)
  @_scientist_context ||= {}
  @_scientist_context.merge!(context) unless context.nil?
  @_scientist_context
end

#fabricate_durations_for_testing_purposes(fabricated_durations = {}) ⇒ Object

Provide predefined durations to use instead of actual timing data. This is here solely as a convenience for developers of libraries that extend Scientist.



298
299
300
 
# File 'lib/scientist/experiment.rb', line 298

def fabricate_durations_for_testing_purposes(fabricated_durations = {})
  @_scientist_fabricated_durations = fabricated_durations
end

#generate_result(name) ⇒ Object

Internal: Generate the observations and create the result from those and the control.



303
304
305
306
307
308
309
310
311
312
313
314
 
# File 'lib/scientist/experiment.rb', line 303

def generate_result(name)
  observations = []

  behaviors.keys.shuffle.each do |key|
    block = behaviors[key]
    fabricated_duration = @_scientist_fabricated_durations && @_scientist_fabricated_durations[key]
    observations << Scientist::Observation.new(key, self, fabricated_duration: fabricated_duration, &block)
  end

  control = observations.detect { |o| o.name == name }
  Scientist::Result.new(self, observations, control)
end

#ignore(&block) ⇒ Object

Configure this experiment to ignore an observation with the given block.

The block takes two arguments, the control observation and the candidate observation which didn’t match the control. If the block returns true, the mismatch is disregarded.

This can be called more than once with different blocks to use.



151
152
153
154
 
# File 'lib/scientist/experiment.rb', line 151

def ignore(&block)
  @_scientist_ignores ||= []
  @_scientist_ignores << block
end

#ignore_mismatched_observation?(control, candidate) ⇒ Boolean

Internal: ignore a mismatched observation?

Iterates through the configured ignore blocks and calls each of them with the given control and mismatched candidate observations.

Returns true or false.

Returns:

  • (Boolean) 


162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/scientist/experiment.rb', line 162

def ignore_mismatched_observation?(control, candidate)
  return false unless @_scientist_ignores
  @_scientist_ignores.any? do |ignore|
    begin
      ignore.call control.value, candidate.value
    rescue StandardError => ex
      raised :ignore, ex
      false
    end
  end
end

#name ⇒ Object

The String name of this experiment. Default is “experiment”. See Scientist::Default for an example of how to override this default.



176
177
178
 
# File 'lib/scientist/experiment.rb', line 176

def name
  "experiment"
end

#observations_are_equivalent?(a, b) ⇒ Boolean

Internal: compare two observations, using the configured compare block if present.

Returns:

  • (Boolean) 


181
182
183
184
185
186
187
188
189
190
 
# File 'lib/scientist/experiment.rb', line 181

def observations_are_equivalent?(a, b)
  if @_scientist_comparator
    a.equivalent_to?(b, &@_scientist_comparator)
  else
    a.equivalent_to? b
  end
rescue StandardError => ex
  raised :compare, ex
  false
end

#raise_on_mismatches? ⇒ Boolean

Whether or not to raise a mismatch error when a mismatch occurs.

Returns:

  • (Boolean) 


288
289
290
291
292
293
294
 
# File 'lib/scientist/experiment.rb', line 288

def raise_on_mismatches?
  if raise_on_mismatches.nil?
    self.class.raise_on_mismatches?
  else
    !!raise_on_mismatches
  end
end

#raise_with(exception) ⇒ Object 



192
193
194
 
# File 'lib/scientist/experiment.rb', line 192

def raise_with(exception)
  @_scientist_custom_mismatch_error = exception
end

#raised(operation, error) ⇒ Object

Called when an exception is raised while running an internal operation, like :publish. Override this method to track these exceptions. The default implementation re-raises the exception.



199
200
201
 
# File 'lib/scientist/experiment.rb', line 199

def raised(operation, error)
  raise error
end

#run(name = nil) ⇒ Object

Internal: Run all the behaviors for this experiment, observing each and publishing the results. Return the result of the named behavior, default “control”.



206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
 
# File 'lib/scientist/experiment.rb', line 206

def run(name = nil)
  behaviors.freeze
  context.freeze

  name = (name || "control").to_s
  block = behaviors[name]

  if block.nil?
    raise Scientist::BehaviorMissing.new(self, name)
  end

  unless should_experiment_run?
    return block.call
  end

  if @_scientist_before_run
    @_scientist_before_run.call
  end

  result = generate_result(name)

  begin
    publish(result)
  rescue StandardError => ex
    raised :publish, ex
  end

  if raise_on_mismatches? && result.mismatched?
    if @_scientist_custom_mismatch_error
      raise @_scientist_custom_mismatch_error.new(self.name, result)
    else
      raise MismatchError.new(self.name, result)
    end
  end

  control = result.control
  raise control.exception if control.raised?
  control.value
end

#run_if(&block) ⇒ Object

Define a block that determines whether or not the experiment should run.



247
248
249
 
# File 'lib/scientist/experiment.rb', line 247

def run_if(&block)
  @_scientist_run_if_block = block
end

#run_if_block_allows? ⇒ Boolean

Internal: does a run_if block allow the experiment to run?

Rescues and reports exceptions in a run_if block if they occur.

Returns:

  • (Boolean) 


254
255
256
257
258
259
 
# File 'lib/scientist/experiment.rb', line 254

def run_if_block_allows?
  (@_scientist_run_if_block ? @_scientist_run_if_block.call : true)
rescue StandardError => ex
  raised :run_if, ex
  return false
end

#should_experiment_run? ⇒ Boolean

Internal: determine whether or not an experiment should run.

Rescues and reports exceptions in the enabled method if they occur.

Returns:

  • (Boolean) 


264
265
266
267
268
269
 
# File 'lib/scientist/experiment.rb', line 264

def should_experiment_run?
  behaviors.size > 1 && enabled? && run_if_block_allows?
rescue StandardError => ex
  raised :enabled, ex
  return false
end

#try(name = nil, &block) ⇒ Object

Register a named behavior for this experiment, default “candidate”.



272
273
274
275
276
277
278
279
280
 
# File 'lib/scientist/experiment.rb', line 272

def try(name = nil, &block)
  name = (name || "candidate").to_s

  if behaviors.include?(name)
    raise Scientist::BehaviorNotUnique.new(self, name)
  end

  behaviors[name] = block
end

#use(&block) ⇒ Object

Register the control behavior for this experiment.



283
284
285
 
# File 'lib/scientist/experiment.rb', line 283

def use(&block)
  try "control", &block
end