Module: LabTech

Extended by:
LabTech
Included in:
LabTech
Defined in:
lib/lab_tech.rb,
lib/lab_tech/engine.rb,
lib/lab_tech/version.rb,
app/models/lab_tech/result.rb,
app/models/lab_tech/speedup.rb,
app/models/lab_tech/summary.rb,
app/models/lab_tech/experiment.rb,
app/models/lab_tech/percentile.rb,
app/models/lab_tech/observation.rb,
app/models/lab_tech/summary/count.rb,
app/models/lab_tech/default_cleaner.rb,
app/models/lab_tech/application_record.rb,
app/models/lab_tech/summary/speedup_line.rb

Defined Under Namespace

Modules: Percentile Classes: ApplicationRecord, DefaultCleaner, Engine, Experiment, Observation, Result, Speedup, Summary

Constant Summary collapse

VERSION = 
'0.1.2'

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.compare_mismatches(experiment_name, limit: nil, io: $stdout, &block) ⇒ Object

…and be annoyed when they’re not 100% correct…

By default, this will simply print the values of all mismatches.
However, if you'd like to pass a block that returns arguments to
IO#puts, you can probably get more useful results.

Here's one example based on an experiment that records the IDs
returned from a search:

  comparison = ->(cont, cand) {
    cont_ids, cand_ids = cont.value, cand.value
    case
    when cont_ids      == cand_ids      ; "EVERYTHING IS FINE" # if this were true, it wouldn't be a mismatch
    when cont_ids.sort == cand_ids.sort ; "ORDER DIFFERS"
    else
      [
        "CONTROL   length: #{ cont_ids.length }",
        "CANDIDATE length: #{ cand_ids.length }",
        "    missing: #{ (cont_ids - cand_ids).inspect }",
        "    extra:   #{ (cand_ids - cont_ids).inspect }",
      ]
    end
  }
  e = Experiment.named "isolate-lead-activities-in-lead-search"
  e.compare_mismatches limit: 10, &comparison

And here's another one that assumes you've recorded a hash of the form:
{ ids: [ 1, 2, ... ], sql: "SELECT FROM ..." }

  comparison = ->(cont, cand) {
    cont_ids, cand_ids = cont.value.fetch(:ids), cand.value.fetch(:ids)
    cont_sql, cand_sql = cont.value.fetch(:sql), cand.value.fetch(:sql)
    sql_strings = [ "", "CONTROL SQL", cont_sql, "", "CANDIDATE SQL", cand_sql ]
    case
    when cont_ids      == cand_ids      ; "EVERYTHING IS FINE" # if this were true, it wouldn't be a mismatch
    when cont_ids.sort == cand_ids.sort ; [ "ORDER DIFFERS" ] + sql_strings
    else
      [
        "CONTROL   length: #{ cont_ids.length }",
        "CANDIDATE length: #{ cand_ids.length }",
        "    missing: #{ (cont_ids - cand_ids).inspect }",
        "    extra:   #{ (cand_ids - cont_ids).inspect }",
      ] + sql_strings
    end
  }
  e = Experiment.named "isolate-lead-activities-in-lead-search"
  e.compare_mismatches limit: 10, &comparison



135
136
137
138
 
# File 'lib/lab_tech.rb', line 135

def self.compare_mismatches(experiment_name, limit: nil, io: $stdout, &block)
  exp = LabTech::Experiment.named( experiment_name )
  exp.compare_mismatches limit: limit, io: io, &block
end

.disable(*experiment_names) ⇒ Object 



51
52
53
 
# File 'lib/lab_tech.rb', line 51

def self.disable(*experiment_names)
  experiments_named( experiment_names, &:disable )
end

.enable(*experiment_names, percent: 100) ⇒ Object

This here is how you turn individual experiments on and off…



45
46
47
48
49
 
# File 'lib/lab_tech.rb', line 45

def self.enable(*experiment_names, percent: 100)
  experiments_named( experiment_names ) do |exp|
    exp.enable percent_enabled: percent
  end
end

.experiments_named(*experiment_names, &block) ⇒ Object

Sometimes we want to act on a batch of experiments

(this is mostly just plumbing; feel free to ignore it)



169
170
171
172
173
174
 
# File 'lib/lab_tech.rb', line 169

def self.experiments_named(*experiment_names, &block)
  names = experiment_names.flatten.compact
  names.each do |exp_name|
    LabTech::Experiment.named(exp_name, &block)
  end
end

.publish_results_in_test_mode ⇒ Object 



63
64
65
66
67
68
69
70
71
 
# File 'lib/lab_tech.rb', line 63

def self.publish_results_in_test_mode
  fail ArgumentError, "a block is required for this method" unless block_given?

  old_value = self.publish_results_in_test_mode?
  self.publish_results_in_test_mode = true
  yield
ensure
  self.publish_results_in_test_mode = old_value
end

.publish_results_in_test_mode=(value) ⇒ Object 



62
 
# File 'lib/lab_tech.rb', line 62

def self.publish_results_in_test_mode=(value) ;   @publish_results_in_test_mode = !!value ; end

.publish_results_in_test_mode? ⇒ Boolean

…with an additional step if you want to record results in the Rails

test environment.

Returns:

  • (Boolean) 


61
 
# File 'lib/lab_tech.rb', line 61

def self.publish_results_in_test_mode?        ; !!@publish_results_in_test_mode           ; end

.reset_run_count! ⇒ Object

Sometimes specs might want to see that an experiment ran, the silly paranoid things



155
156
157
 
# File 'lib/lab_tech.rb', line 155

def self.reset_run_count!
  run_count.clear
end

.run_count ⇒ Object 



158
159
160
 
# File 'lib/lab_tech.rb', line 158

def self.run_count
  @_experiment_run_count ||= Hash.new(0)
end

.summarize_errors(experiment_name, limit: nil, io: $stdout) ⇒ Object

…and be curious about the errors…



145
146
147
148
 
# File 'lib/lab_tech.rb', line 145

def self.summarize_errors(experiment_name, limit: nil, io: $stdout)
  exp = LabTech::Experiment.named( experiment_name )
  exp.summarize_errors( limit: limit, io: io )
end

.summarize_results(*experiment_names) ⇒ Object

You’ll probably want to see how your experiments are doing…



78
79
80
 
# File 'lib/lab_tech.rb', line 78

def self.summarize_results(*experiment_names)
  experiments_named( experiment_names, &:summarize_results )
end

Instance Method Details

#science(experiment_name, opts = {}) {|experiment| ... } ⇒ Object

So, you’ve come here for science? EXCELLENT.

TL;DR:

LabTech.science "experiment-name" do |exp|
  exp.use { STABLE_CODE } # this is the "control"
  exp.try { BETTER_CODE } # this is the "candidate"

  # Optional, but often useful:
  exp.context foo: "spam", bar: "eggs", yak: "bacon"
  exp.compare {|control, candidate| control.map(&:id) == candidate.map(&:id) }
  exp.clean { |records| records.map(&:id) }
end

See https://github.com/github/scientist for an *extremely* detailed
README that explains how to use this.  For those purposes, the thing
passed to the block as `exp` is a Scientist::Experiment.

NOTE: You'll probably want to check out the .enable and .disable methods
below if you want your candidate code to actually *run*...

Yields:

  • (experiment) 


31
32
33
34
35
36
37
38
 
# File 'lib/lab_tech.rb', line 31

def science(experiment_name, opts = {}, &block)
  experiment = Experiment.named( experiment_name )

  yield experiment

  candidate_name = opts[:run]
  experiment.run(candidate_name)
end