Class: Langsmith::Evaluation::ExperimentRunner

Inherits:
Object
  • Object
show all
Defined in:
lib/langsmith/evaluation/experiment_runner.rb

Overview

Orchestrates running an evaluation experiment against a dataset.

Fetches examples, creates an experiment, runs each example through the user-provided block with evaluation context set, scores outputs with evaluators, and returns a summary of results.

Instance Method Summary collapse

Constructor Details

#initialize(dataset_id:, experiment_name:, description: nil, metadata: nil, evaluators: {}, tenant_id: nil, &block) ⇒ ExperimentRunner

Returns a new instance of ExperimentRunner.

Parameters:

  • dataset_id (String) —

    the dataset to evaluate against

  • experiment_name (String) —

    name for the experiment

  • description (String, nil) (defaults to: nil) —

    optional experiment description

  • metadata (Hash, nil) (defaults to: nil) —

    optional experiment metadata

  • evaluators (Hash) (defaults to: {}) —

    map of evaluator key to callable

  • tenant_id (String, nil) (defaults to: nil) —

    tenant ID for dataset/session/feedback API calls

  • block (Proc) —

    block that receives each example and produces a result



18
19
20
21
22
23
24
25
26
27
 
# File 'lib/langsmith/evaluation/experiment_runner.rb', line 18

def initialize(dataset_id:, experiment_name:, description: nil, metadata: nil, evaluators: {}, tenant_id: nil,
               &block)
  @dataset_id = dataset_id
  @experiment_name = experiment_name
  @description = description
  @metadata = metadata
  @evaluators = evaluators
  @tenant_id = tenant_id
  @block = block
end

Instance Method Details

#run ⇒ Hash

Run the evaluation experiment.

Returns:

  • (Hash) —

    summary with :experiment_id, :total, :succeeded, :failed, :results



32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
 
# File 'lib/langsmith/evaluation/experiment_runner.rb', line 32

def run
  experiment_id = nil
  examples = client.list_examples(dataset_id: @dataset_id, tenant_id: @tenant_id)

  experiment = client.create_experiment(
    name: @experiment_name,
    dataset_id: @dataset_id,
    description: @description,
    metadata: @metadata,
    tenant_id: @tenant_id
  )
  experiment_id = experiment[:id]

  results = examples.map { |example| run_example(example, experiment_id) }
  build_summary(experiment_id, results)
ensure
  close_experiment(experiment_id) if experiment_id
end