Module: Langsmith::Evaluation

Defined in:
lib/langsmith/evaluation.rb,
lib/langsmith/evaluation/experiment_runner.rb

Overview

Public API for running evaluations against LangSmith datasets.

Examples:

Langsmith::Evaluation.run(
  dataset_id: "dataset-uuid",
  experiment_name: "my-experiment"
) do |example|
  Langsmith.trace("eval", run_type: "chain", inputs: example[:inputs]) do
    my_app.call(example[:inputs])
  end
end

Defined Under Namespace

Classes: ExperimentRunner

Class Method Summary collapse

Class Method Details

.run(dataset_id:, experiment_name:, description: nil, metadata: nil, evaluators: {}, tenant_id: nil) {|Hash| ... } ⇒ Hash

Run an evaluation experiment against a dataset.

Parameters:

  • dataset_id (String) —

    the dataset to evaluate against

  • experiment_name (String) —

    name for the experiment

  • description (String, nil) (defaults to: nil) —

    optional experiment description

  • metadata (Hash, nil) (defaults to: nil) —

    optional experiment metadata

  • evaluators (Hash) (defaults to: {}) —

    map of evaluator key to callable (see ExperimentRunner)

  • tenant_id (String, nil) (defaults to: nil) —

    tenant ID for dataset/session/feedback API calls

Yields:

  • (Hash) —

    each dataset example

Returns:

  • (Hash) —

    summary with :experiment_id, :total, :succeeded, :failed, :results



28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/langsmith/evaluation.rb', line 28

def self.run(dataset_id:, experiment_name:, description: nil, metadata: nil, evaluators: {}, tenant_id: nil, &block)
  ExperimentRunner.new(
    dataset_id: dataset_id,
    experiment_name: experiment_name,
    description: description,
    metadata: metadata,
    evaluators: evaluators,
    tenant_id: tenant_id,
    &block
  ).run
end