Class: Braintrust::Eval::Runner

Inherits:
Object
  • Object
show all
Defined in:
lib/braintrust/eval/runner.rb

Overview

Internal runner class that performs the execution of the Eval and returns the result. Receives a fully-normalized Context — all callables are already typed wrappers.

Defined Under Namespace

Classes: CaseContext

Constant Summary collapse

MAX_PARALLELISM =

Maximum parallelism allowed (mirrors Internal::ThreadPool::MAX_PARALLELISM)

Internal::ThreadPool::MAX_PARALLELISM

Instance Method Summary collapse

Constructor Details

#initialize(eval_context) ⇒ Runner

Returns a new instance of Runner.

Parameters:

  • eval_context (Context) —

    Normalized eval context



26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/braintrust/eval/runner.rb', line 26

def initialize(eval_context)
  @eval_context = eval_context
  @tracer = eval_context.tracer_provider.tracer("braintrust-eval")

  # Whether any scorer/classifier can receive `trace:`. Computed once: the
  # per-case flush that makes traces queryable over BTQL is a real network
  # wait, so it's only worth paying when something will actually consume it.
  @needs_trace = (eval_context.scorers + eval_context.classifiers).any? { |c| wants_trace?(c) }

  # Mutexes for thread-safe result collection
  @score_mutex = Mutex.new
  @classification_mutex = Mutex.new
end

Instance Method Details

#run(parallelism: 1) ⇒ Result

Run evaluation and return Result

Parameters:

  • parallelism (Integer) (defaults to: 1) —

    Number of parallel workers (default: 1)

Returns:



43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/braintrust/eval/runner.rb', line 43

def run(parallelism: 1)
  start_time = Time.now
  eval_cases = eval_context.cases
  errors = Queue.new
  @scores = {} # Reset for each run: { scorer_name => Array<Numeric> }
  @classifications = {} # Reset for each run: { classifier_name => Array<ClassificationItem> }

  if parallelism && parallelism > 1
    Internal::ThreadPool.each(eval_cases, parallelism: parallelism) do |eval_case|
      run_eval_case(build_case_context(eval_case), errors)
    end
  else
    eval_cases.each do |eval_case|
      run_eval_case(build_case_context(eval_case), errors)
    end
  end

  # Convert Queue to Array after all threads complete
  error_array = [].tap { |a| a << errors.pop until errors.empty? }

  # Deliver any spans still buffered. Callers that supply their own
  # tracer_provider get no at_exit hook (see Trace.setup), so without this
  # the tail of the run would wait on the processor's schedule delay.
  flush_spans

  # Calculate duration
  duration = Time.now - start_time

  # Generate permalink (only when state and experiment are available)
  permalink = if eval_context.state && eval_context.experiment_id
    eval_context.state.object_permalink(object_type: "experiment", object_id: eval_context.experiment_id)
  end

  Result.new(
    experiment_id: eval_context.experiment_id,
    experiment_name: eval_context.experiment_name,
    project_id: eval_context.project_id,
    project_name: eval_context.project_name,
    permalink: permalink,
    errors: error_array,
    duration: duration,
    scores: @scores,
    classifications: @classifications.empty? ? nil : @classifications
  )
end