Module: EvalRuby

Defined in:
lib/eval_ruby.rb,
lib/eval_ruby/rspec.rb,
lib/eval_ruby/report.rb,
lib/eval_ruby/result.rb,
lib/eval_ruby/dataset.rb,
lib/eval_ruby/version.rb,
lib/eval_ruby/minitest.rb,
lib/eval_ruby/evaluator.rb,
lib/eval_ruby/comparison.rb,
lib/eval_ruby/judges/base.rb,
lib/eval_ruby/metrics/mrr.rb,
lib/eval_ruby/metrics/base.rb,
lib/eval_ruby/metrics/ndcg.rb,
lib/eval_ruby/configuration.rb,
lib/eval_ruby/judges/openai.rb,
lib/eval_ruby/embedders/base.rb,
lib/eval_ruby/embedders/openai.rb,
lib/eval_ruby/judges/anthropic.rb,
lib/eval_ruby/metrics/relevance.rb,
lib/eval_ruby/metrics/correctness.rb,
lib/eval_ruby/metrics/recall_at_k.rb,
lib/eval_ruby/metrics/faithfulness.rb,
lib/eval_ruby/metrics/context_recall.rb,
lib/eval_ruby/metrics/precision_at_k.rb,
lib/eval_ruby/metrics/context_precision.rb,
lib/eval_ruby/metrics/semantic_similarity.rb

Overview

Evaluation framework for LLM and RAG applications. Measures quality metrics like faithfulness, relevance, context precision, and answer correctness. Think Ragas or DeepEval for Ruby.

Examples:

Quick evaluation

result = EvalRuby.evaluate(
  question: "What is Ruby?",
  answer: "A programming language",
  context: ["Ruby is a dynamic, open source programming language."],
  ground_truth: "Ruby is a programming language created by Matz."
)
puts result.faithfulness  # => 0.95
puts result.overall       # => 0.87

Retrieval evaluation

result = EvalRuby.evaluate_retrieval(
  question: "What is Ruby?",
  retrieved: ["doc_a", "doc_b", "doc_c"],
  relevant: ["doc_a", "doc_c"]
)
puts result.precision_at_k(3) # => 0.67

Defined Under Namespace

Modules: Assertions, Embedders, Judges, Metrics, RSpecMatchers Classes: APIError, Comparison, Configuration, Dataset, Error, Evaluator, InvalidResponseError, Progress, Report, Result, RetrievalResult, TimeoutError

Constant Summary collapse

VERSION = 
"0.3.0"

Class Method Summary collapse

Class Method Details

.compare(report_a, report_b) ⇒ Comparison

Compares two evaluation reports with statistical significance testing.

Parameters:

  • report_a (Report) —

    baseline report

  • report_b (Report) —

    comparison report

Returns:



166
167
168
 
# File 'lib/eval_ruby.rb', line 166

def compare(report_a, report_b)
  Comparison.new(report_a, report_b)
end

.configuration ⇒ Configuration

Returns the current configuration.

Returns:



67
68
69
 
# File 'lib/eval_ruby.rb', line 67

def configuration
  @configuration ||= Configuration.new
end

.configure {|config| ... } ⇒ void

This method returns an undefined value.

Yields the configuration for modification.

Yield Parameters:



75
76
77
 
# File 'lib/eval_ruby.rb', line 75

def configure
  yield(configuration)
end

.evaluate(question:, answer:, context: [], ground_truth: nil) ⇒ Result

Evaluates an LLM response across multiple quality metrics.

Parameters:

  • question (String) —

    the input question

  • answer (String) —

    the LLM-generated answer

  • context (Array<String>) (defaults to: []) —

    retrieved context chunks

  • ground_truth (String, nil) (defaults to: nil) —

    expected correct answer

Returns:



93
94
95
96
97
98
99
100
 
# File 'lib/eval_ruby.rb', line 93

def evaluate(question:, answer:, context: [], ground_truth: nil)
  Evaluator.new.evaluate(
    question: question,
    answer: answer,
    context: context,
    ground_truth: ground_truth
  )
end

.evaluate_batch(dataset, pipeline: nil) {|progress| ... } ⇒ Report

Evaluates a batch of samples, optionally running them through a pipeline.

If a block is given, it is called after each sample with a Progress snapshot, useful for rendering progress bars or writing incremental logs.

Examples:

With progress callback

EvalRuby.evaluate_batch(dataset) do |progress|
  puts "#{progress.current}/#{progress.total} (#{progress.percent}%)"
end

Parameters:

  • dataset (Dataset, Array<Hash>) —

    samples to evaluate

  • pipeline (#query, nil) (defaults to: nil) —

    optional RAG pipeline to run queries through

Yield Parameters:

  • progress (Progress) —

    progress snapshot after each sample

Returns:



130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/eval_ruby.rb', line 130

def evaluate_batch(dataset, pipeline: nil, &progress_block)
  samples = dataset.is_a?(Dataset) ? dataset.samples : dataset
  evaluator = Evaluator.new
  start_time = Time.now
  total = samples.size

  results = samples.each_with_index.map do |sample, i|
    result = if pipeline
      response = pipeline.query(sample[:question])
      evaluator.evaluate(
        question: sample[:question],
        answer: response.respond_to?(:text) ? response.text : response.to_s,
        context: response.respond_to?(:context) ? response.context : sample[:context],
        ground_truth: sample[:ground_truth]
      )
    else
      evaluator.evaluate(**sample.slice(:question, :answer, :context, :ground_truth))
    end

    progress_block&.call(Progress.new(
      current: i + 1,
      total: total,
      elapsed: Time.now - start_time
    ))

    result
  end

  Report.new(results: results, samples: samples, duration: Time.now - start_time)
end

.evaluate_retrieval(question:, retrieved:, relevant:) ⇒ RetrievalResult

Evaluates retrieval quality using IR metrics.

Parameters:

  • question (String) —

    the input question

  • retrieved (Array<String>) —

    retrieved document IDs

  • relevant (Array<String>) —

    ground-truth relevant document IDs

Returns:



108
109
110
111
112
113
114
 
# File 'lib/eval_ruby.rb', line 108

def evaluate_retrieval(question:, retrieved:, relevant:)
  Evaluator.new.evaluate_retrieval(
    question: question,
    retrieved: retrieved,
    relevant: relevant
  )
end

.reset_configuration! ⇒ Configuration

Resets configuration to defaults.

Returns:



82
83
84
 
# File 'lib/eval_ruby.rb', line 82

def reset_configuration!
  @configuration = Configuration.new
end