Class: EvalRuby::Metrics::SemanticSimilarity

Inherits:
Base
  • Object
show all
Defined in:
lib/eval_ruby/metrics/semantic_similarity.rb

Overview

Cosine similarity between an answer and its ground truth via an injected embedder. A judge-free alternative to Correctness when you want fast, deterministic, reference-based scoring — ideal for chatbot regression testing.

Examples:

embedder = EvalRuby::Embedders::OpenAI.new(EvalRuby.configuration)
metric = EvalRuby::Metrics::SemanticSimilarity.new(embedder: embedder)
metric.call(answer: "Paris is in France", ground_truth: "Paris, France")
# => { score: 0.91, details: { cosine: 0.91, model: "text-embedding-3-small" } }

Instance Attribute Summary collapse

Attributes inherited from Base

#judge

Instance Method Summary collapse

Constructor Details

#initialize(embedder: nil, judge: nil) ⇒ SemanticSimilarity

Returns a new instance of SemanticSimilarity.

Parameters:



22
23
24
25
 
# File 'lib/eval_ruby/metrics/semantic_similarity.rb', line 22

def initialize(embedder: nil, judge: nil)
  super(judge: judge)
  @embedder = embedder
end

Instance Attribute Details

#embedder ⇒ EvalRuby::Embedders::Base? (readonly)

Returns the embedder instance.

Returns:



17
18
19
 
# File 'lib/eval_ruby/metrics/semantic_similarity.rb', line 17

def embedder
  @embedder
end

Instance Method Details

#call(answer:, ground_truth:, **_kwargs) ⇒ Hash

Returns :score (Float 0.0–1.0) and :details (Hash).

Parameters:

  • answer (String) —

    candidate text (typically the model's answer)

  • ground_truth (String) —

    reference text

Returns:

  • (Hash) —

    :score (Float 0.0–1.0) and :details (Hash)

Raises:



31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
 
# File 'lib/eval_ruby/metrics/semantic_similarity.rb', line 31

def call(answer:, ground_truth:, **_kwargs)
  raise EvalRuby::Error, "SemanticSimilarity requires an embedder. Pass `embedder:` in the constructor." unless @embedder

  if answer.to_s.strip.empty? || ground_truth.to_s.strip.empty?
    return {score: 0.0, details: {reason: :empty_input}}
  end

  vectors = @embedder.call([answer.to_s, ground_truth.to_s])
  unless vectors.is_a?(Array) && vectors.length == 2
    raise EvalRuby::Error, "Embedder returned #{vectors.is_a?(Array) ? vectors.length : vectors.class} vectors; expected 2"
  end

  cosine = cosine_similarity(vectors[0], vectors[1])

  {
    score: cosine.clamp(0.0, 1.0),
    details: {cosine: cosine, model: @embedder.model}
  }
end