Module: RubyLLM::Tribunal::Assertions::Embedding

Defined in:
lib/ruby_llm/tribunal/assertions/embedding.rb

Overview

Embedding-based semantic similarity assertions.

Uses sentence embeddings to determine if two texts are semantically similar.

Constant Summary collapse

DEFAULT_THRESHOLD = 
0.7

Class Method Summary collapse

Class Method Details

.available ⇒ Array<Symbol>

Returns list of available embedding assertion types.

Returns:

  • (Array<Symbol>) 


16
17
18
 
# File 'lib/ruby_llm/tribunal/assertions/embedding.rb', line 16

def available
  [:similar]
end

.evaluate(test_case, opts = {}) ⇒ Array

Evaluates semantic similarity between actual and expected output.

Examples:

test_case = TestCase.new(
  actual_output: "The cat is sleeping",
  expected_output: "A feline is resting"
)

Embedding.evaluate(test_case, threshold: 0.8)
#=> [:pass, { similarity: 0.85, threshold: 0.8 }]

Parameters:

  • test_case (TestCase) —

    The test case

  • opts (Hash) (defaults to: {}) —

    Options

Options Hash (opts):

  • :threshold (Float) —

    Similarity threshold (0.0 to 1.0). Default: 0.7

  • :similarity_fn (Proc) —

    Custom similarity function for testing

Returns:

  • (Array) —

    [:pass, details], [:fail, details], or [:error, message]



36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/ruby_llm/tribunal/assertions/embedding.rb', line 36

def evaluate(test_case, opts = {})
  if test_case.expected_output.nil?
    return [:error,
            'Similar assertion requires expected_output to be provided']
  end

  threshold = opts[:threshold] || DEFAULT_THRESHOLD
  similarity_fn = opts[:similarity_fn] || method(:default_similarity)

  result = similarity_fn.call(test_case.actual_output, test_case.expected_output, opts)

  case result
  in [:ok, similarity] if similarity >= threshold
    [:pass, { similarity:, threshold: }]
  in [:ok, similarity]
    [:fail, {
      similarity:,
      threshold:,
      reason: "Output is not semantically similar to expected (#{similarity.round(2)} < #{threshold})"
    }]
  in [:error, reason]
    [:error, "Failed to compute similarity: #{reason}"]
  end
end