Module: RubyLLM::Tribunal::Assertions

Defined in:
lib/ruby_llm/tribunal/assertions.rb,
lib/ruby_llm/tribunal/assertions/judge.rb,
lib/ruby_llm/tribunal/assertions/embedding.rb,
lib/ruby_llm/tribunal/assertions/deterministic.rb

Overview

Assertion evaluation engine.

Routes assertions to the appropriate implementation:

  • Deterministic: contains, regex, is_json, etc.
  • Judge (requires ruby_llm): faithful, relevant, etc.
  • Embedding (requires neighbor): similar

Defined Under Namespace

Modules: Deterministic, Embedding, Judge

Constant Summary collapse

DETERMINISTIC_ASSERTIONS = 
i[
  contains
  not_contains
  contains_any
  contains_all
  regex
  is_json
  max_tokens
  latency_ms
  starts_with
  ends_with
  equals
  min_length
  max_length
  word_count
  is_url
  is_email
  levenshtein
].freeze
EMBEDDING_ASSERTIONS = 
i[similar].freeze

Class Method Summary collapse

Class Method Details

.all_passed?(results) ⇒ Boolean

Checks if all assertions passed.

Parameters:

  • results (Hash) —

    Results from evaluate_all

Returns:

  • (Boolean) —

    True if all passed



69
70
71
 
# File 'lib/ruby_llm/tribunal/assertions.rb', line 69

def all_passed?(results)
  results.all? { |_type, result| result.first == :pass }
end

.available ⇒ Array<Symbol>

Returns list of available assertion types based on loaded dependencies.

Returns:

  • (Array<Symbol>) —

    Available assertion types



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/ruby_llm/tribunal/assertions.rb', line 76

def available
  base = DETERMINISTIC_ASSERTIONS.dup

  # Add judge assertions
  base.concat(Tribunal::Judge.all_judge_names)

  # Add embedding assertions if neighbor is available
  begin
    require 'neighbor'
    base.concat(EMBEDDING_ASSERTIONS)
  rescue LoadError
    # neighbor not available
  end

  base
end

.evaluate(assertion_type, test_case, opts = {}) ⇒ Array

Evaluates a single assertion against a test case.

Parameters:

  • assertion_type (Symbol) —

    The type of assertion

  • test_case (TestCase) —

    The test case to evaluate

  • opts (Hash) (defaults to: {}) —

    Options for the assertion

Returns:

  • (Array) —

    [:pass, details] or [:fail, details]



41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/ruby_llm/tribunal/assertions.rb', line 41

def evaluate(assertion_type, test_case, opts = {})
  if DETERMINISTIC_ASSERTIONS.include?(assertion_type)
    Deterministic.evaluate(assertion_type, test_case.actual_output, opts)
  elsif Tribunal::Judge.builtin_judge?(assertion_type) || Tribunal::Judge.custom_judge?(assertion_type)
    evaluate_judge(assertion_type, test_case, opts)
  elsif EMBEDDING_ASSERTIONS.include?(assertion_type)
    evaluate_embedding(assertion_type, test_case, opts)
  else
    [:error, "Unknown assertion type: #{assertion_type}"]
  end
end

.evaluate_all(assertions, test_case) ⇒ Hash

Evaluates multiple assertions against a test case.

Parameters:

  • assertions (Array, Hash) —

    Assertions to evaluate

  • test_case (TestCase) —

    The test case to evaluate

Returns:

  • (Hash) —

    Map of assertion_type => result



58
59
60
61
62
63
 
# File 'lib/ruby_llm/tribunal/assertions.rb', line 58

def evaluate_all(assertions, test_case)
  normalized = normalize_assertions(assertions)
  normalized.each_with_object({}) do |(type, opts), results|
    results[type] = evaluate(type, test_case, opts)
  end
end