Module: RubyLLM::Tribunal

Defined in:
lib/ruby_llm/tribunal.rb,
lib/ruby_llm/tribunal/judge.rb,
lib/ruby_llm/tribunal/dataset.rb,
lib/ruby_llm/tribunal/version.rb,
lib/ruby_llm/tribunal/red_team.rb,
lib/ruby_llm/tribunal/reporter.rb,
lib/ruby_llm/tribunal/test_case.rb,
lib/ruby_llm/tribunal/assertions.rb,
lib/ruby_llm/tribunal/judges/pii.rb,
lib/ruby_llm/tribunal/judges/bias.rb,
lib/ruby_llm/tribunal/eval_helpers.rb,
lib/ruby_llm/tribunal/configuration.rb,
lib/ruby_llm/tribunal/judges/harmful.rb,
lib/ruby_llm/tribunal/judges/refusal.rb,
lib/ruby_llm/tribunal/reporters/html.rb,
lib/ruby_llm/tribunal/reporters/json.rb,
lib/ruby_llm/tribunal/reporters/text.rb,
lib/ruby_llm/tribunal/judges/faithful.rb,
lib/ruby_llm/tribunal/judges/relevant.rb,
lib/ruby_llm/tribunal/judges/toxicity.rb,
lib/ruby_llm/tribunal/reporters/junit.rb,
lib/ruby_llm/tribunal/assertions/judge.rb,
lib/ruby_llm/tribunal/judges/jailbreak.rb,
lib/ruby_llm/tribunal/reporters/github.rb,
lib/ruby_llm/tribunal/reporters/console.rb,
lib/ruby_llm/tribunal/judges/correctness.rb,
lib/ruby_llm/tribunal/assertions/embedding.rb,
lib/ruby_llm/tribunal/judges/hallucination.rb,
lib/ruby_llm/tribunal/assertions/deterministic.rb

Overview

LLM evaluation framework for Ruby.

Tribunal provides tools for evaluating LLM outputs, detecting hallucinations, and measuring response quality.

Examples:

Quick Start

test_case = RubyLLM::Tribunal.test_case(
  input: "What's the return policy?",
  actual_output: "Returns within 30 days.",
  context: ["Return policy: 30 days with receipt."]
)

assertions = [
  [:contains, { value: "30 days" }],
  [:faithful, { threshold: 0.8 }]
]

results = RubyLLM::Tribunal.evaluate(test_case, assertions)

Defined Under Namespace

Modules: Assertions, Dataset, EvalHelpers, Judge, Judges, RedTeam, Reporter, Reporters Classes: Configuration, Error, TestCase

Constant Summary collapse

VERSION = 
'0.1.1'

Class Method Summary collapse

Class Method Details

.available_assertions ⇒ Array<Symbol>

Returns available assertion types based on loaded dependencies.

Returns:

  • (Array<Symbol>) —

    List of available assertion types



96
97
98
 
# File 'lib/ruby_llm/tribunal.rb', line 96

def available_assertions
  Assertions.available
end

.configuration ⇒ Object

Configuration for Tribunal



61
62
63
 
# File 'lib/ruby_llm/tribunal.rb', line 61

def configuration
  @configuration ||= Configuration.new
end

.configure {|configuration| ... } ⇒ Object

Yields:



65
66
67
 
# File 'lib/ruby_llm/tribunal.rb', line 65

def configure
  yield(configuration)
end

.evaluate(test_case, assertions) ⇒ Hash

Evaluates a test case against assertions.

Examples:

test_case = TestCase.new(
  input: "What's the return policy?",
  actual_output: "Returns within 30 days.",
  context: ["Return policy: 30 days with receipt."]
)

assertions = [
  [:contains, { value: "30 days" }],
  [:faithful, { threshold: 0.8 }]
]

Tribunal.evaluate(test_case, assertions)
#=> { contains: [:pass, {...}], faithful: [:pass, {...}] }

Parameters:

  • test_case (TestCase) —

    The test case to evaluate

  • assertions (Array, Hash) —

    Assertions to run

Returns:

  • (Hash) —

    Results map of assertion_type => result



89
90
91
 
# File 'lib/ruby_llm/tribunal.rb', line 89

def evaluate(test_case, assertions)
  Assertions.evaluate_all(assertions, test_case)
end

.judge_names ⇒ Array<Symbol>

Returns all registered judge names.

Returns:

  • (Array<Symbol>) —

    List of judge names



125
126
127
 
# File 'lib/ruby_llm/tribunal.rb', line 125

def judge_names
  Judge.all_judge_names
end

.register_judge(judge_class) ⇒ Object

Registers a custom judge.

Parameters:

  • judge_class (Class) —

    A class implementing the Judge interface



118
119
120
 
# File 'lib/ruby_llm/tribunal.rb', line 118

def register_judge(judge_class)
  Judge.register(judge_class)
end

.test_case(attrs) ⇒ TestCase

Creates a new test case.

Examples:

Tribunal.test_case(
  input: "What's the price?",
  actual_output: "The price is $29.99.",
  context: ["Product costs $29.99"]
)

Parameters:

  • attrs (Hash) —

    Test case attributes

Returns:



111
112
113
 
# File 'lib/ruby_llm/tribunal.rb', line 111

def test_case(attrs)
  TestCase.new(attrs)
end