Module: RubyLLM::Tribunal::Assertions::Judge

Defined in:
lib/ruby_llm/tribunal/assertions/judge.rb

Overview

LLM-as-judge assertions for evaluating LLM outputs.

Uses RubyLLM to get structured verdicts from the judge model.

Constant Summary collapse

DEFAULT_MODEL = 
'anthropic:claude-3-5-haiku-latest'
DEFAULT_THRESHOLD = 
0.8
SYSTEM_PROMPT = 
<<~PROMPT
  You are a precise evaluator of LLM outputs. Your task is to assess outputs
  based on specific criteria and provide structured verdicts.

  Always respond with valid JSON containing:
  - verdict: "yes", "no", or "partial"
  - reason: A brief explanation
  - score: A float from 0.0 to 1.0

  Be objective and consistent in your evaluations.
PROMPT

Class Method Summary collapse

Class Method Details

.available ⇒ Array<Symbol>

Returns list of available judge assertion types.

Returns:

  • (Array<Symbol>) 


29
30
31
 
# File 'lib/ruby_llm/tribunal/assertions/judge.rb', line 29

def available
  Tribunal::Judge.all_judge_names
end

.evaluate(type, test_case, opts) ⇒ Array

Evaluates a judge assertion against a test case.

Parameters:

  • type (Symbol) —

    The judge type

  • test_case (TestCase) —

    The test case

  • opts (Hash) —

    Options

Returns:

  • (Array) —

    [:pass, details], [:fail, details], or [:error, message]



39
40
41
42
43
44
45
46
47
48
 
# File 'lib/ruby_llm/tribunal/assertions/judge.rb', line 39

def evaluate(type, test_case, opts)
  judge_class = Tribunal::Judge.find(type)
  return [:error, "Unknown judge assertion: #{type}"] unless judge_class

  # Validate test case
  error = judge_class.validate(test_case) if judge_class.respond_to?(:validate)
  return [:error, error] if error

  run_judge(judge_class, test_case, opts)
end