Module: RubyLLM::Tribunal::Judge

Defined in:
lib/ruby_llm/tribunal/judge.rb

Overview

Base module for LLM-as-judge assertions.

All judges (built-in and custom) implement this interface. This provides a consistent interface for evaluation criteria.

Examples:

Creating a custom judge

class BrandVoiceJudge
  include RubyLLM::Tribunal::Judge

  def self.judge_name
    :brand_voice
  end

  def self.prompt(test_case, opts)
    <<~PROMPT
      Evaluate if the response matches our brand voice guidelines:

      - Friendly but professional tone
      - No jargon or technical terms
      - Empathetic and helpful

      Response to evaluate:
      #{test_case.actual_output}

      Query: #{test_case.input}
    PROMPT
  end
end

RubyLLM::Tribunal.register_judge(BrandVoiceJudge)

Constant Summary collapse

BUILTIN_JUDGES =

Built-in judge classes

[].freeze

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.custom_judges ⇒ Object (readonly)

Returns the value of attribute custom_judges.



42
43
44
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 42

def custom_judges
  @custom_judges
end

Class Method Details

.all_judge_names ⇒ Array<Symbol>

Returns list of all judge names (built-in + custom).

Returns:

  • (Array<Symbol>) —

    Judge names



87
88
89
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 87

def all_judge_names
  all_judges.map(&:judge_name)
end

.all_judges ⇒ Array<Class>

Returns all judge modules (built-in + custom).

Returns:

  • (Array<Class>) —

    All judge classes



72
73
74
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 72

def all_judges
  builtin_judges + @custom_judges
end

.builtin_judge?(name) ⇒ Boolean

Checks if a name is a built-in judge.

Parameters:

  • name (Symbol) —

    The judge name

Returns:

  • (Boolean) 


117
118
119
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 117

def builtin_judge?(name)
  builtin_judge_names.include?(name)
end

.builtin_judge_names ⇒ Array<Symbol>

Returns list of built-in judge names.

Returns:

  • (Array<Symbol>) —

    Built-in judge names



94
95
96
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 94

def builtin_judge_names
  builtin_judges.map(&:judge_name)
end

.builtin_judges ⇒ Array<Class>

Returns all built-in judge modules.

Returns:

  • (Array<Class>) —

    Built-in judge classes



54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 54

def builtin_judges
  [
    Judges::Faithful,
    Judges::Relevant,
    Judges::Hallucination,
    Judges::Correctness,
    Judges::Bias,
    Judges::Toxicity,
    Judges::Harmful,
    Judges::Jailbreak,
    Judges::PII,
    Judges::Refusal
  ]
end

.custom_judge?(name) ⇒ Boolean

Checks if a name is a registered custom judge.

Parameters:

  • name (Symbol) —

    The judge name

Returns:

  • (Boolean) 


109
110
111
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 109

def custom_judge?(name)
  custom_judge_names.include?(name)
end

.custom_judge_names ⇒ Array<Symbol>

Returns list of custom judge names.

Returns:

  • (Array<Symbol>) —

    Custom judge names



101
102
103
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 101

def custom_judge_names
  @custom_judges.map(&:judge_name)
end

.evaluate_result(_result, _opts) ⇒ Array

Optional: customize how the LLM result is interpreted.

Parameters:

  • _result (Hash) —

    The LLM response

  • _opts (Hash) —

    Options

Returns:

  • (Array) —

    [:pass, details] or [:fail, details]



161
162
163
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 161

def self.evaluate_result(_result, _opts)
  nil # Return nil to use default interpretation
end

.find(name) ⇒ Class?

Finds a judge module by name.

Parameters:

  • name (Symbol) —

    The judge name

Returns:

  • (Class, nil) —

    The judge class or nil



80
81
82
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 80

def find(name)
  all_judges.find { |judge| judge.judge_name == name }
end

.judge_name ⇒ Symbol

Returns the atom name for this judge. This name is used to invoke the judge in assertions.

Returns:

  • (Symbol) —

    The judge name

Raises:

  • (NotImplementedError) 


128
129
130
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 128

def self.judge_name
  raise NotImplementedError, 'Judge classes must implement .judge_name'
end

.negative_metric? ⇒ Boolean

Optional: whether "no" verdict means pass (for negative metrics like toxicity).

Returns:

  • (Boolean) 


152
153
154
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 152

def self.negative_metric?
  false
end

.prompt(test_case, opts) ⇒ String

Builds the evaluation prompt for the LLM judge.

Parameters:

  • test_case (TestCase) —

    The test case

  • opts (Hash) —

    Options

Returns:

  • (String) —

    The prompt

Raises:

  • (NotImplementedError) 


137
138
139
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 137

def self.prompt(test_case, opts)
  raise NotImplementedError, 'Judge classes must implement .prompt(test_case, opts)'
end

.register(judge_class) ⇒ Object

Registers a custom judge class.

Parameters:

  • judge_class (Class) —

    A class implementing the Judge interface



47
48
49
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 47

def register(judge_class)
  @custom_judges << judge_class unless @custom_judges.include?(judge_class)
end

.validate(_test_case) ⇒ nil, String

Optional: validate that the test case has required fields.

Parameters:

  • _test_case (TestCase) —

    The test case

Returns:

  • (nil, String) —

    nil if valid, error message if not



145
146
147
 
# File 'lib/ruby_llm/tribunal/judge.rb', line 145

def self.validate(_test_case)
  nil
end