Class: RubyLLM::Tribunal::Judges::Faithful

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby_llm/tribunal/judges/faithful.rb

Overview

Evaluates whether LLM output is grounded in provided context.

Faithfulness means the output only contains information that can be derived from the context. Use for RAG systems, documentation assistants, etc.

Uses claim extraction approach: breaks output into claims, verifies each against context, scores based on proportion of supported claims.

Class Method Summary collapse

Class Method Details

.judge_name ⇒ Object 



15
16
17
 
# File 'lib/ruby_llm/tribunal/judges/faithful.rb', line 15

def judge_name
  :faithful
end

.prompt(test_case, _opts) ⇒ Object 



25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/ruby_llm/tribunal/judges/faithful.rb', line 25

def prompt(test_case, _opts)
  context = format_context(test_case.context)

  "    You are evaluating whether an LLM output is faithful to the provided context.\n    Faithfulness means every claim in the output can be derived from the context.\n\n    ## Context\n    \#{context}\n\n    ## Question\n    \#{test_case.input}\n\n    ## Output to Evaluate\n    \#{test_case.actual_output}\n\n    ## Evaluation Process\n    1. Extract each distinct claim or statement from the output\n    2. For each claim, determine if it can be inferred from the context\n    3. Calculate the proportion of supported claims\n\n    ## Criteria\n    - A claim is SUPPORTED if it can be logically inferred from the context\n    - A claim is UNSUPPORTED if it adds information not in the context\n    - A claim is CONTRADICTED if it conflicts with the context\n    - General knowledge (e.g., \"the sky is blue\") doesn't count against faithfulness\n    - Paraphrasing context is acceptable if meaning is preserved\n\n    ## Response Format\n    Respond with JSON:\n    - verdict: \"yes\" if all substantive claims are supported, \"no\" if any claim contradicts\n      or significantly adds to the context, \"partial\" if most but not all claims are supported\n    - reason: List which claims are supported vs unsupported/contradicted\n    - score: (supported claims) / (total claims), ranging 0.0 to 1.0\n  PROMPT\nend\n"

.validate(test_case) ⇒ Object 



19
20
21
22
23
 
# File 'lib/ruby_llm/tribunal/judges/faithful.rb', line 19

def validate(test_case)
  return unless test_case.context.nil? || test_case.context.empty?

  'Faithful assertion requires context to be provided'
end