Class: RubyLLM::Tribunal::Judges::Relevant

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby_llm/tribunal/judges/relevant.rb

Overview

Evaluates whether LLM output is relevant to the input query.

Relevance means the output directly addresses and answers the question. Uses statement extraction: breaks output into statements, classifies each as relevant or irrelevant, scores based on proportion of relevant content.

Class Method Summary collapse

Class Method Details

.judge_name ⇒ Object 



13
14
15
 
# File 'lib/ruby_llm/tribunal/judges/relevant.rb', line 13

def judge_name
  :relevant
end

.prompt(test_case, _opts) ⇒ Object 



17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/ruby_llm/tribunal/judges/relevant.rb', line 17

def prompt(test_case, _opts)
  "    You are evaluating whether an LLM output is relevant to the question asked.\n    Relevance means the output directly addresses what the user is asking.\n\n    ## Question/Input\n    \#{test_case.input}\n\n    ## Output to Evaluate\n    \#{test_case.actual_output}\n\n    ## Evaluation Process\n    1. Identify the user's core question or intent\n    2. Extract each statement from the output\n    3. Classify each statement as relevant or irrelevant to the question\n    4. Calculate the proportion of relevant statements\n\n    ## Relevance Criteria\n    A statement is RELEVANT if it:\n    - Directly answers the question\n    - Provides necessary context for the answer\n    - Clarifies or elaborates on the answer\n    - Addresses a reasonable interpretation of the question\n\n    A statement is IRRELEVANT if it:\n    - Goes off-topic entirely\n    - Addresses a different question\n    - Provides unnecessary information unrelated to the query\n    - Contains filler or padding that doesn't serve the answer\n\n    ## Edge Cases\n    - Partial answers: If the output addresses some but not all aspects of a multi-part\n      question, it's \"partial\"\n    - Refusals: A polite refusal that explains why is relevant if the question is inappropriate\n    - Tangential info: Brief relevant context is acceptable; extensive tangents are not\n\n    ## Response Format\n    Respond with JSON:\n    - verdict: \"yes\" if output addresses the question, \"no\" if off-topic,\n      \"partial\" if only partly addresses it\n    - reason: Explain what makes the output relevant or irrelevant\n    - score: (relevant statements) / (total statements), ranging 0.0 to 1.0\n  PROMPT\nend\n"