23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
|
# File 'lib/ruby_llm/tribunal/judges/correctness.rb', line 23
def prompt(test_case, _opts)
" You are evaluating whether an LLM output is correct compared to an expected answer.\n Correctness means the output conveys the same factual content and meaning.\n\n ## Question\n \#{test_case.input}\n\n ## Expected Answer (Ground Truth)\n \#{test_case.expected_output}\n\n ## Output to Evaluate\n \#{test_case.actual_output}\n\n ## Evaluation Criteria\n\n ### Factual Correctness\n - Does the output contain the same facts as the expected answer?\n - Are numerical values, dates, names, and specifics accurate?\n - Does it avoid stating anything that contradicts the expected answer?\n\n ### Semantic Equivalence\n - Does the output convey the same meaning, even if worded differently?\n - Paraphrasing is acceptable if the meaning is preserved\n - Additional true information doesn't reduce correctness\n - Missing important information from the expected answer reduces correctness\n\n ### Scoring Guide\n - 1.0: Output is factually equivalent to expected answer\n - 0.7-0.9: Mostly correct with minor omissions or additions\n - 0.4-0.6: Partially correct, missing key information or has some errors\n - 0.1-0.3: Mostly incorrect but has some accurate elements\n - 0.0: Completely wrong or contradicts the expected answer\n\n ## Response Format\n Respond with JSON:\n - verdict: \"yes\" if correct, \"no\" if incorrect, \"partial\" if partially correct\n - reason: Explain what matches and what differs from the expected answer\n - score: 0.0 to 1.0 based on the scoring guide above\n PROMPT\nend\n"
|