Module: RubyLLM::Tribunal::Dataset

Defined in:
lib/ruby_llm/tribunal/dataset.rb

Overview

Loads evaluation datasets from JSON or YAML files.

Examples:

Dataset Format (JSON)

[
  {
    "input": "What's the return policy?",
    "context": "Returns accepted within 30 days.",
    "expected": {
      "contains": ["30 days"],
      "faithful": {"threshold": 0.8}
    }
  }
]

Dataset Format (YAML)

- input: What's the return policy?
  context: Returns accepted within 30 days.
  expected:
    contains:
      - 30 days
    faithful:
      threshold: 0.8

Class Method Summary collapse

Class Method Details

.load(path) ⇒ Array<TestCase>

Loads a dataset from a file path.

Parameters:

  • path (String) —

    Path to the dataset file

Returns:

  • (Array<TestCase>) —

    Array of test cases

Raises:

  • (Error) —

    If file cannot be loaded or parsed



34
35
36
37
38
 
# File 'lib/ruby_llm/tribunal/dataset.rb', line 34

def load(path)
  content = File.read(path)
  data = parse(path, content)
  data.map { |item| to_test_case(item) }
end

.load_with_assertions(path) ⇒ Array<Array(TestCase, Array)>

Loads a dataset and extracts assertions per test case.

Parameters:

  • path (String) —

    Path to the dataset file

Returns:

  • (Array<Array(TestCase, Array)>) —

    Array of [test_case, assertions] pairs



44
45
46
47
48
49
50
51
52
53
 
# File 'lib/ruby_llm/tribunal/dataset.rb', line 44

def load_with_assertions(path)
  content = File.read(path)
  data = parse(path, content)

  data.map do |item|
    test_case = to_test_case(item)
    assertions = extract_assertions(item)
    [test_case, assertions]
  end
end