Class: EvalRuby::Dataset

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/eval_ruby/dataset.rb

Overview

Collection of evaluation samples with import/export support. Supports CSV, JSON, and programmatic construction.

Examples:

dataset = EvalRuby::Dataset.new("my_test_set")
dataset.add(question: "What is Ruby?", answer: "A language", ground_truth: "A language")
report = EvalRuby.evaluate_batch(dataset)

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name = "default") ⇒ Dataset

Returns a new instance of Dataset.

Parameters:

  • name (String) (defaults to: "default") —

    dataset name



24
25
26
27
 
# File 'lib/eval_ruby/dataset.rb', line 24

def initialize(name = "default")
  @name = name
  @samples = []
end

Instance Attribute Details

#name ⇒ String (readonly)

Returns dataset name.

Returns:

  • (String) —

    dataset name



18
19
20
 
# File 'lib/eval_ruby/dataset.rb', line 18

def name
  @name
end

#samples ⇒ Array<Hash> (readonly)

Returns sample entries.

Returns:

  • (Array<Hash>) —

    sample entries



21
22
23
 
# File 'lib/eval_ruby/dataset.rb', line 21

def samples
  @samples
end

Class Method Details

.expand_document_paths(paths) ⇒ Array<String>

Expands a list of file/directory paths into a flat list of file paths. Validates existence — missing paths raise an Error.

Parameters:

  • paths (String, Array<String>)

Returns:

  • (Array<String>) —

    absolute-or-relative file paths, each verified to exist

Raises:



188
189
190
191
192
193
194
195
196
197
198
199
200
 
# File 'lib/eval_ruby/dataset.rb', line 188

def self.expand_document_paths(paths)
  result = []
  Array(paths).each do |path|
    if File.directory?(path)
      result.concat(Dir.glob(File.join(path, "**/*")).select { |p| File.file?(p) }.sort)
    elsif File.file?(path)
      result << path
    else
      raise Error, "Document path does not exist: #{path}"
    end
  end
  result
end

.from_csv(path) ⇒ Dataset

Loads a dataset from a CSV file.

Parameters:

  • path (String) —

    path to CSV file

Returns:



67
68
69
70
71
72
73
74
75
76
77
78
 
# File 'lib/eval_ruby/dataset.rb', line 67

def self.from_csv(path)
  dataset = new(File.basename(path, ".*"))
  CSV.foreach(path, headers: true) do |row|
    dataset.add(
      question: row["question"],
      answer: row["answer"],
      context: parse_array_field(row["context"]),
      ground_truth: row["ground_truth"]
    )
  end
  dataset
end

.from_json(path) ⇒ Dataset

Loads a dataset from a JSON file.

Parameters:

  • path (String) —

    path to JSON file

Returns:



84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/eval_ruby/dataset.rb', line 84

def self.from_json(path)
  dataset = new(File.basename(path, ".*"))
  data = JSON.parse(File.read(path))
  samples = data.is_a?(Array) ? data : data["samples"] || data["data"] || []
  samples.each do |sample|
    dataset.add(
      question: sample["question"],
      answer: sample["answer"],
      context: Array(sample["context"]),
      ground_truth: sample["ground_truth"]
    )
  end
  dataset
end

.generate(documents:, questions_per_doc: 5, llm: :openai, judge: nil) ⇒ Dataset

Generates a dataset from documents using an LLM.

Each document is read, passed to the LLM with a prompt asking for questions_per_doc QA pairs, and the resulting pairs are appended to the dataset. Directory paths are expanded via Dir.glob. Missing paths and malformed LLM responses raise or are skipped gracefully rather than crashing the whole generation.

Parameters:

  • documents (String, Array<String>) —

    file paths or directory paths

  • questions_per_doc (Integer) (defaults to: 5) —

    number of QA pairs per document (must be > 0)

  • llm (Symbol) (defaults to: :openai) —

    LLM provider (:openai or :anthropic)

  • judge (Judges::Base, nil) (defaults to: nil) —

    inject a pre-built judge (mainly for testing)

Returns:

Raises:



139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
 
# File 'lib/eval_ruby/dataset.rb', line 139

def self.generate(documents:, questions_per_doc: 5, llm: :openai, judge: nil)
  unless questions_per_doc.is_a?(Integer) && questions_per_doc.positive?
    raise Error, "questions_per_doc must be a positive integer, got #{questions_per_doc.inspect}"
  end

  document_paths = expand_document_paths(documents)
  raise Error, "No documents found in the provided paths" if document_paths.empty?

  judge ||= build_judge_for(llm)

  dataset = new("generated")
  document_paths.each do |doc_path|
    content = File.read(doc_path)
    prompt = <<~PROMPT
      Given the following document, generate #{questions_per_doc} question-answer pairs
      that can be answered using the document content.

      Document:
      #{content}

      Respond in JSON: {"pairs": [{"question": "...", "answer": "...", "context": "relevant excerpt"}]}
    PROMPT

    begin
      result = judge.call(prompt)
    rescue StandardError
      next # keep generating from remaining docs; individual failure should not abort the batch
    end

    extract_pairs(result).each do |pair|
      next unless valid_pair?(pair)

      dataset.add(
        question: pair["question"],
        answer: pair["answer"],
        context: [pair["context"].is_a?(String) && !pair["context"].empty? ? pair["context"] : content],
        ground_truth: pair["answer"]
      )
    end
  end
  dataset
end

Instance Method Details

#[](index) ⇒ Hash

Returns sample at index.

Parameters:

  • index (Integer)

Returns:

  • (Hash) —

    sample at index



59
60
61
 
# File 'lib/eval_ruby/dataset.rb', line 59

def [](index)
  @samples[index]
end

#add(question:, ground_truth: nil, relevant_contexts: [], answer: nil, context: []) ⇒ self

Adds a sample to the dataset.

Parameters:

  • question (String)
  • ground_truth (String, nil) (defaults to: nil)
  • relevant_contexts (Array<String>) (defaults to: []) —

    alias for context

  • answer (String, nil) (defaults to: nil)
  • context (Array<String>) (defaults to: [])

Returns:

  • (self) 


37
38
39
40
41
42
43
44
45
 
# File 'lib/eval_ruby/dataset.rb', line 37

def add(question:, ground_truth: nil, relevant_contexts: [], answer: nil, context: [])
  @samples << {
    question: question,
    answer: answer,
    context: context.empty? ? relevant_contexts : context,
    ground_truth: ground_truth
  }
  self
end

#each {|Hash| ... } ⇒ Object

Yields:

  • (Hash) —

    each sample



48
49
50
 
# File 'lib/eval_ruby/dataset.rb', line 48

def each(&block)
  @samples.each(&block)
end

#size ⇒ Integer

Returns number of samples.

Returns:

  • (Integer) —

    number of samples



53
54
55
 
# File 'lib/eval_ruby/dataset.rb', line 53

def size
  @samples.size
end

#to_csv(path) ⇒ void

This method returns an undefined value.

Exports dataset to CSV.

Parameters:

  • path (String) —

    output file path



103
104
105
106
107
108
109
110
111
112
113
114
115
 
# File 'lib/eval_ruby/dataset.rb', line 103

def to_csv(path)
  CSV.open(path, "w") do |csv|
    csv << %w[question answer context ground_truth]
    @samples.each do |sample|
      csv << [
        sample[:question],
        sample[:answer],
        JSON.generate(sample[:context]),
        sample[:ground_truth]
      ]
    end
  end
end

#to_json(path) ⇒ void

This method returns an undefined value.

Exports dataset to JSON.

Parameters:

  • path (String) —

    output file path



121
122
123
 
# File 'lib/eval_ruby/dataset.rb', line 121

def to_json(path)
  File.write(path, JSON.pretty_generate({name: @name, samples: @samples}))
end