Class: Asimov::Utils::TextEntryFileValidator

Inherits:
JsonlValidator show all
Defined in:
lib/asimov/utils/text_entry_file_validator.rb

Overview

Validates that a file is in the “answers” or “search” format used by OpenAI. The file is a JSONL file, with a “text” key for each line that has a string value and an optional “metadata” key that can have any value. No other keys are permitted.

Instance Method Summary collapse

Methods inherited from JsonlValidator

#validate

Instance Method Details

#text_entry?(parsed) ⇒ Boolean

Returns:

  • (Boolean) 


26
27
28
29
30
31
32
33
34
35
 
# File 'lib/asimov/utils/text_entry_file_validator.rb', line 26

def text_entry?(parsed)
  return false unless parsed.is_a?(Hash)

  keys = parsed.keys
  return false unless keys.size >= 1 && keys.size <= 2
  return false unless keys.include?("text")
  return false unless parsed["text"].is_a?(String)

  keys.size == 1 ? true : keys.include?("metadata")
end

#validate_line(line, idx) ⇒ Object 



13
14
15
16
 
# File 'lib/asimov/utils/text_entry_file_validator.rb', line 13

def validate_line(line, idx)
  parsed = JSON.parse(line)
  validate_text_entry(parsed, idx)
end

#validate_text_entry(parsed, idx) ⇒ Object

Raises:



18
19
20
21
22
23
24
 
# File 'lib/asimov/utils/text_entry_file_validator.rb', line 18

def validate_text_entry(parsed, idx)
  return if text_entry?(parsed)

  raise InvalidTextEntryError,
        "Expected file to have the JSONL format with 'text' key and (optional) " \
        "'metadata' key. Invalid format on line #{idx + 1}."
end