Class: CAFrame::CSVParser::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/carray/frame/csv_parser.rb

Overview

Splits records into fields. Regexps are compiled once and reused across every record, so per-row cost stays low.

Instance Method Summary collapse

Constructor Details

#initialize(sep, quote, strip) ⇒ Tokenizer

Returns a new instance of Tokenizer.



83
84
85
86
87
88
89
90
91
92
 
# File 'lib/carray/frame/csv_parser.rb', line 83

def initialize(sep, quote, strip)
  @sep      = sep
  @quote    = quote
  @strip    = strip
  @sep_re   = /#{Regexp.escape(sep)}/
  @quote_re = /#{Regexp.escape(quote)}/
  @unquoted = /[^#{Regexp.escape(sep)}]*/
  @inner    = /[^#{Regexp.escape(quote)}]*/
  @recno    = 0
end

Instance Method Details

#read(io, blank_is_row: false) ⇒ Object

Fields of the next record, or nil at EOF. A blank line carries no separator, so it cannot be a row of a file with more than one column and is skipped as noise between records. In a single-column file it is the only spelling a missing single field has -- which is what to_csv writes for a masked cell -- so the caller passes blank_is_row: true once the column count is known to be one, and the empty record becomes a row of no fields for build_frame to pad.



101
102
103
104
105
106
107
108
109
 
# File 'lib/carray/frame/csv_parser.rb', line 101

def read(io, blank_is_row: false)
  loop do
    rec = CSVParser.read_record(io, @quote)
    return nil if rec.nil?
    @recno += 1
    next if rec.empty? && !blank_is_row
    return rec.count(@quote).zero? ? simple(rec) : scan(rec)
  end
end