Class: CAFrame

Inherits:
Object
  • Object
show all
Defined in:
lib/carray/frame/join.rb,
lib/carray/frame/io.rb,
lib/carray/frame/sort.rb,
lib/carray/frame/frame.rb,
lib/carray/frame/group.rb,
lib/carray/frame/verbs.rb,
lib/carray/frame/concat.rb,
lib/carray/frame/convert.rb,
lib/carray/frame/records.rb,
lib/carray/frame/csv_parser.rb

Overview

CAFrame record (row-oriented) input (memo §11.2, §11.5).

from_records takes an Array of row Hashes -- the shape JSON.parse yields for a JSON array of objects -- and arranges it into columns. Unlike from_csv, the cell values are already typed Ruby objects (Float / Integer / DateTime / String), so a homogeneous column is built at its native leaf type rather than left as strings ("arrange by the value's own type", not string inference -- §4.2 stays intact: no date-like string is parsed, mixed columns stay object).

An Array-valued cell that is the same length across every record becomes an N-D column (§11.5): { "temp" => [min, mean, max] } over N records is one (N, 3) column. Ragged or non-numeric arrays fall back to an object column.

Missing keys and explicit nils become UNDEF for numeric columns (mask, not a float promotion): an int column with a hole stays int + UNDEF.

Defined Under Namespace

Modules: CSVParser Classes: CSVReader

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(columns = {}, **opts) ⇒ CAFrame

Build a frame from a Hash of name => column. Columns may be CArrays or anything that answers to_ca (Ruby Array, lazy view). All columns must share axis-0 length N.

The column hash may be passed either explicitly (+CAFrame.new(hash, axis_name: ...)+) or as bare inline pairs (+CAFrame.new("a" => x, "b" => y)+). :axis_name / :index are control options; any remaining (string-keyed) options are treated as columns.

Column names are normalized to Strings, so a Symbol key in an explicit column hash is stringified. The Symbol rejection below applies only to the keyword channel, which doubles as the :axis_name / :index control-option channel: a stray Symbol there is a mistyped control option, not a column.



35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/carray/frame/frame.rb', line 35

def initialize(columns = {}, **opts)
  axis_name = opts.delete(:axis_name)
  index     = opts.delete(:index)
  stray = opts.keys.reject { |k| k.is_a?(String) }
  unless stray.empty?
    raise ArgumentError,
          "column keys must be Strings (Symbols are reserved); got #{stray.inspect}"
  end
  columns = columns.merge(opts) unless opts.empty?

  @columns   = {}
  @axis_name = axis_name || DEFAULT_AXIS_NAME
  @index     = nil

  n = nil
  columns.each do |name, col|
    key = name.to_s
    ca  = coerce_column(col)
    len = ca.shape[0]
    if n.nil?
      n = len
    elsif len != n
      raise ArgumentError,
            "column #{key.inspect} has axis-0 length #{len}, expected #{n}"
    end
    @columns[key] = ca
  end
  @nrow = n || 0

  if index
    idx = coerce_column(index)
    unless idx.ndim == 1
      raise ArgumentError, "index must be a 1-D column (got ndim #{idx.ndim})"
    end
    if n && idx.shape[0] != n
      raise ArgumentError,
            "index length #{idx.shape[0]} does not match nrow #{n}"
    end
    @index = idx
    @nrow  = idx.shape[0] if n.nil?
  end

  # With an index, the row axis is named after it; a column of the same name
  # would shadow the index in +row+ and +reset_index+. Reject the collision at
  # construction so those paths never silently drop one for the other.
  if @index && @columns.key?(@axis_name)
    raise ArgumentError,
          "axis_name #{@axis_name.inspect} collides with a column of the same name"
  end
end

Instance Attribute Details

#axis_name ⇒ Object (readonly)

Row-axis name.



427
428
429
 
# File 'lib/carray/frame/frame.rb', line 427

def axis_name
  @axis_name
end

#index ⇒ Object (readonly)

Index column (CArray) or nil.



430
431
432
 
# File 'lib/carray/frame/frame.rb', line 430

def index
  @index
end

#nrow ⇒ Object (readonly)

Number of rows (axis-0 length N).



424
425
426
 
# File 'lib/carray/frame/frame.rb', line 424

def nrow
  @nrow
end

Class Method Details

.concatenate(*frames) ⇒ Object

Concatenate frames along the row axis, eagerly. Each output column is CArray.concatenate of that column across the input frames, so per-column data types auto-promote to a common type. The result is a fresh, independent frame — writes to it do not propagate back to the input frames.

For a view frame that shares storage with the inputs (strict same data type per column, chain composability preserved) use meld.

Column matching, index handling, and column-set / index-mix rules match meld.

CAFrame.concatenate(jan, feb, mar)     # eager, independent result
CAFrame.concatenate([jan, feb, mar])   # an Array is accepted too


62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/carray/frame/concat.rb', line 62

def self.concatenate(*frames)
  frames = frames.flatten
  check_concat_inputs(frames, verb: "concatenate")
  first = frames.first
  names = first.variable_names
  check_column_sets(frames, names, verb: "concatenate")
  cols = {}
  names.each do |name|
    cols[name] = CArray.concatenate(frames.map { |f| f[name] })
  end
  new(cols, axis_name: first.axis_name, index: concatenate_index(frames))
end

.from_csv(path, types: nil, sep: ",", quote: '"', strip: false, encoding: "bom|utf-8", parser: nil, &block) ⇒ Object

Read a CSV into a frame. The header row supplies column names (Strings, §3.7); every column is built raw as an object CArray of the cell strings (§4.2 -- read and arrange only, no type-inference engine). Casting is a separate step: pass types: ({ "temp" => :float64 }, or the array-key / reverse forms of cast) to cast named columns on load, or call cast later. Broken cells fail to_type and become UNDEF automatically (parse-mask, §6-2).

Parsing uses the built-in fast tokenizer (CSVParser). Options:

sep:      field separator (default ",")
quote:    quote character (default '"')
strip:    trim spaces from unquoted fields (default false, RFC spacing)
+encoding+: IO open-mode encoding (default "bom|utf-8", strips a BOM)
parser:   a callable path -> [headers, rows] to inject another parser
        (e.g. the stdlib +csv+, or a typed-table source); when given,
        sep/quote/strip/encoding and any block are that parser's concern.

A block gives reading control for files with preamble lines, a units row, or no header (memo §11.2), using skip / header / column_names / body (see CSVReader). Without a block the default is header then body.

CAFrame.from_csv("obs.csv") do
skip 2; header; skip 1; body
end

Columns are handed to the frame as CABlock views over one backing object array (§3.6 view-by-default); casting a column materializes it, and copy gives an independent frame.



34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/carray/frame/io.rb', line 34

def self.from_csv(path, types: nil,
                  sep: ",", quote: '"', strip: false,
                  encoding: "bom|utf-8", parser: nil, &block)
  names, rows =
    if parser
      parser.call(path)
    else
      File.open(path, "r:#{encoding}") do |io|
        reader = CSVReader.new(io, sep: sep, quote: quote, strip: strip)
        if block
          block.arity == 1 ? block.call(reader) : reader.instance_exec(&block)
        else
          reader.header
          reader.body
        end
        reader.result
      end
    end

  frame = build_frame(names, rows)
  frame.cast(types) if types
  frame
end

.from_records(records, types: nil) ⇒ Object

Build a frame from an Array of row Hashes. Column set is the union of keys in first-appearance order; keys are stringified. types: casts named columns afterward (same map / array-key forms as cast).



21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/carray/frame/records.rb', line 21

def self.from_records(records, types: nil)
  unless records.is_a?(Array) && records.all? { |r| r.is_a?(Hash) }
    raise ArgumentError, "from_records expects an Array of Hashes"
  end
  return new if records.empty?

  keys = record_key_union(records)
  n = records.size
  cols = {}
  keys.each do |key|
    values = records.map { |r| r[key] }
    cols[key.to_s] = build_record_column(values, n)
  end

  frame = new(cols)
  frame.cast(types) if types
  frame
end

.meld(*frames) ⇒ Object

Weld frames along the row axis, view-style. Each output column is CArray.meld of that column across the input frames, so the result is a view frame that shares storage with the inputs (chain composability preserved: writes to the result flow back to whichever input frame owns the target segment, and vice versa).

Per-column data_type must match across frames (CArray.meld is a view constructor that refuses to auto-cast — silent promotion here would hide schema drift). For an eager, auto-casting alternative use concatenate.

Column matching is by name (output order follows the first frame); every frame must carry the same column-name set. N-D columns carry their trailing dimensions along; masks are preserved.

The index is welded too when every frame has one (their axis_name must agree); if none do, the result has no index; a mix raises. Column-set mismatch raises — a union-with-UNDEF mode is a possible future opt-in, kept out here to stay explicit (memo §4.2).

CAFrame.meld(jan, feb, mar)      # view over three months
CAFrame.meld([jan, feb, mar])    # an Array is accepted too


36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/carray/frame/concat.rb', line 36

def self.meld(*frames)
  frames = frames.flatten
  check_concat_inputs(frames, verb: "meld")
  first = frames.first
  names = first.variable_names
  check_column_sets(frames, names, verb: "meld")
  cols = {}
  names.each do |name|
    cols[name] = CArray.meld(frames.map { |f| f[name] }, axis: 0)
  end
  new(cols, axis_name: first.axis_name, index: meld_index(frames))
end

.paste(*frames, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

Paste N frames side by side by row position (symmetric N-ary sibling of the instance #paste; see the instance verb's docstring for the pair semantics). All frames must have the same nrow. The result's axis_name and index come from the first frame (peers, but a base is needed for the index). Column names that appear in more than one frame are colliding — pass suffixes: as an Array of exactly K strings (one per input frame) to disambiguate them; otherwise the collision raises.

CAFrame.paste(obs, fcst)
CAFrame.paste(obs, ecmwf, gfs, suffixes: ["_obs", "_ecmwf", "_gfs"])
CAFrame.paste([obs, fcst])                              # Array is accepted too

Raises:

  • (ArgumentError) 


121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
 
# File 'lib/carray/frame/join.rb', line 121

def self.paste(*frames, suffixes: DEFAULT_JOIN_SUFFIXES)
  frames = frames.flatten
  raise ArgumentError, "paste requires at least one frame" if frames.empty?
  unless frames.all? { |f| f.is_a?(CAFrame) }
    raise ArgumentError, "paste expects CAFrame arguments"
  end
  first = frames.first
  nrow  = first.nrow
  frames.each_with_index do |f, i|
    next if i.zero?
    unless f.nrow == nrow
      raise ArgumentError,
            "paste: row-count mismatch (frame #{i}: #{f.nrow} vs #{nrow}); " \
            "use join for key alignment"
    end
  end

  # Detect columns that appear in 2+ frames (collisions across the K inputs).
  name_count = Hash.new(0)
  frames.each { |f| f.variable_names.each { |name| name_count[name] += 1 } }
  collisions = name_count.select { |_, n| n > 1 }.keys

  if collisions.any?
    if suffixes == false
      raise ArgumentError,
            "paste column name collision: #{collisions.inspect} " \
            "(pass suffixes: [\"_a\", \"_b\", ...] with one entry per frame, " \
            "or rename first)"
    end
    # For the 2-frame case DEFAULT_JOIN_SUFFIXES ("_left", "_right") works
    # as-is; for K > 2 the caller must pass a K-length Array (or use the
    # pair-only default indirectly via the instance method).
    unless suffixes.is_a?(Array) && suffixes.size == frames.size &&
           suffixes.all? { |s| s.is_a?(String) }
      raise ArgumentError,
            "paste: suffixes must be an Array of #{frames.size} Strings " \
            "(one per frame), got #{suffixes.inspect}"
    end
  end

  cols = {}
  frames.each_with_index do |f, i|
    f.variable_names.each do |name|
      key = collisions.include?(name) ? "#{name}#{suffixes[i]}" : name
      if cols.key?(key)
        raise ArgumentError,
              "paste: column name #{key.inspect} produced twice " \
              "(collision after suffix from frame #{i})"
      end
      cols[key] = f[name]
    end
  end
  new(cols, axis_name: first.axis_name, index: first.index)
end

Instance Method Details

#[](*keys) ⇒ Object

String -> a column (raw CArray, the escape unit) String x2+ -> Array (the escape unit, plural) Integer -> a row (Ruby Hash) Range(int)-> positional row slice (view-frame) boolean CA-> row filter (view-frame) integer CA-> row gather (view-frame)

String keys escape: df hands back the raw column(s), never a frame. One name collapses to a bare CArray; several give an Array of them (the "single collapses, plural is an array" rule of ca vs ca), so t, rh = df["temp", "rh"] destructures. A column-subset frame comes from select (memo §13.2).



100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/carray/frame/frame.rb', line 100

def [](*keys)
  if keys.size > 1
    unless keys.all? { |k| k.is_a?(String) }
      raise ArgumentError,
            "multi-key df[...] escapes columns; every key must be a String " \
            "(use df.select(...) for a subset frame)"
    end
    return keys.map { |k| @columns.fetch(k) { raise KeyError, "no column #{k.inspect}" } }
  end

  key = keys.first
  case key
  when String
    @columns.fetch(key) { raise KeyError, "no column #{key.inspect}" }
  when Integer
    row(key)
  when Range
    unless positional_range?(key)
      raise ArgumentError,
            "df[range] takes positional (integer) ranges only; " \
            "use filter { |f| f.index ... } for label ranges"
    end
    select_rows(key)
  when CArray
    case key.data_type
    when :boolean
      select_rows(key)
    when *INTEGER_TYPES
      select_rows(key)
    else
      raise ArgumentError,
            "CArray df[] key must be boolean or integer (got #{key.data_type})"
    end
  when Symbol
    raise NotImplementedError, "df[symbol] is reserved for predicate keys"
  else
    raise ArgumentError, "unsupported df[] key: #{key.class}"
  end
end

#[]=(*keys, value) ⇒ Object

--- df = value : row-level assignment ---------------------------

The RHS value decides the operation; the key selects the rows:

df[sel] = UNDEF  -> mask the selected rows across every column, in place.
                  Shape is unchanged and the write goes through to the
                  column storage; the index stays, so the rows remain
                  identifiable.
df[sel] = nil    -> delete the selected rows. The frame shrinks and the
                  survivors keep their order.
df[sel] = other  -> splice: replace the selected (contiguous) rows with
                  +other+'s rows. +other+ may carry any number of rows,
                  so the row count changes (Ruby Array#[]= splice
                  semantics); its column set must match exactly.

sel is a row-axis indexer key, classified exactly as CArray classifies a 1-D column key (docs/topics/Indexer_decision_tree.md): a slice (BLOCK — Range / ArithmeticSequence / [start, count, step]), a boolean CArray (SELECT), an integer CArray (GRID), or an Integer (a single row). Mask and delete take any of them and are forwarded to the column indexer, so their errors match the rest of CArray. Splice reuses the same classifier (CArray.scan_index) but needs a contiguous slice, so it takes an Integer or a step-1 BLOCK (Range, step-1 ArithmeticSequence, or [start, count]); a strided or scattered selector raises.

Delete and splice rebuild the column set, so an alias previously taken from this frame no longer tracks the frame's new column identity (@columns is rebound to a fresh Hash). Reads through the old alias see the ORIGINAL column's data (unchanged by splice construction), matching the pre-splice snapshot. Writes flow along the CAMeld chain:

  • Writes reaching the HEAD or TAIL segment (the parts of the original column that survived the splice) land back in that column's storage. External aliases into the original column see those writes — this is the chain composability that CAMeld deliberately preserves: a reference stays connected to what it references.
  • Writes reaching the spliced MIDDLE segment (the rows from other) do NOT reach +other+'s storage: other is snapshotted via .copy at splice time so it behaves like a value that was handed over. df1[...] = df2 followed by writes to df1's spliced rows leaves df2 untouched, which matches the usual "assignment" intuition for the RHS of []=.

Callers who want write isolation on the head/tail segments too can materialise explicitly with col_view.copy before the splice, or df["c"] = df["c"].copy afterwards. The shape-preserving UNDEF write path always propagates to derived views (unchanged from previous behaviour).



188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
 
# File 'lib/carray/frame/frame.rb', line 188

def []=(*keys, value)
  if keys.size != 1
    raise ArgumentError,
          "df[...] = takes one key (a column name or a row selector); " \
          "got #{keys.size}"
  end
  selector = keys.first

  # The key picks the axis, exactly as it does when reading (§13.2): a
  # String names a column, anything else selects rows.
  if selector.is_a?(String)
    column_assign(selector, value)
  elsif UNDEF.equal?(value)
    mask_rows(selector)
  elsif value.nil?
    delete_rows(selector)
  elsif value.is_a?(CAFrame)
    splice_rows(selector, value)
  else
    raise ArgumentError,
          "df[...] = expects UNDEF (mask rows), nil (delete rows), or a " \
          "CAFrame (splice rows); got #{value.class}"
  end
  value
end

#align(key, reference) ⇒ Object

Conform every variable to an externally supplied reference key set (the asymmetric sibling of join; pandas reindex). Given a reference array of key values, each column is gathered onto it by exact key match (+locate_addr+): the aligned key column (or index) becomes the reference itself, and a reference key absent from the source comes back UNDEF in every other column.

CAFrame measures no interval and generates nothing -- the caller owns the reference. This is the primitive for reindexing to a caller-built axis (e.g. a complete 10-minute time grid): supply that axis as reference and the gaps fill with UNDEF rows carrying only the reference key.

df.align("time", reftime)   # reftime = a CArray of reference key values

key may be a column name or the index axis name. reference is a CArray (or Array); its keys are matched against the source key by value, so their data types must be comparable (a DateTime object key matches by eql?/hash).



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/carray/frame/join.rb', line 76

def align(key, reference)
  key = key.to_s
  ref = reference.is_a?(CArray) ? reference : reference.to_ca
  on_index = !@columns.key?(key) && @axis_name == key && @index
  addr = ref.locate_addr(on_index ? @index : self[key])

  cols = {}
  variable_names.each { |name| cols[name] = project_rows(self[name], addr) }
  if on_index
    CAFrame.new(cols, axis_name: key, index: ref)
  else
    cols[key] = ref
    CAFrame.new(cols, axis_name: @axis_name,
                      index: @index && project_rows(@index, addr))
  end
end

#append(name, col) ⇒ Object

Add (or replace) a column, returning a new frame — the column set changes, so the result is a different table (memo §3.8). The column is shared by reference; the envelope is cheap. Chain or reassign: df = df.append(...).



12
13
14
15
16
 
# File 'lib/carray/frame/verbs.rb', line 12

def append(name, col)
  key = name.to_s
  ca  = coerce_column(col)
  rebuild(@columns.merge(key => ca))
end

#at(label) ⇒ Object

The single row whose index label equals label, as a Ruby Hash (memo §13.2b). label is matched exactly against the index (+index.eq+), so any orderable / object / datetime / categorical index works. The return type is a row Hash and stays that way: zero matches raise KeyError, and duplicate labels raise (go through filter { |f| f.index.eq(label) } for the multi-row, frame-returning path). Positional access is df[i].

Raises:

  • (ArgumentError) 


371
372
373
374
375
376
377
378
379
380
381
382
383
384
 
# File 'lib/carray/frame/frame.rb', line 371

def at(label)
  raise ArgumentError, "at requires an index (set one with set_index)" unless @index
  pos = @index.eq(label).where
  case pos.elements
  when 0
    raise KeyError, "no row with index label #{label.inspect}"
  when 1
    row(pos[0])
  else
    raise ArgumentError,
          "index label #{label.inspect} matches #{pos.elements} rows; " \
          "use filter { |f| f.index.eq(label) } for duplicate labels"
  end
end

#cast(name_or_map, type = nil) ⇒ Object

Cast columns to a data type and rebind them (memo §11.4). Uses to_type, so parse failures on string columns become UNDEF (parse-mask, §6-2). Three call shapes, disambiguated by the fact that column names are always Strings and types always Symbols (§3.7):

cast("temp", :float64)                     # one column (chains)
cast("temp" => :float64, "rh" => :int32)   # name => type map
cast(["temp", "rh"] => :float64)           # names sharing one type

A map key may be a single name or an Array of names; the value is the target type. Returns self so calls chain.



69
70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/carray/frame/verbs.rb', line 69

def cast(name_or_map, type = nil)
  if name_or_map.is_a?(Hash)
    unless type.nil?
      raise ArgumentError, "cast(map) takes no positional type argument"
    end
    name_or_map.each do |names, t|
      Array(names).each { |name| cast_one(name, t) }
    end
  else
    cast_one(name_or_map, type)
  end
  self
end

#copy ⇒ Object

An independent frame — every column and the index materialized (§3.6).



387
388
389
390
391
 
# File 'lib/carray/frame/frame.rb', line 387

def copy
  cols = {}
  @columns.each { |k, v| cols[k] = v.copy }
  CAFrame.new(cols, axis_name: @axis_name, index: @index && @index.copy)
end

#data_types ⇒ Object

name => data_type, freshly derived.



433
434
435
436
437
 
# File 'lib/carray/frame/frame.rb', line 433

def data_types
  h = {}
  @columns.each { |k, v| h[k] = v.data_type }
  h
end

#drop(*names) ⇒ Object

Remove one or more columns, returning a new frame (column set changes, memo §3.8). Column data is untouched and shared with the original.



20
21
22
23
24
25
26
27
28
 
# File 'lib/carray/frame/verbs.rb', line 20

def drop(*names)
  cols = @columns.dup
  names.each do |name|
    key = name.to_s
    raise KeyError, "no column #{key.inspect}" unless cols.key?(key)
    cols.delete(key)
  end
  rebuild(cols)
end

#each_row ⇒ Object

Iterate rows as Ruby Hashes (memo §11.11). This is an escape path — the primary idiom is column-vectorized work (§4.3); each_row is for touching heterogeneous / Face-carrying rows now and then, not hot loops. Without a block, returns an Enumerator.



8
9
10
11
12
 
# File 'lib/carray/frame/convert.rb', line 8

def each_row
  return enum_for(:each_row) unless block_given?
  nrow.times { |i| yield row(i) }
  self
end

#fill(name, method_or_value) ⇒ Object

Fill masked cells of a column (memo §6, §11.8). A Symbol selects a scan method; anything else is a constant fill value. The fill is write-through: it edits the live shared column rather than rebinding a filled copy (memo §3.8). Returns self so calls chain.

fill("temp", :ffill)   # forward hold  (carry last valid value)
fill("temp", :bfill)   # backward hold (carry next valid value)
fill("temp", :linear)  # linear interpolation; x = this frame's index
                     # coordinate when present, else the cell position
fill("temp", 0.0)      # constant fill

:ffill / :bfill work for any writable column data_type (numeric, time, object, fixlen, …); :linear needs a numeric or time (CATime / CATimedelta) column. A categorical column's codes are read-only (memo §13.4), so any fill on it raises — rebind a filled copy instead: df.append(name, df[name].strip_mask(method: :forward)).

Raises:

  • (KeyError) 


191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
 
# File 'lib/carray/frame/verbs.rb', line 191

def fill(name, method_or_value)
  key = name.to_s
  raise KeyError, "no column #{key.inspect}" unless @columns.key?(key)
  col = @columns[key]
  case method_or_value
  when :ffill  then col.unmask(method: :forward)
  when :bfill  then col.unmask(method: :backward)
  when :linear then fill_linear_column(key, col)
  when Symbol
    raise ArgumentError,
          "fill: unknown method #{method_or_value.inspect} " \
          "(:ffill | :bfill | :linear, or a constant fill value)"
  else
    col.unmask(method_or_value)   # constant fill, in place
  end
  self
end

#filter(keep_masked: false) ⇒ Object

Filter rows with a block that receives the frame and returns a boolean column (memo §15). The block builds the mask from f["col"] / f.index. Keep the rows where the block's boolean selector is true.

A masked (UNDEF) selector cell means the row's membership is genuinely undetermined (e.g. the predicate read a masked input). By default such rows are silently dropped, exactly as a false cell would be. Pass keep_masked: true to carry the UNDEF forward instead: the undetermined rows survive into the result with their data cells masked (and their index value kept, so the row stays identifiable), leaving a later, better-informed pass to re-judge them. Definitely-true rows carry their values through unchanged in both modes.



310
311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/carray/frame/frame.rb', line 310

def filter(keep_masked: false)
  mask = yield(self)
  unless mask.is_a?(CArray) && mask.data_type == :boolean
    raise ArgumentError,
          "filter block must return a boolean CArray (got #{mask.class})"
  end
  unless mask.shape[0] == @nrow
    raise ArgumentError,
          "filter mask has axis-0 length #{mask.shape[0]}, expected nrow #{@nrow}"
  end
  select_rows(mask, keep_masked: keep_masked)
end

#group_by(*keys) ⇒ Object

Group rows by one or more keys. Each key is a column name (String) or an external length-N CArray. Returns a GroupedFrame.

Raises:

  • (ArgumentError) 


12
13
14
15
16
17
 
# File 'lib/carray/frame/group.rb', line 12

def group_by(*keys)
  raise ArgumentError, "group_by needs at least one key" if keys.empty?
  cat  = grouping_categorical(keys)
  axis = keys.size == 1 && keys.first.is_a?(String) ? keys.first : "group"
  GroupedFrame.new(self, cat, axis)
end

#head(n = 5) ⇒ Object

The first n rows as a positional view-frame (memo §11.3). n larger than nrow yields the whole frame; n of 0 yields an empty frame.

Raises:

  • (ArgumentError) 


353
354
355
356
 
# File 'lib/carray/frame/frame.rb', line 353

def head(n = 5)
  raise ArgumentError, "head count must be non-negative (got #{n})" if n < 0
  row_span(0, [n, @nrow].min)
end

#inspect ⇒ String

The summary line (nrow, variable data types, index) followed by the middle-elided table.

Returns:

  • (String) 


158
159
160
161
162
163
164
165
 
# File 'lib/carray/frame/io.rb', line 158

def inspect
  parts = @columns.map { |k, v| "#{k}:#{v.data_type}#{v.ndim > 1 ? v.shape[1..].inspect : ''}" }
  idx = @index ? " index=#{@axis_name.inspect}" : ""
  head = "#<CAFrame nrow=#{@nrow} vars=[#{parts.join(', ')}]#{idx}>"
  return head if @columns.empty?
  head + "\n" + render_table(head: 8, tail: 2, index: true, precision: 6,
                             footer: false)
end

#join(other, on:, how: :left, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

Join other on a shared key column.

how: :left  (default) — keep all left rows; right columns gathered per
                      left row (locate_addr), misses become UNDEF.
how: :inner/:outer/:right — set-align both key sets (align_addr) and
                      gather both sides; the aligned key values form
                      the +on+ column.

A non-key column present on both sides collides. The key column (+on+) is kept once, never suffixed. By default the collision is resolved by suffixing both sides (+suffixes: ["_left", "_right"]+); pass a 2-element array to pick meaningful names up front (+["_obs", "_fcst"]+), or suffixes: false to raise instead. Rename afterward with rename if needed (§12-C).



29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/carray/frame/join.rb', line 29

def join(other, on:, how: :left, suffixes: DEFAULT_JOIN_SUFFIXES)
  on   = on.to_s
  lkey = self[on]
  rkey = other[on]
  plan = join_name_plan(other, on, suffixes)

  case how
  when :left
    join_left(other, on, lkey, rkey, plan)
  when :inner, :outer, :right
    join_align(other, on, lkey, rkey, how, plan)
  else
    raise ArgumentError, "unknown join mode #{how.inspect}"
  end
end

#join_asof(other, on:, direction: :floor, tolerance: nil, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

As-of (nearest-key) left join for irregular series (memo §11.7). Each left row is matched to the nearest other row by on via locate_nearest_addr (the per-row, row-preserving counterpart of align_nearest_addr); direction: follows CArray (:floor = most recent at-or-before, :ceil = next, :round = nearest), and rows with no match in range or beyond tolerance: come back UNDEF. Same wiring as the left join, only the address primitive differs.



52
53
54
55
56
57
 
# File 'lib/carray/frame/join.rb', line 52

def join_asof(other, on:, direction: :floor, tolerance: nil, suffixes: DEFAULT_JOIN_SUFFIXES)
  on   = on.to_s
  plan = join_name_plan(other, on, suffixes)
  addr = self[on].locate_nearest_addr(other[on], direction: direction, tolerance: tolerance)
  join_by_addr(other, on, addr, plan)
end

#mask_eq(name, value) ⇒ Object

Mask cells of a column that equal value (sentinel -> mask, memo §11.4). In-place on the column (write-through, §4.3): numeric/object columns mask in place; a categorical column's codes are read-only (§13.4) so this raises — recode by rebuilding and rebinding instead.



53
54
55
56
 
# File 'lib/carray/frame/verbs.rb', line 53

def mask_eq(name, value)
  self[name][:eq, value] = UNDEF
  self
end

#nvar ⇒ Object

Number of variables (columns).



419
420
421
 
# File 'lib/carray/frame/frame.rb', line 419

def nvar
  @columns.size
end

#parse_to_time(name, format = nil, unit: :s) ⇒ Object

Parse a string column into a time column and rebind it (memo §11.2). The column must be string-bearing (an object CArray of Strings, or a CAString / CAConstString / CAFixlenString) — this is the "text -> time" mode, distinct from the integer-serial mode of to_time. Delegates to CArray.time(col, on_error: :mask): format picks strptime parsing (auto-detect when nil), unit the storage resolution; masked / nil and unparseable cells become UNDEF (bulk column parse tolerates bad cells). Make it the index with set_index afterward.

df.parse_to_time("time").set_index("time")


122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/carray/frame/verbs.rb', line 122

def parse_to_time(name, format = nil, unit: :s)
  key = name.to_s
  col = @columns.fetch(key) { raise KeyError, "no column #{key.inspect}" }
  unless string_column?(col)
    raise ArgumentError,
          "parse_to_time needs a string column (object / CAString / " \
          "CAConstString / CAFixlenString); #{key.inspect} is #{col.data_type}"
  end
  @columns[key] = CArray.time(col, format: format, unit: unit, on_error: :mask)
  self
end

#paste(other, suffixes: DEFAULT_JOIN_SUFFIXES) ⇒ Object

Paste +other+'s variables beside this frame's, matched by row position (memo §12-C) — a keyless column merge (the column-direction counterpart of the row-stacking concat, and the positional counterpart of the key-aligned join; named after the UNIX paste). Both frames must have the same nrow; rows are assumed to already correspond (no key alignment, consistent with the no-implicit-align stance). Colliding column names are disambiguated by the same policy as join (default suffix +_left+/+_right+, suffixes: to override, suffixes: false to raise). This frame's index is kept; +other+'s index, if any, is not carried (only its columns are pasted). Returns a new frame.

obs.paste(fcst)                          # side by side, same rows
obs.paste(fcst, suffixes: ["_obs", "_fcst"])


106
107
108
 
# File 'lib/carray/frame/join.rb', line 106

def paste(other, suffixes: DEFAULT_JOIN_SUFFIXES)
  CAFrame.paste(self, other, suffixes: suffixes)
end

#promote(type = nil) ⇒ Object

Bring every column to one common data type and rebind them (memo §11.4). The frame-level counterpart of CArray.promote_list: where cast forces named columns to a type, promote widens the whole table until it has a single type. Returns self so calls chain (§3.8 -- a type-interpretation change), and like cast it allocates fresh columns, so a parent frame sharing the old ones is untouched. The index is not a column and is left alone.

df.promote             # the common type CArray.result_type would pick
df.promote(:object)    # force the widest type: anything fits

Without an argument the common type is whatever promote_list picks -- the same decision to_ca / CArray.stack make internally, so df.promote is exactly "make this frame stackable". Columns that are already uniform (including a uniformly Face-typed frame) are left as they are.

With an argument the type must be a widening for every column: promoting is not the place to lose values, so a narrowing target raises and points at cast, which is the verb that forces. :object is the widest type and therefore always accepted -- it is the way a frame mixing text, Face-typed and numeric columns becomes a single-type table (and so the way to_ca can hand back a matrix for it).



106
107
108
109
110
 
# File 'lib/carray/frame/verbs.rb', line 106

def promote(type = nil)
  return self if @columns.empty?
  type.nil? ? promote_to_common : promote_to_type(type)
  self
end

#rename(mapping) ⇒ Object

Rename columns, returning a new frame (column names change, memo §3.8). Column order is preserved (memo §13.2 "column order = insertion order"); columns are shared by reference.



33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/carray/frame/verbs.rb', line 33

def rename(mapping)
  norm = {}
  mapping.each do |old, new|
    o = old.to_s
    n = new.to_s
    raise KeyError, "no column #{o.inspect}" unless @columns.key?(o)
    if n != o && @columns.key?(n)
      raise ArgumentError, "rename target #{n.inspect} already exists"
    end
    norm[o] = n
  end
  rebuilt = {}
  @columns.each { |k, v| rebuilt[norm[k] || k] = v }
  rebuild(rebuilt)
end

#reset_index ⇒ Object

Drop the index back into an ordinary column (the inverse of set_index). Also an index-role change, so it mutates self (memo §3.8). The former index becomes the first column, named after the row axis.



343
344
345
346
347
348
349
 
# File 'lib/carray/frame/frame.rb', line 343

def reset_index
  return self unless @index
  @columns   = { @axis_name => @index }.merge(@columns)
  @index     = nil
  @axis_name = DEFAULT_AXIS_NAME
  self
end

#select(*names) ⇒ Object

Column projection (memo §13.2): a view-frame holding the named columns as aliases (zero-copy, sharing storage with the parent frame). Distinct from df[...], which escapes columns to raw CArrays, and from filter, which selects rows. select takes no block — row conditions go through filter (old carray-dataframe fused the two into select(cols) { cond }; here they are orthogonal and compose by chaining: df.select(...).filter { ... }).

Raises:

  • (ArgumentError) 


287
288
289
290
291
292
293
294
295
296
 
# File 'lib/carray/frame/frame.rb', line 287

def select(*names)
  raise ArgumentError, "select requires at least one column name" if names.empty?
  cols = {}
  names.each do |name|
    key = name.to_s
    raise KeyError, "no column #{key.inspect}" unless @columns.key?(key)
    cols[key] = @columns[key]
  end
  rebuild(cols)
end

#set_index(name) ⇒ Object

Move a column to the index and name the row axis after it (memo §3). An index-role change: the data is unchanged, so this mutates self and returns it (memo §3.8). Any existing index is replaced.

Raises:

  • (KeyError) 


326
327
328
329
330
331
332
333
334
335
336
337
338
 
# File 'lib/carray/frame/frame.rb', line 326

def set_index(name)
  key = name.to_s
  raise KeyError, "no column #{key.inspect}" unless @columns.key?(key)
  idx = @columns[key]
  unless idx.ndim == 1
    raise ArgumentError, "index must be a 1-D column (got ndim #{idx.ndim})"
  end
  @columns.delete(key)
  @index     = idx
  @axis_name = key
  @nrow      = idx.shape[0]
  self
end

#sort_by(masked_position: :last) ⇒ Object

Sort rows by key columns the block builds (memo §4.3 escape). The block receives the frame and returns a key CArray, or an Array of them for a multi-key lexicographic sort; rows are sorted ascending by CArray.sort_addr over those keys and gathered into a view-frame. This is the sort sibling of filter: where sort_by_key takes plain column names, the block form lets you build any key with CArray ops -- a descending key is its dense descending rank, derived or composite keys are ordinary column math. masked_position: places masked key rows first or last (default last).

df.sort_by { |f| (f["temp"] - target).abs }   # nearest-to-target first
df.sort_by { |f| f["temp"].order(descending: true, method: :dense) }

Raises:

  • (ArgumentError) 


60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/carray/frame/sort.rb', line 60

def sort_by(masked_position: :last)
  raise ArgumentError, "sort_by requires a block" unless block_given?
  keys = yield(self)
  keys = [keys] unless keys.is_a?(Array)
  raise ArgumentError, "sort_by block must return a key CArray or an Array of them" if keys.empty?
  keys.each_with_index do |k, i|
    unless k.is_a?(CArray)
      raise ArgumentError, "sort_by key #{i} must be a CArray (got #{k.class})"
    end
    unless k.ndim == 1 && k.shape[0] == nrow
      raise ArgumentError,
            "sort_by key #{i} must be a 1-D column of length #{nrow} (got shape #{k.shape.inspect})"
    end
  end
  select_rows(CArray.sort_addr(*keys, masked_position: masked_position))
end

#sort_by_key(*specs, order: :asc, masked_position: :last) ⇒ Object

Sort rows by one or more key columns, lexicographic (the first key is primary), returning a view-frame. Delegates to CArray.sort_addr for the row permutation and gathers every column and the index by it.

Each key is a column name (or the index axis name), or a [name, :asc | :desc] pair for a per-key direction. The order: keyword is the direction for bare-name keys (default :asc). masked_position: sends masked key rows to the :last (default) or :first end.

df.sort_by_key("temp")                              # one column, ascending
df.sort_by_key("station", "temp")                   # lexicographic
df.sort_by_key("temp", order: :desc)                # all keys descending
df.sort_by_key(["station", :asc], ["temp", :desc])  # per-key direction
df.sort_by_key("temp", masked_position: :first)     # masked rows first
df.sort_by_key("time")                              # by the index

Raises:

  • (ArgumentError) 


38
39
40
41
42
43
44
45
46
47
 
# File 'lib/carray/frame/sort.rb', line 38

def sort_by_key(*specs, order: :asc, masked_position: :last)
  raise ArgumentError, "sort_by_key requires at least one key" if specs.empty?
  validate_sort_order(order)
  keys = specs.map do |spec|
    name, dir = parse_sort_key_spec(spec, order)
    col = sort_key_column(name)
    dir == :desc ? col.order(descending: true, method: :dense) : col
  end
  select_rows(CArray.sort_addr(*keys, masked_position: masked_position))
end

#tail(n = 5) ⇒ Object

The last n rows as a positional view-frame (memo §11.3). n larger than nrow yields the whole frame; n of 0 yields an empty frame.

Raises:

  • (ArgumentError) 


360
361
362
363
 
# File 'lib/carray/frame/frame.rb', line 360

def tail(n = 5)
  raise ArgumentError, "tail count must be non-negative (got #{n})" if n < 0
  row_span(@nrow - [n, @nrow].min, @nrow)
end

#to_ca(writable: false) ⇒ Object

Hand the frame over as a CArray with the minimum work (memo §11.9): a 2-D view of shape (nrow, nvar), one column per variable in column order (§12-C). Nothing is materialised — the result is a CAStack over the stored columns, so reads gather from them and writes flow back (§3.6). Call copy on it for an independent, owned matrix; CArray.tabulate is the eager sibling that builds one directly.

Only same-shape scalar (1-D) columns qualify. An N-D column has no single matrix form and raises — escape it per column with df["name"]. A mixed data type set is promoted to a common type (+result_type+, §12-F) through lazy cast lanes, so the promotion costs no buffer either.

writable: true demands a result whose writes reach this frame's own columns (the 3.0 to_ca contract). That holds only when every column enters the stack unchanged, so it is refused when a column is read-only or when the common type promotes it: the promoted column is stacked through a cast lane, and a write there is no longer the value the caller handed over. Cast the frame first (+df.cast+) when the promoted matrix is what should be written to.

Raises:

  • (ArgumentError) 


44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/carray/frame/convert.rb', line 44

def to_ca(writable: false)
  cols = @columns.values
  raise ArgumentError, "frame has no columns to stack into a matrix" if cols.empty?
  nd = @columns.find { |_, c| c.ndim != 1 }
  if nd
    raise ArgumentError,
          "to_ca needs all-scalar (1-D) columns; #{nd.first.inspect} is " \
          "#{nd.last.ndim}-D — escape per column with df[name]"
  end
  refuse_unshared_columns(cols) if writable
  begin
    CArray.stack(cols, axis: 1)
  rescue ArgumentError, RuntimeError => e
    # The columns have no common type (a text or Face column beside a
    # numeric one). Point at the frame-level verb that gives them one.
    raise e.class,
          "#{e.message} -- df.promote(:object) brings every column to its " \
          "surface values, and that frame stacks"
  end
end

#to_csv(path = nil, sep: ",", quote: '"', header: true, index: true) ⇒ Object

Build a frame from parsed [names, rows]. When names is nil (headerless and no column_names) positional names "c0".."cN" are generated from the widest row. Rows are squared off to the column count (short rows padded with nil, over-long rows raise), one 2-D object array is bulk-filled, and each column is a view into it (§3.6). Write the frame as CSV. CSV is a flat table of scalar cells, so this is the text form of the same all-scalar subset to_ca requires (§11.9): every column must be 1-D. Unlike to_ca it does not promote to a common data type -- each column is formatted to text independently, so mixed data types (numbers, strings, datetime / categorical Faces) sit side by side. An N-D column has no flat CSV cell and raises; export it per column, or use to_records + JSON for the structured shape (memo §11.9, the N-D escape).

With path, writes the file and returns self; without it, returns the CSV String. The index (if any) is written as the first column under axis_name unless index: false. A masked cell (UNDEF) becomes an empty field, which from_csv reads back as UNDEF (parse-mask, §6-2) -- so mask round-trips. A genuine empty string is written quoted (+""+) to stay distinct from missing, matching the tokenizer's own unquoted-empty vs quoted-empty distinction.

df.to_csv("out.csv")          # write file
csv = df.to_csv               # get a String

Options: sep / quote mirror from_csv; header writes the name row (default true); index writes the index column (default true).



83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/carray/frame/io.rb', line 83

def to_csv(path = nil, sep: ",", quote: '"', header: true, index: true)
  nd = @columns.find { |_, c| c.ndim != 1 }
  if nd
    raise ArgumentError,
          "to_csv needs all-scalar (1-D) columns; #{nd.first.inspect} is " \
          "#{nd.last.ndim}-D — export it per column or via to_records + JSON"
  end

  names     = []
  formatted = []
  if index && @index
    names     << @axis_name
    formatted << format_csv_column(@index)
  end
  @columns.each do |name, col|
    names     << name
    formatted << format_csv_column(col)
  end

  out = +""
  if header
    out << names.map { |t| quote_csv_field(t, sep, quote) }.join(sep) << "\n"
  end
  @nrow.times do |i|
    out << formatted.map { |fcol| quote_csv_field(fcol[i], sep, quote) }.join(sep) << "\n"
  end

  if path
    File.write(path, out)
    self
  else
    out
  end
end

#to_records ⇒ Object

Export rows as an Array of plain Ruby Hashes (memo §11.11) — the inverse of from_records and the shape JSON.generate wants. Each row is one Hash of column name => value; a scalar cell is a Ruby value, an N-D cell is a Ruby Array, and a masked cell (UNDEF) is nil. Normalizing UNDEF -> nil and CArray -> Array (unlike each_row, which yields the raw view with UNDEF / CArray slices) is what makes it round-trip: from_records(df.to_records) rebuilds the same typing, and the result is JSON-serializable.



21
22
23
 
# File 'lib/carray/frame/convert.rb', line 21

def to_records
  each_row.map { |row| row.transform_values { |v| record_value(v) } }
end

#to_s ⇒ Object

to_s is the whole frame, inspect the middle-elided one -- so puts df dumps everything and p df stays a screenful. inspect leads with the same summary line it always had (nrow, variable data types, index), so the table under it needs no row-count footer.



151
152
153
 
# File 'lib/carray/frame/io.rb', line 151

def to_s
  to_table(rows: nil)
end

#to_table(rows: 20, index: true, precision: 6) ⇒ Object

Render the frame as an aligned text table for reading:

puts df.to_table

time        temp  station
----------  ----  -------
2026-01-01   1.5  Tokyo
2026-01-02     _  Osaka

This is the display counterpart of to_csv and shares none of its constraints: it is text meant to be looked at, not read back. Numeric columns are right-aligned, everything else left-aligned; a masked cell shows as _, the same marker CArray's own inspect uses. An N-D column (which to_csv rejects, having no flat cell) shows each row's slice as an Array literal.

Float cells are rounded to precision decimal places for display only (default 6); precision: nil prints them at full precision, which is faithful but lets one long value set the column width.

Long frames are truncated in the middle: rows caps how many rows are printed (default 20, split evenly around an ellipsis row), and rows: nil prints every row. index: false drops the index column.



141
142
143
144
145
 
# File 'lib/carray/frame/io.rb', line 141

def to_table(rows: 20, index: true, precision: 6)
  head = rows && (rows + 1) / 2
  render_table(head: head, tail: rows && rows - head,
               index: index, precision: precision, footer: true)
end

#to_time(name, grid = nil, unit: :s, epoch: nil) ⇒ Object

Reinterpret an integer column as time serial counts and rebind it (memo §11.2). This is the "serial -> time" mode: each value is a count of unit resolution since epoch (default the Unix epoch, 1970-01-01 UTC). epoch takes any time literal (String / Time / Integer), so a column measured from another origin — a netCDF "hours since 1990-01-01" axis, an Excel serial date (epoch "1899-12-30", unit :D) — converts directly.

A float column is accepted only when every value is whole (no fractional part); a fractional serial has sub-unit precision that a finer unit should carry, so it raises rather than silently truncate. Make it the index with set_index afterward.

A CATime::Grid carries the same (unit, epoch) pair as one value, so a netCDF units attribute goes straight in. It also carries a phase the keyword form cannot: the keyword epoch is read on the unit grid, so an epoch off that grid ("days since 1980-01-01 12:00") loses its time-of-day, while a grid resolves the finer storage that holds it.

df.to_time("time", unit: :h, epoch: "1990-01-01").set_index("time")
df.to_time("time", CATime::Grid.parse("hours since 1990-01-01"))
df.to_time("time", CATime::Grid.parse("days since 1980-01-01 12:00"))


155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
 
# File 'lib/carray/frame/verbs.rb', line 155

def to_time(name, grid = nil, unit: :s, epoch: nil)
  key = name.to_s
  col = @columns.fetch(key) { raise KeyError, "no column #{key.inspect}" }
  raw = integer_serial_column(col, key)
  grid = unit if unit.is_a?(CATime::Grid)
  if grid.is_a?(CATime::Grid)
    @columns[key] = grid.at(raw)
    return self
  end
  unless grid.nil?
    raise ArgumentError,
          "the positional argument must be a CATime::Grid (got #{grid.class})"
  end
  if epoch
    raw = raw + CArray.time(epoch, unit: unit).ticks[0]
  end
  @columns[key] = raw.time(unit: unit)
  self
end

#variable_names ⇒ Object

Variable (column) names in column order. Returns Array<String>.



407
408
409
 
# File 'lib/carray/frame/frame.rb', line 407

def variable_names
  @columns.keys
end

#variables ⇒ Object

Variables (raw column CArrays) in column order. Returns Array<CArray>. Equivalent to df[*variable_names] but built directly from the columns Hash.



414
415
416
 
# File 'lib/carray/frame/frame.rb', line 414

def variables
  @columns.values
end