Class: CACategorical

Inherits:
CAObject show all
Defined in:
lib/carray/categorical.rb

Overview

Categorical column: dense integer codes plus a label vocabulary, so each element is an index into a small set of categories. Structurally the same idea as a pandas Categorical or an Arrow dictionary array.

Implemented as a read-only non-numeric Face over the codes array — the storage is the integer codes, while the surface is CA_FIXLEN so numeric kernels are gated off (cat + 1 raises; arithmetic on category codes is not meaningful).

Constant Summary collapse

UNPACK_FORMAT =

raw-byte unpack format per storage (codes) data_type, native endian. The FIXLEN surface delivers a per-cell fetch as an N-byte String; this decodes it back into the integer code.

{
  CA_INT8  => "c", CA_UINT8  => "C",
  CA_INT16 => "s", CA_UINT16 => "S",
  CA_INT32 => "l", CA_UINT32 => "L",
  CA_INT64 => "q", CA_UINT64 => "Q",
}.freeze
SENTINEL =

The exclusion sentinel per codes data_type: the all-ones bit pattern, read as type-max for an unsigned data type and as -1 for a signed one. Either way it is out of every valid [0, k) range and byte-identical to a pandas / Arrow missing code.

{
  CA_UINT8  => 0xFF,               CA_INT8  => -1,
  CA_UINT16 => 0xFFFF,             CA_INT16 => -1,
  CA_UINT32 => 0xFFFFFFFF,         CA_INT32 => -1,
  CA_UINT64 => 0xFFFFFFFFFFFFFFFF, CA_INT64 => -1,
}.freeze
MAX_LABELS =

The largest vocabulary each codes data type can carry: every valid code in 0...k has to be representable AND distinct from the exclusion sentinel. An unsigned type spends its top value on the sentinel; a signed one spends -1, which was never a valid index, so a signed type carries one more label than the unsigned type of the same width. categorize picks a width by this rule when it builds codes; from_codes has to check it when it receives them.

{
  CA_UINT8  => 0xFF,               CA_INT8  => 0x80,
  CA_UINT16 => 0xFFFF,             CA_INT16 => 0x8000,
  CA_UINT32 => 0xFFFFFFFF,         CA_INT32 => 0x80000000,
  CA_UINT64 => 0xFFFFFFFFFFFFFFFF, CA_INT64 => 0x8000000000000000,
}.freeze

Constants inherited from CArray

CArray::HAVE_COMPLEX, CArray::LAZY_BINCMP_OP_IDS, CArray::LAZY_BINCMP_TOL_OP_IDS, CArray::LAZY_BINOP_OP_IDS, CArray::LAZY_MONOP_OP_IDS, CArray::LAZY_TRIOP_OP_IDS, CArray::UNSPECIFIED, CArray::VERSION

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from CArray

#%, #&, #*, #**, #+, #-, #-@, #/, #<, #<<, #<=, #<=>, #==, #>, #>=, #>>, #T, #[], #[]=, #^, _scan_float, _scan_int, #abs, #abs!, #abs_i, #abs_i!, #acos, #acos!, #acosh, #acosh!, #add, #add!, addr2index, #addr2index, #address, align_addr, align_nearest_addr, #all, #all_masked?, #ancestors, #and, #and!, #any, #any_masked?, #arg, #arg!, #as_boolean, #as_byte, #as_cmplx128, #as_cmplx256, #as_cmplx64, #as_complex, #as_dcomplex, #as_double, #as_fixlen, #as_float, #as_float128, #as_float32, #as_float64, #as_int, #as_int16, #as_int32, #as_int64, #as_int8, #as_object, #as_short, #as_strided, #as_type, #as_uint16, #as_uint32, #as_uint64, #as_uint8, #asin, #asin!, #asinh, #asinh!, #atan, #atan!, #atan2, #atan2!, #atanh, #atanh!, #attached?, #attr, #attrs, #axis2addr, #axis_group, big_endian?, #bin, #bin_to, #bincount_nd, #bit_and, #bit_lshift, #bit_lshift!, #bit_neg, #bit_neg!, #bit_or, #bit_rshift, #bit_rshift!, #bit_xor, #bitarray, #bitfield, #bits, #block_view, #blocks, #boolean, boolean, #boolean?, #broadcast_to, #bsearch, #bsearch_addr, byte, #byte, #bytes, cast, #cast_with, #categorize, #ceil, #ceil!, #choose, #clip, #clip_int16, #clip_int32, #clip_int64, #clip_int8, #clip_uint16, #clip_uint32, #clip_uint64, #clip_uint8, cmplx128, #cmplx128, cmplx64, #cmplx64, #coerce, #collect!, #collect_addr!, #collect_index!, #collect_with_addr!, #collect_with_index!, #complex, complex, #complex?, concatenate, #concatenate, #conditional, #conj, #conj!, const_string, #convert, #copy, #copysign, #copysign!, #cos, #cos!, #cosh, #cosh!, #count_masked, #count_not_masked, #crop, #data_class, #data_class=, #data_type, data_type_code, #data_type_name, data_type_name, dcomplex, #dcomplex, #deg, #deg!, #deg_180, #deg_180!, #deg_360, #deg_360!, #delete_block, #diagonal, #dim, #dim0, #dim1, #dim2, #dim3, #dim_view, #div, #div!, #divmod, #double, double, #drop_axis, dump, #dump_binary, #each, #each_addr, each_index, #each_index, #each_slab, #each_with_addr, #each_with_index, #elem_copy, #elem_decr, #elem_fetch, #elem_incr, #elem_mask, #elem_masked?, #elem_max, #elem_min, #elem_store, #elem_swap, #elem_unmask, #elements, empty, #empty?, #endian, endian, #entity?, #eql?, #exp, #exp!, #exp10, #exp10!, #exp2, #exp2!, #expm1, #expm1!, #face?, #factorize, #fake, #false, #farray, #feq, #field, #fields, #fields_at, #fill, #fill_copy, #first, #fixlen, fixlen, #fixlen?, fixlen_string, #flags, #flatten, #flip, #float, float, #float32, float32, #float64, float64, #float?, #floor, #floor!, #fma, #fma!, #fmax, #fmin, #fmod, #fmod!, #fms, #fms!, format, #format, #frac, #freeze, from_bit_string, #from_bit_string, from_memory_view, fuse, #gather_nd, #ge, #grid, #group_by_category, #group_by_run, #gt, guard_undef, guess_type_and_bytes, #has_attr?, #has_data_class?, #has_mask?, #hash, #histogram, #histogram1d, #histogram2d, #hypot, #hypot!, #imag, #imag=, #imag_i, #imag_i!, #index, index2addr, #index2addr, #indices, #inherit_mask, #inherit_mask_replace, #initialize_copy, #insert_axis, #insert_block, #inspect_full, int, #int, #int16, int16, int32, #int32, #int64, int64, #int8, int8, #integer?, #invert_mask, #is_close, #is_equiv, #is_finite, #is_inf, #is_invalid, #is_masked, #is_mode, #is_nan, #is_not_masked, #is_real, #join, #last, #lazy, #le, #length, #linear_fetch, #linear_section, little_endian?, load, load_arrow_tensor, #load_binary, #locate_nearest_addr, #log, #log!, #log10, #log10!, #log1p, #log1p!, #log2, #log2!, #logaddexp, #logaddexp!, #logb, #logb!, #lookup, #lt, #map!, #map_addr!, #map_index!, #map_slab, #map_with_addr!, #map_with_index!, #marshal_dump, #marshal_load, #mask, #mask=, #mask_array?, #mask_duplicates, #mask_eq, #mask_invalid, #mask_where, #maximum, #maximum!, #median, meld, #meld, #members, memory_view_available?, memory_view_reject_reason, meshgrid, #minimum, #minimum!, #mod, #mod!, montage, mosaic, #mul, #mul!, #ndim, #neg, #neg!, #nextafter, #nextafter!, #none, #normalize_axes, #normalize_axis, normalize_axis, #normalize_index, #not, #not!, #numeric?, #nunique, #obj_type, #object, object, #object?, #one, #one!, #or, #or!, #order, #pack_bits, #parent, #partition, #partition_copy, #paste, #percentile, #pmax, #pmax!, #pmin, #pmin!, #pow, #power, #power!, #project, promote_list, #put_nd, #quantile, #rad, #rad!, #rad_2pi, #rad_2pi!, #rad_pi, #rad_pi!, #random, #random!, #randomn, #randomn!, #rank, #rcp, #rcp!, #rcp_mul, #rcp_mul!, #read_only?, #real, #real=, #real?, #reduce_slab, #refer, #repeat, #replace_where, #reshape, #resize, result_type, #roll, #root_array, #round, #rsqrt, #rsqrt!, #same_shape?, save, #save_arrow_tensor, #scalar?, #scale, #scale!, #scatter_add!, #scatter_max!, #scatter_min!, #scatter_mul!, #scatter_replace!, #scatter_sub!, #search, #search_addr, #search_nearest, #search_nearest_addr, select, #seq, #seq!, #set_attr, #set_read_only_flag, #shape, #shift, short, #short, #shuffle, #shuffle!, #sign, #sign!, #signbit, #sin, #sin!, #sinh, #sinh!, #size, sizeof, #sliding_windows, #snap, #snap_to, #sort, sort_addr, #sort_copy, #source_code, #span, #span!, #split, #sqrt, #sqrt!, #square, #square!, #st, #stack, stack, string, #strip_mask, struct, #sub, #sub!, #swap_bytes, tabulate, #tan, #tan!, #tanh, #tanh!, #template, #then_else, #tile, #time, time, time_range, time_series, #timedelta, #to_a, #to_bit_string, #to_ca, #to_s, #to_type, #transpose, #true, #trunc, #trunc!, uint16, #uint16, uint32, #uint32, uint64, #uint64, uint8, #uint8, #unfold, union, #unmask, #valid_addr?, #valid_index?, #validity_bits, #value, #value_array?, #virtual?, #where, #window, #windows, wrap, wrap_memory_view, wrap_readonly, wrap_writable, #xor, #xor!, #zero, #zero!, #|, #~

Methods included from CArray::DataTypeExtension

#arange, #empty, #eye, #full, #identity, #linspace, #ones, #zeros

Constructor Details

#initialize(codes, labels) ⇒ CACategorical

codes : integer CArray, the storage parent, already normalised by the caller (categorize builds it that way; from_codes rewrites what it receives). Excluded cells are both masked AND store the type-max sentinel value (= the all-ones bit pattern, which is signed -1 byte-for-byte — the pandas / Arrow missing code). Marking the codes read-only below keeps the two in step from here on, so consumers may rely on either: the mask (CArray-native) or the sentinel (axis-group's out-of-range skip, zero-copy export). labels: Array | CArray, the vocabulary; labels = category.

Allocates a READONLY CACategorical Face whose storage is codes and whose vocabulary is labels. The label list is copied and frozen so codes always index a stable vocabulary.

Parameters:

  • codes (CArray) —

    integer code storage.

  • labels (Array, CArray) —

    category vocabulary.



232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
 
# File 'lib/carray/categorical.rb', line 232

def initialize(codes, labels)
  # Own a frozen copy of the vocabulary: the categorical is READONLY and its
  # codes index into labels, so the label list must not change under it. We
  # copy first so a caller's array is never frozen as a side effect; the
  # label objects themselves are left untouched (container-level freeze).
  @labels = (labels.respond_to?(:to_a) ? labels.to_a : Array(labels)).dup.freeze
  super(CA_FIXLEN, codes.dim,
        bytes:     codes.bytes,
        storage:   codes.data_type,
        parent:    codes,
        read_only: true,
        face:      true)
  # Mark the codes storage read-only so the READONLY guarantee holds at the
  # root, not just on the Face. Without this the Face is read_only but its
  # parent is writable, so `cat.codes[i] = x` silently mutates the categorical
  # (and any grouping cache derived from it). We set the CA_FLAG_READ_ONLY flag
  # rather than #freeze: freeze also freezes the Ruby object, which propagates
  # through views/Faces (a reshape of frozen codes is frozen) and would block
  # the grouping cache from memoising. The flag gives the same write protection
  # (mutations raise) while keeping the object non-frozen. One-way: it takes
  # ownership of `codes`, which from_codes has already validated, materialised
  # if it was a view, and normalised; a caller keeping a mutable entity of its
  # own must pass `.copy`.
  codes.set_read_only_flag
end

Instance Attribute Details

#labels ⇒ Object (readonly)

Returns the value of attribute labels.



258
259
260
 
# File 'lib/carray/categorical.rb', line 258

def labels
  @labels
end

Class Method Details

.from_codes(codes, labels) ⇒ CACategorical

Wrap already-dense codes + labels with no discovery — the import receiver for a pandas Categorical or an Arrow dictionary. from_codes takes ownership of codes, which becomes the Face's storage parent: verbatim when it is an entity that needs no rewriting, so a wrapped memory view stays a view, and materialised when it is a view over an array the caller still holds, since a view owns no bytes to take ownership of.

This is the one door through which an already-built encoding enters, so it validates rather than assumes, and it normalises before handing over: a cell is excluded when it arrives masked OR holds the all-ones sentinel (type-max for unsigned codes, -1 for signed — both the pandas / Arrow missing code), and every excluded cell leaves here holding the sentinel AND masked. Writing both matters because the two encodings have different readers: the axis-group kernel classifies on the code byte, the materialising paths classify on the mask. A categorical that carries only one of them answers membership two ways, silently. An Arrow dictionary carries its missingness in a validity bitmap with arbitrary code bytes — frequently 0 — so arriving masked-only is the normal import, not an edge.

#initialize marks the codes read-only, so this is also the last point at which they can be written; a read-only argument is copied rather than refused.

Returns a CACategorical wrapping already-dense integer codes with the given labels, without discovery. codes becomes the Face's storage parent, materialised first if it is a view. A cell that is masked or holds the type-max sentinel is excluded, and leaves as both.

Parameters:

  • codes (CArray) —

    integer code storage.

  • labels (Array, CArray) —

    category vocabulary indexed by code. Must be unique, and must fit the codes data type with the sentinel reserved.

Returns:

Raises:

  • (ArgumentError) —

    when codes is not an integer CArray, when labels holds duplicates or is too large for the codes data type, or when an unmasked code is outside 0...labels.size and is not the sentinel.



134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
 
# File 'lib/carray/categorical.rb', line 134

def from_codes(codes, labels)
  unless codes.is_a?(CArray) && SENTINEL.key?(codes.data_type)
    got = codes.is_a?(CArray) ? codes.data_type : codes.class
    raise ArgumentError, "from_codes: codes must be an integer CArray (got #{got})"
  end

  labels_arr = labels.respond_to?(:to_a) ? labels.to_a : Array(labels)
  if labels_arr.uniq.size != labels_arr.size
    raise ArgumentError, "from_codes: labels must be unique (got duplicates)"
  end
  k        = labels_arr.size
  sentinel = SENTINEL[codes.data_type]
  max      = MAX_LABELS[codes.data_type]
  if k > max
    raise ArgumentError,
          "from_codes: #{k} labels do not fit #{CArray.data_type_name(codes.data_type)} " \
          "codes, which carry at most #{max} (the top value is reserved as the " \
          "exclusion sentinel); widen the codes data type"
  end

  # Classify on the code bytes with the mask set aside (`.value`): a
  # comparison against a masked cell yields UNDEF, which would read as
  # "not excluded" and let the cell through carrying a valid-looking code.
  raw = codes.value
  if sentinel == -1                 # signed codes: -1 is the sentinel
    out = raw.lt(0)
    # Only test the upper bound when k is representable in the codes data
    # type. At the very top of the range (k == max) no value can reach k
    # anyway, and comparing against an unrepresentable literal would wrap
    # and flag every cell as out of range.
    out = out | raw.ge(k) if k <= max - 1
  else                              # unsigned codes: type-max is the sentinel
    out = raw.ge(k)                 # k <= max here, so always representable
  end
  masked   = codes.has_mask? ? codes.is_masked : nil
  excluded = masked ? (out | masked) : out
  # A masked cell may hold any byte at all — that is the CArray contract —
  # so it is never corrupt, only in need of normalising. An *unmasked* cell
  # holding an out-of-range code that is not the sentinel is neither a
  # category nor missingness; refuse it here, where the input is still in
  # the caller's hands, instead of letting it surface later as an IndexError
  # from the grouping plan or as a wrong label from a decode.
  corrupt = out & raw.ne(sentinel)
  corrupt = corrupt & masked.not if masked
  if corrupt.any
    bad = raw[corrupt].to_a.uniq.sort
    shown = bad.first(4).join(", ") + (bad.size > 4 ? ", ..." : "")
    raise ArgumentError,
          "from_codes: code#{bad.size == 1 ? "" : "s"} #{shown} outside " \
          "0...#{k} for #{k} label#{k == 1 ? "" : "s"} " \
          "(use #{sentinel} to exclude a cell, or mask it)"
  end

  # Take real ownership, then normalise so the byte reader and the mask
  # reader agree from here on.
  #
  # A view owns no bytes — its root does, and the caller still holds that
  # root. #initialize marks what it is given read-only, which would stop
  # `cat.codes[i] = x` but not `root[i] = x`, so the codes could still
  # change underneath a Face that is supposed to be immutable (and
  # underneath the grouping plan memoised against them). Marking the root
  # instead is worse: it would freeze bytes outside the window the caller
  # handed over. So a view is materialised. Read-only codes are copied for
  # the adjacent reason — the normalising writes need somewhere to land.
  #
  # An entity that needs no rewriting is adopted verbatim, which is what
  # keeps a wrapped memory view a view. That buffer's producer can still
  # write it; that is the borrowed-buffer bargain, and not something this
  # constructor can close.
  needs_mask = masked ? (excluded & masked.not).any : excluded.any
  needs_byte = (excluded & raw.ne(sentinel)).any
  if !codes.entity? || ((needs_mask || needs_byte) && codes.read_only?)
    codes = codes.copy
  end
  if needs_mask || needs_byte
    codes.value[excluded] = sentinel if needs_byte
    codes.mask = excluded
  end

  new(codes, labels_arr)
end

Instance Method Details

#build_grouping ⇒ self

Force-build (and cache) the whole grouping plan up front — sort_addr, reduceat_index, category_sizes — for prepare-ahead use before a batch of groupings. Lazy building already covers correctness; this is the explicit "pay the counting sort now" handle (e.g. right after df.group_by(col)). Returns self so it chains.

Eagerly builds and caches the grouping plan (sort_addr / reduceat_index / category_sizes). Optional — the plan is built lazily on first grouping access — but useful to pay the counting sort once ahead of a batch.

Returns:

  • (self) 


520
521
522
523
524
 
# File 'lib/carray/categorical.rb', line 520

def build_grouping
  sort_addr        # pulls category_sizes; reduceat_index shares category_sizes
  reduceat_index
  self
end

#category_sizes ⇒ CArray Also known as: bincount

Per-category counts as a length-k array aligned to #labels. Trailing empty categories are kept as 0 (unlike codes.bincount, which truncates them), so labels.zip(category_sizes.to_a) always pairs up.

Memoised: the codes are read-only (immutable storage), so the counts are a pure function of the categorical and stay valid for its lifetime. The same cached array backs #reduceat_index and #sort_addr, so a wide aggregate over the same categorical pays the count once. (Do not mutate the returned array — it is shared; take .copy for a scratch buffer.) A derived-view / composite categorical caches too: read-only rides from the codes as a flag, not a Ruby freeze, so the object stays non-frozen and the memo ivar sticks.

Returns per-category counts as a CArray with one entry per #labels aligned to #labels. Trailing empty categories are kept as 0 so labels.zip(category_sizes.to_a) always pairs up.

Returns:



358
359
360
361
362
363
364
365
 
# File 'lib/carray/categorical.rb', line 358

def category_sizes
  return @_category_sizes if @_category_sizes
  bc  = codes.bincount
  out = CArray.new(bc.data_type, [@labels.size])   # new zero-fills
  out[0...bc.elements] = bc if bc.elements > 0
  @_category_sizes = out
  out
end

#codes ⇒ CArray

The raw integer codes (= the storage parent). On a derived view this is the correspondingly sliced/reshaped codes, since codes ride the chain. Excluded cells are masked and store the type-max sentinel; the same array serves the axis-group kernel (out-of-range skip) and the pandas / Arrow bridge (byte-reinterpret to signed -1) with no conversion.

Returns the raw integer code CArray backing self. Excluded cells are masked and store the type-max sentinel. The array is read-only — the categorical owns immutable codes, so codes[i] = x raises; use codes.copy for a mutable copy.

Returns:



271
272
273
 
# File 'lib/carray/categorical.rb', line 271

def codes
  parent
end

#copy_state(src) ⇒ Object

Face hook: carry the vocabulary across lifted views (slice / reshape / …). The codes ride along automatically as the Face's parent.



277
278
279
 
# File 'lib/carray/categorical.rb', line 277

def copy_state(src)
  @labels = src.labels
end

#count(label) ⇒ Integer

Returns the number of cells whose category equals label (0 for an unknown label).

Parameters:

  • label (Object) —

    category to count.

Returns:

  • (Integer) 


337
338
339
340
 
# File 'lib/carray/categorical.rb', line 337

def count(label)
  code = @labels.index(label)
  code ? codes.count(code) : 0
end

#difference(other, sort: false) ⇒ CArray

Returns object CArray of the labels only self has.

Returns:

  • (CArray) —

    object CArray of the labels only self has.



440
441
442
 
# File 'lib/carray/categorical.rb', line 440

def difference (other, sort: false)
  unique.difference(label_array(other), sort: sort)
end

#eq(label) ⇒ CArray

Boolean mask of cells whose category == label. Excluded cells stay UNDEF (their category is unknown); an unknown label yields an all-false mask.

Returns a boolean CArray marking cells whose category equals label. Excluded cells stay masked; an unknown label yields an all-false result.

Parameters:

  • label (Object) —

    category to match.

Returns:



320
321
322
 
# File 'lib/carray/categorical.rb', line 320

def eq(label)
  codes.eq(@labels.index(label) || @labels.size)
end

#face_state_compatible?(other) ⇒ Boolean

Face hook: the homogeneity gate for multi-parent constructions (CArray.promote_list / CArray.stack / anything that Face-lifts a list). A code only means anything against the vocabulary it was assigned from, so two categoricals may share one lifted Face only when they index the same labels in the same code order.

Same labels in a different code order is refused rather than re-coded, for the same reason CATime refuses a unit mismatch it knows how to convert: this is a predicate consulted after the parents are assembled, with no channel to rewrite storage — and the codes are read-only by construction, so agreeing would mean silently materialising fresh codes inside what the caller asked for as a view. Build the shared vocabulary up front instead: keys.categorize(labels: shared).

Returns:

  • (Boolean) 


294
295
296
 
# File 'lib/carray/categorical.rb', line 294

def face_state_compatible?(other)
  other.is_a?(CACategorical) && @labels == other.labels
end

#inspect ⇒ String

Returns a short summary showing element count, category count, and vocabulary.

Returns:

  • (String) 


601
602
603
 
# File 'lib/carray/categorical.rb', line 601

def inspect
  "#<CACategorical n=#{elements} categories=#{@labels.size} labels=#{@labels.inspect}>"
end

#intersection(other, sort: false) ⇒ CArray

Returns object CArray of the labels present in both.

Returns:

  • (CArray) —

    object CArray of the labels present in both.



434
435
436
 
# File 'lib/carray/categorical.rb', line 434

def intersection (other, sort: false)
  unique.intersection(label_array(other), sort: sort)
end

#is_in(values) ⇒ CArray

Returns a boolean CArray, true where the cell's label is in values. An unknown label matches nothing. Masked cells stay masked.

Parameters:

Returns:



427
428
429
430
 
# File 'lib/carray/categorical.rb', line 427

def is_in (values)
  wanted = label_list(values).filter_map { |l| @labels.index(l) }
  codes.is_in(CArray.int64(wanted.size) { |i| wanted[i] })
end

#locate_addr(ref) ⇒ CArray

Returns for each cell, its label's address in ref (UNDEF when the label does not occur there).

Returns:

  • (CArray) —

    for each cell, its label's address in ref (UNDEF when the label does not occur there).



453
454
455
 
# File 'lib/carray/categorical.rb', line 453

def locate_addr (ref)
  label_values.locate_addr(label_array(ref))
end

#mode(axis: nil) ⇒ CArray+

Returns the most frequent label(s). Runs in label space because the modal values come back sorted ascending, which for labels is not the code order.

Returns:



418
419
420
 
# File 'lib/carray/categorical.rb', line 418

def mode (axis: nil)
  label_values.mode(axis: axis)
end

#ne(label) ⇒ CArray

Returns the complement of #eq.

Parameters:

  • label (Object) —

    category to compare against.

Returns:



328
329
330
 
# File 'lib/carray/categorical.rb', line 328

def ne(label)
  codes.ne(@labels.index(label) || @labels.size)
end

#reduceat_index ⇒ CArray

Segment start offsets into the category-contiguous layout produced by #sort_addr: reduceat_index is where category c's block begins, so block c spans reduceat_index[c] ... reduceat_index[c] + category_sizes[c]. Empty categories repeat the following start (a zero-width block). Length k, aligned to #labels; pair with #category_sizes for the block lengths. On data already laid out in category order the offsets index it directly.

Returns an int64 CArray, one entry per #labels, of segment start offsets aligned to #labels. Pair with #category_sizes for lengths.

Returns:



583
584
585
586
587
588
589
590
591
592
593
594
595
 
# File 'lib/carray/categorical.rb', line 583

def reduceat_index
  return @_reduceat_index if @_reduceat_index
  k = @labels.size
  out = CArray.int64(k > 0 ? k : 0)
  if k > 1
    out[0] = 0
    out[1..-1] = category_sizes.cumsum.int64[0..-2]
  elsif k == 1
    out[0] = 0
  end
  @_reduceat_index = out
  out
end

#sort_addr ⇒ CArray

Flat addresses that gather self into category-contiguous order: every cell of category 0 first, then 1, ..., then k-1, with excluded cells last. Built by a counting sort over the codes (O(n + k), stable): the segment starts (an exclusive prefix scan of #category_sizes) drive a scatter that places each source index into its category's block in source order; excluded cells (masked, or code out of range 0...k) are appended at the tail in source order. The first #category_sizes.sum addresses are exactly the classified cells in category order — CACategoricalIterator's permutation. For an N-D categorical the addresses are into the raveled storage, so value.reshape(elements)[cat.sort_addr] produces the contiguous blocks. Memoised (see the grouping-plan note above): the counting sort is the dominant grouping cost, so it is computed once and shared by every payload column and by CACategoricalIterator. Do not mutate the returned array.

Returns a length-CArray#elements integer CArray of flat storage addresses that order the cells by category (excluded cells last).

Returns:



543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
 
# File 'lib/carray/categorical.rb', line 543

def sort_addr
  return @_sort_addr if @_sort_addr
  n      = elements
  k      = @labels.size
  cs     = category_sizes                       # per-category counts
  nvalid = cs.sum
  cur    = CArray.int64(k > 0 ? k : 0)          # segment starts, consumed as cursor
  cur[1..-1] = cs.cumsum.int64[0..-2] if k > 1
  flat   = codes.reshape(n)
  seq    = CArray.int64(n).seq!                 # source indices, scattered as payload
  out    = CArray.int64(n)
  if nvalid == n
    flat.send(:__categorical_scatter__, seq, cur, out, k) # all valid: scatter straight in
  else
    if nvalid > 0
      valid = CArray.int64(nvalid)              # scatter target must be an entity, not a view
      flat.send(:__categorical_scatter__, seq, cur, valid, k)
      out[0...nvalid] = valid
    end
    # Excluded = every cell the scatter skips: code out of [0, k), OR masked
    # (a from_codes pre-masked cell keeps a valid code but is excluded). Both
    # must be caught or the tail slot count would not add up to n - nvalid.
    excluded = flat.value.ge(k).or(flat.value.lt(0))
    excluded = excluded.or(flat.is_masked) if flat.has_mask?
    out[nvalid..-1] = seq[excluded]             # excluded cells, source order
  end
  @_sort_addr = out
  out
end

#storage_to_scalar(raw) ⇒ Object

Face hook: decode a per-cell code into its category label, through the one decode every code-to-label path shares (see #label_at).



300
301
302
303
 
# File 'lib/carray/categorical.rb', line 300

def storage_to_scalar(raw)
  code = raw.is_a?(String) ? raw.unpack1(UNPACK_FORMAT.fetch(parent.data_type)) : raw
  label_at(code)
end

#union(other, sort: false) ⇒ CArray

Returns object CArray of the labels of either side.

Returns:

  • (CArray) —

    object CArray of the labels of either side.



446
447
448
 
# File 'lib/carray/categorical.rb', line 446

def union (other, sort: false)
  unique.union(label_array(other), sort: sort)
end

#unique(sort: false) ⇒ CArray

Returns the labels that occur, in first-appearance order. A category with no cells is not included (use #labels for the vocabulary).

Parameters:

  • sort (Boolean) (defaults to: false) —

    when true, sort the labels ascending.

Returns:

  • (CArray) —

    object CArray of labels.



389
390
391
392
 
# File 'lib/carray/categorical.rb', line 389

def unique (sort: false)
  u = labels_for(codes.unique)
  sort ? u.sort : u
end

#value_counts(sort: false) ⇒ Array(CArray, CArray)

Returns [labels, counts] for the categories that occur. #category_sizes is the aligned-to-#labels counterpart, which keeps the empty ones.

Parameters:

  • sort (false, :count, :value) (defaults to: false) —

    pair ordering.

Returns:



399
400
401
402
403
404
405
406
407
408
409
410
411
 
# File 'lib/carray/categorical.rb', line 399

def value_counts (sort: false)
  unless [false, :count, :value].include?(sort)
    raise ArgumentError, "value_counts: sort must be false, :count, or :value"
  end
  code_values, counts = codes.value_counts(sort: sort == :count ? :count : false)
  values = labels_for(code_values)
  return [values, counts] unless sort == :value
  # Ascending *label*: code order is the vocabulary's, so reorder here.
  la    = values.to_a
  order = (0...la.size).sort_by { |i| [la[i], i] }
  idx   = CArray.int64(order.size) { |i| order[i] }
  [values[idx], counts[idx]]
end