Class: DaruLite::DataFrame

Inherits:
Object show all
Extended by:
Gem::Deprecate
Includes:
Aggregatable, Calculatable, Convertible, Duplicatable, Fetchable, Filterable, IOAble, Indexable, Iterable, Joinable, Missable, Pivotable, Queryable, Setable, Sortable, Maths::Arithmetic::DataFrame, Maths::Statistics::DataFrame
Defined in:
lib/daru_lite/dataframe.rb,
lib/daru_lite/data_frame/setable.rb,
lib/daru_lite/data_frame/i_o_able.rb,
lib/daru_lite/data_frame/iterable.rb,
lib/daru_lite/data_frame/joinable.rb,
lib/daru_lite/data_frame/missable.rb,
lib/daru_lite/data_frame/sortable.rb,
lib/daru_lite/data_frame/fetchable.rb,
lib/daru_lite/data_frame/indexable.rb,
lib/daru_lite/data_frame/pivotable.rb,
lib/daru_lite/data_frame/queryable.rb,
lib/daru_lite/extensions/which_dsl.rb,
lib/daru_lite/data_frame/filterable.rb,
lib/daru_lite/data_frame/convertible.rb,
lib/daru_lite/data_frame/aggregatable.rb,
lib/daru_lite/data_frame/calculatable.rb,
lib/daru_lite/data_frame/duplicatable.rb

Overview

rubocop:disable Metrics/ClassLength

Defined Under Namespace

Modules: Aggregatable, Calculatable, Convertible, Duplicatable, Fetchable, Filterable, IOAble, Indexable, Iterable, Joinable, Missable, Pivotable, Queryable, Setable, Sortable

Constant Summary collapse

AXES = 
i[row vector].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Maths::Statistics::DataFrame

#acf, #correlation, #count, #covariance, #cumsum, #describe, #ema, #max, #mean, #median, #min, #mode, #percent_change, #product, #range, #rolling_count, #rolling_max, #rolling_mean, #rolling_median, #rolling_min, #rolling_std, #rolling_variance, #standardize, #std, #sum, #variance_sample

Methods included from Maths::Arithmetic::DataFrame

#%, #*, #**, #+, #-, #/, #exp, #round, #sqrt

Methods included from Queryable

#all?, #any?, #has_vector?, #include_values?

Methods included from Sortable

#order=, #rotate_vectors, #sort, #sort!

Methods included from Setable

#[]=, #add_row, #add_vector, #insert_vector, #set_at, #set_row_at

Methods included from Pivotable

#pivot_table

Methods included from Missable

#has_missing_data?, #missing_values_rows, #rolling_fillna, #rolling_fillna!

Methods included from Joinable

#concat, #join, #merge, #one_to_many, #union

Methods included from IOAble

#_dump, included, #save, #write_csv, #write_excel, #write_sql

Methods included from Iterable

#apply_method, #collect, #collect_matrix, #collect_row_with_index, #collect_rows, #collect_vector_with_index, #collect_vectors, #each, #each_index, #each_row, #each_row_with_index, #each_vector, #each_vector_with_index, #map, #map!, #map_rows, #map_rows!, #map_rows_with_index, #map_vectors, #map_vectors!, #map_vectors_with_index, #recode, #recode_rows, #recode_vectors, #replace_values, #verify

Methods included from Indexable

#index=, #reindex, #reindex_vectors, #reset_index, #set_index, #vectors=

Methods included from Filterable

#filter, #filter_rows, #filter_vector, #filter_vectors, #keep_row_if, #keep_vector_if, #reject_values, #uniq, #where

Methods included from Fetchable

#[], #access_row_tuples_by_indexs, #at, #get_sub_dataframe, #get_vector_anyways, #head, #numeric_vector_names, #numeric_vectors, #only_numerics, #row_at, #split_by_category, #tail

Methods included from Duplicatable

#clone, #clone_only_valid, #clone_structure, #dup, #dup_only_valid

Methods included from Convertible

#create_sql, #to_a, #to_df, #to_h, #to_html, #to_html_tbody, #to_html_thead, #to_json, #to_matrix, #to_s

Methods included from Calculatable

#compute, #summary, #vector_by_calculation, #vector_count_characters, #vector_mean, #vector_sum

Methods included from Aggregatable

#aggregate, #group_by, #group_by_and_aggregate

Constructor Details

#initialize(source = {}, opts = {}) ⇒ DataFrame

DataFrame basically consists of an Array of Vector objects. These objects are indexed by row and column by vectors and index Index objects.

Arguments

  • source - Source from the DataFrame is to be initialized. Can be a Hash

of names and vectors (array or DaruLite::Vector), an array of arrays or array of DaruLite::Vectors.

Options

:order - An Array/DaruLite::Index/DaruLite::MultiIndex containing the order in which Vectors should appear in the DataFrame.

:index - An Array/DaruLite::Index/DaruLite::MultiIndex containing the order in which rows of the DataFrame will be named.

:name - A name for the DataFrame.

:clone - Specify as true or false. When set to false, and Vector objects are passed for the source, the Vector objects will not duplicated when creating the DataFrame. Will have no effect if Array is passed in the source, or if the passed DaruLite::Vectors have different indexes. Default to true.

Usage

df = DaruLite::DataFrame.new
# =>
# <DaruLite::DataFrame(0x0)>
# Creates an empty DataFrame with no rows or columns.

df = DaruLite::DataFrame.new({}, order: [:a, :b])
#<DaruLite::DataFrame(0x2)>
  a   b
# Creates a DataFrame with no rows and columns :a and :b

df = DaruLite::DataFrame.new({a: [1,2,3,4], b: [6,7,8,9]}, order: [:b, :a],
  index: [:a, :b, :c, :d], name: :spider_man)

# =>
# <DaruLite::DataFrame:80766980 @name = spider_man @size = 4>
#             b          a
#  a          6          1
#  b          7          2
#  c          8          3
#  d          9          4

df = DaruLite::DataFrame.new([[1,2,3,4],[6,7,8,9]], name: :bat_man)

# =>
# #<DaruLite::DataFrame: bat_man (4x2)>
#             0          1
#  0          1          6
#  1          2          7
#  2          3          8
#  3          4          9

# Dataframe having Index name

df = DaruLite::DataFrame.new({a: [1,2,3,4], b: [6,7,8,9]}, order: [:b, :a],
  index: DaruLite::Index.new([:a, :b, :c, :d], name: 'idx_name'),
  name: :spider_man)

# =>
# <DaruLite::DataFrame:80766980 @name = spider_man @size = 4>
# idx_name            b          a
#        a          6          1
#        b          7          2
#        c          8          3
#        d          9          4

idx = DaruLite::Index.new [100, 99, 101, 1, 2], name: "s1"
=> #<DaruLite::Index(5): s1 {100, 99, 101, 1, 2}>

df = DaruLite::DataFrame.new({b: [11,12,13,14,15], a: [1,2,3,4,5],
  c: [11,22,33,44,55]},
  order: [:a, :b, :c],
  index: idx)
 # =>
 #<DaruLite::DataFrame(5x3)>
 #   s1   a   b   c
 #  100   1  11  11
 #   99   2  12  22
 #  101   3  13  33
 #    1   4  14  44
 #    2   5  15  55



228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
 
# File 'lib/daru_lite/dataframe.rb', line 228

def initialize(source = {}, opts = {})
  vectors = opts[:order]
  index = opts[:index] # FIXME: just keyword arges after Ruby 2.1
  @data = []
  @name = opts[:name]

  case source
  when [], {}
    create_empty_vectors(vectors, index)
  when Array
    initialize_from_array source, vectors, index, opts
  when Hash
    initialize_from_hash source, vectors, index, opts
  end

  set_size
  validate
  update
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args) ⇒ Object 



483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
 
# File 'lib/daru_lite/dataframe.rb', line 483

def method_missing(name, *args, &)
  stringified_name = name.to_s

  if /^([^=]+)=/.match?(stringified_name)
    name = stringified_name[/^([^=]+)=/].delete('=')
    name = name.to_sym unless has_vector?(name)
    insert_or_modify_vector [name], args[0]
  elsif has_vector?(name)
    self[name]
  elsif has_vector?(stringified_name)
    self[stringified_name]
  else
    super
  end
end

Instance Attribute Details

#data ⇒ Object (readonly)

TOREMOVE



128
129
130
 
# File 'lib/daru_lite/dataframe.rb', line 128

def data
  @data
end

#index ⇒ Object (readonly)

The index of the rows of the DataFrame



131
132
133
 
# File 'lib/daru_lite/dataframe.rb', line 131

def index
  @index
end

#name ⇒ Object (readonly)

The name of the DataFrame



134
135
136
 
# File 'lib/daru_lite/dataframe.rb', line 134

def name
  @name
end

#size ⇒ Object (readonly)

The number of rows present in the DataFrame



137
138
139
 
# File 'lib/daru_lite/dataframe.rb', line 137

def size
  @size
end

#vectors ⇒ Object (readonly)

The vectors (columns) index of the DataFrame



126
127
128
 
# File 'lib/daru_lite/dataframe.rb', line 126

def vectors
  @vectors
end

Class Method Details

.crosstab_by_assignation(rows, columns, values) ⇒ Object

Generates a new dataset, using three vectors

  • Rows

  • Columns

  • Values

For example, you have these values

x   y   v
a   a   0
a   b   1
b   a   1
b   b   0

You obtain

id  a   b
 a  0   1
 b  1   0

Useful to process outputs from databases



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/daru_lite/dataframe.rb', line 84

def crosstab_by_assignation(rows, columns, values)
  raise 'Three vectors should be equal size' if
    rows.size != columns.size || rows.size != values.size

  data = Hash.new do |h, col|
    h[col] = rows.factors.map { |r| [r, nil] }.to_h
  end
  columns.zip(rows, values).each { |c, r, v| data[c][r] = v }

  # FIXME: in fact, WITHOUT this line you'll obtain more "right"
  # data: with vectors having "rows" as an index...
  data = data.transform_values(&:values)
  data[:_id] = rows.factors

  DataFrame.new(data)
end

.rows(source, opts = {}) ⇒ Object

Create DataFrame by specifying rows as an Array of Arrays or Array of DaruLite::Vector objects.

Raises:



50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/daru_lite/dataframe.rb', line 50

def rows(source, opts = {})
  raise SizeError, 'All vectors must have same length' \
    unless source.all? { |v| v.size == source.first.size }

  opts[:order] ||= guess_order(source)

  if ArrayHelper.array_of?(source, Array) || source.empty?
    DataFrame.new(source.transpose, opts)
  elsif ArrayHelper.array_of?(source, Vector)
    from_vector_rows(source, opts)
  else
    raise ArgumentError, "Can't create DataFrame from #{source}"
  end
end

Instance Method Details

#==(other) ⇒ Object 



458
459
460
461
462
463
464
 
# File 'lib/daru_lite/dataframe.rb', line 458

def ==(other)
  self.class == other.class   &&
    @size    == other.size    &&
    @index   == other.index   &&
    @vectors == other.vectors &&
    @vectors.to_a.all? { |v| self[v] == other[v] }
end

#add_level_to_vectors(top_level_label) ⇒ Object

Converts the vectors to a DaruLite::MultiIndex. The argument passed is used as the MultiIndex’s top level



399
400
401
402
 
# File 'lib/daru_lite/dataframe.rb', line 399

def add_level_to_vectors(top_level_label)
  tuples = vectors.map { |label| [top_level_label, *label] }
  self.vectors = DaruLite::MultiIndex.from_tuples(tuples)
end

#add_vectors_by_split(name, join = '-', sep = DaruLite::SPLIT_TOKEN) ⇒ Object 



336
337
338
339
340
 
# File 'lib/daru_lite/dataframe.rb', line 336

def add_vectors_by_split(name, join = '-', sep = DaruLite::SPLIT_TOKEN)
  self[name]
    .split_by_separator(sep)
    .each { |k, v| self[:"#{name}#{join}#{k}"] = v }
end

#add_vectors_by_split_recode(nm, join = '-', sep = DaruLite::SPLIT_TOKEN) ⇒ Object 



404
405
406
407
408
409
410
411
 
# File 'lib/daru_lite/dataframe.rb', line 404

def add_vectors_by_split_recode(nm, join = '-', sep = DaruLite::SPLIT_TOKEN)
  self[nm]
    .split_by_separator(sep)
    .each_with_index do |(k, v), i|
      v.rename "#{nm}:#{k}"
      self[:"#{nm}#{join}#{i + 1}"] = v
    end
end

#bootstrap(n = nil) ⇒ DaruLite::DataFrame

Creates a DataFrame with the random data, of n size. If n not given, uses original number of rows.

Returns:



304
305
306
307
308
309
310
311
312
 
# File 'lib/daru_lite/dataframe.rb', line 304

def bootstrap(n = nil)
  n ||= nrows
  DaruLite::DataFrame.new({}, order: @vectors).tap do |df_boot|
    n.times do
      df_boot.add_row(row[rand(n)])
    end
    df_boot.update
  end
end

#delete_at_position(position) ⇒ Object

Delete a row based on its position More robust than #delete_row when working with a CategoricalIndex or when the Index includes integers

Raises:

  • (IndexError) 


291
292
293
294
295
296
297
298
 
# File 'lib/daru_lite/dataframe.rb', line 291

def delete_at_position(position)
  raise IndexError, "Position #{position} does not exist." unless position < size

  @index = @index.delete_at(position)
  each_vector { |vector| vector.delete_at_position(position) }

  set_size
end

#delete_row(index) ⇒ Object

Delete a row

Raises:

  • (IndexError) 


275
276
277
278
279
280
281
282
283
284
285
286
 
# File 'lib/daru_lite/dataframe.rb', line 275

def delete_row(index)
  idx = named_index_for index

  raise IndexError, "Index #{index} does not exist." unless @index.include? idx

  @index = DaruLite::Index.new(@index.to_a - [idx])
  each_vector do |vector|
    vector.delete_at idx
  end

  set_size
end

#delete_vector(vector) ⇒ Object

Delete a vector

Raises:

  • (IndexError) 


258
259
260
261
262
263
264
265
 
# File 'lib/daru_lite/dataframe.rb', line 258

def delete_vector(vector)
  raise IndexError, "Vector #{vector} does not exist." unless @vectors.include?(vector)

  @data.delete_at @vectors[vector]
  @vectors = DaruLite::Index.new @vectors.to_a - [vector]

  self
end

#delete_vectors(*vectors) ⇒ Object

Deletes a list of vectors



268
269
270
271
272
 
# File 'lib/daru_lite/dataframe.rb', line 268

def delete_vectors(*vectors)
  Array(vectors).each { |vec| delete_vector vec }

  self
end

#inspect(spacing = DaruLite.spacing, threshold = DaruLite.max_rows) ⇒ Object

Pretty print in a nice table format for the command line (irb/pry/iruby)



441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
 
# File 'lib/daru_lite/dataframe.rb', line 441

def inspect(spacing = DaruLite.spacing, threshold = DaruLite.max_rows)
  name_part = @name ? ": #{@name} " : ''
  spacing = [
    headers.to_a.map { |header| header.try(:length) || header.to_s.length }.max,
    spacing
  ].max

  "#<#{self.class}#{name_part}(#{nrows}x#{ncols})>#{$INPUT_RECORD_SEPARATOR}" +
    Formatters::Table.format(
      each_row.lazy,
      row_headers: row_headers,
      headers: headers,
      threshold: threshold,
      spacing: spacing
    )
end

#interact_code(vector_names, full) ⇒ Object 



503
504
505
506
507
508
509
510
511
 
# File 'lib/daru_lite/dataframe.rb', line 503

def interact_code(vector_names, full)
  dfs = vector_names.zip(full).map do |vec_name, f|
    self[vec_name].contrast_code(full: f).each.to_a
  end

  all_vectors = recursive_product(dfs)
  DaruLite::DataFrame.new all_vectors,
                          order: all_vectors.map(&:name)
end

#ncols ⇒ Object

The number of vectors



353
354
355
 
# File 'lib/daru_lite/dataframe.rb', line 353

def ncols
  @vectors.size
end

#nest(*tree_keys, &block) ⇒ Object

Return a nested hash using vector names as keys and an array constructed of hashes with other values. If block provided, is used to provide the values, with parameters row of dataset, current last hash on hierarchy and name of the key to include



318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
 
# File 'lib/daru_lite/dataframe.rb', line 318

def nest(*tree_keys, &block)
  tree_keys = tree_keys[0] if tree_keys[0].is_a? Array

  each_row.with_object({}) do |row, current|
    # Create tree
    *keys, last = tree_keys
    current = keys.inject(current) { |c, f| c[row[f]] ||= {} }
    name = row[last]

    if block
      current[name] = yield(row, current, name)
    else
      current[name] ||= []
      current[name].push(row.to_h.delete_if { |key, _value| tree_keys.include? key })
    end
  end
end

#nrows ⇒ Object

The number of rows



348
349
350
 
# File 'lib/daru_lite/dataframe.rb', line 348

def nrows
  @index.size
end

#rename(new_name) ⇒ Object Also known as: name=

Rename the DataFrame.



423
424
425
426
 
# File 'lib/daru_lite/dataframe.rb', line 423

def rename(new_name)
  @name = new_name
  self
end

#rename_vectors(name_map) ⇒ Object

Renames the vectors

Arguments

  • name_map - A hash where the keys are the exising vector names and

    the values are the new names.  If a vector is renamed
to a vector name that is already in use, the existing
one is overwritten.

Usage

df = DaruLite::DataFrame.new({ a: [1,2,3,4], b: [:a,:b,:c,:d], c: [11,22,33,44] })
df.rename_vectors :a => :alpha, :c => :gamma
df.vectors.to_a #=> [:alpha, :b, :gamma]



371
372
373
374
375
376
377
 
# File 'lib/daru_lite/dataframe.rb', line 371

def rename_vectors(name_map)
  existing_targets = name_map.reject { |k, v| k == v }.values & vectors.to_a
  delete_vectors(*existing_targets)

  new_names = vectors.to_a.map { |v| name_map[v] || v }
  self.vectors = DaruLite::Index.new new_names
end

#rename_vectors!(name_map) ⇒ Object

Renames the vectors and returns itself

Arguments

  • name_map - A hash where the keys are the exising vector names and

    the values are the new names.  If a vector is renamed
to a vector name that is already in use, the existing
one is overwritten.

Usage

df = DaruLite::DataFrame.new({ a: [1,2,3,4], b: [:a,:b,:c,:d], c: [11,22,33,44] })
df.rename_vectors! :a => :alpha, :c => :gamma # df



392
393
394
395
 
# File 'lib/daru_lite/dataframe.rb', line 392

def rename_vectors!(name_map)
  rename_vectors(name_map)
  self
end

#respond_to_missing?(name, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


499
500
501
 
# File 'lib/daru_lite/dataframe.rb', line 499

def respond_to_missing?(name, include_private = false)
  name.to_s.end_with?('=') || has_vector?(name) || super
end

#row ⇒ Object

Access a row or set/create a row. Refer #[] and #[]= docs for details.

Usage

df.row[:a] # access row named ':a'
df.row[:b] = [1,2,3] # set row ':b' to [1,2,3]



253
254
255
 
# File 'lib/daru_lite/dataframe.rb', line 253

def row
  DaruLite::Accessors::DataFrameByRow.new(self)
end

#shape ⇒ Object

Return the number of rows and columns of the DataFrame in an Array.



343
344
345
 
# File 'lib/daru_lite/dataframe.rb', line 343

def shape
  [nrows, ncols]
end

#to_category(*names) ⇒ DaruLite::DataFrame

Converts the specified non category type vectors to category type vectors

Examples:

df = DaruLite::DataFrame.new({
  a: [1, 2, 3],
  b: ['a', 'a', 'b']
})
df.to_category :b
df[:b].type
# => :category

Parameters:

  • names (Array) —

    of non category type vectors to be converted

Returns:

  • (DaruLite::DataFrame) —

    data frame in which specified vectors have been converted to category type



478
479
480
481
 
# File 'lib/daru_lite/dataframe.rb', line 478

def to_category(*names)
  names.each { |n| self[n] = self[n].to_category }
  self
end

#transpose ⇒ Object

Transpose a DataFrame, tranposing elements and row, column indexing.



430
431
432
433
434
435
436
437
438
 
# File 'lib/daru_lite/dataframe.rb', line 430

def transpose
  DaruLite::DataFrame.new(
    each_vector.map(&:to_a).transpose,
    index: @vectors,
    order: @index,
    dtype: @dtype,
    name: @name
  )
end

#update ⇒ Object

Method for updating the metadata (i.e. missing value positions) of the after assingment/deletion etc. are complete. This is provided so that time is not wasted in creating the metadata for the vector each time assignment/deletion of elements is done. Updating data this way is called lazy loading. To set or unset lazy loading, see the .lazy_update= method.



418
419
420
 
# File 'lib/daru_lite/dataframe.rb', line 418

def update
  @data.each(&:update) if DaruLite.lazy_update
end

#which ⇒ Object

a simple query DSL for accessing where(), inspired by gem “squeel” e.g.: df.which{ FamilySize == FamilySize.max } equals df.where( df.eq( df.max ) )

e.g.: df.which{ (NameTitle == ‘Dr’) & (Sex == ‘female’) } equals df.where( df.eq(‘Dr’) & df.eq(‘female’) )



15
16
17
 
# File 'lib/daru_lite/extensions/which_dsl.rb', line 15

def which(&)
  WhichQuery.new(self, &).exec
end