Class: Polars::DataFrame
- Inherits:
-
Object
- Object
- Polars::DataFrame
- Includes:
- Plot
- Defined in:
- lib/polars/data_frame.rb
Overview
Two-dimensional data structure representing data as a table with rows and columns.
Instance Method Summary collapse
-
#!=(other) ⇒ DataFrame
Not equal.
-
#%(other) ⇒ DataFrame
Returns the modulo.
-
#*(other) ⇒ DataFrame
Performs multiplication.
-
#+(other) ⇒ DataFrame
Performs addition.
-
#-(other) ⇒ DataFrame
Performs subtraction.
-
#/(other) ⇒ DataFrame
Performs division.
-
#<(other) ⇒ DataFrame
Less than.
-
#<=(other) ⇒ DataFrame
Less than or equal.
-
#==(other) ⇒ DataFrame
Equal.
-
#>(other) ⇒ DataFrame
Greater than.
-
#>=(other) ⇒ DataFrame
Greater than or equal.
-
#[](*args) ⇒ Object
Returns subset of the DataFrame.
-
#[]=(*key, value) ⇒ Object
Set item.
-
#apply(return_dtype: nil, inference_size: 256, &f) ⇒ Object
Apply a custom/user-defined function (UDF) over the rows of the DataFrame.
-
#clear(n = 0) ⇒ DataFrame
(also: #cleared)
Create an empty copy of the current DataFrame.
-
#columns ⇒ Array
Get column names.
-
#columns=(columns) ⇒ Object
Change the column names of the DataFrame.
-
#delete(name) ⇒ Series
Drop in place if exists.
-
#describe ⇒ DataFrame
Summary statistics for a DataFrame.
-
#drop(columns) ⇒ DataFrame
Remove column from DataFrame and return as new.
-
#drop_in_place(name) ⇒ Series
Drop in place.
-
#drop_nulls(subset: nil) ⇒ DataFrame
Return a new DataFrame where the null values are dropped.
-
#dtypes ⇒ Array
Get dtypes of columns in DataFrame.
-
#each(&block) ⇒ Object
Returns an enumerator.
-
#each_row(named: true, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
-
#equals(other, null_equal: true) ⇒ Boolean
(also: #frame_equal)
Check if DataFrame is equal to other.
-
#estimated_size(unit = "b") ⇒ Numeric
Return an estimation of the total (heap) allocated size of the DataFrame.
-
#explode(columns) ⇒ DataFrame
Explode
DataFrame
to long format by exploding a column with Lists. -
#extend(other) ⇒ DataFrame
Extend the memory backed by this
DataFrame
with the values fromother
. -
#fill_nan(fill_value) ⇒ DataFrame
Fill floating point NaN values by an Expression evaluation.
-
#fill_null(value = nil, strategy: nil, limit: nil, matches_supertype: true) ⇒ DataFrame
Fill null values using the specified value or strategy.
-
#filter(predicate) ⇒ DataFrame
Filter the rows in the DataFrame based on a predicate expression.
-
#fold(&operation) ⇒ Series
Apply a horizontal reduction on a DataFrame.
-
#gather_every(n, offset = 0) ⇒ DataFrame
(also: #take_every)
Take every nth row in the DataFrame and return as a new DataFrame.
-
#get_column(name) ⇒ Series
Get a single column as Series by name.
-
#get_column_index(name) ⇒ Series
(also: #find_idx_by_name)
Find the index of a column by name.
-
#get_columns ⇒ Array
Get the DataFrame as a Array of Series.
-
#group_by(by, maintain_order: false) ⇒ GroupBy
(also: #groupby, #group)
Start a group by operation.
-
#group_by_dynamic(index_column, every:, period: nil, offset: nil, truncate: true, include_boundaries: false, closed: "left", by: nil, start_by: "window") ⇒ DataFrame
(also: #groupby_dynamic)
Group based on a time value (or index value of type
:i32
,:i64
). -
#group_by_rolling(index_column:, period:, offset: nil, closed: "right", by: nil, check_sorted: true) ⇒ RollingGroupBy
(also: #groupby_rolling)
Create rolling groups based on a time column.
-
#hash_rows(seed: 0, seed_1: nil, seed_2: nil, seed_3: nil) ⇒ Series
Hash and combine the rows in this DataFrame.
-
#head(n = 5) ⇒ DataFrame
Get the first
n
rows. -
#height ⇒ Integer
(also: #count, #length, #size)
Get the height of the DataFrame.
-
#hstack(columns, in_place: false) ⇒ DataFrame
Return a new DataFrame grown horizontally by stacking multiple Series to it.
-
#include?(name) ⇒ Boolean
Check if DataFrame includes column.
-
#initialize(data = nil, schema: nil, columns: nil, schema_overrides: nil, orient: nil, infer_schema_length: 100, nan_to_null: false) ⇒ DataFrame
constructor
Create a new DataFrame.
-
#insert_column(index, series) ⇒ DataFrame
(also: #insert_at_idx)
Insert a Series at a certain column index.
-
#interpolate ⇒ DataFrame
Interpolate intermediate values.
-
#is_duplicated ⇒ Series
Get a mask of all duplicated rows in this DataFrame.
-
#is_empty ⇒ Boolean
(also: #empty?)
Check if the dataframe is empty.
-
#is_unique ⇒ Series
Get a mask of all unique rows in this DataFrame.
-
#item ⇒ Object
Return the dataframe as a scalar.
-
#iter_rows(named: false, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
-
#join(other, left_on: nil, right_on: nil, on: nil, how: "inner", suffix: "_right", join_nulls: false) ⇒ DataFrame
Join in SQL-like fashion.
-
#join_asof(other, left_on: nil, right_on: nil, on: nil, by_left: nil, by_right: nil, by: nil, strategy: "backward", suffix: "_right", tolerance: nil, allow_parallel: true, force_parallel: false) ⇒ DataFrame
Perform an asof join.
-
#lazy ⇒ LazyFrame
Start a lazy query from this point.
-
#limit(n = 5) ⇒ DataFrame
Get the first
n
rows. -
#max(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their maximum value.
-
#mean(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their mean value.
-
#median ⇒ DataFrame
Aggregate the columns of this DataFrame to their median value.
-
#melt(id_vars: nil, value_vars: nil, variable_name: nil, value_name: nil) ⇒ DataFrame
Unpivot a DataFrame from wide to long format.
-
#merge_sorted(other, key) ⇒ DataFrame
Take two sorted DataFrames and merge them by the sorted key.
-
#min(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their minimum value.
-
#n_chunks(strategy: "first") ⇒ Object
Get number of chunks used by the ChunkedArrays of this DataFrame.
-
#n_unique(subset: nil) ⇒ DataFrame
Return the number of unique rows, or the number of unique row-subsets.
-
#null_count ⇒ DataFrame
Create a new DataFrame that shows the null counts per column.
-
#partition_by(groups, maintain_order: true, include_key: true, as_dict: false) ⇒ Object
Split into multiple DataFrames partitioned by groups.
-
#pipe(func, *args, **kwargs, &block) ⇒ Object
Offers a structured way to apply a sequence of user-defined functions (UDFs).
-
#pivot(values:, index:, columns:, aggregate_fn: "first", maintain_order: true, sort_columns: false, separator: "_") ⇒ DataFrame
Create a spreadsheet-style pivot table as a DataFrame.
-
#plot(x = nil, y = nil, type: nil, group: nil, stacked: nil) ⇒ Vega::LiteChart
included
from Plot
Plot data.
-
#product ⇒ DataFrame
Aggregate the columns of this DataFrame to their product values.
-
#quantile(quantile, interpolation: "nearest") ⇒ DataFrame
Aggregate the columns of this DataFrame to their quantile value.
-
#rechunk ⇒ DataFrame
This will make sure all subsequent operations have optimal and predictable performance.
-
#rename(mapping) ⇒ DataFrame
Rename column names.
-
#replace(column, new_col) ⇒ DataFrame
Replace a column by a new Series.
-
#replace_column(index, series) ⇒ DataFrame
(also: #replace_at_idx)
Replace a column at an index location.
-
#reverse ⇒ DataFrame
Reverse the DataFrame.
-
#row(index = nil, by_predicate: nil, named: false) ⇒ Object
Get a row as tuple, either by index or by predicate.
-
#rows(named: false) ⇒ Array
Convert columnar data to rows as Ruby arrays.
-
#sample(n: nil, frac: nil, with_replacement: false, shuffle: false, seed: nil) ⇒ DataFrame
Sample from this DataFrame.
-
#schema ⇒ Hash
Get the schema.
-
#select(*exprs, **named_exprs) ⇒ DataFrame
Select columns from this DataFrame.
-
#set_sorted(column, *more_columns, descending: false) ⇒ DataFrame
Indicate that one or multiple columns are sorted.
-
#shape ⇒ Array
Get the shape of the DataFrame.
-
#shift(n, fill_value: nil) ⇒ DataFrame
Shift values by the given period.
-
#shift_and_fill(periods, fill_value) ⇒ DataFrame
Shift the values by a given period and fill the resulting null values.
-
#shrink_to_fit(in_place: false) ⇒ DataFrame
Shrink DataFrame memory usage.
-
#slice(offset, length = nil) ⇒ DataFrame
Get a slice of this DataFrame.
-
#sort(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column.
-
#sort!(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column in-place.
-
#std(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their standard deviation value.
-
#sum(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their sum value.
-
#tail(n = 5) ⇒ DataFrame
Get the last
n
rows. -
#to_a ⇒ Array
Returns an array representing the DataFrame.
-
#to_csv(**options) ⇒ String
Write to comma-separated values (CSV) string.
-
#to_dummies(columns: nil, separator: "_", drop_first: false) ⇒ DataFrame
Get one hot encoded dummy variables.
-
#to_h(as_series: true) ⇒ Hash
Convert DataFrame to a hash mapping column name to values.
-
#to_hashes ⇒ Array
Convert every row to a dictionary.
-
#to_numo ⇒ Numo::NArray
Convert DataFrame to a 2D Numo array.
-
#to_s ⇒ String
(also: #inspect)
Returns a string representing the DataFrame.
-
#to_series(index = 0) ⇒ Series
Select column as Series at index location.
-
#to_struct(name) ⇒ Series
Convert a
DataFrame
to aSeries
of typeStruct
. -
#transpose(include_header: false, header_name: "column", column_names: nil) ⇒ DataFrame
Transpose a DataFrame over the diagonal.
-
#unique(maintain_order: true, subset: nil, keep: "first") ⇒ DataFrame
Drop duplicate rows from this DataFrame.
-
#unnest(names) ⇒ DataFrame
Decompose a struct into its fields.
-
#unstack(step:, how: "vertical", columns: nil, fill_values: nil) ⇒ DataFrame
Unstack a long table to a wide form without doing an aggregation.
-
#upsample(time_column:, every:, offset: nil, by: nil, maintain_order: false) ⇒ DataFrame
Upsample a DataFrame at a regular frequency.
-
#var(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their variance value.
-
#vstack(df, in_place: false) ⇒ DataFrame
Grow this DataFrame vertically by stacking a DataFrame to it.
-
#width ⇒ Integer
Get the width of the DataFrame.
-
#with_column(column) ⇒ DataFrame
Return a new DataFrame with the column added or replaced.
-
#with_columns(*exprs, **named_exprs) ⇒ DataFrame
Add columns to this DataFrame.
-
#with_row_index(name: "row_nr", offset: 0) ⇒ DataFrame
(also: #with_row_count)
Add a column at index 0 that counts the rows.
-
#write_avro(file, compression = "uncompressed") ⇒ nil
Write to Apache Avro file.
-
#write_csv(file = nil, has_header: true, include_header: nil, sep: ",", quote: '"', batch_size: 1024, datetime_format: nil, date_format: nil, time_format: nil, float_precision: nil, null_value: nil) ⇒ String?
Write to comma-separated values (CSV) file.
-
#write_ipc(file, compression: "uncompressed") ⇒ nil
Write to Arrow IPC binary stream or Feather file.
-
#write_json(file, pretty: false, row_oriented: false) ⇒ nil
Serialize to JSON representation.
-
#write_ndjson(file) ⇒ nil
Serialize to newline delimited JSON representation.
-
#write_parquet(file, compression: "zstd", compression_level: nil, statistics: false, row_group_size: nil) ⇒ nil
Write to Apache Parquet file.
Constructor Details
#initialize(data = nil, schema: nil, columns: nil, schema_overrides: nil, orient: nil, infer_schema_length: 100, nan_to_null: false) ⇒ DataFrame
Create a new DataFrame.
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
# File 'lib/polars/data_frame.rb', line 21 def initialize(data = nil, schema: nil, columns: nil, schema_overrides: nil, orient: nil, infer_schema_length: 100, nan_to_null: false) schema ||= columns if defined?(ActiveRecord) && (data.is_a?(ActiveRecord::Relation) || data.is_a?(ActiveRecord::Result)) raise ArgumentError, "Use read_database instead" end if data.nil? self._df = self.class.hash_to_rbdf({}, schema: schema, schema_overrides: schema_overrides) elsif data.is_a?(Hash) data = data.transform_keys { |v| v.is_a?(Symbol) ? v.to_s : v } self._df = self.class.hash_to_rbdf(data, schema: schema, schema_overrides: schema_overrides, nan_to_null: nan_to_null) elsif data.is_a?(::Array) self._df = self.class.sequence_to_rbdf(data, schema: schema, schema_overrides: schema_overrides, orient: orient, infer_schema_length: infer_schema_length) elsif data.is_a?(Series) self._df = self.class.series_to_rbdf(data, schema: schema, schema_overrides: schema_overrides) else raise ArgumentError, "DataFrame constructor called with unsupported type; got #{data.class.name}" end end |
Instance Method Details
#!=(other) ⇒ DataFrame
Not equal.
447 448 449 |
# File 'lib/polars/data_frame.rb', line 447 def !=(other) _comp(other, "neq") end |
#%(other) ⇒ DataFrame
Returns the modulo.
530 531 532 533 534 535 536 537 |
# File 'lib/polars/data_frame.rb', line 530 def %(other) if other.is_a?(DataFrame) return _from_rbdf(_df.rem_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.rem(other._s)) end |
#*(other) ⇒ DataFrame
Performs multiplication.
482 483 484 485 486 487 488 489 |
# File 'lib/polars/data_frame.rb', line 482 def *(other) if other.is_a?(DataFrame) return _from_rbdf(_df.mul_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.mul(other._s)) end |
#+(other) ⇒ DataFrame
Performs addition.
506 507 508 509 510 511 512 513 |
# File 'lib/polars/data_frame.rb', line 506 def +(other) if other.is_a?(DataFrame) return _from_rbdf(_df.add_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.add(other._s)) end |
#-(other) ⇒ DataFrame
Performs subtraction.
518 519 520 521 522 523 524 525 |
# File 'lib/polars/data_frame.rb', line 518 def -(other) if other.is_a?(DataFrame) return _from_rbdf(_df.sub_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.sub(other._s)) end |
#/(other) ⇒ DataFrame
Performs division.
494 495 496 497 498 499 500 501 |
# File 'lib/polars/data_frame.rb', line 494 def /(other) if other.is_a?(DataFrame) return _from_rbdf(_df.div_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.div(other._s)) end |
#<(other) ⇒ DataFrame
Less than.
461 462 463 |
# File 'lib/polars/data_frame.rb', line 461 def <(other) _comp(other, "lt") end |
#<=(other) ⇒ DataFrame
Less than or equal.
475 476 477 |
# File 'lib/polars/data_frame.rb', line 475 def <=(other) _comp(other, "lt_eq") end |
#==(other) ⇒ DataFrame
Equal.
440 441 442 |
# File 'lib/polars/data_frame.rb', line 440 def ==(other) _comp(other, "eq") end |
#>(other) ⇒ DataFrame
Greater than.
454 455 456 |
# File 'lib/polars/data_frame.rb', line 454 def >(other) _comp(other, "gt") end |
#>=(other) ⇒ DataFrame
Greater than or equal.
468 469 470 |
# File 'lib/polars/data_frame.rb', line 468 def >=(other) _comp(other, "gt_eq") end |
#[](*args) ⇒ Object
Returns subset of the DataFrame.
571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 |
# File 'lib/polars/data_frame.rb', line 571 def [](*args) if args.size == 2 row_selection, col_selection = args # df[.., unknown] if row_selection.is_a?(Range) # multiple slices # df[.., ..] if col_selection.is_a?(Range) raise Todo end end # df[2, ..] (select row as df) if row_selection.is_a?(Integer) if col_selection.is_a?(::Array) df = self[0.., col_selection] return df.slice(row_selection, 1) end # df[2, "a"] if col_selection.is_a?(::String) || col_selection.is_a?(Symbol) return self[col_selection][row_selection] end end # column selection can be "a" and ["a", "b"] if col_selection.is_a?(::String) || col_selection.is_a?(Symbol) col_selection = [col_selection] end # df[.., 1] if col_selection.is_a?(Integer) series = to_series(col_selection) return series[row_selection] end if col_selection.is_a?(::Array) # df[.., [1, 2]] if Utils.is_int_sequence(col_selection) series_list = col_selection.map { |i| to_series(i) } df = self.class.new(series_list) return df[row_selection] end end df = self[col_selection] return df[row_selection] elsif args.size == 1 item = args[0] # select single column # df["foo"] if item.is_a?(::String) || item.is_a?(Symbol) return Utils.wrap_s(_df.get_column(item.to_s)) end # df[idx] if item.is_a?(Integer) return slice(_pos_idx(item, 0), 1) end # df[..] if item.is_a?(Range) return Slice.new(self).apply(item) end if item.is_a?(::Array) && item.all? { |v| Utils.strlike?(v) } # select multiple columns # df[["foo", "bar"]] return _from_rbdf(_df.select(item.map(&:to_s))) end if Utils.is_int_sequence(item) item = Series.new("", item) end if item.is_a?(Series) dtype = item.dtype if dtype == String return _from_rbdf(_df.select(item)) elsif dtype == UInt32 return _from_rbdf(_df.take_with_series(item._s)) elsif [UInt8, UInt16, UInt64, Int8, Int16, Int32, Int64].include?(dtype) return _from_rbdf( _df.take_with_series(_pos_idxs(item, 0)._s) ) end end end # Ruby-specific if item.is_a?(Expr) || item.is_a?(Series) return filter(item) end raise ArgumentError, "Cannot get item of type: #{item.class.name}" end |
#[]=(*key, value) ⇒ Object
Set item.
673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 |
# File 'lib/polars/data_frame.rb', line 673 def []=(*key, value) if key.length == 1 key = key.first elsif key.length != 2 raise ArgumentError, "wrong number of arguments (given #{key.length + 1}, expected 2..3)" end if Utils.strlike?(key) if value.is_a?(::Array) || (defined?(Numo::NArray) && value.is_a?(Numo::NArray)) value = Series.new(value) elsif !value.is_a?(Series) value = Polars.lit(value) end self._df = with_column(value.alias(key.to_s))._df elsif key.is_a?(::Array) row_selection, col_selection = key if Utils.strlike?(col_selection) s = self[col_selection] elsif col_selection.is_a?(Integer) raise Todo else raise ArgumentError, "column selection not understood: #{col_selection}" end s[row_selection] = value if col_selection.is_a?(Integer) replace_column(col_selection, s) elsif Utils.strlike?(col_selection) replace(col_selection, s) end else raise Todo end end |
#apply(return_dtype: nil, inference_size: 256, &f) ⇒ Object
The frame-level apply
cannot track column names (as the UDF is a black-box
that may arbitrarily drop, rearrange, transform, or add new columns); if you
want to apply a UDF such that column names are preserved, you should use the
expression-level apply
syntax instead.
Apply a custom/user-defined function (UDF) over the rows of the DataFrame.
The UDF will receive each row as a tuple of values: udf(row)
.
Implementing logic using a Ruby function is almost always significantly slower and more memory intensive than implementing the same logic using the native expression API because:
- The native expression engine runs in Rust; UDFs run in Ruby.
- Use of Ruby UDFs forces the DataFrame to be materialized in memory.
- Polars-native expressions can be parallelised (UDFs cannot).
- Polars-native expressions can be logically optimised (UDFs cannot).
Wherever possible you should strongly prefer the native expression API to achieve the best performance.
2595 2596 2597 2598 2599 2600 2601 2602 |
# File 'lib/polars/data_frame.rb', line 2595 def apply(return_dtype: nil, inference_size: 256, &f) out, is_df = _df.apply(f, return_dtype, inference_size) if is_df _from_rbdf(out) else _from_rbdf(Utils.wrap_s(out).to_frame._df) end end |
#clear(n = 0) ⇒ DataFrame Also known as: cleared
Create an empty copy of the current DataFrame.
Returns a DataFrame with identical schema but no data.
2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 |
# File 'lib/polars/data_frame.rb', line 2896 def clear(n = 0) if n == 0 _from_rbdf(_df.clear) elsif n > 0 || len > 0 self.class.new( schema.to_h { |nm, tp| [nm, Series.new(nm, [], dtype: tp).extend_constant(nil, n)] } ) else clone end end |
#columns ⇒ Array
Get column names.
364 365 366 |
# File 'lib/polars/data_frame.rb', line 364 def columns _df.columns end |
#columns=(columns) ⇒ Object
Change the column names of the DataFrame.
397 398 399 |
# File 'lib/polars/data_frame.rb', line 397 def columns=(columns) _df.set_column_names(columns) end |
#delete(name) ⇒ Series
Drop in place if exists.
2856 2857 2858 |
# File 'lib/polars/data_frame.rb', line 2856 def delete(name) drop_in_place(name) if include?(name) end |
#describe ⇒ DataFrame
Summary statistics for a DataFrame.
1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 |
# File 'lib/polars/data_frame.rb', line 1341 def describe describe_cast = lambda do |stat| columns = [] self.columns.each_with_index do |s, i| if self[s].is_numeric || self[s].is_boolean columns << stat[0.., i].cast(:f64) else # for dates, strings, etc, we cast to string so that all # statistics can be shown columns << stat[0.., i].cast(:str) end end self.class.new(columns) end summary = _from_rbdf( Polars.concat( [ describe_cast.( self.class.new(columns.to_h { |c| [c, [height]] }) ), describe_cast.(null_count), describe_cast.(mean), describe_cast.(std), describe_cast.(min), describe_cast.(max), describe_cast.(median) ] )._df ) summary.insert_column( 0, Polars::Series.new( "describe", ["count", "null_count", "mean", "std", "min", "max", "median"], ) ) summary end |
#drop(columns) ⇒ DataFrame
Remove column from DataFrame and return as new.
2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 |
# File 'lib/polars/data_frame.rb', line 2810 def drop(columns) if columns.is_a?(::Array) df = clone columns.each do |n| df._df.drop_in_place(n) end df else _from_rbdf(_df.drop(columns)) end end |
#drop_in_place(name) ⇒ Series
Drop in place.
2846 2847 2848 |
# File 'lib/polars/data_frame.rb', line 2846 def drop_in_place(name) Utils.wrap_s(_df.drop_in_place(name)) end |
#drop_nulls(subset: nil) ⇒ DataFrame
Return a new DataFrame where the null values are dropped.
1722 1723 1724 1725 1726 1727 |
# File 'lib/polars/data_frame.rb', line 1722 def drop_nulls(subset: nil) if subset.is_a?(::String) subset = [subset] end _from_rbdf(_df.drop_nulls(subset)) end |
#dtypes ⇒ Array
Get dtypes of columns in DataFrame. Dtypes can also be found in column headers when printing the DataFrame.
415 416 417 |
# File 'lib/polars/data_frame.rb', line 415 def dtypes _df.dtypes end |
#each(&block) ⇒ Object
Returns an enumerator.
564 565 566 |
# File 'lib/polars/data_frame.rb', line 564 def each(&block) get_columns.each(&block) end |
#each_row(named: true, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
4671 4672 4673 |
# File 'lib/polars/data_frame.rb', line 4671 def each_row(named: true, buffer_size: 500, &block) iter_rows(named: named, buffer_size: buffer_size, &block) end |
#equals(other, null_equal: true) ⇒ Boolean Also known as: frame_equal
Check if DataFrame is equal to other.
1534 1535 1536 |
# File 'lib/polars/data_frame.rb', line 1534 def equals(other, null_equal: true) _df.equals(other._df, null_equal) end |
#estimated_size(unit = "b") ⇒ Numeric
Return an estimation of the total (heap) allocated size of the DataFrame.
Estimated size is given in the specified unit (bytes by default).
This estimation is the sum of the size of its buffers, validity, including nested arrays. Multiple arrays may share buffers and bitmaps. Therefore, the size of 2 arrays is not the sum of the sizes computed from this function. In particular, StructArray's size is an upper bound.
When an array is sliced, its allocated size remains constant because the buffer unchanged. However, this function will yield a smaller number. This is because this function returns the visible size of the buffer, not its total capacity.
FFI buffers are included in this estimation.
1088 1089 1090 1091 |
# File 'lib/polars/data_frame.rb', line 1088 def estimated_size(unit = "b") sz = _df.estimated_size Utils.scale_bytes(sz, to: unit) end |
#explode(columns) ⇒ DataFrame
Explode DataFrame
to long format by exploding a column with Lists.
3094 3095 3096 |
# File 'lib/polars/data_frame.rb', line 3094 def explode(columns) lazy.explode(columns).collect(no_optimization: true) end |
#extend(other) ⇒ DataFrame
Extend the memory backed by this DataFrame
with the values from other
.
Different from vstack
which adds the chunks from other
to the chunks of this
DataFrame
extend
appends the data from other
to the underlying memory
locations and thus may cause a reallocation.
If this does not cause a reallocation, the resulting data structure will not have any extra chunks and thus will yield faster queries.
Prefer extend
over vstack
when you want to do a query after a single append.
For instance during online operations where you add n
rows and rerun a query.
Prefer vstack
over extend
when you want to append many times before doing a
query. For instance when you read in multiple files and when to store them in a
single DataFrame
. In the latter case, finish the sequence of vstack
operations with a rechunk
.
2778 2779 2780 2781 |
# File 'lib/polars/data_frame.rb', line 2778 def extend(other) _df.extend(other._df) self end |
#fill_nan(fill_value) ⇒ DataFrame
Note that floating point NaNs (Not a Number) are not missing values!
To replace missing values, use fill_null
.
Fill floating point NaN values by an Expression evaluation.
3059 3060 3061 |
# File 'lib/polars/data_frame.rb', line 3059 def fill_nan(fill_value) lazy.fill_nan(fill_value).collect(no_optimization: true) end |
#fill_null(value = nil, strategy: nil, limit: nil, matches_supertype: true) ⇒ DataFrame
Fill null values using the specified value or strategy.
3019 3020 3021 3022 3023 3024 3025 3026 |
# File 'lib/polars/data_frame.rb', line 3019 def fill_null(value = nil, strategy: nil, limit: nil, matches_supertype: true) _from_rbdf( lazy .fill_null(value, strategy: strategy, limit: limit, matches_supertype: matches_supertype) .collect(no_optimization: true) ._df ) end |
#filter(predicate) ⇒ DataFrame
Filter the rows in the DataFrame based on a predicate expression.
1307 1308 1309 |
# File 'lib/polars/data_frame.rb', line 1307 def filter(predicate) lazy.filter(predicate).collect end |
#fold(&operation) ⇒ Series
Apply a horizontal reduction on a DataFrame.
This can be used to effectively determine aggregations on a row level, and can be applied to any DataType that can be supercasted (casted to a similar parent type).
An example of the supercast rules when applying an arithmetic operation on two DataTypes are for instance:
i8 + str = str f32 + i64 = f32 f32 + f64 = f64
4480 4481 4482 4483 4484 4485 4486 4487 |
# File 'lib/polars/data_frame.rb', line 4480 def fold(&operation) acc = to_series(0) 1.upto(width - 1) do |i| acc = operation.call(acc, to_series(i)) end acc end |
#gather_every(n, offset = 0) ⇒ DataFrame Also known as: take_every
Take every nth row in the DataFrame and return as a new DataFrame.
4708 4709 4710 |
# File 'lib/polars/data_frame.rb', line 4708 def gather_every(n, offset = 0) select(Utils.col("*").gather_every(n, offset)) end |
#get_column(name) ⇒ Series
Get a single column as Series by name.
2936 2937 2938 |
# File 'lib/polars/data_frame.rb', line 2936 def get_column(name) self[name] end |
#get_column_index(name) ⇒ Series Also known as: find_idx_by_name
Find the index of a column by name.
1394 1395 1396 |
# File 'lib/polars/data_frame.rb', line 1394 def get_column_index(name) _df.get_column_index(name) end |
#get_columns ⇒ Array
Get the DataFrame as a Array of Series.
2914 2915 2916 |
# File 'lib/polars/data_frame.rb', line 2914 def get_columns _df.get_columns.map { |s| Utils.wrap_s(s) } end |
#group_by(by, maintain_order: false) ⇒ GroupBy Also known as: groupby, group
Start a group by operation.
1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 |
# File 'lib/polars/data_frame.rb', line 1833 def group_by(by, maintain_order: false) if !Utils.bool?(maintain_order) raise TypeError, "invalid input for group_by arg `maintain_order`: #{maintain_order}." end GroupBy.new( self, by, maintain_order: maintain_order ) end |
#group_by_dynamic(index_column, every:, period: nil, offset: nil, truncate: true, include_boundaries: false, closed: "left", by: nil, start_by: "window") ⇒ DataFrame Also known as: groupby_dynamic
Group based on a time value (or index value of type :i32
, :i64
).
Time windows are calculated and rows are assigned to windows. Different from a normal group by is that a row can be member of multiple groups. The time/index window could be seen as a rolling window, with a window size determined by dates/times/values instead of slots in the DataFrame.
A window is defined by:
- every: interval of the window
- period: length of the window
- offset: offset of the window
The every
, period
and offset
arguments are created with
the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 day)
- 1w (1 week)
- 1mo (1 calendar month)
- 1y (1 calendar year)
- 1i (1 index count)
Or combine them: "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
In case of a group_by_dynamic on an integer column, the windows are defined by:
- "1i" # length 1
- "10i" # length 10
2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 |
# File 'lib/polars/data_frame.rb', line 2174 def group_by_dynamic( index_column, every:, period: nil, offset: nil, truncate: true, include_boundaries: false, closed: "left", by: nil, start_by: "window" ) DynamicGroupBy.new( self, index_column, every, period, offset, truncate, include_boundaries, closed, by, start_by ) end |
#group_by_rolling(index_column:, period:, offset: nil, closed: "right", by: nil, check_sorted: true) ⇒ RollingGroupBy Also known as: groupby_rolling
Create rolling groups based on a time column.
Also works for index values of type :i32
or :i64
.
Different from a dynamic_group_by
the windows are now determined by the
individual values and are not of constant intervals. For constant intervals use
group_by_dynamic
The period
and offset
arguments are created either from a timedelta, or
by using the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 day)
- 1w (1 week)
- 1mo (1 calendar month)
- 1y (1 calendar year)
- 1i (1 index count)
Or combine them: "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
In case of a group_by_rolling on an integer column, the windows are defined by:
- "1i" # length 1
- "10i" # length 10
1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 |
# File 'lib/polars/data_frame.rb', line 1936 def group_by_rolling( index_column:, period:, offset: nil, closed: "right", by: nil, check_sorted: true ) RollingGroupBy.new(self, index_column, period, offset, closed, by, check_sorted) end |
#hash_rows(seed: 0, seed_1: nil, seed_2: nil, seed_3: nil) ⇒ Series
Hash and combine the rows in this DataFrame.
The hash value is of type :u64
.
4745 4746 4747 4748 4749 4750 4751 |
# File 'lib/polars/data_frame.rb', line 4745 def hash_rows(seed: 0, seed_1: nil, seed_2: nil, seed_3: nil) k0 = seed k1 = seed_1.nil? ? seed : seed_1 k2 = seed_2.nil? ? seed : seed_2 k3 = seed_3.nil? ? seed : seed_3 Utils.wrap_s(_df.hash_rows(k0, k1, k2, k3)) end |
#head(n = 5) ⇒ DataFrame
Get the first n
rows.
1661 1662 1663 |
# File 'lib/polars/data_frame.rb', line 1661 def head(n = 5) _from_rbdf(_df.head(n)) end |
#height ⇒ Integer Also known as: count, length, size
Get the height of the DataFrame.
331 332 333 |
# File 'lib/polars/data_frame.rb', line 331 def height _df.height end |
#hstack(columns, in_place: false) ⇒ DataFrame
Return a new DataFrame grown horizontally by stacking multiple Series to it.
2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 |
# File 'lib/polars/data_frame.rb', line 2680 def hstack(columns, in_place: false) if !columns.is_a?(::Array) columns = columns.get_columns end if in_place _df.hstack_mut(columns.map(&:_s)) self else _from_rbdf(_df.hstack(columns.map(&:_s))) end end |
#include?(name) ⇒ Boolean
Check if DataFrame includes column.
557 558 559 |
# File 'lib/polars/data_frame.rb', line 557 def include?(name) columns.include?(name) end |
#insert_column(index, series) ⇒ DataFrame Also known as: insert_at_idx
Insert a Series at a certain column index. This operation is in place.
1260 1261 1262 1263 1264 1265 1266 |
# File 'lib/polars/data_frame.rb', line 1260 def insert_column(index, series) if index < 0 index = columns.length + index end _df.insert_column(index, series._s) self end |
#interpolate ⇒ DataFrame
Interpolate intermediate values. The interpolation method is linear.
4778 4779 4780 |
# File 'lib/polars/data_frame.rb', line 4778 def interpolate select(Utils.col("*").interpolate) end |
#is_duplicated ⇒ Series
Get a mask of all duplicated rows in this DataFrame.
3585 3586 3587 |
# File 'lib/polars/data_frame.rb', line 3585 def is_duplicated Utils.wrap_s(_df.is_duplicated) end |
#is_empty ⇒ Boolean Also known as: empty?
Check if the dataframe is empty.
4792 4793 4794 |
# File 'lib/polars/data_frame.rb', line 4792 def is_empty height == 0 end |
#is_unique ⇒ Series
Get a mask of all unique rows in this DataFrame.
3610 3611 3612 |
# File 'lib/polars/data_frame.rb', line 3610 def is_unique Utils.wrap_s(_df.is_unique) end |
#item ⇒ Object
Return the dataframe as a scalar.
Equivalent to df[0,0]
, with a check that the shape is (1,1).
721 722 723 724 725 726 |
# File 'lib/polars/data_frame.rb', line 721 def item if shape != [1, 1] raise ArgumentError, "Can only call .item if the dataframe is of shape (1,1), dataframe is of shape #{shape}" end self[0, 0] end |
#iter_rows(named: false, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 4642 4643 4644 4645 4646 4647 4648 4649 4650 4651 4652 4653 4654 4655 |
# File 'lib/polars/data_frame.rb', line 4624 def iter_rows(named: false, buffer_size: 500, &block) return to_enum(:iter_rows, named: named, buffer_size: buffer_size) unless block_given? # load into the local namespace for a modest performance boost in the hot loops columns = self.columns # note: buffering rows results in a 2-4x speedup over individual calls # to ".row(i)", so it should only be disabled in extremely specific cases. if buffer_size offset = 0 while offset < height zerocopy_slice = slice(offset, buffer_size) rows_chunk = zerocopy_slice.rows(named: false) if named rows_chunk.each do |row| yield columns.zip(row).to_h end else rows_chunk.each(&block) end offset += buffer_size end elsif named height.times do |i| yield columns.zip(row(i)).to_h end else height.times do |i| yield row(i) end end end |
#join(other, left_on: nil, right_on: nil, on: nil, how: "inner", suffix: "_right", join_nulls: false) ⇒ DataFrame
Join in SQL-like fashion.
2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 |
# File 'lib/polars/data_frame.rb', line 2521 def join(other, left_on: nil, right_on: nil, on: nil, how: "inner", suffix: "_right", join_nulls: false) lazy .join( other.lazy, left_on: left_on, right_on: right_on, on: on, how: how, suffix: suffix, join_nulls: join_nulls ) .collect(no_optimization: true) end |
#join_asof(other, left_on: nil, right_on: nil, on: nil, by_left: nil, by_right: nil, by: nil, strategy: "backward", suffix: "_right", tolerance: nil, allow_parallel: true, force_parallel: false) ⇒ DataFrame
Perform an asof join.
This is similar to a left-join except that we match on nearest key rather than equal keys.
Both DataFrames must be sorted by the asof_join key.
For each row in the left DataFrame:
- A "backward" search selects the last row in the right DataFrame whose 'on' key is less than or equal to the left's key.
- A "forward" search selects the first row in the right DataFrame whose 'on' key is greater than or equal to the left's key.
The default is "backward".
2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 |
# File 'lib/polars/data_frame.rb', line 2391 def join_asof( other, left_on: nil, right_on: nil, on: nil, by_left: nil, by_right: nil, by: nil, strategy: "backward", suffix: "_right", tolerance: nil, allow_parallel: true, force_parallel: false ) lazy .join_asof( other.lazy, left_on: left_on, right_on: right_on, on: on, by_left: by_left, by_right: by_right, by: by, strategy: strategy, suffix: suffix, tolerance: tolerance, allow_parallel: allow_parallel, force_parallel: force_parallel ) .collect(no_optimization: true) end |
#lazy ⇒ LazyFrame
Start a lazy query from this point.
3617 3618 3619 |
# File 'lib/polars/data_frame.rb', line 3617 def lazy wrap_ldf(_df.lazy) end |
#limit(n = 5) ⇒ DataFrame
Get the first n
rows.
Alias for #head.
1630 1631 1632 |
# File 'lib/polars/data_frame.rb', line 1630 def limit(n = 5) head(n) end |
#max(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their maximum value.
3877 3878 3879 3880 3881 3882 3883 3884 3885 |
# File 'lib/polars/data_frame.rb', line 3877 def max(axis: 0) if axis == 0 lazy.max.collect(_eager: true) elsif axis == 1 Utils.wrap_s(_df.max_horizontal) else raise ArgumentError, "Axis should be 0 or 1." end end |
#mean(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their mean value.
3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 |
# File 'lib/polars/data_frame.rb', line 3995 def mean(axis: 0, null_strategy: "ignore") case axis when 0 lazy.mean.collect(_eager: true) when 1 Utils.wrap_s(_df.mean_horizontal(null_strategy)) else raise ArgumentError, "Axis should be 0 or 1." end end |
#median ⇒ DataFrame
Aggregate the columns of this DataFrame to their median value.
4110 4111 4112 |
# File 'lib/polars/data_frame.rb', line 4110 def median lazy.median.collect(_eager: true) end |
#melt(id_vars: nil, value_vars: nil, variable_name: nil, value_name: nil) ⇒ DataFrame
Unpivot a DataFrame from wide to long format.
Optionally leaves identifiers set.
This function is useful to massage a DataFrame into a format where one or more columns are identifier variables (id_vars), while all other columns, considered measured variables (value_vars), are "unpivoted" to the row axis, leaving just two non-identifier columns, 'variable' and 'value'.
3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 |
# File 'lib/polars/data_frame.rb', line 3241 def melt(id_vars: nil, value_vars: nil, variable_name: nil, value_name: nil) if value_vars.is_a?(::String) value_vars = [value_vars] end if id_vars.is_a?(::String) id_vars = [id_vars] end if value_vars.nil? value_vars = [] end if id_vars.nil? id_vars = [] end _from_rbdf( _df.melt(id_vars, value_vars, value_name, variable_name) ) end |
#merge_sorted(other, key) ⇒ DataFrame
Take two sorted DataFrames and merge them by the sorted key.
The output of this operation will also be sorted. It is the callers responsibility that the frames are sorted by that key otherwise the output will not make sense.
The schemas of both DataFrames must be equal.
4907 4908 4909 |
# File 'lib/polars/data_frame.rb', line 4907 def merge_sorted(other, key) lazy.merge_sorted(other.lazy, key).collect(_eager: true) end |
#min(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their minimum value.
3909 3910 3911 3912 3913 3914 3915 3916 3917 |
# File 'lib/polars/data_frame.rb', line 3909 def min(axis: 0) if axis == 0 lazy.min.collect(_eager: true) elsif axis == 1 Utils.wrap_s(_df.min_horizontal) else raise ArgumentError, "Axis should be 0 or 1." end end |
#n_chunks(strategy: "first") ⇒ Object
Get number of chunks used by the ChunkedArrays of this DataFrame.
3845 3846 3847 3848 3849 3850 3851 3852 3853 |
# File 'lib/polars/data_frame.rb', line 3845 def n_chunks(strategy: "first") if strategy == "first" _df.n_chunks elsif strategy == "all" get_columns.map(&:n_chunks) else raise ArgumentError, "Strategy: '{strategy}' not understood. Choose one of {{'first', 'all'}}" end end |
#n_unique(subset: nil) ⇒ DataFrame
Return the number of unique rows, or the number of unique row-subsets.
4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 |
# File 'lib/polars/data_frame.rb', line 4283 def n_unique(subset: nil) if subset.is_a?(StringIO) subset = [Polars.col(subset)] elsif subset.is_a?(Expr) subset = [subset] end if subset.is_a?(::Array) && subset.length == 1 expr = Utils.expr_to_lit_or_expr(subset[0], str_to_lit: false) else struct_fields = subset.nil? ? Polars.all : subset expr = Polars.struct(struct_fields) end df = lazy.select(expr.n_unique).collect df.is_empty ? 0 : df.row(0)[0] end |
#null_count ⇒ DataFrame
Create a new DataFrame that shows the null counts per column.
4333 4334 4335 |
# File 'lib/polars/data_frame.rb', line 4333 def null_count _from_rbdf(_df.null_count) end |
#partition_by(groups, maintain_order: true, include_key: true, as_dict: false) ⇒ Object
Split into multiple DataFrames partitioned by groups.
3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 |
# File 'lib/polars/data_frame.rb', line 3458 def partition_by(groups, maintain_order: true, include_key: true, as_dict: false) if groups.is_a?(::String) groups = [groups] elsif !groups.is_a?(::Array) groups = Array(groups) end if as_dict out = {} if groups.length == 1 _df.partition_by(groups, maintain_order, include_key).each do |df| df = _from_rbdf(df) out[df[groups][0, 0]] = df end else _df.partition_by(groups, maintain_order, include_key).each do |df| df = _from_rbdf(df) out[df[groups].row(0)] = df end end out else _df.partition_by(groups, maintain_order, include_key).map { |df| _from_rbdf(df) } end end |
#pipe(func, *args, **kwargs, &block) ⇒ Object
It is recommended to use LazyFrame when piping operations, in order to fully take advantage of query optimization and parallelization. See #lazy.
Offers a structured way to apply a sequence of user-defined functions (UDFs).
1765 1766 1767 |
# File 'lib/polars/data_frame.rb', line 1765 def pipe(func, *args, **kwargs, &block) func.call(self, *args, **kwargs, &block) end |
#pivot(values:, index:, columns:, aggregate_fn: "first", maintain_order: true, sort_columns: false, separator: "_") ⇒ DataFrame
Create a spreadsheet-style pivot table as a DataFrame.
3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 |
# File 'lib/polars/data_frame.rb', line 3135 def pivot( values:, index:, columns:, aggregate_fn: "first", maintain_order: true, sort_columns: false, separator: "_" ) if values.is_a?(::String) values = [values] end if index.is_a?(::String) index = [index] end if columns.is_a?(::String) columns = [columns] end if aggregate_fn.is_a?(::String) case aggregate_fn when "first" aggregate_expr = Polars.element.first._rbexpr when "sum" aggregate_expr = Polars.element.sum._rbexpr when "max" aggregate_expr = Polars.element.max._rbexpr when "min" aggregate_expr = Polars.element.min._rbexpr when "mean" aggregate_expr = Polars.element.mean._rbexpr when "median" aggregate_expr = Polars.element.median._rbexpr when "last" aggregate_expr = Polars.element.last._rbexpr when "len" aggregate_expr = Polars.len._rbexpr when "count" warn "`aggregate_function: \"count\"` input for `pivot` is deprecated. Use `aggregate_function: \"len\"` instead." aggregate_expr = Polars.len._rbexpr else raise ArgumentError, "Argument aggregate fn: '#{aggregate_fn}' was not expected." end elsif aggregate_fn.nil? aggregate_expr = nil else aggregate_expr = aggregate_function._rbexpr end _from_rbdf( _df.pivot_expr( index, columns, values, maintain_order, sort_columns, aggregate_expr, separator ) ) end |
#plot(x = nil, y = nil, type: nil, group: nil, stacked: nil) ⇒ Vega::LiteChart Originally defined in module Plot
Plot data.
#product ⇒ DataFrame
Aggregate the columns of this DataFrame to their product values.
4136 4137 4138 |
# File 'lib/polars/data_frame.rb', line 4136 def product select(Polars.all.product) end |
#quantile(quantile, interpolation: "nearest") ⇒ DataFrame
Aggregate the columns of this DataFrame to their quantile value.
4167 4168 4169 |
# File 'lib/polars/data_frame.rb', line 4167 def quantile(quantile, interpolation: "nearest") lazy.quantile(quantile, interpolation: interpolation).collect(_eager: true) end |
#rechunk ⇒ DataFrame
This will make sure all subsequent operations have optimal and predictable performance.
4307 4308 4309 |
# File 'lib/polars/data_frame.rb', line 4307 def rechunk _from_rbdf(_df.rechunk) end |
#rename(mapping) ⇒ DataFrame
Rename column names.
1209 1210 1211 |
# File 'lib/polars/data_frame.rb', line 1209 def rename(mapping) lazy.rename(mapping).collect(no_optimization: true) end |
#replace(column, new_col) ⇒ DataFrame
Replace a column by a new Series.
1563 1564 1565 1566 |
# File 'lib/polars/data_frame.rb', line 1563 def replace(column, new_col) _df.replace(column.to_s, new_col._s) self end |
#replace_column(index, series) ⇒ DataFrame Also known as: replace_at_idx
Replace a column at an index location.
1429 1430 1431 1432 1433 1434 1435 |
# File 'lib/polars/data_frame.rb', line 1429 def replace_column(index, series) if index < 0 index = columns.length + index end _df.replace_column(index, series._s) self end |
#reverse ⇒ DataFrame
Reverse the DataFrame.
1178 1179 1180 |
# File 'lib/polars/data_frame.rb', line 1178 def reverse select(Polars.col("*").reverse) end |
#row(index = nil, by_predicate: nil, named: false) ⇒ Object
The index
and by_predicate
params are mutually exclusive. Additionally,
to ensure clarity, the by_predicate
parameter must be supplied by keyword.
When using by_predicate
it is an error condition if anything other than
one row is returned; more than one row raises TooManyRowsReturned
, and
zero rows will raise NoRowsReturned
(both inherit from RowsException
).
Get a row as tuple, either by index or by predicate.
4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 |
# File 'lib/polars/data_frame.rb', line 4528 def row(index = nil, by_predicate: nil, named: false) if !index.nil? && !by_predicate.nil? raise ArgumentError, "Cannot set both 'index' and 'by_predicate'; mutually exclusive" elsif index.is_a?(Expr) raise TypeError, "Expressions should be passed to the 'by_predicate' param" end if !index.nil? row = _df.row_tuple(index) if named columns.zip(row).to_h else row end elsif !by_predicate.nil? if !by_predicate.is_a?(Expr) raise TypeError, "Expected by_predicate to be an expression; found #{by_predicate.class.name}" end rows = filter(by_predicate).rows n_rows = rows.length if n_rows > 1 raise TooManyRowsReturned, "Predicate #{by_predicate} returned #{n_rows} rows" elsif n_rows == 0 raise NoRowsReturned, "Predicate #{by_predicate} returned no rows" end row = rows[0] if named columns.zip(row).to_h else row end else raise ArgumentError, "One of 'index' or 'by_predicate' must be set" end end |
#rows(named: false) ⇒ Array
Convert columnar data to rows as Ruby arrays.
4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 |
# File 'lib/polars/data_frame.rb', line 4585 def rows(named: false) if named columns = self.columns _df.row_tuples.map do |v| columns.zip(v).to_h end else _df.row_tuples end end |
#sample(n: nil, frac: nil, with_replacement: false, shuffle: false, seed: nil) ⇒ DataFrame
Sample from this DataFrame.
4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 |
# File 'lib/polars/data_frame.rb', line 4373 def sample( n: nil, frac: nil, with_replacement: false, shuffle: false, seed: nil ) if !n.nil? && !frac.nil? raise ArgumentError, "cannot specify both `n` and `frac`" end if n.nil? && !frac.nil? frac = Series.new("frac", [frac]) unless frac.is_a?(Series) _from_rbdf( _df.sample_frac(frac._s, with_replacement, shuffle, seed) ) end if n.nil? n = 1 end n = Series.new("", [n]) unless n.is_a?(Series) _from_rbdf(_df.sample_n(n._s, with_replacement, shuffle, seed)) end |
#schema ⇒ Hash
Get the schema.
433 434 435 |
# File 'lib/polars/data_frame.rb', line 433 def schema columns.zip(dtypes).to_h end |
#select(*exprs, **named_exprs) ⇒ DataFrame
Select columns from this DataFrame.
3709 3710 3711 |
# File 'lib/polars/data_frame.rb', line 3709 def select(*exprs, **named_exprs) lazy.select(*exprs, **named_exprs).collect(_eager: true) end |
#set_sorted(column, *more_columns, descending: false) ⇒ DataFrame
Indicate that one or multiple columns are sorted.
4921 4922 4923 4924 4925 4926 4927 4928 4929 |
# File 'lib/polars/data_frame.rb', line 4921 def set_sorted( column, *more_columns, descending: false ) lazy .set_sorted(column, *more_columns, descending: descending) .collect(no_optimization: true) end |
#shape ⇒ Array
Get the shape of the DataFrame.
319 320 321 |
# File 'lib/polars/data_frame.rb', line 319 def shape _df.shape end |
#shift(n, fill_value: nil) ⇒ DataFrame
Shift values by the given period.
3527 3528 3529 |
# File 'lib/polars/data_frame.rb', line 3527 def shift(n, fill_value: nil) lazy.shift(n, fill_value: fill_value).collect(_eager: true) end |
#shift_and_fill(periods, fill_value) ⇒ DataFrame
Shift the values by a given period and fill the resulting null values.
3560 3561 3562 |
# File 'lib/polars/data_frame.rb', line 3560 def shift_and_fill(periods, fill_value) shift(periods, fill_value: fill_value) end |
#shrink_to_fit(in_place: false) ⇒ DataFrame
Shrink DataFrame memory usage.
Shrinks to fit the exact capacity needed to hold the data.
4680 4681 4682 4683 4684 4685 4686 4687 4688 4689 |
# File 'lib/polars/data_frame.rb', line 4680 def shrink_to_fit(in_place: false) if in_place _df.shrink_to_fit self else df = clone df._df.shrink_to_fit df end end |
#slice(offset, length = nil) ⇒ DataFrame
Get a slice of this DataFrame.
1597 1598 1599 1600 1601 1602 |
# File 'lib/polars/data_frame.rb', line 1597 def slice(offset, length = nil) if !length.nil? && length < 0 length = height - offset + length end _from_rbdf(_df.slice(offset, length)) end |
#sort(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column.
1486 1487 1488 1489 1490 |
# File 'lib/polars/data_frame.rb', line 1486 def sort(by, reverse: false, nulls_last: false) lazy .sort(by, reverse: reverse, nulls_last: nulls_last) .collect(no_optimization: true) end |
#sort!(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column in-place.
1502 1503 1504 |
# File 'lib/polars/data_frame.rb', line 1502 def sort!(by, reverse: false, nulls_last: false) self._df = sort(by, reverse: reverse, nulls_last: nulls_last)._df end |
#std(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their standard deviation value.
4043 4044 4045 |
# File 'lib/polars/data_frame.rb', line 4043 def std(ddof: 1) lazy.std(ddof: ddof).collect(_eager: true) end |
#sum(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their sum value.
3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 |
# File 'lib/polars/data_frame.rb', line 3957 def sum(axis: 0, null_strategy: "ignore") case axis when 0 lazy.sum.collect(_eager: true) when 1 Utils.wrap_s(_df.sum_horizontal(null_strategy)) else raise ArgumentError, "Axis should be 0 or 1." end end |
#tail(n = 5) ⇒ DataFrame
Get the last n
rows.
1692 1693 1694 |
# File 'lib/polars/data_frame.rb', line 1692 def tail(n = 5) _from_rbdf(_df.tail(n)) end |
#to_a ⇒ Array
Returns an array representing the DataFrame
550 551 552 |
# File 'lib/polars/data_frame.rb', line 550 def to_a rows(named: true) end |
#to_csv(**options) ⇒ String
Write to comma-separated values (CSV) string.
962 963 964 |
# File 'lib/polars/data_frame.rb', line 962 def to_csv(**) write_csv(**) end |
#to_dummies(columns: nil, separator: "_", drop_first: false) ⇒ DataFrame
Get one hot encoded dummy variables.
4198 4199 4200 4201 4202 4203 |
# File 'lib/polars/data_frame.rb', line 4198 def to_dummies(columns: nil, separator: "_", drop_first: false) if columns.is_a?(::String) columns = [columns] end _from_rbdf(_df.to_dummies(columns, separator, drop_first)) end |
#to_h(as_series: true) ⇒ Hash
Convert DataFrame to a hash mapping column name to values.
733 734 735 736 737 738 739 |
# File 'lib/polars/data_frame.rb', line 733 def to_h(as_series: true) if as_series get_columns.to_h { |s| [s.name, s] } else get_columns.to_h { |s| [s.name, s.to_a] } end end |
#to_hashes ⇒ Array
Convert every row to a dictionary.
Note that this is slow.
752 753 754 755 756 757 758 759 |
# File 'lib/polars/data_frame.rb', line 752 def to_hashes rbdf = _df names = columns height.times.map do |i| names.zip(rbdf.row_tuple(i)).to_h end end |
#to_numo ⇒ Numo::NArray
Convert DataFrame to a 2D Numo array.
This operation clones data.
773 774 775 776 777 778 779 780 |
# File 'lib/polars/data_frame.rb', line 773 def to_numo out = _df.to_numo if out.nil? Numo::NArray.vstack(width.times.map { |i| to_series(i).to_numo }).transpose else out end end |
#to_s ⇒ String Also known as: inspect
Returns a string representing the DataFrame.
542 543 544 |
# File 'lib/polars/data_frame.rb', line 542 def to_s _df.to_s end |
#to_series(index = 0) ⇒ Series
Select column as Series at index location.
808 809 810 811 812 813 |
# File 'lib/polars/data_frame.rb', line 808 def to_series(index = 0) if index < 0 index = columns.length + index end Utils.wrap_s(_df.select_at_idx(index)) end |
#to_struct(name) ⇒ Series
Convert a DataFrame
to a Series
of type Struct
.
4822 4823 4824 |
# File 'lib/polars/data_frame.rb', line 4822 def to_struct(name) Utils.wrap_s(_df.to_struct(name)) end |
#transpose(include_header: false, header_name: "column", column_names: nil) ⇒ DataFrame
This is a very expensive operation. Perhaps you can do it differently.
Transpose a DataFrame over the diagonal.
1150 1151 1152 1153 |
# File 'lib/polars/data_frame.rb', line 1150 def transpose(include_header: false, header_name: "column", column_names: nil) keep_names_as = include_header ? header_name : nil _from_rbdf(_df.transpose(keep_names_as, column_names)) end |
#unique(maintain_order: true, subset: nil, keep: "first") ⇒ DataFrame
Note that this fails if there is a column of type List
in the DataFrame or
subset.
Drop duplicate rows from this DataFrame.
4243 4244 4245 4246 4247 4248 4249 4250 |
# File 'lib/polars/data_frame.rb', line 4243 def unique(maintain_order: true, subset: nil, keep: "first") self._from_rbdf( lazy .unique(maintain_order: maintain_order, subset: subset, keep: keep) .collect(no_optimization: true) ._df ) end |
#unnest(names) ⇒ DataFrame
Decompose a struct into its fields.
The fields will be inserted into the DataFrame
on the location of the
struct
type.
4858 4859 4860 4861 4862 4863 |
# File 'lib/polars/data_frame.rb', line 4858 def unnest(names) if names.is_a?(::String) names = [names] end _from_rbdf(_df.unnest(names)) end |
#unstack(step:, how: "vertical", columns: nil, fill_values: nil) ⇒ DataFrame
This functionality is experimental and may be subject to changes without it being considered a breaking change.
Unstack a long table to a wide form without doing an aggregation.
This can be much faster than a pivot, because it can skip the grouping phase.
3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 |
# File 'lib/polars/data_frame.rb', line 3330 def unstack(step:, how: "vertical", columns: nil, fill_values: nil) if !columns.nil? df = select(columns) else df = self end height = df.height if how == "vertical" n_rows = step n_cols = (height / n_rows.to_f).ceil else n_cols = step n_rows = (height / n_cols.to_f).ceil end n_fill = n_cols * n_rows - height if n_fill > 0 if !fill_values.is_a?(::Array) fill_values = [fill_values] * df.width end df = df.select( df.get_columns.zip(fill_values).map do |s, next_fill| s.extend_constant(next_fill, n_fill) end ) end if how == "horizontal" df = ( df.with_column( (Polars.arange(0, n_cols * n_rows, eager: true) % n_cols).alias( "__sort_order" ) ) .sort("__sort_order") .drop("__sort_order") ) end zfill_val = Math.log10(n_cols).floor + 1 slices = df.get_columns.flat_map do |s| n_cols.times.map do |slice_nbr| s.slice(slice_nbr * n_rows, n_rows).alias("%s_%0#{zfill_val}d" % [s.name, slice_nbr]) end end _from_rbdf(DataFrame.new(slices)._df) end |
#upsample(time_column:, every:, offset: nil, by: nil, maintain_order: false) ⇒ DataFrame
Upsample a DataFrame at a regular frequency.
The every
and offset
arguments are created with
the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 day)
- 1w (1 week)
- 1mo (1 calendar month)
- 1y (1 calendar year)
- 1i (1 index count)
Or combine them: "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 |
# File 'lib/polars/data_frame.rb', line 2265 def upsample( time_column:, every:, offset: nil, by: nil, maintain_order: false ) if by.nil? by = [] end if by.is_a?(::String) by = [by] end if offset.nil? offset = "0ns" end every = Utils._timedelta_to_pl_duration(every) offset = Utils._timedelta_to_pl_duration(offset) _from_rbdf( _df.upsample(by, time_column, every, offset, maintain_order) ) end |
#var(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their variance value.
4084 4085 4086 |
# File 'lib/polars/data_frame.rb', line 4084 def var(ddof: 1) lazy.var(ddof: ddof).collect(_eager: true) end |
#vstack(df, in_place: false) ⇒ DataFrame
Grow this DataFrame vertically by stacking a DataFrame to it.
2729 2730 2731 2732 2733 2734 2735 2736 |
# File 'lib/polars/data_frame.rb', line 2729 def vstack(df, in_place: false) if in_place _df.vstack_mut(df._df) self else _from_rbdf(_df.vstack(df._df)) end end |
#width ⇒ Integer
Get the width of the DataFrame.
346 347 348 |
# File 'lib/polars/data_frame.rb', line 346 def width _df.width end |
#with_column(column) ⇒ DataFrame
Return a new DataFrame with the column added or replaced.
2644 2645 2646 2647 2648 |
# File 'lib/polars/data_frame.rb', line 2644 def with_column(column) lazy .with_column(column) .collect(no_optimization: true, string_cache: false) end |
#with_columns(*exprs, **named_exprs) ⇒ DataFrame
Add columns to this DataFrame.
Added columns will replace existing columns with the same name.
3821 3822 3823 |
# File 'lib/polars/data_frame.rb', line 3821 def with_columns(*exprs, **named_exprs) lazy.with_columns(*exprs, **named_exprs).collect(_eager: true) end |
#with_row_index(name: "row_nr", offset: 0) ⇒ DataFrame Also known as: with_row_count
Add a column at index 0 that counts the rows.
1797 1798 1799 |
# File 'lib/polars/data_frame.rb', line 1797 def with_row_index(name: "row_nr", offset: 0) _from_rbdf(_df.with_row_index(name, offset)) end |
#write_avro(file, compression = "uncompressed") ⇒ nil
Write to Apache Avro file.
974 975 976 977 978 979 980 981 982 983 |
# File 'lib/polars/data_frame.rb', line 974 def write_avro(file, compression = "uncompressed") if compression.nil? compression = "uncompressed" end if Utils.pathlike?(file) file = Utils.normalise_filepath(file) end _df.write_avro(file, compression) end |
#write_csv(file = nil, has_header: true, include_header: nil, sep: ",", quote: '"', batch_size: 1024, datetime_format: nil, date_format: nil, time_format: nil, float_precision: nil, null_value: nil) ⇒ String?
Write to comma-separated values (CSV) file.
899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 |
# File 'lib/polars/data_frame.rb', line 899 def write_csv( file = nil, has_header: true, include_header: nil, sep: ",", quote: '"', batch_size: 1024, datetime_format: nil, date_format: nil, time_format: nil, float_precision: nil, null_value: nil ) include_header = has_header if include_header.nil? if sep.length > 1 raise ArgumentError, "only single byte separator is allowed" elsif quote.length > 1 raise ArgumentError, "only single byte quote char is allowed" elsif null_value == "" null_value = nil end if file.nil? buffer = StringIO.new buffer.set_encoding(Encoding::BINARY) _df.write_csv( buffer, include_header, sep.ord, quote.ord, batch_size, datetime_format, date_format, time_format, float_precision, null_value ) return buffer.string.force_encoding(Encoding::UTF_8) end if Utils.pathlike?(file) file = Utils.normalise_filepath(file) end _df.write_csv( file, include_header, sep.ord, quote.ord, batch_size, datetime_format, date_format, time_format, float_precision, null_value, ) nil end |
#write_ipc(file, compression: "uncompressed") ⇒ nil
Write to Arrow IPC binary stream or Feather file.
993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 |
# File 'lib/polars/data_frame.rb', line 993 def write_ipc(file, compression: "uncompressed") return_bytes = file.nil? if return_bytes file = StringIO.new file.set_encoding(Encoding::BINARY) end if Utils.pathlike?(file) file = Utils.normalise_filepath(file) end if compression.nil? compression = "uncompressed" end _df.write_ipc(file, compression) return_bytes ? file.string : nil end |
#write_json(file, pretty: false, row_oriented: false) ⇒ nil
Serialize to JSON representation.
827 828 829 830 831 832 833 834 835 836 837 838 |
# File 'lib/polars/data_frame.rb', line 827 def write_json( file, pretty: false, row_oriented: false ) if Utils.pathlike?(file) file = Utils.normalise_filepath(file) end _df.write_json(file, pretty, row_oriented) nil end |
#write_ndjson(file) ⇒ nil
Serialize to newline delimited JSON representation.
846 847 848 849 850 851 852 853 |
# File 'lib/polars/data_frame.rb', line 846 def write_ndjson(file) if Utils.pathlike?(file) file = Utils.normalise_filepath(file) end _df.write_ndjson(file) nil end |
#write_parquet(file, compression: "zstd", compression_level: nil, statistics: false, row_group_size: nil) ⇒ nil
Write to Apache Parquet file.
1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 |
# File 'lib/polars/data_frame.rb', line 1036 def write_parquet( file, compression: "zstd", compression_level: nil, statistics: false, row_group_size: nil ) if compression.nil? compression = "uncompressed" end if Utils.pathlike?(file) file = Utils.normalise_filepath(file) end _df.write_parquet( file, compression, compression_level, statistics, row_group_size ) end |