Class: Daru::IO::Exporters::Avro

Inherits:
Base
  • Object
show all
Defined in:
lib/daru/io/exporters/avro.rb

Overview

Avro Exporter Class, that extends to_avro_string and write_avro methods to Daru::DataFrame instance variables

Instance Method Summary collapse

Methods inherited from Base

#optional_gem

Constructor Details

#initialize(dataframe, schema = nil) ⇒ Object

Initializes an Avro Exporter instance.

Examples:

Initializing an Avro Exporter

schema = {
  "type" => "record",
  "name" => "User",
  "fields" => [
    {"name" => "name", "type" => "string"},
    {"name" => "points", "type" => "int"},
    {"name"=> "winner", "type"=> "boolean", "default"=> "false"}
  ]
}

df = Daru::DataFrame.new(
  [
    {"name"=> "Dany", "points"=> 100, "winner"=> true},
    {"name"=> "Jon", "points"=> 100, "winner"=> true},
    {"name"=> "Tyrion", "points"=> 100, "winner"=> true}
  ]
)

#=> #<Daru::DataFrame(3x3)>
#        name points winner
#    0   Dany    100   true
#    1    Jon    100   true
#    2 Tyrion    100   true

instance = Daru::IO::Exporters::Avro.new(df, schema)

Parameters:

  • dataframe (Daru::DataFrame) —

    A dataframe to export

  • schema (Avro::Schema or Hash) (defaults to: nil) —

    The schema should contain details such as :type, :name and :fields



46
47
48
49
50
51
52
 
# File 'lib/daru/io/exporters/avro.rb', line 46

def initialize(dataframe, schema=nil)
  optional_gem 'avro'
  require 'json'

  super(dataframe)
  @schema = schema
end

Instance Method Details

#to_s ⇒ String

Exports an Avro Exporter instance to a file-writable String.

Examples:

Getting a file-writable string from Avro Exporter instance

instance.to_s

#=> "Obj\u0001\u0004\u0014avro.codec\bnull\u0016avro.schema\xBC\u0002{\"type\":\"record\"..."

Returns:

  • (String) —

    A file-writable string



62
63
64
 
# File 'lib/daru/io/exporters/avro.rb', line 62

def to_s
  super
end

#write(path) ⇒ Object

Exports an Avro Exporter instance to an avro file.

Examples:

Writing an Avro Exporter instance to an Avro file

instance.write('azor_ahai.avro')

Parameters:

  • path (String) —

    Path of Avro file where the dataframe is to be saved



72
73
74
75
76
77
78
79
80
81
 
# File 'lib/daru/io/exporters/avro.rb', line 72

def write(path)
  @schema_obj = process_schema
  @writer     = ::Avro::IO::DatumWriter.new(@schema_obj)
  @buffer     = StringIO.new
  @writer     = ::Avro::DataFile::Writer.new(@buffer, @writer, @schema_obj)
  @dataframe.each_row { |row| @writer << row.to_h }
  @writer.close

  File.open(path, 'w') { |file| file.write(@buffer.string) }
end