Class: Chicago::ETL::Transformations::WrittenRowFilter

Inherits:
Chicago::ETL::Transformation show all
Defined in:
lib/chicago/etl/transformations.rb

Overview

Filters rows so they only get output once, based on a :key.

Instance Method Summary collapse

Methods inherited from Chicago::ETL::Transformation

#added_fields, added_fields, adds_fields, #applies_to_stream?, #downstream_fields, #flush, #output_streams, #process, removed_fields, #removed_fields, removes_fields, #required_options, required_options, requires_options, #upstream_fields

Constructor Details

#initialize(*args) ⇒ WrittenRowFilter

Returns a new instance of WrittenRowFilter.



8
9
10
11
 
# File 'lib/chicago/etl/transformations.rb', line 8

def initialize(*args)
  super(*args)
  @written_rows = Set.new
end

Instance Method Details

#key_field ⇒ Object 



22
23
24
 
# File 'lib/chicago/etl/transformations.rb', line 22

def key_field
  @options[:key]
end

#process_row(row) ⇒ Object 



13
14
15
16
17
18
19
20
 
# File 'lib/chicago/etl/transformations.rb', line 13

def process_row(row)
  key = row[key_field]
  # puts "Checking on #{key}"
  unless @written_rows.include?(key)
    @written_rows << key
    row
  end
end