Class: BulkProcessor::FileSplitter

Inherits:
Object
  • Object
show all
Defined in:
lib/bulk_processor/file_splitter.rb

Overview

Split a CSV file on S3 using the specified chunker

Instance Method Summary collapse

Constructor Details

#initialize(key:, row_chunker:) ⇒ FileSplitter

Returns a new instance of FileSplitter.



4
5
6
7
 
# File 'lib/bulk_processor/file_splitter.rb', line 4

def initialize(key:, row_chunker:)
  @key = key
  @row_chunker = row_chunker
end

Instance Method Details

#split! ⇒ Array<String>

Generate multiple files on S3, composed of chunks of the input file.

Returns:

  • (Array<String>) —

    the S3 keys for each new file



12
13
14
15
16
17
18
19
20
21
 
# File 'lib/bulk_processor/file_splitter.rb', line 12

def split!
  return @keys if instance_variable_defined?('@keys')
  ranges = row_chunker.ranges_for(input_csv)
  @keys = ranges.map.with_index do |range, index|
    chunk_key = key_from_index(index, ranges.count)
    contents = csv_from_range(range)
    BulkProcessor.config.file_class.new(chunk_key).write(contents)
    chunk_key
  end
end