Class: ETL::Control::FileSource

Inherits:
Source show all
Defined in:
lib/etl/control/source/file_source.rb

Overview

A File source.

Instance Attribute Summary collapse

Attributes inherited from Source

#configuration, #control, #definition, #local_base, #store_locally

Instance Method Summary collapse

Methods inherited from Source

class_for_name, #errors, #last_local_file, #last_local_file_trigger, #local_file, #local_file_trigger, #read_locally, #timestamp

Constructor Details

#initialize(control, configuration, definition) ⇒ FileSource

Initialize the source

Configuration options:

  • :file: The source file

  • :parser: One of the following: a parser name as a String or symbol, a class which extends from Parser, a Hash with :name and optionally an :options key. Whether or not the parser uses the options is dependent on which parser is used. See the documentation for each parser for information on what options it accepts.

  • :skip_lines: The number of lines to skip (defaults to 0)

  • :store_locally: Set to false to not store a copy of the source data locally for archival



26
27
28
29
 
# File 'lib/etl/control/source/file_source.rb', line 26

def initialize(control, configuration, definition)
  super
  configure
end

Instance Attribute Details

#file ⇒ Object

The source file



12
13
14
 
# File 'lib/etl/control/source/file_source.rb', line 12

def file
  @file
end

#parser ⇒ Object

Accessor for the underlying parser



9
10
11
 
# File 'lib/etl/control/source/file_source.rb', line 9

def parser
  @parser
end

#skip_lines ⇒ Object

The number of lines to skip, default is 0



6
7
8
 
# File 'lib/etl/control/source/file_source.rb', line 6

def skip_lines
  @skip_lines
end

Instance Method Details

#each ⇒ Object

Returns each row from the source



42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/etl/control/source/file_source.rb', line 42

def each
  count = 0
  copy_sources if @store_locally
  @parser.each do |row|
    if ETL::Engine.offset && count < ETL::Engine.offset
      count += 1
    else
      row = ETL::Row[row]
      row.source = self
      yield row
    end
  end
end

#local_directory ⇒ Object

Get the local storage directory



37
38
39
 
# File 'lib/etl/control/source/file_source.rb', line 37

def local_directory
  File.join(local_base, File.basename(file, File.extname(file)))
end

#to_s ⇒ Object

Get a String identifier for the source



32
33
34
 
# File 'lib/etl/control/source/file_source.rb', line 32

def to_s
  file
end