Module: NdrImport::UniversalImporterHelper

Defined in:
lib/ndr_import/universal_importer_helper.rb

Overview

This mixin provides file importer helper methods that abstract away some of the complexity of enumerating over files and tables (which should be universally useful).

Instance Method Summary collapse

Instance Method Details

#extract(source_file, unzip_path, &block) ⇒ Object

Iterate through the file(s) line by line, yielding each one in turn, using get_table_mapping to select the mapping relevant to this file.



26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/ndr_import/universal_importer_helper.rb', line 26

def extract(source_file, unzip_path, &block)
  return enum_for(:extract, source_file, unzip_path) unless block

  files = NdrImport::File::Registry.files(source_file,
                                          'unzip_path' => unzip_path)
  files.each do |filename|
    # now at the individual file level, can we find the table mapping?
    table_mapping = get_table_mapping(filename, nil)

    options = {
      'unzip_path'       => unzip_path,
      'col_sep'          => table_mapping.try(:delimiter),
      'liberal_parsing'  => table_mapping.try(:liberal_parsing),
      'xml_record_xpath' => table_mapping.try(:xml_record_xpath)
    }

    tables = NdrImport::File::Registry.tables(filename, table_mapping.try(:format), options)
    yield_tables_and_their_content(filename, tables, &block)
  end
end

#get_notifier(_total_records) ⇒ Object

This method needs to be implemented where this mixin is used.



62
63
64
 
# File 'lib/ndr_import/universal_importer_helper.rb', line 62

def get_notifier(_total_records)
  fail 'Implement get_notifier'
end

#get_table_mapping(filename, tablename) ⇒ Object

This method returns the correct NdrImport::,NonTabular::Table for the given filename/tablename. It requires all the mappings to be stored in the table_mappings instance variable.



20
21
22
 
# File 'lib/ndr_import/universal_importer_helper.rb', line 20

def get_table_mapping(filename, tablename)
  @table_mappings.find { |mapping| mapping.match(filename, tablename) }
end

#table_enumerators(filename) ⇒ Object 



7
8
9
10
11
12
13
14
15
 
# File 'lib/ndr_import/universal_importer_helper.rb', line 7

def table_enumerators(filename)
  table_enumerators = {}

  extract(filename).each do |table, rows|
    table_enumerators[table.canonical_name] = table.transform(rows)
  end

  table_enumerators
end

#yield_tables_and_their_content(filename, tables, &block) ⇒ Object

This method does the table row yielding for the extract method, setting the notifier so that we can monitor progress



49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/ndr_import/universal_importer_helper.rb', line 49

def yield_tables_and_their_content(filename, tables, &block)
  tables.each do |tablename, table_content|
    mapping = get_table_mapping(filename, tablename)
    next if mapping.nil?

    total_records = table_content.count unless table_content.is_a?(Enumerator)
    mapping.notifier = get_notifier(total_records)

    yield(mapping, table_content)
  end
end