Module: NdrImport::UniversalImporterHelper
- Defined in:
- lib/ndr_import/universal_importer_helper.rb
Overview
This mixin provides file importer helper methods that abstract away some of the complexity of enumerating over files and tables (which should be universally useful). It is assumed that the host module/class defines ‘unzip_path`.
Instance Method Summary collapse
-
#extract(source_file, &block) ⇒ Object
Iterate through the file(s) line by line, yielding each one in turn, using get_table_mapping to select the mapping relevant to this file.
-
#get_notifier(_total_records) ⇒ Object
This method needs to be implemented where this mixin is used.
-
#get_table_mapping(filename, tablename) ⇒ Object
This method returns the correct NdrImport::,NonTabular::Table for the given filename/tablename.
- #mapped_tables(filename) ⇒ Object
- #record_total(filename, table_content) ⇒ Object
- #table_enumerators(filename) ⇒ Object
-
#yield_tables_and_their_content(filename, tables, &block) ⇒ Object
This method does the table row yielding for the extract method, setting the notifier so that we can monitor progress.
Instance Method Details
#extract(source_file, &block) ⇒ Object
Iterate through the file(s) line by line, yielding each one in turn, using get_table_mapping to select the mapping relevant to this file.
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 27 def extract(source_file, &block) return enum_for(:extract, source_file) unless block files = NdrImport::File::Registry.files(source_file, 'unzip_path' => unzip_path) files.each do |filename| # now at the individual file level, can we find the table mapping? table_mapping = get_table_mapping(filename, nil) = { 'unzip_path' => unzip_path, 'col_sep' => table_mapping.try(:delimiter), 'file_password' => table_mapping.try(:file_password), 'liberal_parsing' => table_mapping.try(:liberal_parsing), 'xml_record_xpath' => table_mapping.try(:xml_record_xpath) } tables = NdrImport::File::Registry.tables(filename, table_mapping.try(:format), ) yield_tables_and_their_content(filename, tables, &block) end end |
#get_notifier(_total_records) ⇒ Object
This method needs to be implemented where this mixin is used.
69 70 71 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 69 def get_notifier(_total_records) raise NotImplementedError, 'get_notifier must be defined!' end |
#get_table_mapping(filename, tablename) ⇒ Object
This method returns the correct NdrImport::,NonTabular::Table for the given filename/tablename. It requires all the mappings to be stored in the table_mappings instance variable.
21 22 23 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 21 def get_table_mapping(filename, tablename) @table_mappings.find { |mapping| mapping.match(filename, tablename) } end |
#mapped_tables(filename) ⇒ Object
64 65 66 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 64 def mapped_tables(filename) @mapped_tables ||= table_enumerators(filename) end |
#record_total(filename, table_content) ⇒ Object
73 74 75 76 77 78 79 80 81 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 73 def record_total(filename, table_content) if '.csv' == ::File.extname(filename).downcase return `wc -l #{Shellwords.escape(filename)}`.strip.match(/\A(\d+)/)[1].to_i elsif table_content.is_a?(Enumerator) nil # Avoid slurping else table_content.size end end |
#table_enumerators(filename) ⇒ Object
8 9 10 11 12 13 14 15 16 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 8 def table_enumerators(filename) table_enumerators = {} extract(filename).each do |table, rows| table_enumerators[table.canonical_name] = table.transform(rows) end table_enumerators end |
#yield_tables_and_their_content(filename, tables, &block) ⇒ Object
This method does the table row yielding for the extract method, setting the notifier so that we can monitor progress
51 52 53 54 55 56 57 58 59 60 61 62 |
# File 'lib/ndr_import/universal_importer_helper.rb', line 51 def yield_tables_and_their_content(filename, tables, &block) return enum_for(:yield_tables_and_their_content, filename, tables) unless block_given? tables.each do |tablename, table_content| mapping = get_table_mapping(filename, tablename) next if mapping.nil? mapping.notifier = get_notifier(record_total(filename, table_content)) yield(mapping, table_content) end end |