Class: DS::Util::CsvValidator

Inherits:
Object
  • Object
show all
Defined in:
lib/ds/util/csv_validator.rb

Constant Summary collapse

ERROR_UNBALANCED_SUBFIELDS = 
'Row has subfields of different lengths'
ERROR_BLANK_SUBFIELDS = 
'Row has blank subfields'
ERROR_MISSING_REQUIRED_COLUMNS = 
"CSV is missing required column(s)"
ERROR_TRAILING_WHITESPACE = 
'Row contains trailing whitespace'
PIPE_SPLIT_REGEXP =

split on pipes that are not escaped with ''

%r{(?<!\\)\|}
PIPE_SEMICOLON_REGEXP =

split on pipes and semicolons that are not escaped with ''

%r{(?<!\\)[;|]}
MAX_SPLITS =

Maximum number of subfields to allow in a row; this number is arbitrarily set to 100,000 to ensure all trailing empty values are included in the array output by split.

100000

Class Method Summary collapse

Class Method Details

.validate_all_rows(rows, required_columns: [], balanced_columns: {}, nested_columns: {}, allow_blank: false) ⇒ Array<String>

Validates all rows of data against a set of required columns, balanced columns, and nested columns.

Parameters:

  • rows (Array<Hash,CSV::Row>) —

    The rows of data to be validated.

  • required_columns (Array<Symbol>) (defaults to: []) —

    The required columns for each row.

  • balanced_columns (Hash<Symbol, Array<Symbol>>) (defaults to: {}) —

    A hash of groups of balanced columns.

  • nested_columns (Hash<Symbol, Array<Symbol>>) (defaults to: {}) —

    A hash of nested columns.

  • allow_blank (Boolean) (defaults to: false) —

    Whether to allow blank subfields in balanced columns.

Returns:

  • (Array<String>) —

    An array of error messages, if any.



26
27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/ds/util/csv_validator.rb', line 26

def self.validate_all_rows rows, required_columns: [], balanced_columns: {}, nested_columns: {}, allow_blank: false
  errors = validate_required_columns(rows.first, row_num: 1, required_columns: required_columns)
  return errors unless errors.blank?
  rows.each_with_index do |row, row_num|
    errors += validate_row(
      row, row_num: row_num + 1,
      required_columns: required_columns,
      balanced_columns: balanced_columns,
      nested_columns: nested_columns,
      allow_blank: allow_blank
    )
  end
  errors
end

.validate_balanced_columns(row, row_num:, balanced_columns: {}, allow_blank: false) ⇒ Array<String>

Validates the balanced columns in a given row of data.

balanced_columns is a hash of groups of balanced columns.

Examples:

# row has unbalanced columns :a and :b
row = { a: 'a', b: 'b|b', c: 'c', d: 'd' }
balanced_columns = { group1: [:a, :b] }
csv_validator.validate_balanced_columns(
    row, balanced_columns: balanced_columns
)  # => ["Row has subfields of different lengths: group: :group1, sizes: [1, 2], row: [\"a\", \"b|b\"]"]

Parameters:

  • row (Hash) —

    The row of data to be validated.

  • balanced_columns (Hash<Symbol, Array<Symbol>>) (defaults to: {}) —

    A hash of groups of balanced columns.

  • allow_blank (Boolean) (defaults to: false) —

    Whether to allow blank subfields in balanced columns.

Returns:

  • (Array<String>) —

    An array of error messages, if any; otherwise, an empty array.



94
95
96
97
98
99
100
101
102
 
# File 'lib/ds/util/csv_validator.rb', line 94

def self.validate_balanced_columns row, row_num:, balanced_columns: {}, allow_blank: false
  return [] if balanced_columns.blank?
  errors = []
  balanced_columns.each { |group, columns|
    values = columns.map { |column| row[column.to_s] || row[column.to_sym] }
    errors += validate_row_splits(group: group, row_num: row_num, row_values: values, allow_blank: allow_blank)
  }
errors
end

.validate_required_columns(row, row_num:, required_columns:) ⇒ Array<String>

Validates the presence of required columns in a given row of data.

Parameters:

  • row (Hash, CSV::Row) —

    The row of data to be validated.

  • required_columns (Array<Symbol>) —

    The required columns for the row.

Returns:

  • (Array<String>) —

    An array of error messages, if any; otherwise, an empty array.



70
71
72
73
74
75
 
# File 'lib/ds/util/csv_validator.rb', line 70

def self.validate_required_columns row, row_num:, required_columns:
  return [] if required_columns.blank?
  missing = required_columns - row.to_h.keys
  return [] if missing.empty?
  ["#{ERROR_MISSING_REQUIRED_COLUMNS}: #{missing.map(&:inspect).join(', ')} row #{row_num}"]
end

.validate_row(row, row_num:, required_columns: [], balanced_columns: {}, nested_columns: {}, allow_blank: false) ⇒ Array<String>

Validates a row of data against a set of required columns and balanced columns.

# validate a CSV row for required columns and balanced columns
# columns a and b are required,
# columns a and b, and c and d are balanced
# balanced_columns keys are used as labels for the error messages
required_columns = [:a, :b]
balanced_columns = { group1: [:a, :b], group2: [:c: :d] }
csv_validator.validate(row, required_columns: required_columns, balanced_columns: balanced_columns)

Parameters:

  • row (Hash, CSV::Row) —

    The row of data to be validated.

  • required_columns (Array<Symbol>) (defaults to: []) —

    The required columns for the row.

  • balanced_columns (Hash<Symbol, Array<Symbol>>) (defaults to: {}) —

    a hash of groups of balanced columns; see example above

  • allow_blank (Boolean) (defaults to: false) —

    Whether to allow blank subfields in balanced columns

Returns:

  • (Array<String>) —

    An array of error messages, if any.



56
57
58
59
60
61
62
63
 
# File 'lib/ds/util/csv_validator.rb', line 56

def self.validate_row row, row_num:, required_columns: [], balanced_columns: {}, nested_columns: {}, allow_blank: false
  errors = []
  errors += validate_required_columns(row, row_num: row_num, required_columns: required_columns)
  return errors unless errors.blank?
  errors += validate_balanced_columns(row, row_num: row_num, balanced_columns: balanced_columns, allow_blank: allow_blank)
  errors += validate_whitespace(row, row_num: row_num, nested_columns: nested_columns)
  errors
end

.validate_row_splits(row_values: [], row_num:, separators: '|;', allow_blank: false, group: nil) ⇒ Array<String>

Return an error if each value in row_values has the same number of subfields and none of the subfields are blank; otherwise, return nil.

If allow_blank is true, ignore blanks, only check for balanced subfields.

Note: It is always allowed for every value to be blank (empty string). When row values are nil they are treated as empty strings. Blank values are treated a single values

So:

[ 'a|b|c', '1|2|3' ]   # => valid, return []
[ '', '' ]             # => valid, return []
[ 'a', '']             # => valid, return []
[ 'a|b|c', '1|2' ]     # => not valid, return ERROR_UNBALANCED_SUBFIELDS
[ 'a|b', '']           # => not valid, return ERROR_UNBALANCED_SUBFIELDS
[ 'a||c', '1|2|3' ]    # => not valid, return ERROR_BLANK_SUBFIELDS
[ 'a||c', '1|2|3' ]    # => valid if allow_blank == true, return []

Parameters:

  • row_values (Array<String>) (defaults to: []) —

    an array of strings from one or more columns

  • separators (String) (defaults to: '|;') —

    a list of allowed subfield separators; e.g., ';', '|', ';|'

  • allow_blank (Boolean) (defaults to: false) —

    whether any of the subfields may be blank

Returns:

  • (Array<String>) —

    the row errors, or [] if there are no errors



136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/ds/util/csv_validator.rb', line 136

def self.validate_row_splits row_values: [], row_num:, separators: '|;', allow_blank: false, group: nil
  errors = []
  return errors if row_values.all? { |val| val.blank? }
  # Input array is an array of two or more strings that must split into
  # equal numbers of subfields.
  #
  #   ['a|bc', '1|2|3'] => [['a', 'b', 'c'],
  #                         ['1', '2', '3']]
  #   ['a|b|c', '1|2']  => [['a', 'b', 'c'],
  #                         ['1' '2']]
  #
  # Count the subfields and make sure there's an equal number in each field
  #
  #    ['a|bc', '1|2|3'] => # 3 subfields each; => valid
  #    ['a|b|c', '1|2']  => # 2 and 3 subfields; => not valid
  splits = row_values.map { |v|
    v.to_s.split %r{[#{Regexp.escape separators}]}, MAX_SPLITS
  }

  # all sizes should 0 or 1; or there should be only one
  # subfield length
  sizes = splits.map { |vals| vals.size }
  if sizes.all? { |size| [0,1].include? size }
    return errors
  elsif sizes.uniq.size > 1
    errors << "#{ERROR_UNBALANCED_SUBFIELDS}: group: #{group.inspect}, sizes: #{sizes.inspect}, row: #{row_values.inspect} (row #{row_num})"
  end

  # return true if we don't have check for blanks
  return errors if allow_blank

  # return an error if any of the subfields are blank
  if splits.flatten.any? &:blank?
    errors << "#{ERROR_BLANK_SUBFIELDS}: group: #{group.inspect}, row: #{row_values.inspect} (row #{row_num})"
  end
  errors
end

.validate_whitespace(row, row_num:, nested_columns: []) ⇒ Array<String>

Validates a row of data for trailing whitespace. Returns an error for each column that contains trailing whitespace.

Nested columns is a hash with column names as keys and group names as values; e.g.,

nested_columns = {
  "subject_label" => :subjects,
  "subject" => :subjects,
  "genre_label" => :genres
  "genre" => :genres
}

Parameters:

  • row (Hash) —

    The row of data to be validated.

  • nested_columns (Array<Symbol>) (defaults to: []) —

    A hash of nested columns.

Returns:

  • (Array<String>) —

    An array of error messages, if any.



190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
 
# File 'lib/ds/util/csv_validator.rb', line 190

def self.validate_whitespace row, row_num:, nested_columns: []
  errors = []

  row.each do |column, value|
    # Assume all columns can have subfields delimited by pipes;
    # some columns are "nested"; that is, they can be be further
    # subdivided by semicolons. Select the regexp for the
    # subfield type
    split_chars = nested_columns.include?(column) ? PIPE_SEMICOLON_REGEXP : PIPE_SPLIT_REGEXP
    if value.to_s.split(split_chars).any? { |sub| sub =~ %r{\s+$} }
      errors << "#{ERROR_TRAILING_WHITESPACE}: column #{column.inspect}, value: #{value.inspect} (row #{row_num})"
    end
  end

  errors
end