Conformist

Bend CSVs to your will with declarative schemas. Map one or many columns, preprocess cells and lazily enumerate. Declarative schemas are easier to understand, quicker to setup and independent of I/O. Use CSV (Formally FasterCSV), Spreadsheet or any array of array-like data structure.

Quick and Dirty Examples

Open a CSV file and declare a schema. A schema compromises of columns. A column takes an arbitrary name followed by its position in the input. A column may be derived from multiple positions.

require 'conformist'
require 'csv'

csv    = CSV.open '~/transmitters.csv'
schema = Conformist.new do
  column :callsign, 1
  column :latitude, 1, 2, 3
  column :longitude, 3, 4, 5
  column :name, 0 do |value|
    value.upcase
  end
end

Insert the transmitters into a SQLite database.

require 'sqlite3'

db = SQLite3::Database.new 'transmitters.db'
schema.conform(csv).each do |transmitter|
  db.execute "INSERT INTO transmitters (callsign, ...) VALUES ('#{transmitter.callsign}', ...);"
end

Only insert the transmitters with the name "Mount Cooth-tha" using ActiveRecord or DataMapper.

transmitters = schema.conform(csv).select do |transmitter|
  transmitter.name == 'Mount Coot-tha'
end
transmitters.each do |transmitter|
  Transmitter.create! transmitter.attributes
end

Source from multiple, different input files and insert transmitters together into a single database.

require 'conformist'
require 'csv'
require 'sqlite3'

au_schema = Conformist.new do
  column :callsign, 8
  column :latitude, 10
end
us_schema = Conformist.new do
  column :callsign, 1
  column :latitude, 1, 2, 3
end

au_csv = CSV.open '~/au/transmitters.csv'
us_csv = CSV.open '~/us/transmitters.csv'

db = SQLite3::Database.new 'transmitters.db'

[au_schema.conform(au_csv), us_schema.conform(us_csv)].each do |schema|
  schema.each do |transmitter|
    db.execute "INSERT INTO transmitters (callsign, ...) VALUES ('#{transmitter.callsign}', ...);"
  end
end

Open a Microsoft Excel spreadsheet and declare a schema.

require 'conformist'
require 'spreadsheet'

book   = Spreadsheet.open '~/states.xls'
sheet  = book.worksheet 0
schema = Conformist.new do
  column :state, 0, 1 do |values|
    "#{values.first}, #{values.last}"
  end
  column :capital, 2
end

Print each state's attributes to standard out.

schema.conform(sheet).each do |state|
  $stdout.puts state.attributes
end

For more examples see test/fixtures, test/schemas and test/unit/integration_test.rb.

Installation

Conformist is available as a gem. Install it at the command line.

$ [sudo] gem install conformist

Or add it to your Gemfile and run $ bundle install.

gem 'conformist'

Usage

Anonymous Schema

Anonymous schemas are quick to declare and don't have the overhead of creating an explicit class.

citizen = Conformist.new do
  column :name, 0, 1
  column :email, 2
end

citizen.conform [['Tate', 'Johnson', '[email protected]']]

Class Schema

Class schemas are explicit. Class schemas were the only type available in earlier versions of Conformist.

class Citizen
  extend Conformist

  column :name, 0, 1
  column :email, 2
end

Citizen.conform [['Tate', 'Johnson', '[email protected]']]

Implicit Indexing

Column indexes are implicitly incremented when the index argument is omitted. Implicit indexing is all or nothing.

column :account_number                              # => 0
column :date { |v| Time.new *v.split('/').reverse } # => 1
column :description                                 # => 2
column :debit                                       # => 3
column :credit                                      # => 4

Conform

Conform is the principle method for lazily applying a schema to the given input.

enumerator = schema.conform CSV.open('~/file.csv')
enumerator.each do |row|
  puts row.attributes
end

Input

#conform expects any object that responds to #each to return an array-like object.

CSV.open('~/file.csv').responds_to? :each # => true
[[], [], []].responds_to? :each           # => true

Header Row

#conform takes an option to skip the first row of input. Given a typical CSV document, the first row is the header row and irrelevant for enumeration.

schema.conform CSV.open('~/file_with_headers.csv'), :skip_first => true

Named Columns

Strings can be used as column indexes instead of integers. These strings will be matched against the first row to determine the appropriate numerical index.

citizen = Conformist.new do
  column :email, 'EM'
  column :name, 'FN', 'LN'
end

citizen.conform [['FN', 'LN', 'EM'], ['Tate', 'Johnson', '[email protected]']], :skip_first => true

Enumerator

#conform is lazy, returning an Enumerator. Input is not parsed until you call #each, #map or any method defined in Enumerable. That means schemas can be assigned now and evaluated later. #each has the lowest memory footprint because it does not build a collection.

Struct

The argument passed into the block is a struct-like object. You can access columns as methods or keys. Columns were only accessible as keys in earlier versions of Conformist. Methods are now the preferred syntax.

citizen[:name] # => "Tate Johnson"
citizen.name   # => "Tate Johnson"

For convenience the #attributes method returns a hash of key-value pairs suitable for creating ActiveRecord or DataMapper records.

citizen.attributes # => {:name => "Tate Johnson", :email => "[email protected]"}

One Column

Maps the first column in the input file to :first_name. Column indexing starts at zero.

column :first_name, 0

Many Columns

Maps the first and second columns in the input file to :name.

column :name, 0, 1

Indexing is completely arbitrary and you can map any combination.

column :name_and_city 0, 1, 2

Many columns are implicitly concatenated. Behaviour can be changed by passing a block. See preprocessing.

Preprocessing

Sometimes values need to be manipulated before they're conformed. Passing a block gets access to values. The return value of the block becomes the conformed output.

column :name, 0, 1 do |values|
  values.map(&:upcase) * ' '
end

Works with one column too. Instead of getting a collection of objects, one object is passed to the block.

column :first_name, 0 do |value|
  value.upcase
end

It's also possible to provide a context object that is made available during preprocessing.

citizen = Conformist.new do
  column :name, 0, 1 do |values, context|
    (context[:upcase?] ? values.map(&:upcase) : values) * ' '
  end
end

citizen.conform [['tate', 'johnson']], context: {upcase?: true}

Virtual Columns

Virtual columns are not sourced from input. Omit the index to create a virtual column. Like real columns, virtual columns are included in the conformed output.

column :day do
  1
end

Inheritance

Inheriting from a schema gives access to all of the parent schema's columns.

Anonymous Schema

Anonymous inheritance takes inspiration from Ruby's syntax for instantiating new classes.

parent = Conformist.new do
  column :name, 0, 1
end

child = Conformist.new parent do
  column :category do
    'Child'
  end
end

Class Schema

Classical inheritance works as expected.

class Parent
  extend Conformist

  column :name, 0, 1
end

class Child < Parent
  column :category do
    'Child'
  end
end

Upgrading from <= 0.0.3 to >= 0.1.0

Where previously you had

class Citizen
  include Conformist::Base

  column :name, 0, 1
end

Citizen.load('~/file.csv').foreach do |citizen|
  # ...
end

You should now do

require 'fastercsv'

class Citizen
  extend Conformist

  column :name, 0, 1
end

Citizen.conform(FasterCSV.open('~/file.csv')).each do |citizen|
  # ...
end

See CHANGELOG.md for a full list of changes.

Compatibility

MRI 2.4.0, 2.3.1, 2.2.0, 2.1.0, 2.0.0, 1.9.3
JRuby

Dependencies

No explicit dependencies, although CSV and Spreadsheet are commonly used.

Contributing

Fork
Install dependancies by running $ bundle install
Write tests and code
Make sure the tests pass locally by running $ bundle exec rake
Push to GitHub and make sure continuous integration tests pass at https://travis-ci.org/tatey/conformist/pull_requests
Send a pull request on GitHub

Please do not increment the version number in lib/conformist/version.rb. The version number will be incremented by the maintainer after the patch is accepted.

Motivation

Motivation for this project came from the desire to simplify importing data from various government organisations into Antenna Mate. The data from each government was similar, but had completely different formatting. Some pieces of data needed preprocessing while others simply needed to be concatenated together. Not wanting to write a parser for each new government organisation, I created Conformist.

Conformist

Quick and Dirty Examples

Installation

Usage

Anonymous Schema

Class Schema

Implicit Indexing

Conform

Input

Header Row

Named Columns

Enumerator

Struct

One Column

Many Columns

Preprocessing

Virtual Columns

Inheritance

Anonymous Schema

Class Schema

Upgrading from <= 0.0.3 to >= 0.1.0

Compatibility

Dependencies

Contributing

Motivation

Copyright