Cid

Cid is Continuous Integraton for Data.

There are two facets to Cid:

The first takes a Datapackage with CSVs in a Github repo and validates each one against a schema (in JSON table schema format) which lives in the same folder.

The second updates the datapackage.json and pushes the changes to Github.

With these two things in place, we can continuously integrate our data!

Installation

Add this line to your application's Gemfile:

gem 'cid'

And then execute:

$ bundle

Or install it yourself as:

$ gem install cid

Usage

Currently, Cid assumes your Datapackage lives in a cloned Github repo and has a structure like this:

datapackage.json
- folder
  |
  |_ schema.json
  |_ file.csv
  |_ ..

The schema.json file must match the JSON table schema.

You can have any number of folders, and each schema must correspond to the files within it.

Alternatively, you can specify your schema in your datapackage.json like so:


{
  "name": "my-dataset",
  # here we list the data files in this dataset
  "resources": [
    {
      "path": "data.csv",
      "schema": {
        "fields": [
          {
            "name": "var1",
            "type": "string"
          },
          {
            "name": "var2",
            "type": "integer"
          },
          {
            "name": "var3",
            "type": "number"
          }
        ]
      }
    }
  ]
}

For more information, check out the Tabular data package specification

When you run cid validate on the command line, Cid loops through each folder and validates each csv against the schema. It also checks for any common errors like ragged rows and missing characters using csvlint.

When you run cid publish on the command line, Cid (again), loops through each folder and adds each csv to the datapackage.json. It then pushes the changes to the GitHub repo. For this to happen sucessfully, you must have a Github API key, which you can specify either as an environment variable like so:

export GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE"

Or as a command line option like so:

cid publish --github-token=YOUR_TOKEN_HERE

If you just want to skip the GitHub push altogether, just run

cid publish --skip-github

Getting this in Travis

Obviously, Cid is at its most powerful when used in a CI build. To get Cid working in Travis, simply run:

cid bootstrap --github-token=YOUR_TOKEN_HERE

This will create a .travis.yml to your repo and add your encrypted Github key.

Then just add your repo to Travis, and push your changes.

Now whenever someone makes a pull request on your data, Cid will validate the data against the schema, and you'll get a nice build status telling you if it's good to go!

If the branch is master, it will also generate a new datapackage.json and push that to Github.

If you would rather generate the datapackage.json on a different branch (for example, gh-pages), simply add the option branch like so:

cid bootstrap --github-token=YOUR_TOKEN_HERE --branch=gh-pages

You can also do this manually if you prefer - simply add a .travis.yml file to your repo that looks a bit like this:

before_script:
    - gem install cid
script:
    - cid validate
after_success:
    - '[ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && cid publish'

You'll also need to add your Github token to the .travis.yml file. To do this, just run:

travis encrypt GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE" --add

Examples

There's only two repos that use Cid so far (it is a new thing after all), and you can see it here:

If you decide to start using Cid, and want it listing here, open a PR and add it to this list!

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request