Cid
Cid is Continuous Integraton for Data.
There are two facets to Cid:
The first takes a Datapackage with CSVs in a Github repo and validates each one against a schema (in JSON table schema format) which lives in the same folder.
The second updates the datapackage.json and pushes the changes to Github.
With these two things in place, we can continuously integrate our data!
Installation
Add this line to your application's Gemfile:
gem 'cid'
And then execute:
$ bundle
Or install it yourself as:
$ gem install cid
Usage
Currently, Cid assumes your Datapackage lives in a cloned Github repo and has a structure like this:
datapackage.json
- folder
|
|_ schema.json
|_ file.csv
|_ ..
The schema.json file must match the JSON table schema.
You can have any number of folders, and each schema must correspond to the files within it.
When you run cid validate
on the command line, Cid loops through each folder and validates each csv against the schema. It also checks for any common errors like ragged rows and missing characters using csvlint.
When you run cid publish
on the command line, Cid (again), loops through each folder and adds each csv to the datapackage.json
. It then pushes the changes to the GitHub repo. For this to happen sucessfully, you must have a Github API key, which you can specify either as an environment variable like so:
export GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE"
Or as a command line option like so:
cid publish --github-token=YOUR_TOKEN_HERE
If you just want to skip the GitHub push altogether, just run
cid publish --skip-github
Getting this in Travis
Obviously, Cid is at its most powerful when used in a CI build. To get Cid working in Travis, simply add a .travis.yml
file to your repo that looks a bit like this:
before_script:
- gem install cid
script:
- cid validate
after_success:
- '[ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && cid publish'
This installs cid, runs the validation, and then (if the branch is master) generates a new datapackage.json
and pushes it to github.
You'll also need to add your Github token to the .travis.yml
file. To do this, just run:
gem install travis
travis encrypt GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE" --add
Then just add your repo to Travis, and push your changes.
Now whenever someone makes a pull request on your data, Cid will validate the data against the schema, and you'll get a nice build status telling you if it's good to go!
Examples
There's only two repos that use Cid so far (it is a new thing after all), and you can see it here:
If you decide to start using Cid, and want it listing here, open a PR and add it to this list!
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request