Sluice
Sluice is a Ruby gem (built with Bundler) to help you build cloud-friendly ETL (extract, transform, load) processes.
Currently Sluice provides the following very robust, very parallel S3 operations:
- File upload to S3
- File download from S3
- File delete from S3
- File move within S3 (from/to the same or different AWS accounts)
- File copy within S3 (from/to the same or different AWS accounts; optionally using a manifest)
Sluice has been extracted from a pair of Ruby ETL applications built by the Snowplow Analytics team, specifically:
- EmrEtlRunner, a Ruby application to run the Snowplow ETL process on Elastic MapReduce
- StorageLoader, a Ruby application to load Snowplow event files from Amazon S3 into databases such as Redshift and Postgres
Installation
$ gem install sluice
Or in your Gemfile:
gem 'sluice', '~> 0.1.0'
Usage
Rubydoc and usage examples to come.
Hacking and contributing
To hack on Sluice locally:
$ gem build sluice.gemspec
$ sudo gem install sluice-0.1.0.gem
To contribute:
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Added some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
Credits and thanks
Sluice was developed by Alex Dean (Snowplow Analytics) and Michael Tibben (99designs).
Copyright and license
Sluice is copyright 2012-2013 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.