S3 Twin
Take a mirror of a S3 bucket. It leverages aws-sdk's copy_to
function to do the heavy lifting at Amazon, similar to s3cmd.
Supports local usage or remote workers, some of which may support scheduling - to keep a bucket's twin up to date.
One way copy only i.e. will not keep two buckets in sync. Carries across Public ACLs only, no support for S3 features like server side encryption.
If an object exists in the source bucket and does not exist in the destiantion bucket it's copied. If an object exists in both bucket and the etags are identical the object is not copied, if the etags do not match the object is copied. Does not remove object that have been deleted from the source bucket.
Please note; while this works it's the first release so the API may change drastically.
Installation
$ gem install s3twin
Usage
$ s3twin help
$ s3twin go
Example with payload:
$ s3twin go --payload=source_bucket:bar source_access_key:foo source_secret_key:world destination_bucket:bar destination_access_key:foo destination_secret_key:world
The payload can be set via an argument --payload=key:value key2:value2
or .env
file, if neither of those are present the user will be prompted for input.
Remote Workers
Currently only [iron.io] IronWorker tasks (including scheduled) are supported.
IronWorker Usage
Create a new project at https://hud.iron.io, collect your Iron Token and Project ID.
$ s3twin ironworker upload
$ s3twin ironworker go
$ s3twin ironworker schedule --time
Optionally create, or append, your .env
file:
IRON_TOKEN='foo'
IRON_PROJECT_ID='bar'
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request