BucketClient
Bucket Client is a ruby gem that allows programmers to interact with popular Blob Storage cloud services. This intends to act as a layer of abstraction, much like ORM is to databases.
With this, you may easily change the blob storage provider or even defer them.
The supported cloud storage include:
- Google Cloud Platform Cloud Storage
- Amazon Web Service S3 Bucket
- Digital Ocean Spaces
- Azure Blob Storage (Microsoft)
Installation
Add this line to your application's Gemfile:
gem 'bucket_client'
And then execute:
$ bundle
Or install it yourself as:
$ gem install bucket_client
Usage
Creation of Client
To begin using BucketClient, you have to create a Client object based on the cloud service you want to use.
It is advised to use the LocalClient for development and test, as it uses the local disk to simulate how
online blob storage will work, except for set_get_cors
and set_read_policy
functions. This ensure fast
tests and development without polluting the actual bucket.
The creation of client use the generate
class-level (or rather, module-level) method of the BucketClient
module.
Local Client
Local Client uses a local path to simulate storage of blob. The path
variable is the relative path from
terminal that it uses to simulate storage. Please be careful as it may delete files within that folder.
To create a local client:
require "bucket_client"
client = BucketClient::generate type: :local, path: "./public/sample-bucket"
client # => the client used to perform CRUD on bucket and blobs
Amazon Web Service S3 Bucket
AWS Client requires 3 values, the access id
, access key
, which is the secret, and the region
.
To create a AWS S3 client:
require "bucket_client"
client = BucketClient::generate type: :aws, id: ENV["AWS_ID"], secret: ENV["AWS_SECRET"], region: ENV["AWS_REGION"]
client # => the client used to perform CRUD on bucket and blobs
Digital Ocean Spaces
Digital Ocean spaces requires 3 values, the access id
, access key
, which is the secret, and the region
.
To create a Digital Ocean client:
require "bucket_client"
client = BucketClient::generate type: :spaces, id: ENV["AWS_ID"], secret: ENV["AWS_SECRET"], region: ENV["AWS_REGION"]
client #=> the client used to perform CRUD on bucket and blobs
Azure Blob Storage
Azure Blob Storage require 2 values, the account name
and the key
, which is the secret.
To create a Azure client:
require "bucket_client"
client = BucketClient::generate type: :azure, id: ENV["AZURE_ACC_NAME"], secret: ENV["AZURE_KEY"]
client #=> the client used to perform CRUD on bucket and blobs
Google Cloud Platform Storage Service
GCP Cloud Storage require 2 value, the project_id
and the secret
The secret can be passed in via 2 methods,
- serialized Hash object of the secret
- path to the secret json file on disk
To create a GCP client using the Hash object (assume the JSON value is stored as environment variable):
require "json"
require "bucket_client"
secret = JSON.parse(ENV["GOOGLE_KEY"])
client = BucketClient::generate type: :gcp, id: ENV["GOOGLE_ID"], secret: secret
client #=> the client used to perform CRUD on bucket and blobs
To create a GCP client using the path to json secret:
require "bucket_client"
client = BucketClient::generate type: :gcp, id: ENV["GOOGLE_ID"], secret: "path/to/secret.json"
client #=> the client used to perform CRUD on bucket and blobs
Operation Result
OperationResult is the object you obtain from the normal operations. It contains details of the operation where you can check:
Property | Description |
---|---|
success | whether the operation was successful. Boolean value |
code | the status code of the operation |
message | the message of the operation. Error messages can be checked here |
value | the usable value of the operation. May be url or binary |
If you rather immediately obtain the value, and raise error when it is unsuccessful, you may use the ! version of the method.
# Using OperationResult Object
result = bucket.create_blob binary, "path/to/bucket/file.bin"
if result.success
p result.code #=> prints the status code
p result. #=> prints the success message, if any
p result.value # => prints the URI obtain from "create_blob" method, when successful
else
p result.code #=> check what HTTP error code is obtained
p result. #=> check error message
end
# Or use ! method to immediate capture the method
begin
result = bucket.create_blob! binary, "path/to/bucket/file.bin"
p result #=> prints the URI obtained from "create_blob" method, when successful
rescue StandardError => e
p e. #=> prints the error message. This will include the status code
end
Using Client object for Bucket CRUD
The client object obtain via the generate
method can be used to perform Bucket CRUD actions. It works across
all platforms.
bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
exist_bucket key:string
=> boolean
Checks whether the bucket of a certain key exist. Raises exception when the HTTP request underneath fails
# client from above
exist = client.exist_bucket "sample-bucket"
exist #=> true if exist, false if it does not exist
create_bucket key:string
=> OperationResult<Bucket>
Creates a bucket using the provided key. Fails if bucket already exist.
value
of OperationResult
if successful is the Bucket
object that has been created.
# client from above
result = client.create_bucket "sample-bucket"
if result.success
bucket = result.value
bucket #=> obtains the bucket
else
p result. #=> prints the error message
p result.code #=> prints the status code
end
create_bucket! key:string
=> Bucket
Creates a bucket using the provided key. Fails if bucket already exist.
Raises exception if fail, returns Bucket
object that has been created if it succeeds
# client from above
result = client.create_bucket "sample-bucket"
result #=> obtains bucket
delete_bucket key:string
=> OperationResult<nil>
Deletes the bucket provided in the key. Fails if bucket does not exist.
To prevent that behaviour, use delete_bucket_if_exist
to not fail even if
bucket does not exist.
Does not return anything on success. value
will always return nil
# client from above
result = client.delete_bucket "sample-bucket"
result.success #=> whether the bucket has been successfully deleted
result. #=> Error message or success message
result.code #=> status code of the operation
delete_bucket! key:string
=> nil
Deletes the bucket provided in the key. Fails if bucket does not exist.
To prevent that behaviour, use delete_bucket_if_exist!
to not fail even if
bucket does not exist.
Raises exception if fail. Returns nil
# client from above
client.delete_bucket! "sample-bucket" #=> nil
delete_bucket_if_exist key:string
=> OperationResult<nil>
Deletes the bucket provided in the key. Will succeed even if bucket does not exist.
Does not return anything on success. value
will always return nil
# client from above
result = client.delete_bucket_if_exist "sample-bucket"
result.success #=> whether the bucket has been successfully deleted
result. #=> Error message or success message
result.code #=> status code of the operation
delete_bucket_if_exist! key:string
=> nil
Deletes the bucket provided in the key. Will succeed even if bucket does not exist.
Raises exception if the operation fails
# client from above
client.delete_bucket_if_exist! "sample-bucket" #=> nil
put_bucket key:string
=> OperationResult<Bucket>
Creates the bucket provided in key if it does not exist.
This method will succeed and return the bucket even if the bucket exist, unlike create_bucket
value
of the OperationResult
is the Bucket
object if the operation succeeds
# client from above
result = client.put_bucket "sample-bucket"
if result.success
bucket = result.value
bucket #=> obtains the bucket
else
p result. #=> prints the error message
p result.code #=> prints the status code
end
put_bucket! key:string
=> Bucket
Creates the bucket provided in key if it does not exist.
This method will succeed and return the bucket even if the bucket exist, unlike create_bucket
Returns the Bucket
object
Raises exception if the operation fails
# client from above
bucket = client.put_bucket! "sample-bucket"
bucket #=> obtains the bucket that has been creted
set_read_policy key:string
, access:symbol
=> OperationResult<nil>
Sets the read policy of the bucket. This does not work for LocalBucket
as LocalBucket
does not have
concept of "access".
Only two values are accepted: :public
and :private
.
:public
allows everyone with access to the link to read the blobs within the bucket
:private
only allows people with authorization (with secret key) to read the blob within the bucket
# client from above
result = client.set_read_policy "sample-bucket", :public
result.success #=> whether the bucket has been made public
result. #=> Error message or success message
result.code #=> status code of the operation
set_read_policy! key:string
, access:symbol
=> nil
Sets the read policy of the bucket. This does not work for LocalBucket
as LocalBucket
does not have
concept of "access".
Raises exception if the operation fails.
Only two values are accepted: :public
and :private
.
:public
allows everyone with access to the link to read the blobs within the bucket
:private
only allows people with authorization (with secret key) to read the blob within the bucket
# client from above
client.set_read_policy! "sample-bucket", :public #=> nil
set_get_cors key:string
, cors:array<string>
=> OperationResult<nil>
Sets the GET CORS of the bucket. This is limits the Cross Origin Resource Sharing to the domains
within the cors
array you input. To allow all origin, please use ["*"]
as cors value.
This does not work for LocalBucket
as it does not have concept of cors
.
This is one-level higher for AzureBucket
, where it modifies the whole accounts' CORS
, not just the bucket.
# client from above
result = client.set_get_cors "sample-bucket", ["*"]
result.success #=> whether it has succeeded allowing all origin to read
result. #=> Error message or success message
result.code #=> status code of the operation
set_get_cors! key:string
, cors:array<string>
=> nil
Sets the GET CORS of the bucket. This is limits the Cross Origin Resource Sharing to the domains
within the cors
array you input. To allow all origin, please use ["*"]
as cors value.
This does not work for LocalBucket
as it does not have concept of cors
.
This is one-level higher for AzureBucket
, where it modifies the whole accounts' CORS
, not just the bucket.
Raises exception if the operation fails
#client from above
client.set_get_cors! "sample-bucket", ["*"] #=> nil
get_bucket key:string
=> Bucket
Obtains the Bucket
instance with the key.
The bucket instance can be used to perform CRUD for blobs within the bucket.
This method will raise exception if the bucket does not exist. To improve speed as you are sure that the bucket
already exist, please use the bang version, get_bucket!
, where it will not do a look up.
#client from above
bucket = get_bucket "sample-bucket"
bucket #=> bucket instance obtained.
get_bucket! key:string
=> Bucket
Obtains the Bucket
instance with the key.
The bucket instance can be used to perform CRUD for blobs within the bucket.
This method will not do a look up, so you instance's blob CRUD operation may fail if you did not verify the existence of this bucket. This performs faster than the non-bang version as it does not spend operation to check existence of the bucket, making the assumption that it exist.
#client from above
bucket = get_bucket! "sample-bucket"
bucket #=> bucket instance obtained.
Using Client object for Blob CRUD
The client object can perform Blob CRUD if it has access to the full URI or URL of the blob.
Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
get_blob uri:string
=> OperationResult<array<byte>>
Obtains the binary of the blob via the URI of the blob.
value
of the OperationResult
is the byte array of the binary if the operation succeeds
# client from above
result = client.get_blob "https://host.com/bucket/blob.bin"
if result.success #=> whether the obtaining of the blob succeeded
binary = result.value #=> obtain the binary value
IO.binwrite "blob.bin", binary #=> writes it to disk
else
p result. #=> Error message or success message
p result.code #=> status code of the operation
end
get_blob! uri:string
=> <array<byte>>
Obtains the binary of the blob via the URI of the blob
Raises exception if it fails
binary = client.get_blob! "https://host.com/bucket/blob.bin"
IO.binwrite "blob.bin", binary
exist_blob uri:string
=> boolean
Checks whether the blob exist
exist = client.exist_blob "https://host.com/bucket/blob.bin"
exist #=> true if blob exist, false if it doesn't
update_blob payload:array<byte>
, uri:string
=> Operation<string>
Updates a blob with new payload in byte array
value
of the OperationResult
will return URI of the blob if success
Fails if blob with the URI doesn't exist
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.update_blob img, uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> Uri of update blob
update_blob! payload:array<byte>
, uri:string
=> string
Updates a blob with new payload in byte array Fails if blob doesnt exist
Raises exception if operation fails
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.update_blob! img, uri
result #=> URI of update blob
put_blob payload:array<byte>
, uri:string
=> OperationResult<string>
Creates the blob with the payload if it does not exist, updates the blob with the new payload if it exist
value
of the OperationResult
will return URI of the blob if success
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.put_blob img, uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> Uri of blob
put_blob! payload:array<byte>
, uri:string
=> string
Creates the blob with the payload if it does not exist, updates the blob with the new payload if it exist
Raises exception if the operation fails
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.put_blob! img, uri
result #=> returns URI of updated blob
delete_blob uri:string
=> OperationResult<nil>
Deletes the blob in the provided URI
Fails if the blob does not exist. Use delete_blob_if_exist if you do not want this behaviour
value
of OperationResult
is always nil
uri = "https://host.com/folder/pic.png"
result = client.delete_blob uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> nil
delete_blob! uri:string
=> nil
Deletes the blob in the provided URI Fails if the blob does not exist. Use delete_blob_if_exist if you do not want this behaviour
Raises exception if the operation fails
uri = "https://host.com/folder/pic.png"
client.delete_blob! uri
delete_blob_if_exist uri:string
=> OperationResult<nil>
Deletes the blob if it exist, else does nothing
value
of OperationResult
is always nil
uri = "https://host.com/folder/pic.png"
result = client.delete_blob uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> nil
delete_blob_if_exist! uri:string
=> nil
Deletes the blob if it exist, else does nothing
Raises exception if the operation fails
uri = "https://host.com/folder/pic.png"
client.delete_blob! uri
Using Bucket object to perform blob CRUD with blob keys
The bucket instance is able to perform CRUD operations on blobs it owns.
Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
In this section, we assume we obtain a Bucket
instance from the Client
instance using the get_bucket
method.
bucket = client.get_bucket! "first-bucket"
bucket #=> bucket instance used to illustrate the examples below
get_blob key:string
=> OperationResult<array<byte>>
Get blob as byte array
value
of the OperationResult
is the blob's byte array, if the operation succeeds
result = bucket.get_blob "image.png"
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> the byte array of the blob
get_blob! key:string
=> array<byte>
Get blob as byte array
Raises exception if the operation fails.
img = bucket.get_blob! "image.png"
IO.binwrite "image.png", img
exist_blob key:string
=> boolean
Checks if the blob with the given key exist.
exist = bucket.exist_blob "image.png"
exist #=> true if exist, false if it does not exist
create_blob payload:byte<array>
,key:string
=> Operation<string>
Create blob with payload. Fails if blob already exist.
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.create_blob img, "image.png"
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> URI of the blob
create_blob! payload:array<byte>
,key:string
=> string
Create blob with payload. Fails if blob already exist.
Raises exception if operation fails
img = IO.binread "image.png"
uri = bucket.create_blob! img, "image.png"
uri #=> URI of the created blob
update_blob payload:array<byte>
, key:string
=> OperationResult<string>
Updates the blob with new payload. Fails if blob does not exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.update_blob img, "image.png"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> URI of the blob
update_blob! payload:array<byte>
, key:string
=> string
Updates the blob with new payload. Fails if blob does not exist
Raises exception if the operation fails
img = IO.binread "image.png"
result = bucket.update_blob!(img, "image.png")
result #=> URI of updated blob
put_blob payload:array<byte>
, key:string
=> OperationResult<string>
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.put_blob(img, "image.png")
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> URI of the blob
put_blob! payload:array<byte>
, key:string
=> string
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
Raises exception if operation fails
img = IO.binread "image.png"
uri = bucket.put_blob! img, "image.png"
uri #=> uri of the blob
delete_blob key:string
=> Operation<nil>
Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist method
value
of OperationResult
will always return nil
result = bucket.delete_blob "image.png"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> nil
delete_blob! key:string
=> nil
Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist method
Raises exception if the operation fails
bucket.delete_blob! "image.png"
delete_blob_if_exist key:string
=> Operation<nil>
Deletes a blob if it exist.
value
of OperationResult
will always return nil
result = bucket.delete_blob_if_exist "image.png"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> nil
delete_blob_if_exist! key:string
=> nil
Deletes a blob if it exist.
Raises exception if the operation fails
bucket.delete_blob_if_exist! "image.png"
Using Bucket object to perform blob CRUD with full URI
The bucket instance is able to perform CRUD operations on blobs it owns, via the full URI of the blob.
Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
In this section, we assume we obtain a Bucket
instance from the Client
instance using the get_bucket!
method.
bucket = client.get_bucket! "first-bucket"
bucket #=> bucket instance used to illustrate the examples below
get_blob_with_uri uri:string
=> OperationResult<array<byte>>
Get blob in target URI as byte array
value
of the OperationResult
is the blob's byte array, if the operation succeeds
result = bucket.get_blob_with_uri "https://domain.com/bucket/binary.ext"
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> the byte array of the blob
get_blob_with_uri! key:string
=> array<byte>
Get blob in target URI as byte array
Raises exception if the operation fails.
img = bucket.get_blob_with_uri! "https://domain.com/bucket/binary.ext"
IO.binwrite "image.png", img
exist_blob_uri uri:string
=> boolean
Checks if the blob with the given URI exist.
exist = bucket.exist_blob_with_uri "https://domain.com/bucket/binary.ext"
exist #=> true if exist, false if it does not exist
update_blob_with_uri payload:array<byte>
, uri:string
=> OperationResult<string>
Updates the blob with new payload to the uri. Fails if blob does not exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.update_blob_with_uri img, "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> URI of the blob
update_blob_with_uri! payload:array<byte>
, uri:string
=> string
Updates the blob with new payload. Fails if blob does not exist
Raises exception if the operation fails
img = IO.binread "image.png"
result = bucket.update_blob_with_uri! img, "https://domain.com/bucket/binary.ext"
result #=> URI of updated blob
put_blob_with_uri payload:array<byte>
, uri:string
=> OperationResult<string>
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.put_blob_with_uri img, "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> URI of the blob
put_blob_with_uri! payload:array<byte>
, uri:string
=> string
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
Raises exception if operation fails
img = IO.binread "image.png"
uri = bucket.put_blob_with_uri! img, "https://domain.com/bucket/binary.ext"
uri #=> uri of the blob
delete_blob_with_uri uri:string
=> Operation<nil>
Deletes a blob in the uri. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist_with_uri method
value
of OperationResult
will always return nil
result = bucket.delete_blob_with_uri "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> nil
delete_blob_with_uri! uri:string
=> nil
Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist_with_uri! method
Raises exception if the operation fails
bucket.delete_blob_with_uri! "https://domain.com/bucket/binary.ext"
delete_blob_if_exist_with_uri uri:string
=> Operation<nil>
Deletes a blob in the uri if it exist.
value
of OperationResult
will always return nil
result = bucket.delete_blob_if_exist_with_uri "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result. #=> Error message if it failed
result.value #=> nil
delete_blob_if_exist_with_uri! uri:string
=> nil
Deletes a blob in the uri if it exist.
Raises exception if the operation fails
bucket.delete_blob_if_exist_with_uri! "https://domain.com/bucket/binary.ext"
Development
After checking out the repo, run bin/setup
to install dependencies.
Then, run bundle exec rspec
to run the unit tests, or if you are on RubyMime, the run configuration of
Unit Test is configured. You can run the test by selecting the Unit Test
configuration.
To run the integration test with the cloud, please provide your cloud credentials in a .env
file. The
credentials required are:
Env Variable | Description |
---|---|
AWS_ID | The AWS access key ID |
AWS_SECRET | The AWS secret |
AWS_REGION | The 3region of the AWS bucket |
DO_ID | Digital Ocean Spaces access key ID |
DO_SECRET | Digital Ocean Spaces secret |
DO_REGION | Digital Ocean Spaces region |
AZURE_ACC_ID | The Azure Blob Storage account name |
AZURE_KEY | The Azure Blob Storage secret |
GOOGLE_ID | The project ID for the Blob Storage account |
GOOGLE_KEY | The content of the secret JSON file in 1 line |
After setting up the .env file, you can now run the command bundle exec rspec ./integration
.
Alternatively, the gitlab ci has prepared a set of blob storage account online to run integration test. You can activate the test by pushing a commit to your branch if setting up the environment it too complicated.
To install this gem onto your local machine, run bundle exec rake install
.
Contributing
Bug reports and pull requests are welcome on GitLab at https://gitlab.com/ruby-gem/bucket_client. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
License
The gem is available as open source under the terms of the MIT License.
Code of Conduct
Everyone interacting in the BucketClient project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.