BucketClient

Bucket Client is a ruby gem that allows programmers to interact with popular Blob Storage cloud services. This intends to act as a layer of abstraction, much like ORM is to databases.

With this, you may easily change the blob storage provider or even defer them.

The supported cloud storage include:

  • Google Cloud Platform Cloud Storage
  • Amazon Web Service S3 Bucket
  • Digital Ocean Spaces
  • Azure Blob Storage (Microsoft)

Installation

Add this line to your application's Gemfile:

gem 'bucket_client'

And then execute:

$ bundle

Or install it yourself as:

$ gem install bucket_client

Usage

Creation of Client

To begin using BucketClient, you have to create a Client object based on the cloud service you want to use.

It is advised to use the LocalClient for development and test, as it uses the local disk to simulate how online blob storage will work, except for set_get_cors and set_read_policy functions. This ensure fast tests and development without polluting the actual bucket.

The creation of client use the generate class-level (or rather, module-level) method of the BucketClient module.

Local Client

Local Client uses a local path to simulate storage of blob. The path variable is the relative path from terminal that it uses to simulate storage. Please be careful as it may delete files within that folder.

To create a local client:

require "bucket_client"

client = BucketClient::generate type: :local, path: "./public/sample-bucket"
client # => the client used to perform CRUD on bucket and blobs

Amazon Web Service S3 Bucket

AWS Client requires 3 values, the access id, access key, which is the secret, and the region.

To create a AWS S3 client:

require "bucket_client"

client = BucketClient::generate type: :aws, id: ENV["AWS_ID"], secret: ENV["AWS_SECRET"], region: ENV["AWS_REGION"]
client # => the client used to perform CRUD on bucket and blobs

Digital Ocean Spaces

Digital Ocean spaces requires 3 values, the access id, access key, which is the secret, and the region.

To create a Digital Ocean client:

require "bucket_client"

client = BucketClient::generate type: :spaces, id: ENV["AWS_ID"], secret: ENV["AWS_SECRET"], region: ENV["AWS_REGION"]
client #=> the client used to perform CRUD on bucket and blobs

Azure Blob Storage

Azure Blob Storage require 2 values, the account name and the key, which is the secret.

To create a Azure client:

require "bucket_client"

client = BucketClient::generate type: :azure, id: ENV["AZURE_ACC_NAME"], secret: ENV["AZURE_KEY"]
client #=> the client used to perform CRUD on bucket and blobs 

Google Cloud Platform Storage Service

GCP Cloud Storage require 2 value, the project_id and the secret The secret can be passed in via 2 methods,

  • serialized Hash object of the secret
  • path to the secret json file on disk

To create a GCP client using the Hash object (assume the JSON value is stored as environment variable):

require "json"
require "bucket_client"

secret = JSON.parse(ENV["GOOGLE_KEY"])
client = BucketClient::generate type: :gcp, id: ENV["GOOGLE_ID"], secret: secret
client #=> the client used to perform CRUD on bucket and blobs 

To create a GCP client using the path to json secret:

require "bucket_client"
client = BucketClient::generate type: :gcp, id: ENV["GOOGLE_ID"], secret: "path/to/secret.json"
client #=> the client used to perform CRUD on bucket and blobs 

Operation Result

OperationResult is the object you obtain from the normal operations. It contains details of the operation where you can check:

Property Description
success whether the operation was successful. Boolean value
code the status code of the operation
message the message of the operation. Error messages can be checked here
value the usable value of the operation. May be url or binary

If you rather immediately obtain the value, and raise error when it is unsuccessful, you may use the ! version of the method.

# Using OperationResult Object
result = bucket.create_blob binary, "path/to/bucket/file.bin"
if result.success
    p result.code #=> prints the status code
    p result.message #=> prints the success message, if any
    p result.value # => prints the URI obtain from "create_blob" method, when successful
else
    p result.code #=> check what HTTP error code is obtained
    p result.message #=> check error message   
end

# Or use ! method to immediate capture the method
begin
    result = bucket.create_blob! binary, "path/to/bucket/file.bin"
    p result #=> prints the URI obtained from "create_blob" method, when successful
rescue StandardError => e
    p e.message #=> prints the error message. This will include the status code
end


Using Client object for Bucket CRUD

The client object obtain via the generate method can be used to perform Bucket CRUD actions. It works across all platforms.

bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.

exist_bucket key:string => boolean

Checks whether the bucket of a certain key exist. Raises exception when the HTTP request underneath fails

# client from above

exist = client.exist_bucket "sample-bucket" 
exist #=> true if exist, false if it does not exist

create_bucket key:string => OperationResult<Bucket>

Creates a bucket using the provided key. Fails if bucket already exist.

value of OperationResult if successful is the Bucket object that has been created.

# client from above
result = client.create_bucket "sample-bucket" 
if result.success
    bucket = result.value 
    bucket #=> obtains the bucket
else
    p result.message #=> prints the error message
    p result.code #=> prints the status code    
end

create_bucket! key:string => Bucket

Creates a bucket using the provided key. Fails if bucket already exist.

Raises exception if fail, returns Bucket object that has been created if it succeeds

# client from above
result = client.create_bucket "sample-bucket" 
result #=> obtains bucket 

delete_bucket key:string => OperationResult<nil>

Deletes the bucket provided in the key. Fails if bucket does not exist. To prevent that behaviour, use delete_bucket_if_exist to not fail even if bucket does not exist.

Does not return anything on success. value will always return nil

# client from above
result = client.delete_bucket "sample-bucket"
result.success #=> whether the bucket has been successfully deleted
result.message #=> Error message or success message
result.code #=> status code of the operation 

delete_bucket! key:string => nil

Deletes the bucket provided in the key. Fails if bucket does not exist. To prevent that behaviour, use delete_bucket_if_exist! to not fail even if bucket does not exist.

Raises exception if fail. Returns nil

# client from above
 client.delete_bucket! "sample-bucket" #=> nil

delete_bucket_if_exist key:string => OperationResult<nil>

Deletes the bucket provided in the key. Will succeed even if bucket does not exist.

Does not return anything on success. value will always return nil

# client from above
result = client.delete_bucket_if_exist "sample-bucket"
result.success #=> whether the bucket has been successfully deleted
result.message #=> Error message or success message
result.code #=> status code of the operation 

delete_bucket_if_exist! key:string => nil

Deletes the bucket provided in the key. Will succeed even if bucket does not exist.

Raises exception if the operation fails

# client from above
 client.delete_bucket_if_exist! "sample-bucket" #=> nil

put_bucket key:string => OperationResult<Bucket>

Creates the bucket provided in key if it does not exist.

This method will succeed and return the bucket even if the bucket exist, unlike create_bucket

value of the OperationResult is the Bucket object if the operation succeeds

# client from above
result = client.put_bucket "sample-bucket" 
if result.success
    bucket = result.value 
    bucket #=> obtains the bucket
else
    p result.message #=> prints the error message
    p result.code #=> prints the status code    
end

put_bucket! key:string => Bucket

Creates the bucket provided in key if it does not exist.

This method will succeed and return the bucket even if the bucket exist, unlike create_bucket

Returns the Bucket object

Raises exception if the operation fails

# client from above
bucket = client.put_bucket! "sample-bucket" 
bucket #=> obtains the bucket that has been creted 

set_read_policy key:string, access:symbol => OperationResult<nil>

Sets the read policy of the bucket. This does not work for LocalBucket as LocalBucket does not have concept of "access".

Only two values are accepted: :public and :private. :public allows everyone with access to the link to read the blobs within the bucket :private only allows people with authorization (with secret key) to read the blob within the bucket

# client from above
result = client.set_read_policy "sample-bucket", :public 
result.success #=> whether the bucket has been made public
result.message #=> Error message or success message
result.code #=> status code of the operation 

set_read_policy! key:string, access:symbol => nil

Sets the read policy of the bucket. This does not work for LocalBucket as LocalBucket does not have concept of "access".

Raises exception if the operation fails.

Only two values are accepted: :public and :private. :public allows everyone with access to the link to read the blobs within the bucket :private only allows people with authorization (with secret key) to read the blob within the bucket

# client from above
client.set_read_policy! "sample-bucket", :public #=> nil 

set_get_cors key:string, cors:array<string> => OperationResult<nil>

Sets the GET CORS of the bucket. This is limits the Cross Origin Resource Sharing to the domains within the cors array you input. To allow all origin, please use ["*"] as cors value.

This does not work for LocalBucket as it does not have concept of cors. This is one-level higher for AzureBucket, where it modifies the whole accounts' CORS, not just the bucket.

# client from above
result = client.set_get_cors "sample-bucket", ["*"] 
result.success #=> whether it has succeeded allowing all origin to read
result.message #=> Error message or success message
result.code #=> status code of the operation   

set_get_cors! key:string, cors:array<string> => nil

Sets the GET CORS of the bucket. This is limits the Cross Origin Resource Sharing to the domains within the cors array you input. To allow all origin, please use ["*"] as cors value.

This does not work for LocalBucket as it does not have concept of cors. This is one-level higher for AzureBucket, where it modifies the whole accounts' CORS, not just the bucket.

Raises exception if the operation fails

#client from above
client.set_get_cors! "sample-bucket", ["*"]  #=> nil

get_bucket key:string => Bucket

Obtains the Bucket instance with the key.

The bucket instance can be used to perform CRUD for blobs within the bucket.

This method will raise exception if the bucket does not exist. To improve speed as you are sure that the bucket already exist, please use the bang version, get_bucket!, where it will not do a look up.

#client from above
bucket = get_bucket "sample-bucket"
bucket #=> bucket instance obtained.  

get_bucket! key:string => Bucket

Obtains the Bucket instance with the key.

The bucket instance can be used to perform CRUD for blobs within the bucket.

This method will not do a look up, so you instance's blob CRUD operation may fail if you did not verify the existence of this bucket. This performs faster than the non-bang version as it does not spend operation to check existence of the bucket, making the assumption that it exist.

#client from above
bucket = get_bucket! "sample-bucket"
bucket #=> bucket instance obtained.  

Using Client object for Blob CRUD

The client object can perform Blob CRUD if it has access to the full URI or URL of the blob.

Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.

get_blob uri:string => OperationResult<array<byte>>

Obtains the binary of the blob via the URI of the blob.

value of the OperationResult is the byte array of the binary if the operation succeeds

# client from above
result = client.get_blob "https://host.com/bucket/blob.bin"
if result.success #=> whether the obtaining of the blob succeeded
    binary = result.value #=> obtain the binary value
    IO.binwrite "blob.bin", binary #=> writes it to disk
else 
    p result.message #=> Error message or success message
    p result.code #=> status code of the operation   
end  

get_blob! uri:string => <array<byte>>

Obtains the binary of the blob via the URI of the blob

Raises exception if it fails

binary = client.get_blob! "https://host.com/bucket/blob.bin"
IO.binwrite "blob.bin", binary

exist_blob uri:string => boolean

Checks whether the blob exist

exist = client.exist_blob "https://host.com/bucket/blob.bin"
exist #=> true if blob exist, false if it doesn't

update_blob payload:array<byte>, uri:string => Operation<string>

Updates a blob with new payload in byte array

value of the OperationResult will return URI of the blob if success

Fails if blob with the URI doesn't exist

img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.update_blob img, uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> Uri of update blob

update_blob! payload:array<byte>, uri:string => string

Updates a blob with new payload in byte array Fails if blob doesnt exist

Raises exception if operation fails

img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.update_blob! img, uri 
result #=> URI of update blob

put_blob payload:array<byte>, uri:string => OperationResult<string>

Creates the blob with the payload if it does not exist, updates the blob with the new payload if it exist

value of the OperationResult will return URI of the blob if success

img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.put_blob img, uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> Uri of blob

put_blob! payload:array<byte>, uri:string => string

Creates the blob with the payload if it does not exist, updates the blob with the new payload if it exist

Raises exception if the operation fails

img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.put_blob! img, uri 
result #=> returns URI of updated blob

delete_blob uri:string => OperationResult<nil>

Deletes the blob in the provided URI

Fails if the blob does not exist. Use delete_blob_if_exist if you do not want this behaviour

value of OperationResult is always nil

uri = "https://host.com/folder/pic.png"
result = client.delete_blob uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> nil

delete_blob! uri:string => nil

Deletes the blob in the provided URI Fails if the blob does not exist. Use delete_blob_if_exist if you do not want this behaviour

Raises exception if the operation fails

uri = "https://host.com/folder/pic.png"
client.delete_blob! uri

delete_blob_if_exist uri:string => OperationResult<nil>

Deletes the blob if it exist, else does nothing

value of OperationResult is always nil

uri = "https://host.com/folder/pic.png"
result = client.delete_blob uri
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> nil

delete_blob_if_exist! uri:string => nil

Deletes the blob if it exist, else does nothing

Raises exception if the operation fails

uri = "https://host.com/folder/pic.png"
client.delete_blob! uri

Using Bucket object to perform blob CRUD with blob keys

The bucket instance is able to perform CRUD operations on blobs it owns.

Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.

In this section, we assume we obtain a Bucket instance from the Client instance using the get_bucket method.

bucket = client.get_bucket! "first-bucket"
bucket #=> bucket instance used to illustrate the examples below

get_blob key:string => OperationResult<array<byte>>

Get blob as byte array

value of the OperationResult is the blob's byte array, if the operation succeeds

result = bucket.get_blob "image.png"
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> the byte array of the blob

get_blob! key:string => array<byte>

Get blob as byte array

Raises exception if the operation fails.

img = bucket.get_blob! "image.png" 
IO.binwrite "image.png", img 

exist_blob key:string => boolean

Checks if the blob with the given key exist.

exist = bucket.exist_blob "image.png"
exist #=> true if exist, false if it does not exist

create_blob payload:byte<array>,key:string => Operation<string>

Create blob with payload. Fails if blob already exist.

value of OperationResult will return URI of the created blob if operations succeeded

img = IO.binread "image.png"
result = bucket.create_blob img, "image.png"
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> URI of the blob

create_blob! payload:array<byte>,key:string => string

Create blob with payload. Fails if blob already exist.

Raises exception if operation fails

img = IO.binread "image.png"
uri = bucket.create_blob! img, "image.png" 
uri #=> URI of the created blob

update_blob payload:array<byte>, key:string => OperationResult<string>

Updates the blob with new payload. Fails if blob does not exist

value of OperationResult will return URI of the created blob if operations succeeded

img = IO.binread "image.png"
result = bucket.update_blob img, "image.png"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> URI of the blob

update_blob! payload:array<byte>, key:string => string

Updates the blob with new payload. Fails if blob does not exist

Raises exception if the operation fails

img = IO.binread "image.png"
result = bucket.update_blob!(img, "image.png") 
result #=> URI of updated blob

put_blob payload:array<byte>, key:string => OperationResult<string>

Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist

value of OperationResult will return URI of the created blob if operations succeeded

img = IO.binread "image.png"
result = bucket.put_blob(img, "image.png")
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> URI of the blob

put_blob! payload:array<byte>, key:string => string

Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist

Raises exception if operation fails

img = IO.binread "image.png"
uri = bucket.put_blob! img, "image.png" 
uri #=> uri of the blob

delete_blob key:string => Operation<nil>

Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist method

value of OperationResult will always return nil

result = bucket.delete_blob "image.png"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> nil

delete_blob! key:string => nil

Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist method

Raises exception if the operation fails

bucket.delete_blob! "image.png"

delete_blob_if_exist key:string => Operation<nil>

Deletes a blob if it exist.

value of OperationResult will always return nil

result = bucket.delete_blob_if_exist "image.png"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> nil

delete_blob_if_exist! key:string => nil

Deletes a blob if it exist.

Raises exception if the operation fails

bucket.delete_blob_if_exist! "image.png"

Using Bucket object to perform blob CRUD with full URI

The bucket instance is able to perform CRUD operations on blobs it owns, via the full URI of the blob.

Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.

In this section, we assume we obtain a Bucket instance from the Client instance using the get_bucket! method.

bucket = client.get_bucket! "first-bucket"
bucket #=> bucket instance used to illustrate the examples below

get_blob_with_uri uri:string => OperationResult<array<byte>>

Get blob in target URI as byte array

value of the OperationResult is the blob's byte array, if the operation succeeds

result = bucket.get_blob_with_uri "https://domain.com/bucket/binary.ext"
result.success #=> Whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> the byte array of the blob

get_blob_with_uri! key:string => array<byte>

Get blob in target URI as byte array

Raises exception if the operation fails.

img = bucket.get_blob_with_uri! "https://domain.com/bucket/binary.ext"
IO.binwrite "image.png", img 

exist_blob_uri uri:string => boolean

Checks if the blob with the given URI exist.

exist = bucket.exist_blob_with_uri "https://domain.com/bucket/binary.ext"
exist #=> true if exist, false if it does not exist

update_blob_with_uri payload:array<byte>, uri:string => OperationResult<string>

Updates the blob with new payload to the uri. Fails if blob does not exist

value of OperationResult will return URI of the created blob if operations succeeded

img = IO.binread "image.png"
result = bucket.update_blob_with_uri img, "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> URI of the blob

update_blob_with_uri! payload:array<byte>, uri:string => string

Updates the blob with new payload. Fails if blob does not exist

Raises exception if the operation fails

img = IO.binread "image.png"
result = bucket.update_blob_with_uri! img, "https://domain.com/bucket/binary.ext" 
result #=> URI of updated blob

put_blob_with_uri payload:array<byte>, uri:string => OperationResult<string>

Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist

value of OperationResult will return URI of the created blob if operations succeeded

img = IO.binread "image.png"
result = bucket.put_blob_with_uri img, "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> URI of the blob

put_blob_with_uri! payload:array<byte>, uri:string => string

Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist

Raises exception if operation fails

img = IO.binread "image.png"
uri = bucket.put_blob_with_uri! img, "https://domain.com/bucket/binary.ext" 
uri #=> uri of the blob

delete_blob_with_uri uri:string => Operation<nil>

Deletes a blob in the uri. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist_with_uri method

value of OperationResult will always return nil

result = bucket.delete_blob_with_uri "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> nil

delete_blob_with_uri! uri:string => nil

Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use delete_blob_if_exist_with_uri! method

Raises exception if the operation fails

bucket.delete_blob_with_uri! "https://domain.com/bucket/binary.ext"

delete_blob_if_exist_with_uri uri:string => Operation<nil>

Deletes a blob in the uri if it exist.

value of OperationResult will always return nil

result = bucket.delete_blob_if_exist_with_uri "https://domain.com/bucket/binary.ext"
result.success #=> whether the operation succeeded
result.code #=> Status Code of the operation
result.message #=> Error message if it failed
result.value #=> nil

delete_blob_if_exist_with_uri! uri:string => nil

Deletes a blob in the uri if it exist.

Raises exception if the operation fails

bucket.delete_blob_if_exist_with_uri! "https://domain.com/bucket/binary.ext"

Development

After checking out the repo, run bin/setup to install dependencies.

Then, run bundle exec rspec to run the unit tests, or if you are on RubyMime, the run configuration of
Unit Test is configured. You can run the test by selecting the Unit Test configuration.

To run the integration test with the cloud, please provide your cloud credentials in a .env file. The credentials required are:

Env Variable Description
AWS_ID The AWS access key ID
AWS_SECRET The AWS secret
AWS_REGION The 3region of the AWS bucket
DO_ID Digital Ocean Spaces access key ID
DO_SECRET Digital Ocean Spaces secret
DO_REGION Digital Ocean Spaces region
AZURE_ACC_ID The Azure Blob Storage account name
AZURE_KEY The Azure Blob Storage secret
GOOGLE_ID The project ID for the Blob Storage account
GOOGLE_KEY The content of the secret JSON file in 1 line

After setting up the .env file, you can now run the command bundle exec rspec ./integration.

Alternatively, the gitlab ci has prepared a set of blob storage account online to run integration test. You can activate the test by pushing a commit to your branch if setting up the environment it too complicated.

To install this gem onto your local machine, run bundle exec rake install.

Contributing

Bug reports and pull requests are welcome on GitLab at https://gitlab.com/ruby-gem/bucket_client. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the BucketClient project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.