Amazon Extended Client Library for Ruby

Gem Version Build status

Implements the functionality of amazon-sns-java-extended-client-lib and amazon-sqs-java-extended-client-lib in Ruby

The Amazon SQS Extended Client allows clients to manage Amazon SNS and SQS message payloads that exceed the 256 KB message size limit, up to a size of 2 GB. In the event of publishing such large messages, the client accomplishes this feat by storing the actual payload in a S3 bucket and by storing the reference of the stored object in the SQS queue. Similarly, the extended-client is also used for retrieving and dereferencing these references of message objects stored in S3. Thus, the library is used for the following purposes:

  1. Specify whether payloads are always stored in Amazon S3 or only when a payload's size exceeds 256 KB.
  2. Send a message that references a single message object stored in an Amazon S3 bucket.
  3. Get the corresponding payload object from an Amazon S3 bucket.
  4. Delete the corresponding payload object from an Amazon S3 bucket.

Installation

Install the gem and add to the application's Gemfile by executing:

bundle add aws-sdk-extended

If bundler is not being used to manage dependencies, install the gem by executing:

gem install aws-sdk-extended

Usage

The SNS and SQS extended clients are initialized respectively with Aws::SNS::Client or Aws::SQS::Client objects as single mandatory positional arguments. All method calls are delegated to them, so you can use the extended client objects everywhere you'd use the standard AWS SDK client objects.

They can be initialized with the following optional attributes (as keyword arguments):

  • :use_legacy_attribute -- if true, then all published messages use the Legacy reserved message attribute ("SQSLargePayloadSize") instead of the current reserved message attribute ("ExtendedPayloadSize") (defaults to false)
  • :bucket -- the S3 bucket name that will store large messages. It falls back to the bucket identified by the AWS_EXTENDED_CLIENT_S3_BUCKET env var (if the env var is not set, then it becomes a mandatory attribute).
  • :payload_size_threshold -- the threshold for storing the message in the large messages bucket. Cannot be less than 0 or greater than 262144 (defaults to 262144).
  • :always_through_s3 -- if true, then all messages will be serialized to S3 (defaults to false).
  • :s3_client -- the awk-sdk-s3 Aws::S3::Client object to use to store objects to S3. Use this if you want to control the S3 client (for example, custom S3 config or credentials) (defaults to a shared Aws::S3::Client on first use).
  • (SQS only) :delete_payload_from_s3 -- if true, deletes the large payload from the S3 bucket when the corresponding SQS message is deleted (defaults to false).

Note:

The s3 bucket must already exist prior to usage, and be accessible by whatever credentials you have available

# for SNS
require "aws-sdk-sns"
require "aws/sdk/extended"

sns = Aws::SNS::Client.new
ext_sns = Aws::SNS::ExtendedClient.new(sns) # or, Aws::SNS::ExtendedClient.new(sns, payload_size_threshold: 32)

ext_sns.publish(
    topic_arn: "TOPIC_ARN",
    message: "This message should be published to S3 if it exceeds the threshold",
)

# for SQS
require "aws-sdk-sqs"
require "aws/sdk/extended"

sqs = Aws::SQS::Client.new
ext_sqs = Aws::SQS::ExtendedClient.new(sqs) # or, Aws::SQS::ExtendedClient.new(sqs, payload_size_threshold: 32)

queue_url = ext_sqs.get_queue_url(
    queue_name: "demo-preparation-queue"
).queue_url

# sending message
large_message = "a" * 300000 # Shall cross the limit of 256 KB

ext_sqs.send_message(
    queue_url: queue_url,
    message_body: large_message
)

response = ext_sqs.receive_message(
    queue_url: queue_url,
)

response.messages[0].body #=> "aaaaaaaaa...."

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle install. To release a new version, update the version number in version.rb, add the relevant change description to CHANGELOG.md, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

aws-sdk-extended ships with RBS signature files, and uses RBS inline syntax and rbs-inline to keep them in sync. In order to not let them fall out of sync when developing, run fswatch -0 lib | xargs -0 -n1 bundle exec rbs-inline --opt-out --output=sig in the background.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/HoneyryderChuck/aws-sdk-extended. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the Aws::Sdk::Extended project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.