ocean-dynamo

This is the OceanDynamo ruby gem, implementing a highly scalable Amazon DynamoDB near drop-in replacement for ActiveRecord.

OceanDynamo requires Ruby 2.0 and Ruby on Rails 4.0.0 or later.

<img src=“https://badge.fury.io/rb/ocean-dynamo.png” alt=“Gem Version” />

Installation

gem install ocean-dynamo

Features

As one important use case for OceanDynamo is to facilitate the conversion of SQL based ActiveRecord models to DynamoDB based models, it is important that the syntax and semantics of OceanDynamo’s operations are as close as possible to those of ActiveRecord, including callbacks, exceptions and support methods. Ocean-dynamo follows this pattern closely and is of course based on ActiveModel.

The attribute and persistence layer of OceanDynamo is modeled on that of ActiveRecord: there’s save, save!, create, update, update!, update_attributes and all the other methods you’re used to. The design goal is always to implement as much as possible of the ActiveRecord interface, without sacrificing scalability. This makes the task of switching from SQL to no-SQL much easier.

Thanks to its structural similarity to ActiveRecord, OceanDynamo works with FactoryGirl. To facilitate testing, future versions will keep track of and delete instances after tests.

OceanDynamo will use secondary indices to retrieve related table items, which means it will scale without limits.

Example

The following example shows the syntax.

class AsyncJob < OceanDynamo::Base

  dynamo_schema do
    attribute :credentials,          :string
    attribute :token
    attribute :steps,                :serialized, default: []
    attribute :max_seconds_in_queue, :integer,    default: 1.day
    attribute :default_poison_limit, :integer,    default: 5
    attribute :default_step_time,    :integer,    default: 30
    attribute :started_at,           :datetime
    attribute :last_completed_step,  :integer
    attribute :finished_at,          :datetime
    attribute :destroy_at,           :datetime
    attribute :created_by
    attribute :updated_by
    attribute :succeeded,            :boolean,    default: false
    attribute :failed,               :boolean,    default: false
    attribute :poison,               :boolean,    default: false
  end

end

Each attribute has a name, a type (:string, :integer, :float, :datetime, :boolean, or :serialized) where :string is the default. Each attribute also optionally has a default value, which can be a Proc. The hash key attribute is by default :uuid (this can of course be specified) and is a :string. The keys can also be explicitly declared.

The :string, :integer, :float and :datetime types can also store sets of their type. Sets are represented as arrays, may not contain duplicates and may not be empty.

All attributes except the :string type can take the value nil. Storing nil for a string value will return the empty string, "".

Also, dynamo_schema takes args and many options. Here’s the full syntax:

dynamo_schema(
  table_hash_key = :uuid,                 # The name of the hash key attribute
  table_range_key = nil,                  # The name of the range key attribute (or nil)
  table_name: compute_table_name,         # The basename of the DynamoDB table
  table_name_prefix: nil,                 # A basename prefix string or nil
  table_name_suffix: nil,                 # A basename suffix string or nil
  read_capacity_units: 10,                # Used only when creating a table
  write_capacity_units: 5,                # Used only when creating a table
  connect: :late,                         # true, :late, nil/false
  create: false,                          # If true, create the table if nonexistent
  locking: :lock_version,                 # The name of the lock attribute or nil/false
  timestamps: [:created_at, :updated_at]  # An array of timestamp columns or nil/false
) do
  # Attribute definitions
  ...
  ...
end

At the moment, OceanDynamo is fully usable as an ActiveModel and can be used by Rails controllers. Furthermore, OceanDynamo implements much of the infrastructure of ActiveRecord; for instance, read_attribute, write_attribute, and much of the control logic and parameters.

Relations are not yet implemented, but are underway. Relations will use secondary indices, up to the DynamoDB maximum of 5 secondary keys per table. At the moment, only find with a single uuid is implemented. Collections can not yet be obtained.

OceanDynamo is currently used in the Ocean framework (wiki.oceanframework.net) e.g. to implement critical job queues. It will be used increasingly as features are added to OceanDynamo and will eventually replace all ActiveRecord tables.

Documentation

See also Ocean, a Rails framework for creating highly scalable SOAs in the cloud, in which OceanDynamo is used as a central component:

Contributing

Contributions are welcome. Fork in the usual way. OceanDynamo is developed using TDD: the specs are extensive and test coverage is very near to 100 percent. Pull requests will not be considered unless all tests pass and coverage is equally high or higher. All contributed code must therefore also be exhaustively tested.

Running the specs

To run the specs for the OceanDynamo gem, you must first install the bundle. It will download a gem called fake_dynamo, which runs a local, in-memory functional clone of Amazon DynamoDB. We use fake_dynamo during development and testing.

First of all, copy the AWS configuration file from the template:

cp spec/dummy/config/aws.yml.example spec/dummy/config/aws.yml

NB: aws.yml is excluded from source control. This allows you to enter your AWS credentials safely. Note that aws.yml.example is under source control: don’t edit it.

Make sure your have version 0.1.3 of the fake_dynamo gem. It implements the 2011-12-05 version of the DynamoDB API. We’re not yet using the 2012-08-10 version, as the aws-sdk ruby gem doesn’t fully support it. We’ll make the change as soon as aws-sdk is updated. Reportedly, it’s in the works.

Next, start fake_dynamo:

fake_dynamo --port 4567

If this returns errors, make sure that /usr/local/var/fake_dynamo exists and is writable:

sudo mkdir -p /usr/local/var/fake_dynamo
sudo chown peterb:staff /usr/local/var/fake_dynamo

When fake_dynamo runs normally, open another window and issue the following command:

curl -X DELETE http://localhost:4567

This will reset the fake_dynamo database. It’s not a required operation when starting fake_dynamo; we’re just using it here as a test that the installation works. It will be issued automatically as part of the test suite, so don’t expect test data to survive between runs.

With fake_dynamo running, you should now be able to do

rspec

All tests should pass.

Rails console

The Rails console is available from the built-in dummy application:

cd spec/dummy
rails console

This will, amongst other things, also create the CloudModel table if it doesn’t already exist. On Amazon, this will take a little while. With fake_dynamo, it’s practically instant.

When you leave the console, you must navigate back to the top directory (cd ../..) in order to be able to run RSpec again.