dm-adapter-simpledb

What

A DataMapper adapter for Amazon’s SimpleDB service.

Features:

* Uses the RightAWS gem for efficient SimpleDB operations.
* Full set of CRUD operations
* Supports all DataMapper query predicates.
* Can translate many queries into efficient native SELECT operations.
* Migrations
* DataMapper identity map support for record caching
* Lazy-loaded attributes
* DataMapper Serial property support via UUIDs.
* Array properties
* Basic aggregation support (Model.count("..."))
* String "chunking" permits attributes to exceed the 1024-byte limit

Note: as of version 1.0.0, this gem supports supports the DataMapper 0.10.* series and breaks backwards compatibility with DataMapper 0.9.*.

Who

Originally written by Jeremy Boles.

Contributers:

Edward Ocampo-Gooding (edward) 
Dan Mayer (danmayer)
Thomas Olausson (latompa)
Avdi Grimm (avdi)

Where

dm-adapter-simpledb is currently maintained by the Devver team and lives at: github.com/devver/dm-adapter-simpledb/

TODO

* More complete handling of NOT conditions in queries
* Robust quoting in SELECT calls
* Handle exclusive ranges natively
  Implement as inclusive range + filter step
* Tests for associations
* Option for smart lexicographical storage for numbers
  - Zero-pad integers
  - Store floats using exponential notation
* Option to store Date/Time/DateTime as ISO8601
* Full aggregate support (min/max/etc)
* Offset support
  Note, from the SimpleDB documentation:

  "The next token returned by count(*) and select are interchangeable as long
  as the where and order by clauses match. For example, if you want to return
  the 200 items after the first 10,000 (similar to an offset), you can perform
  a count with a limit clause of 10,000 and use the next token to return the
  next 200 items with select."

* Option to use libxml if available
* Parallelized queries for increased throughput
* Support of normalized 1:1 table:domain schemes that works with associations
* Sharding
* Support BatchPutAttributes
* Silence SSL warnings
  See http://pivotallabs.com/users/carl/blog/articles/1079-standup-blog-11-24-2009-model-validations-without-backing-store-associations-to-array-and-ssl-with-aws

Usage

Standalone

require 'rubygems'
require 'dm-core'
require 'dm-adapter-simpledb'

DataMapper.setup(:default, 
  :adapter       => 'simpledb',
  :access_key    => "ACCESS_KEY",
  :secret_key    => "SECRET_KEY",
  :domain        => "DOMAIN",
)

[Same as the following, but skip the database.yml]

In a Merb application

See sample Merb application using Merb-Auth and protected resources on SimpleDB:
http://github.com/danmayer/merb-simpledb-dm_example/tree/master

Setup database.yml with the SimpleDB DataMapper adapter:

  adapter:    simpledb
  access_key: (a 20-character, alphanumeric sequence)
  secret_key: (a 40-character sequence)
  domain:     'my_amazon_sdb_domain'

Create a model

  class Tree
    include DataMapper::Resource

    storage_name[:default] = "trees"

    property :id,   Serial
    property :name, String, :nullable => false
  end

Use interactively (with merb -i)

$ merb -i

  maple = Tree.new
  maple.name = "Acer rubrum"
  maple.save

  all_trees = Tree.all() # calls #read_all
  a_tree = Tree.first(:name => "Acer rubrum")
  yanked_tree = Tree.remote(:name => "Acer rubrum")

Running the tests

Add these two lines to your .bash_profile as the spec_helper relies on them

  $ export AMAZON_ACCESS_KEY_ID='YOUR_ACCESS_KEY'
  $ export AMAZON_SECRET_ACCESS_KEY='YOUR_SECRET_ACCESS_KEY'

Configure the domain to use for integration tests. THIS DOMAIN WILL BE
DELETED AND RECREATED BY THE TESTS, so do not choose a domain which contains
data you care about.  Configure the domain by creating a file named
THROW_AWAY_SDB_DOMAIN in the projet root:

 $ echo dm_simpledb_adapter_test > THROW_AWAY_SDB_DOMAIN

Run the tests:

  rake spec

NOTE: While every attempt has been made to make the tests robust, Amazon
SimpleDB is by it's nature an unreliable service. Sometimes it can take a
very long time for updates to be reflected by queries, and sometimes calls
just time out. If the tests fail, try them again a few times before reporting
it as a bug. Also try running the spec files individually.

Bibliography

Relating to Amazon SimpleDB developer.amazonwebservices.com/connect/entry.jspa?externalID=1292&ref=featured Approaching SimpleDB from a relational database background

Active Record Persistence with Amazon SimpleDB developer.amazonwebservices.com/connect/entry.jspa?externalID=1367&categoryID=152

Building for Performance and Reliability with Amazon SimpleDB developer.amazonwebservices.com/connect/entry.jspa?externalID=1394&categoryID=152

Query 101: Building Amazon SimpleDB Queries developer.amazonwebservices.com/connect/entry.jspa?externalID=1231&categoryID=152

Query 201: Tips & Tricks for Amazon SimpleDB Query developer.amazonwebservices.com/connect/entry.jspa?externalID=1232&categoryID=152 Latter portion describes parallelization advantages of normalized domains – the downside being the added complexity at the application layer (this library’s).

Using SimpleDB and Rails in No Time with ActiveResource developer.amazonwebservices.com/connect/entry.jspa?externalID=1242&categoryID=152 Exemplifies using the Single Table Inheritance pattern within a single SimpleDB domain by storing the model type in an attribute called ‘_resource’ and using a “SHA512 hash function on the request body combined with a timestamp and a configurable salt” for the id.

RightScale Ruby library to access Amazon EC2, S3, SQS, and SDB developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=140&externalID=1014&fromSearchPage=true