Class: Dynamoid::AdapterPlugin::AwsSdkV2

Inherits:
Object
  • Object
show all
Defined in:
lib/dynamoid/adapter_plugin/aws_sdk_v2.rb

Overview

The AwsSdkV2 adapter provides support for the aws-sdk version 2 for ruby.

Defined Under Namespace

Classes: ItemUpdater, Table

Constant Summary collapse

EQ = 
"EQ".freeze
RANGE_MAP = 
{
    range_greater_than: 'GT',
    range_less_than:    'LT',
    range_gte:          'GE',
    range_lte:          'LE',
    range_begins_with:  'BEGINS_WITH',
    range_between:      'BETWEEN',
    range_eq:           'EQ'
}
HASH_KEY = 
"HASH".freeze
RANGE_KEY = 
"RANGE".freeze
STRING_TYPE = 
"S".freeze
NUM_TYPE = 
"N".freeze
BINARY_TYPE = 
"B".freeze
TABLE_STATUSES = 
{
    creating: "CREATING",
    updating: "UPDATING",
    deleting: "DELETING",
    active: "ACTIVE"
}.freeze
PARSE_TABLE_STATUS = 
->(resp, lookup = :table) {
  # lookup is table for describe_table API
  # lookup is table_description for create_table API
  #   because Amazon, damnit.
  resp.send(lookup).table_status
}

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#table_cache ⇒ Object (readonly)

Returns the value of attribute table_cache.



33
34
35
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 33

def table_cache
  @table_cache
end

Instance Method Details

#batch_delete_item(options) ⇒ Object

Delete many items at once from DynamoDB. More efficient than delete each item individually.

or

Dynamoid::Adapter::AwsSdkV2.batch_delete_item('table1' => [['hk1', 'rk2'], ['hk1', 'rk2']]]))

@todo: Provide support for passing options to underlying delete_item docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#delete_item-instance_method

Examples:

Delete IDs 1 and 2 from the table testtable

Dynamoid::Adapter::AwsSdk.batch_delete_item('table1' => ['1', '2'])

Parameters:

  • options (Hash) —

    the hash of tables and IDs to delete

Returns:

  • nil



115
116
117
118
119
120
121
122
123
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 115

def batch_delete_item(options)
  options.each_pair do |table_name, ids|
    table = describe_table(table_name)
    ids.each do |id|
      client.delete_item(table_name: table_name, key: key_stanza(table, *id))
    end
  end
  nil
end

#batch_get_item(table_ids, options = {}) ⇒ Hash

Get many items at once from DynamoDB. More efficient than getting each item individually.

@todo: Provide support for passing options to underlying batch_get_item docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#batch_get_item-instance_method

Examples:

Retrieve IDs 1 and 2 from the table testtable

Dynamoid::Adapter::AwsSdkV2.batch_get_item({'table1' => ['1', '2']})

Parameters:

  • table_ids (Hash) —

    the hash of tables and IDs to retrieve

  • options (Hash) (defaults to: {}) —

    to be passed to underlying BatchGet call

Returns:

  • (Hash) —

    a hash where keys are the table names and the values are the retrieved items

Since:

  • 1.0.0



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 67

def batch_get_item(table_ids, options = {})
  request_items = Hash.new{|h, k| h[k] = []}
  return request_items if table_ids.all?{|k, v| v.empty?}

  table_ids.each do |t, ids|
    next if ids.empty?
    tbl = describe_table(t)
    hk  = tbl.hash_key.to_s
    rng = tbl.range_key.to_s

    keys = if rng.present?
      Array(ids).map do |h,r|
        { hk => h, rng => r }
      end
    else
      Array(ids).map do |id|
        { hk => id }
      end
    end

    request_items[t] = {
      keys: keys
    }
  end

  results = client.batch_get_item(
    request_items: request_items
  )

  ret = Hash.new([].freeze) # Default for tables where no rows are returned
  results.data[:responses].each do |table, rows|
    ret[table] = rows.collect { |r| result_item_to_hash(r) }
  end
  ret
end

#client ⇒ Object

Return the client object.

Since:

  • 1.0.0



50
51
52
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 50

def client
  @client
end

#connect! ⇒ Aws::DynamoDB::Client

Establish the connection to DynamoDB.

Returns:

  • (Aws::DynamoDB::Client) —

    the DynamoDB connection



38
39
40
41
42
43
44
45
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 38

def connect!
  @client = if Dynamoid::Config.endpoint?
    Aws::DynamoDB::Client.new(endpoint: Dynamoid::Config.endpoint)
  else
    Aws::DynamoDB::Client.new
  end
  @table_cache = {}
end

#count(table_name) ⇒ Object 



483
484
485
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 483

def count(table_name)
  describe_table(table_name, true).item_count
end

#create_table(table_name, key = :id, options = {}) ⇒ Object

Create a table on DynamoDB. This usually takes a long time to complete.

Parameters:

  • table_name (String) —

    the name of the table to create

  • key (Symbol) (defaults to: :id) —

    the table’s primary key (defaults to :id)

  • options (Hash) (defaults to: {}) —

    provide a range key here if the table has a composite key

Options Hash (options):

Since:

  • 1.0.0



135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 135

def create_table(table_name, key = :id, options = {})
  Dynamoid.logger.info "Creating #{table_name} table. This could take a while."
  read_capacity = options[:read_capacity] || Dynamoid::Config.read_capacity
  write_capacity = options[:write_capacity] || Dynamoid::Config.write_capacity

  secondary_indexes = options.slice(
    :local_secondary_indexes,
    :global_secondary_indexes
  )
  ls_indexes = options[:local_secondary_indexes]
  gs_indexes = options[:global_secondary_indexes]

  key_schema = {
    :hash_key_schema => { key => (options[:hash_key_type] || :string) },
    :range_key_schema => options[:range_key]
  }
  attribute_definitions = build_all_attribute_definitions(
    key_schema,
    secondary_indexes
  )
  key_schema = aws_key_schema(
    key_schema[:hash_key_schema],
    key_schema[:range_key_schema]
  )

  client_opts = {
    table_name: table_name,
    provisioned_throughput: {
      read_capacity_units: read_capacity,
      write_capacity_units: write_capacity
    },
    key_schema: key_schema,
    attribute_definitions: attribute_definitions
  }

  if ls_indexes.present?
    client_opts[:local_secondary_indexes] = ls_indexes.map do |index|
      index_to_aws_hash(index)
    end
  end

  if gs_indexes.present?
    client_opts[:global_secondary_indexes] = gs_indexes.map do |index|
      index_to_aws_hash(index)
    end
  end
  resp = client.create_table(client_opts)
  options[:sync] = true if !options.has_key?(:sync) && ls_indexes.present? || gs_indexes.present?
  until_past_table_status(table_name) if options[:sync] &&
      (status = PARSE_TABLE_STATUS.call(resp, :table_description)) &&
      status != TABLE_STATUSES[:creating]
  # Response to original create_table, which, if options[:sync]
  #   may have an outdated table_description.table_status of "CREATING"
  resp
rescue Aws::DynamoDB::Errors::ResourceInUseException => e
  Dynamoid.logger.error "Table #{table_name} cannot be created as it already exists"
end

#create_table_synchronously(table_name, key = :id, options = {}) ⇒ Object

Create a table on DynamoDB synchronously. This usually takes a long time to complete. CreateTable is normally an asynchronous operation. You can optionally define secondary indexes on the new table,

as part of the CreateTable operation.

If you want to create multiple tables with secondary indexes on them,

you must create the tables sequentially.

Only one table with secondary indexes can be

in the CREATING state at any given time.

See: docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#create_table-instance_method

Parameters:

  • table_name (String) —

    the name of the table to create

  • key (Symbol) (defaults to: :id) —

    the table’s primary key (defaults to :id)

  • options (Hash) (defaults to: {}) —

    provide a range key here if the table has a composite key

Options Hash (options):

Since:

  • 1.2.0



211
212
213
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 211

def create_table_synchronously(table_name, key = :id, options = {})
  create_table(table_name, key, options.merge(sync: true))
end

#delete_item(table_name, key, options = {}) ⇒ Object

Removes an item from DynamoDB.

@todo: Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#delete_item-instance_method

Parameters:

  • table_name (String) —

    the name of the table

  • key (String) —

    the hash key of the item to delete

  • options (Hash) (defaults to: {}) —

    provide a range key here if the table has a composite key

Since:

  • 1.0.0



224
225
226
227
228
229
230
231
232
233
234
235
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 224

def delete_item(table_name, key, options = {})
  range_key = options[:range_key]
  conditions = options[:conditions]
  table = describe_table(table_name)
  client.delete_item(
    table_name: table_name,
    key: key_stanza(table, key, range_key),
    expected: expected_stanza(conditions)
  )
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
  raise Dynamoid::Errors::ConditionalCheckFailedException, e
end

#delete_table(table_name, options = {}) ⇒ Object

Deletes an entire table from DynamoDB.

Parameters:

  • table_name (String) —

    the name of the table to destroy

  • options (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (options):

  • sync (Boolean) —

    Wait for table status check to raise ResourceNotFoundException

Since:

  • 1.0.0



243
244
245
246
247
248
249
250
251
252
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 243

def delete_table(table_name, options = {})
  resp = client.delete_table(table_name: table_name)
  until_past_table_status(table_name, :deleting) if options[:sync] &&
      (status = PARSE_TABLE_STATUS.call(resp, :table_description)) &&
      status != TABLE_STATUSES[:deleting]
  table_cache.delete(table_name)
rescue Aws::DynamoDB::Errors::ResourceInUseException => e
  Dynamoid.logger.error "Table #{table_name} cannot be deleted as it is in use"
  raise e
end

#delete_table_synchronously(table_name, options = {}) ⇒ Object 



254
255
256
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 254

def delete_table_synchronously(table_name, options = {})
  delete_table(table_name, options.merge(sync: true))
end

#get_item(table_name, key, options = {}) ⇒ Hash

TODO:

Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#get_item-instance_method

Fetches an item from DynamoDB.

Parameters:

  • table_name (String) —

    the name of the table

  • key (String) —

    the hash key of the item to find

  • options (Hash) (defaults to: {}) —

    provide a range key here if the table has a composite key

Returns:

  • (Hash) —

    a hash representing the raw item in DynamoDB

Since:

  • 1.0.0



271
272
273
274
275
276
277
278
279
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 271

def get_item(table_name, key, options = {})
  table    = describe_table(table_name)
  range_key = options.delete(:range_key)

  item = client.get_item(table_name: table_name,
    key: key_stanza(table, key, range_key)
  )[:item]
  item ? result_item_to_hash(item) : nil
end

#list_tables ⇒ Object

TODO:

Provide limit support docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#update_item-instance_method

List all tables on DynamoDB.

Since:

  • 1.0.0



316
317
318
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 316

def list_tables
  client.list_tables[:table_names]
end

#put_item(table_name, object, options = nil) ⇒ Object

Persists an item on DynamoDB.

@todo: Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#put_item-instance_method

Parameters:

  • table_name (String) —

    the name of the table

  • object (Object) —

    a hash or Dynamoid object to persist

Since:

  • 1.0.0



328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 328

def put_item(table_name, object, options = nil)
  item = {}

  object.each do |k, v|
    next if v.nil? || (v.respond_to?(:empty?) && v.empty?)
    item[k.to_s] = v
  end

  begin
    client.put_item(table_name: table_name,
      item: item,
      expected: expected_stanza(options)
    )
  rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
    raise Dynamoid::Errors::ConditionalCheckFailedException, e
  end
end

#query(table_name, opts = {}) ⇒ Enumerable

TODO:

Provide support for various other options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#query-instance_method

Query the DynamoDB table. This employs DynamoDB’s indexes so is generally faster than scanning, but is only really useful for range queries, since it can only find by one hash key at once. Only provide one range key to the hash.

Parameters:

  • table_name (String) —

    the name of the table

  • opts (Hash) (defaults to: {}) —

    the options to query the table with

Options Hash (opts):

  • :hash_value (String) —

    the value of the hash key to find

  • :range_between (Number, Number) —

    find the range key within this range

  • :range_greater_than (Number) —

    find range keys greater than this

  • :range_less_than (Number) —

    find range keys less than this

  • :range_gte (Number) —

    find range keys greater than or equal to this

  • :range_lte (Number) —

    find range keys less than or equal to this

Returns:

  • (Enumerable) —

    matching items

Since:

  • 1.0.0



364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 364

def query(table_name, opts = {})
  table = describe_table(table_name)
  hk    = (opts[:hash_key].present? ? opts[:hash_key] : table.hash_key).to_s
  rng   = (opts[:range_key].present? ? opts[:range_key] : table.range_key).to_s
  q     = opts.slice(
            :consistent_read,
            :scan_index_forward,
            :limit,
            :select,
            :index_name
          )

  opts.delete(:consistent_read)
  opts.delete(:scan_index_forward)
  opts.delete(:limit)
  opts.delete(:select)
  opts.delete(:index_name)

  opts.delete(:next_token).tap do |token|
    break unless token
    q[:exclusive_start_key] = {
      hk  => token[:hash_key_element],
      rng => token[:range_key_element]
    }
  end

  key_conditions = {
    hk => {
      # TODO: Provide option for other operators like NE, IN, LE, etc
      comparison_operator: EQ,
      attribute_value_list: [
        opts.delete(:hash_value).freeze
      ]
    }
  }

  opts.each_pair do |k, v|
    # TODO: ATM, only few comparison operators are supported, provide support for all operators
    next unless(op = RANGE_MAP[k])
    key_conditions[rng] = {
      comparison_operator: op,
      attribute_value_list: [
        opts.delete(k).freeze
      ].flatten # Flatten as BETWEEN operator specifies array of two elements
    }
  end

  q[:table_name]     = table_name
  q[:key_conditions] = key_conditions

  Enumerator.new { |y|
    result = client.query(q)

    result.items.each { |r|
      y << result_item_to_hash(r)
    }
  }
end

#scan(table_name, scan_hash, select_opts = {}) ⇒ Enumerable

Scan the DynamoDB table. This is usually a very slow operation as it naively filters all data on the DynamoDB servers.

@todo: Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#scan-instance_method

Parameters:

  • table_name (String) —

    the name of the table

  • scan_hash (Hash) —

    a hash of attributes: matching records will be returned by the scan

Returns:

  • (Enumerable) —

    matching items

Since:

  • 1.0.0



434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 434

def scan(table_name, scan_hash, select_opts = {})
  limit = select_opts.delete(:limit)
  batch = select_opts.delete(:batch_size)

  request = { table_name: table_name }
  request[:limit] = batch || limit if batch || limit
  request[:scan_filter] = scan_hash.reduce({}) do |memo, kvp|
    memo[kvp[0].to_s] = {
      attribute_value_list: [kvp[1]],
      # TODO: Provide support for all comparison operators
      comparison_operator: EQ
    }
    memo
  end if scan_hash.present?

  Enumerator.new do |y|
    # Batch loop, pulls multiple requests until done using the start_key
    loop do
      results = client.scan(request)

      results.data[:items].each { |row| y << result_item_to_hash(row) }

      if((lk = results[:last_evaluated_key]) && batch)
        request[:exclusive_start_key] = lk
      else
        break
      end
    end
  end
end

#truncate(table_name) ⇒ Object

Truncates all records in the given table

Parameters:

  • table_name (String) —

    the name of the table

Since:

  • 1.0.0



471
472
473
474
475
476
477
478
479
480
481
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 471

def truncate(table_name)
  table = describe_table(table_name)
  hk    = table.hash_key
  rk    = table.range_key

  scan(table_name, {}, {}).each do |attributes|
    opts = {}
    opts[:range_key] = attributes[rk.to_sym] if rk
    delete_item(table_name, attributes[hk], opts)
  end
end

#update_item(table_name, key, options = {}) {|iu = ItemUpdater.new(table, key, range_key)| ... } ⇒ Object

TODO:

Provide support for various options docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html#update_item-instance_method

Edits an existing item’s attributes, or adds a new item to the table if it does not already exist. You can put, delete, or add attribute values

Parameters:

  • table_name (String) —

    the name of the table

  • key (String) —

    the hash key of the item to find

  • options (Hash) (defaults to: {}) —

    provide a range key here if the table has a composite key

Yields:

Returns:

  • new attributes for the record



290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
 
# File 'lib/dynamoid/adapter_plugin/aws_sdk_v2.rb', line 290

def update_item(table_name, key, options = {})
  range_key = options.delete(:range_key)
  conditions = options.delete(:conditions)
  table = describe_table(table_name)

  yield(iu = ItemUpdater.new(table, key, range_key))

  raise "non-empty options: #{options}" unless options.empty?
  begin
    result = client.update_item(table_name: table_name,
      key: key_stanza(table, key, range_key),
      attribute_updates: iu.to_h,
      expected: expected_stanza(conditions),
      return_values: "ALL_NEW"
    )
    result_item_to_hash(result[:attributes])
  rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException => e
    raise Dynamoid::Errors::ConditionalCheckFailedException, e
  end
end