Module: FulfilApi::Relation::Batchable

Included in:
FulfilApi::Relation
Defined in:
lib/fulfil_api/relation/batchable.rb

Overview

The Batchable module includes a set of

helper/query methods that queries resources in Fulfil in batches.

Defined Under Namespace

Classes: RetryLimitExceeded

Instance Method Summary collapse

Instance Method Details

#find_each(batch_size: 500) {|FulfilApi::Resource| ... } ⇒ FulfilApi::Relation

The #find_each is a shorthand for iterating over individual API resources

in a memory effective way.

Under the hood, it uses the #in_batches to find API resources in batches

and process them efficiently.

Examples:

find all resources


FulfilApi::Resource.set(model_name: "sale.sale").find_each do |sales_order|
  process_sales_order(sales_order)
end

Parameters:

  • batch_size (Integer) (defaults to: 500) —

    The default batch forwarded to the #in_batches method.

Yields:

Returns:



27
28
29
30
31
 
# File 'lib/fulfil_api/relation/batchable.rb', line 27

def find_each(batch_size: 500, &block)
  in_batches(of: batch_size) do |batch|
    batch.each(&block)
  end
end

#in_batches(of: 500, retries: :unlimited) {|FulfilApi::Relation| ... } ⇒ FulfilApi::Relation

Note:

the #in_batches automatically retries when it encounters a 429 (TooManyRequests) HTTP error to ensure the lookup can be completed.

Finds API resources in batches. Defaults to the maximum number of resources

Fulfil's API endpoints will return (500 resources).

Examples:

find resources in batches of 10.


FulfilApi::Resource.set(model_name: "sale.sale").in_batches(of: 10) do |batch|
  batch.each do |sales_order|
    process_sales_order(sales_order)
  end
end

Parameters:

  • of (Integer) (defaults to: 500) —

    The maximum number of resources in a batch.

  • retries (Symbol, Integer) (defaults to: :unlimited) —

    The maximum number of retries before raising.

Yields:

  • (FulfilApi::Relation) —

    Yields FulfilApi::Relation objects to work with a batch of records.

Returns:



52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/fulfil_api/relation/batchable.rb', line 52

def in_batches(of: 500, retries: :unlimited) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize
  current_retry = 0
  current_offset = request_offset.presence || 0
  batch_size = of

  loop do
    batch_relation = dup.offset(current_offset * batch_size).limit(batch_size)
    batch_relation.load

    yield(batch_relation)

    break unless batch_relation.size == batch_size

    current_offset += 1
    current_retry = 0 # Reset the retries back to the default
  rescue FulfilApi::Error => e
    if e.details[:response_status] == 429
      if retries != :unlimited && current_retry > retries
        raise RetryLimitExceeded, "the maximum number of #{retries} retries has been reached."
      end

      current_retry += 1
      sleep 0.25
      retry
    end

    raise e
  end

  self
end