Class: Remi::SfBulkHelper::SfBulkOperation

Inherits:
Object
  • Object
show all
Defined in:
lib/remi/sf_bulk_helper.rb

Overview

Public: Class used to execute SF Bulk operations. This class is not meant to be used directly. It is instead meant to be inherited by classes that perform the specific query, update, create, or upsert operations.

Examples

sf_query = SfBulkQuery.query(client, 'Contact', 'SELECT Id, Name FROM Contact')
puts sf_query.result

mydata = [ { 'Id' => '001G000000ncxb8IAA', 'Name' => 'Happy Feet' } ]
sf_update = SfBulkUpdate.update(client, 'Contact', mydata)

Direct Known Subclasses

SfBulkCreate, SfBulkQuery, SfBulkUpdate, SfBulkUpsert

Instance Method Summary collapse

Constructor Details

#initialize(restforce_client, object, data, batch_size: 5000, max_attempts: 2, logger: Logger.new(STDOUT)) ⇒ SfBulkOperation

Public: Initializes a SfBulkOperation (does not execute operation).

restforce_client - An instance of Restforce that is used to authenticate the connection. object - The name of the object to operate on (e.g., Contact, Task, etc). data - For query operations, this is the SOQL query string. For other

operations, this is an array of hashes, where the hash keys are column names
and the values are the data.

batch_size - Batch size to use to download or upload data (default: 10000) max_mattempts - The maximum number of attempts to upload data (default: 2) logger - Logger to use (default: Logger.new(STDOUT)



30
31
32
33
34
35
36
37
38
 
# File 'lib/remi/sf_bulk_helper.rb', line 30

def initialize(restforce_client, object, data, batch_size: 5000, max_attempts: 2, logger: Logger.new(STDOUT))
  @restforce_client = restforce_client
  @object = object
  @data = data
  @batch_size = batch_size
  @max_attempts = max_attempts
  @attempts = Hash.new(0)
  @logger = logger
end

Instance Method Details

#as_lookup(key:, duplicates: false) ⇒ Object

Public: Converts the result into a hash that can be used to lookup the row for a given key (e.g., external id field).

key - A string representing the name of the column to be used as the lookup key. duplicates - Indicates whether duplicate keys are allowed. If they are allowed,

only the first row found will be retained.  If duplicates are not allowed,
an error is raised (default: false).

Returns a hash.



103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/remi/sf_bulk_helper.rb', line 103

def as_lookup(key:, duplicates: false)
  execute if @attempts[:total] == 0

  @as_lookup ||= {}
  @attempts[:as_lookup] = Hash.new(0) if @attempts[:as_lookup] == 0

  return @as_lookup[key] if @as_lookup[key] and @attempts[:as_lookup][key] == @attempts[:total]
  @attempts[:as_lookup][key] += 1

  @as_lookup[key] = result.inject({}) do |lkp,row|
    raise DupeLookupKeyError, "Duplicate key: #{row[key]} found in result of query: #{@data}" if lkp.has_key?(row[key]) and not duplicates
    lkp[row[key]] = row unless lkp.has_key?(row[key])
    lkp
  end
end

#failed_records? ⇒ Boolean

Public: Returns true if any of the records failed to update.

Returns:

  • (Boolean) 


121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/remi/sf_bulk_helper.rb', line 121

def failed_records?
  n_failed_records = result.reduce(0) do |count, row|
    count += 1 if row['success'] != 'true'
    count
  end

  n_failed_batches = raw_result['batches'].reduce(0) do |count, batch|
    count += 1 if batch['state'].first != 'Completed'
    count
  end

  n_failed_records > 0 || n_failed_batches > 0
end

#info ⇒ Object

Public: Returns useful metadata about the batch query.



56
57
58
59
60
61
62
63
64
65
 
# File 'lib/remi/sf_bulk_helper.rb', line 56

def info
  execute if @attempts[:total] == 0

  return @info if @info and @attempts[:info] == @attempts[:total]
  @attempts[:info] += 1

  @info = raw_result.reject { |k,v| k == 'batches' }.tap do |h|
    h['query'] = @data if operation == :query
  end
end

#operation ⇒ Object

Public: A symbol representing the operation to be performed (:query, :update, :create, :upsert)



41
42
43
 
# File 'lib/remi/sf_bulk_helper.rb', line 41

def operation
  :undefined
end

#raw_result ⇒ Object

Public: Returns the raw result from the SalesforceBulkApi query



51
52
53
 
# File 'lib/remi/sf_bulk_helper.rb', line 51

def raw_result
  @raw_result || execute
end

#result ⇒ Object

Public: Collects the results from all of the batches and aggregates them into an array of hashes. Each element of the array represents a record in the result and the hash gives the column-value. Note that if multiple retries are needed, this is just the final result.

Returns an array of hashes.



73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/remi/sf_bulk_helper.rb', line 73

def result
  execute if @attempts[:total] == 0

  return @result if @result and @attempts[:result] == @attempts[:total]
  @attempts[:result] += 1

  @result = []
  raw_result['batches'].each do |batch|
    next unless batch['response']

    batch['response'].each do |record|
      @result << record.inject({}) { |h, (k,v)| h[k] = v.first unless ['xsi:type','type'].include? k; h }
    end

    # delete raw result at end of processing to free memory
    batch['response'] = nil
  end

  @result
end

#sf_bulk ⇒ Object

Public: Returns the instance of SalesforceBulkApi::Api used for bulk operations.



46
47
48
 
# File 'lib/remi/sf_bulk_helper.rb', line 46

def sf_bulk
  @sf_bulk ||= SalesforceBulkApi::Api.new(@restforce_client).tap { |o| o.connection.set_status_throttle(5) }
end