Class: Chimps::Workflows::BatchUpdater

Inherits:
Object
  • Object
show all
Defined in:
lib/chimps/workflows/batch.rb

Overview

A class for performing batch updates/uploads to Infochimps.

It works by taking YAML data describing many updates and performing a single batch API request with this data.

The batch response is then parsed and analyzed and (given success or fearlessness) any necessary uploads are performed.

Examples of the input data format can be found in the /examples directory of the Chimps distribution.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(data, options = {}) ⇒ Chimps::Workflows::BatchUpdater

Create a new BatchUpdater with the given data and options.

The intermediate batch response can be saved at a file named by :output_path, though this isn’t necessary.

Parameters:

  • data (Array) —

    an array of resource updates

  • options (Hash) (defaults to: {})

Options Hash (options):

  • output_path (String) —

    path to store the batch response

  • upload_even_if_errors (true, false) —

    whether to continue uploading in the presence of errors on update

  • fmt (String) —

    the data format to annotate each upload with (see ‘chimps upload’)



44
45
46
47
48
49
 
# File 'lib/chimps/workflows/batch.rb', line 44

def initialize data, options={}
  @data                  = data
  @output_path           = options[:output_path]
  @upload_even_if_errors = options[:upload_even_if_errors]
  @fmt                   = options[:fmt]
end

Instance Attribute Details

#batch_response ⇒ Object (readonly)

The batch update response



20
21
22
 
# File 'lib/chimps/workflows/batch.rb', line 20

def batch_response
  @batch_response
end

#data ⇒ Object (readonly)

The data used sent as a bulk update.



17
18
19
 
# File 'lib/chimps/workflows/batch.rb', line 17

def data
  @data
end

#fmt ⇒ Object (readonly)

The data format to annotate the upload with.

Chimps will try to guess if this isn’t given.



31
32
33
 
# File 'lib/chimps/workflows/batch.rb', line 31

def fmt
  @fmt
end

#output_path ⇒ Object (readonly)

The output file to store the bulk update response.



23
24
25
 
# File 'lib/chimps/workflows/batch.rb', line 23

def output_path
  @output_path
end

#upload_even_if_errors ⇒ Object (readonly)

Whether to upload even if there were errors on update.



26
27
28
 
# File 'lib/chimps/workflows/batch.rb', line 26

def upload_even_if_errors
  @upload_even_if_errors
end

Instance Method Details

#batch_path ⇒ String

The path to submit batch update requests.

Returns:



54
55
56
 
# File 'lib/chimps/workflows/batch.rb', line 54

def batch_path
  "batch.json"
end

#batch_update! ⇒ Object

Perform the batch update.



65
66
67
68
69
 
# File 'lib/chimps/workflows/batch.rb', line 65

def batch_update!
  @batch_response = Request.new(batch_path, :data => { :batch => data }, :authenticate => true).post
  File.open(output_path, 'w') { |f| f.puts batch_response.body } if output_path
  batch_response.print
end

#batch_upload! ⇒ Object

Perform the batch upload.

Will bail if the batch update had an error unless Chimps::Workflows::BatchUpdater#upload_even_if_errors returns true.



95
96
97
98
99
100
101
 
# File 'lib/chimps/workflows/batch.rb', line 95

def batch_upload!
  return unless success? || upload_even_if_errors
  $stderr.puts("WARNING: continuing with uploads even though there were errors") unless success?
  dataset_ids_and_local_paths.each do |id, local_paths|
    Chimps::Workflows::Uploader.new(:dataset => id, :local_paths => local_paths, :fmt => fmt).execute!
  end
end

#error? ⇒ true, false

Were any of the updates performed during the batch update errors?

Returns:

  • (true, false) 


75
76
77
78
79
80
81
 
# File 'lib/chimps/workflows/batch.rb', line 75

def error?
  batch_response['batch'].each do |response|
    status = response['status']
    return true unless ['created', 'updated'].include?(status)
  end
  false
end

#execute! ⇒ Object

Perform this batch update followed by the batch upload.



59
60
61
62
 
# File 'lib/chimps/workflows/batch.rb', line 59

def execute!
  batch_update!
  batch_upload!
end

#success? ⇒ true, false

Did all of the updates performed in the batch update succeed?

Returns:

  • (true, false) 


86
87
88
 
# File 'lib/chimps/workflows/batch.rb', line 86

def success?
  ! error?
end