Class: Bobik::Client

Inherits:
Object
  • Object
show all
Includes:
HTTParty
Defined in:
lib/bobik/client.rb

Overview

Author

Eugene Mirkin

This is the main class for interacting with Bobik platform.

Instance Method Summary collapse

Constructor Details

#initialize(opts) ⇒ Client

Notable parameters:

  • :auth_token - [required] authentication token

  • :timeout_ms - [optional] when to stop waiting for the job to finish

  • :logger - [optional] any logger that conforms to the Log4r interface

Be aware that Timeout::Error may be thrown.



16
17
18
19
20
 
# File 'lib/bobik/client.rb', line 16

def initialize(opts)
  @auth_token = opts[:auth_token] || raise(Error.new("'auth_token' was not provided"))
  @timeout_ms = opts[:timeout_ms] || 60000
  @log = opts[:logger] || (defined?(Rails.logger) && Rails.logger)
end

Instance Method Details

#get_job_data(job_id, with_results) ⇒ Object

A single call to get a given job’s status with or without results



67
68
69
70
71
72
73
 
# File 'lib/bobik/client.rb', line 67

def get_job_data(job_id, with_results)
  job_response = self.class.get('/jobs.json', :body => {
    auth_token: @auth_token,
    no_results: !with_results,
    job:        job_id
  })
end

#log(msg) ⇒ Object 



76
77
78
79
 
# File 'lib/bobik/client.rb', line 76

def log(msg)
  return unless @log
  @log.debug(msg)
end

#scrape(request, block_until_done, &block) ⇒ Object

Submit a scraping request. Request is a JSON composed in accordance with usebobik.com/api/docs The callback block will be invoked when results arrive. If asynchronous mode is used, the method returns right away. Otherwise, it blocks until results arrive.

Raises:



26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
 
# File 'lib/bobik/client.rb', line 26

def scrape(request, block_until_done, &block)
  request = Marshal.load(Marshal.dump(request))
  request[:auth_token] = @auth_token

  job_response = self.class.post('/jobs.json', :body => request)
  raise Error.new(job_response['errors'].join("\n")) if job_response['errors']
  job_id = job_response['job']

  Thread.abort_on_exception = true
  t = Thread.new do
    wait_until_finished(job_id, &block)
  end
  t.join if block_until_done
  true
end

#wait_until_finished(job_id, &block) ⇒ Object

Blocks until the job is finished or timeout is reached. When done, yields results to the optional block. Exceptions thrown: Timeout::Error, Errno::ECONNRESET, Errno::ECONNREFUSED



45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/bobik/client.rb', line 45

def wait_until_finished(job_id, &block)
  log("Waiting for job #{job_id} to finish")
  results = nil
  errors = nil
  Timeout::timeout(@timeout_ms.to_f/1000) do
    while true
      job_response = get_job_data(job_id, false)
      progress = job_response['progress']
      log("Job #{job_id} progress: #{progress*100}%")
      if progress == 1
        job_response = get_job_data(job_id, true)
        results = job_response['results']
        errors = job_response['errors']
        break
      end
      sleep(job_response['estimated_time_left_ms'].to_f/1000)
    end
  end
  block.call(results, errors)
end