Class: Krikri::AsyncUriGetter

Inherits:
Object
  • Object
show all
Defined in:
lib/krikri/async_uri_getter.rb

Overview

Helper class for fetching multiple URLs concurrently.

At this point, 5 threads are launched to fetch the list of URLs. We can wait for them all to finish if we want to make sure we don’t continue until all threads have terminated: ‘requests.map(&:join)`

Or simply access the responses and have our current thread block until they’re available:

requests.each do |request|
  request.with_response do |response|
    if response.status == 200
      puts "Response body: #{response.body}"
    else
      puts "Got return status: #{response.status}"
    end
  end
end

Examples:

to fetch 5 URLs in 5 threads

urls = ['http://example.com/one',
        'http://example.com/two',
        'http://example.com/three',
        'http://example.com/four',
        'http://example.com/five']
       .map { |url| URI.parse(url) }

getter = Krikri::AsyncUriGetter.new

requests = urls.map do |url|
  getter.add_request(uri: url, opts: { follow_redirects: true })
end

Defined Under Namespace

Classes: Request

Constant Summary collapse

MAX_REDIRECTS = 
10

Instance Method Summary collapse

Constructor Details

#initialize(opts: {}) ⇒ AsyncUriGetter

Create a new asynchronous URL fetcher.

Parameters:

  • opts (Hash) (defaults to: {}) —

    a hash of the supported options, which are:

Options Hash (opts:):

  • :follow_redirects (Boolean) —

    Whether to follow HTTP 3xx redirects.

  • :max_redirects (Integer) —

    Number of redirects to follow before giving up. (default: 10)

  • :inline_exceptions (Boolean) —

    If true, pass exceptions as a 5xx response with the exception string in the body. (default: false)



56
57
58
 
# File 'lib/krikri/async_uri_getter.rb', line 56

def initialize(opts: {})
  @default_opts = { max_redirects: MAX_REDIRECTS }.merge(opts)
end

Instance Method Details

#add_request(uri: nil, headers: {}, opts: {}) ⇒ Object

Run a request (in a new thread) and return a promise-like object for the response.

Parameters:

  • uri (URI) (defaults to: nil) —

    the URI to be fetched

  • headers (Hash<String, String>) (defaults to: {}) —

    HTTP headers to include with the request

  • opts (Hash) (defaults to: {}) —

    options to override the ones provided when AsyncUriGetter was initialized. All supported options from ‘#initialize` are available here as well.



70
71
72
73
 
# File 'lib/krikri/async_uri_getter.rb', line 70

def add_request(uri: nil, headers: {}, opts: {})
  fail ArgumentError, "uri must be a URI; got: #{uri}" unless uri.is_a?(URI)
  Request.new(uri, headers, @default_opts.merge(opts))
end