em-http-fetcher

HTTP fetch client based on ruby EventMachne and EM-HTTP-Request that has configureable concurrency regardless of EM’s thread pool.

Example

EM.run do
  trap(:INT) { EM.stop }
  fetcher = EM::HttpFetcher.new
  fetcher.callback do |req| # req is HttpRequest instance
    # Here is global callback block for all request
    p "Fetch success! #{req.last_effective_url} (#{req.response.size} bytes)"
  end

  %w(
    http://www.google.com/
    http://heroku.com/
    http://sourceforge.net/
    http://github.com/
  ).each do |url|
    fetcher.request url
  end

  req = fetcher.request 'http://www.ruby-lang.org/'
  req.callback do
    # Here is appendix callback block for this request.
    # Global callback block will also be called.
    puts "Hello Ruby!"
  end
end

Install

You can install with gem. However em-http-request (=< 1.0.3) has a redirection issue. If you need to handle redirection properly, try following workaround.

$ gem install em-http-fetcher

Workaround with bundler

gem install bundler

And create Gemfile to fetch develop version of em-http-request.

source "http://rubygems.org"
gem 'em-http-request', :git => 'git://github.com/igrigorik/em-http-request.git'

Then run bundle to install gems.

$ bundle

Finally run your script with “bundle exec”.

$ bundle exec YOUR_SCRIPT

Usage

Options for HttpFetcher.new

:concurrency: Concurrency for all request.
:host_concurrency: Concurrency per host.
:host_request_wait: Wait specified seconds after request on each request thread.
(all other keys): Pass through for HttpRequest.new

Options for HttpFetcher#request

:uri: Target URI (String or URI object)
:method: Request method (get/head/put…) (default=:get)
(all other keys): Pass through for HttpRequest#(get/head/put…)

If first argument is not a hash, it will be treated as :uri.

Limitations

:host_concurrency is checked only for initial URI. When request is redirected to another host, number of parallel
```
requests for one host may be over host_concurrency.
```
Redirections will not work until issue #230 of em-http-request is resolved; github.com/igrigorik/em-http-request/pull/230

License

Same as Ruby 2.0 (2-clause BSDL or Ruby original license)