em-http-fetcher

HTTP fetch client based on ruby EventMachne and EM-HTTP-Request that has configureable concurrency regardless of EM's thread pool.

Example

EM.run do
trap(:INT) { EM.stop }
fetcher = EM::HttpFetcher.new
fetcher.callback do |req| # req is HttpRequest instance
  # Here is global callback block for all request
  p "Fetch success! #{req.last_effective_url} (#{req.response.size} bytes)"
end

%w(
  http://www.google.com/
  http://heroku.com/
  http://sourceforge.net/
  http://github.com/
).each do |url|
  fetcher.request url
end

req = fetcher.request 'http://www.ruby-lang.org/'
req.callback do
  # Here is appendix callback block for this request.
  # Global callback block will also be called.
  puts "Hello Ruby!"
end
end

Install

You can install with gem. However em-http-request (=< 1.0.3) has a redirection issue. If you need to handle redirection properly, try following workaround.

$ gem install em-http-fetcher

Workaround with bundler

gem install bundler

And create Gemfile to fetch develop version of em-http-request.

source "http://rubygems.org"
gem 'em-http-request', :git => 'git://github.com/igrigorik/em-http-request.git'

Then run bundle to install gems.

$ bundle

Finally run your script with "bundle exec".

$ bundle exec YOUR_SCRIPT

Usage

Options for HttpFetcher.new

[:concurrency] Concurrency for all request. [:host_concurrency] Concurrency per host. [:host_request_wait] Wait specified seconds after request on each request thread. [(all other keys)] Pass through for HttpRequest.new

Options for HttpFetcher#request

[:uri] Target URI (String or URI object) [:method] Request method (get/head/put...) (default=:get) [(all other keys)] Pass through for HttpRequest#(get/head/put...)

If first argument is not a hash, it will be treated as :uri.

Limitations

:host_concurrency is checked only for initial URI. When request is redirected to another host, number of parallel requests for one host may be over host_concurrency.
Redirections will not work until issue #230 of em-http-request is resolved; https://github.com/igrigorik/em-http-request/pull/230

License

Same as Ruby 2.0 (2-clause BSDL or Ruby original license)