Class: Manticore::Client

Inherits:
Object
  • Object
show all
Includes:
ProxiesInterface
Defined in:
lib/manticore/client.rb,
lib/manticore/client/proxies.rb

Overview

Core Manticore client, with a backing PoolingHttpClientConnectionManager

Defined Under Namespace

Modules: ProxiesInterface Classes: AsyncProxy, BackgroundProxy, BaseProxy, StubProxy

Constant Summary collapse

DEFAULT_MAX_POOL_SIZE =

The default maximum pool size for requests

50
DEFAULT_REQUEST_TIMEOUT = 
60
DEFAULT_SOCKET_TIMEOUT = 
10
DEFAULT_CONNECT_TIMEOUT = 
10
DEFAULT_MAX_REDIRECTS = 
5
DEFAULT_EXPECT_CONTINUE = 
false
DEFAULT_STALE_CHECK = 
false

Instance Method Summary collapse

Methods included from ProxiesInterface

#async, #background, #respond_with

Constructor Details

#initialize(options = {}) {|builder, request_config| ... } ⇒ Client

Create a new HTTP client with a backing request pool. if you pass a block to the initializer, the underlying HttpClientBuilder and RequestConfig.Builder will be yielded so that you can operate on them directly.

Examples:

Simple instantiation and usage

client = Manticore::Client.new
client.get("http://www.google.com")

Instantiation with a block

client = Manticore::Client.new(socket_timeout: 5) do |http_client_builder, request_builder|
  http_client_builder.disable_redirect_handling
end

Parameters:

  • options (Hash) (defaults to: {}) —

    Client pool options

Options Hash (options):

  • user_agent (String) —

    The user agent used in requests.

  • pool_max (Integer) — default: 50 —

    The maximum number of active connections in the pool

  • pool_max_per_route (integer) — default: pool_max —

    Sets the maximum number of active connections for a given target endpoint

  • cookies (boolean) — default: true —

    enable or disable automatic cookie management between requests

  • compression (boolean) — default: true —

    enable or disable transparent gzip/deflate support

  • request_timeout (integer) — default: 60 —

    Sets the timeout for requests. Raises Timeout on failure.

  • connect_timeout (integer) — default: 10 —

    Sets the timeout for connections. Raises Manticore::Timeout on failure.

  • socket_timeout (integer) — default: 10 —

    Sets SO_TIMEOUT for open connections. A value of 0 is an infinite timeout. Raises Manticore::Timeout on failure.

  • tcp_no_delay (boolean) — default: true —

    Enable or disable Nagle’s algorithm

  • request_timeout (integer) — default: 60 —

    Sets the timeout for a given request. Raises Manticore::Timeout on failure.

  • max_redirects (integer) — default: 5 —

    Sets the maximum number of redirects to follow.

  • automatic_retries (integer) — default: 3 —

    Sets the number of times the client will automatically retry failed requests.

  • expect_continue (boolean) — default: false —

    Enable support for HTTP 100

  • stale_check (boolean) — default: false —

    Enable support for stale connection checking. Adds overhead.

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • keepalive (Boolean/Integer) — default: true —

    Whether to allow connections to be reused. Defaults to true. If an integer, then connections will be kept alive for this long when Connection: keep-alive is sent, but no Keep-Alive header is sent.

Yields:

  • (builder, request_config)

See Also:



125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/manticore/client.rb', line 125

def initialize(options = {})
  builder  = client_builder
  builder.set_user_agent options.fetch(:user_agent, "Manticore #{VERSION}")
  @use_cookies = options.fetch(:cookies, false)
  builder.disable_cookie_management unless @use_cookies
  builder.disable_content_compression if options.fetch(:compression, true) == false
  builder.set_proxy get_proxy_host(options[:proxy]) if options.key?(:proxy)

  builder.set_retry_handler do |exception, executionCount, context|
    if (executionCount > options.fetch(:automatic_retries, 3))
      false
    else
      case exception
      when Java::OrgApacheHttp::NoHttpResponseException, Java::JavaNet::SocketException
        context.setAttribute "retryCount", executionCount
        true
      else
        false
      end
    end
  end

  # This should make it easier to reuse connections
  # TODO: Determine what this actually does!
  # builder.disable_connection_state

  keepalive = options.fetch(:keepalive, true)
  if keepalive == false
    builder.set_connection_reuse_strategy {|response, context| false }
  else
    builder.set_connection_reuse_strategy DefaultConnectionReuseStrategy.new
  end

  socket_config_builder = SocketConfig.custom
  socket_config_builder.set_so_timeout( options.fetch(:socket_timeout, DEFAULT_SOCKET_TIMEOUT) * 1000 )
  socket_config_builder.set_tcp_no_delay( options.fetch(:tcp_no_delay, true) )
  builder.set_default_socket_config socket_config_builder.build

  builder.set_connection_manager pool(options)

  request_config = RequestConfig.custom
  request_config.set_connection_request_timeout     options.fetch(:request_timeout, DEFAULT_REQUEST_TIMEOUT) * 1000
  request_config.set_connect_timeout                options.fetch(:connect_timeout, DEFAULT_CONNECT_TIMEOUT) * 1000
  request_config.set_socket_timeout                 options.fetch(:socket_timeout, DEFAULT_SOCKET_TIMEOUT) * 1000
  request_config.set_max_redirects                  options.fetch(:max_redirects, DEFAULT_MAX_REDIRECTS)
  request_config.set_expect_continue_enabled        options.fetch(:expect_continue, DEFAULT_EXPECT_CONTINUE)
  request_config.set_stale_connection_check_enabled options.fetch(:stale_check, DEFAULT_STALE_CHECK)
  request_config.set_circular_redirects_allowed false

  yield builder, request_config if block_given?

  builder.set_default_request_config request_config.build
  @client = builder.build
  @options = options
  @async_requests = []
  @stubs = {}
end

Instance Method Details

#clear_pending ⇒ Object

Remove all pending asynchronous requests.

Returns:

  • nil



264
265
266
267
 
# File 'lib/manticore/client.rb', line 264

def clear_pending
  @async_requests.clear
  nil
end

#clear_stubs! ⇒ Object

Wipe all currently-set stubs.



257
258
259
 
# File 'lib/manticore/client.rb', line 257

def clear_stubs!
  @stubs.clear
end

#delete(url, options = {}, &block) ⇒ Response

Perform a HTTP DELETE request

Examples:

Simple usage

body = client.delete("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.delete("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.delete("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



222
223
224
 
# File 'lib/manticore/client.rb', line 222

def delete(url, options = {}, &block)
  request HttpDelete, url, options, &block
end

#execute! ⇒ Array

Execute all queued async requests

Returns:

  • (Array) —

    An array of the responses from the requests executed.



272
273
274
275
276
277
 
# File 'lib/manticore/client.rb', line 272

def execute!
  method = executor.java_method(:submit, [java.util.concurrent.Callable.java_class])
  result = @async_requests.map {|r| method.call r }
  @async_requests.clear
  result.map(&:get)
end

#executor ⇒ Object

Get at the underlying ExecutorService used to invoke asynchronous calls.



280
281
282
283
 
# File 'lib/manticore/client.rb', line 280

def executor
  create_executor_if_needed
  @executor
end

#get(url, options = {}, &block) ⇒ Response

Perform a HTTP GET request

Examples:

Simple usage

body = client.get("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.get("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.get("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



198
199
200
 
# File 'lib/manticore/client.rb', line 198

def get(url, options = {}, &block)
  request HttpGet, url, options, &block
end

#head(url, options = {}, &block) ⇒ Response

Perform a HTTP HEAD request

Examples:

Simple usage

body = client.head("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.head("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.head("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



210
211
212
 
# File 'lib/manticore/client.rb', line 210

def head(url, options = {}, &block)
  request HttpHead, url, options, &block
end

#options(url, options = {}, &block) ⇒ Response

Perform a HTTP OPTIONS request

Examples:

Simple usage

body = client.options("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.options("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.options("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



228
229
230
 
# File 'lib/manticore/client.rb', line 228

def options(url, options = {}, &block)
  request HttpOptions, url, options, &block
end

#patch(url, options = {}, &block) ⇒ Response

Perform a HTTP PATCH request

Examples:

Simple usage

body = client.patch("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.patch("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.patch("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

  • body (Hash) —

    Hash of options to pass as request body

Returns:

Raises:



234
235
236
 
# File 'lib/manticore/client.rb', line 234

def patch(url, options = {}, &block)
  request HttpPatch, url, options, &block
end

#pool_stats ⇒ Object

Return a hash of statistics about this client’s HTTP connection pool



184
185
186
187
188
189
190
191
192
 
# File 'lib/manticore/client.rb', line 184

def pool_stats
  stats = @pool.get_total_stats
  {
    max: stats.get_max,
    leased: stats.get_leased,
    pending: stats.get_pending,
    available: stats.get_available
  }
end

#post(url, options = {}, &block) ⇒ Response

Perform a HTTP POST request

Examples:

Simple usage

body = client.post("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.post("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.post("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

  • body (Hash) —

    Hash of options to pass as request body

Returns:

Raises:



216
217
218
 
# File 'lib/manticore/client.rb', line 216

def post(url, options = {}, &block)
  request HttpPost, url, options, &block
end

#put(url, options = {}, &block) ⇒ Response

Perform a HTTP PUT request

Examples:

Simple usage

body = client.put("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.put("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.put("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

  • body (Hash) —

    Hash of options to pass as request body

Returns:

Raises:



204
205
206
 
# File 'lib/manticore/client.rb', line 204

def put(url, options = {}, &block)
  request HttpPut, url, options, &block
end

#stub(url, stubs) ⇒ Object

Cause this client to return a stubbed response for this URL

Parameters:

  • url (String) —

    URL to stub for

  • stubs (Hash) —

    Hash of options to return for the stubbed response



247
248
249
 
# File 'lib/manticore/client.rb', line 247

def stub(url, stubs)
  @stubs[url] = stubs
end

#unstub(url) ⇒ Object

Cause this client to unstubbed previously-stubbed URL



252
253
254
 
# File 'lib/manticore/client.rb', line 252

def unstub(url)
  @stubs.delete(url)
end