Class: Manticore::Client

Inherits:
Object
  • Object
show all
Includes:
ProxiesInterface
Defined in:
lib/manticore/client.rb,
lib/manticore/client/proxies.rb

Overview

Core Manticore client, with a backing PoolingHttpClientConnectionManager

Defined Under Namespace

Modules: ProxiesInterface Classes: AsyncProxy, BackgroundProxy, BaseProxy, StubProxy

Constant Summary collapse

DEFAULT_MAX_POOL_SIZE =

The default maximum pool size for requests

50
DEFAULT_REQUEST_TIMEOUT = 
60
DEFAULT_SOCKET_TIMEOUT = 
10
DEFAULT_CONNECT_TIMEOUT = 
10
DEFAULT_MAX_REDIRECTS = 
5
DEFAULT_EXPECT_CONTINUE = 
false
DEFAULT_STALE_CHECK = 
false

Instance Method Summary collapse

Methods included from ProxiesInterface

#async, #background, #respond_with

Constructor Details

#initialize(options = {}) {|builder, request_config| ... } ⇒ Client

Create a new HTTP client with a backing request pool. if you pass a block to the initializer, the underlying HttpClientBuilder and RequestConfig.Builder will be yielded so that you can operate on them directly.

Examples:

Simple instantiation and usage

client = Manticore::Client.new
client.get("http://www.google.com")

Instantiation with a block

client = Manticore::Client.new(socket_timeout: 5) do |http_client_builder, request_builder|
  http_client_builder.disable_redirect_handling
end

Parameters:

  • options (Hash) (defaults to: {}) —

    Client pool options

Options Hash (options):

  • user_agent (String) —

    The user agent used in requests.

  • pool_max (Integer) — default: 50 —

    The maximum number of active connections in the pool

  • pool_max_per_route (integer) — default: pool_max —

    Sets the maximum number of active connections for a given target endpoint

  • cookies (boolean) — default: true —

    enable or disable automatic cookie management between requests

  • compression (boolean) — default: true —

    enable or disable transparent gzip/deflate support

  • request_timeout (integer) — default: 60 —

    Sets the timeout for requests. Raises Timeout on failure.

  • connect_timeout (integer) — default: 10 —

    Sets the timeout for connections. Raises Manticore::Timeout on failure.

  • socket_timeout (integer) — default: 10 —

    Sets SO_TIMEOUT for open connections. A value of 0 is an infinite timeout. Raises Manticore::Timeout on failure.

  • tcp_no_delay (boolean) — default: true —

    Enable or disable Nagle’s algorithm

  • request_timeout (integer) — default: 60 —

    Sets the timeout for a given request. Raises Manticore::Timeout on failure.

  • max_redirects (integer) — default: 5 —

    Sets the maximum number of redirects to follow.

  • automatic_retries (integer) — default: 3 —

    Sets the number of times the client will automatically retry failed requests.

  • expect_continue (boolean) — default: false —

    Enable support for HTTP 100

  • stale_check (boolean) — default: false —

    Enable support for stale connection checking. Adds overhead.

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’[, user: ‘username@host’, password: ‘password’]]]

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org:1234’[, user: ‘username@host’, password: ‘password’]]]

  • proxy (URI) —

    Proxy host as a URI object

  • keepalive (Boolean/Fixnum) — default: true —

    Whether to allow connections to be reused. Defaults to true. If an integer, then connections will be kept alive for this long when Connection: keep-alive is sent, but no Keep-Alive header is sent.

  • ssl (Hash) —

    Hash of options for configuring SSL

  • ssl[:protocols] (Array<String>) — default: nil —

    A list of protocols that Manticore should accept

  • ssl[:cipher_suites] (Array<String>) — default: nil —

    A list of cipher suites that Manticore should accept

  • ssl[:verify] (Symbol) — default: :strict —

    Hostname verification setting. Set to ‘:disable` to turn off hostname verification. Setting to `:browser` will cause Manticore to accept a certificate for *.foo.com for all subdomains and sub-subdomains (eg a.b.foo.com). The default `:strict` is like `:browser` except it’ll only accept a single level of subdomains for wildcards, eg ‘b.foo.com` will be accepted for a `*.foo.com` certificate, but `a.b.foo.com` will not be.

  • ssl[:truststore] (String) — default: nil —

    Path to a custom trust store to use the verifying SSL connections

  • ssl[:truststore_password] (String) — default: nil —

    Password used for decrypting the server trust store

  • ssl[:truststore_type] (String) — default: nil —

    Format of the trust store, ie “JKS” or “PKCS12”. If left nil, the type will be inferred from the truststore filename.

  • ssl[:keystore] (String) — default: nil —

    Path to a custom key store to use for client certificate authentication

  • ssl[:keystore_password] (String) — default: nil —

    Password used for decrypting the client auth key store

  • ssl[:keystore_type] (String) — default: nil —

    Format of the key store, ie “JKS” or “PKCS12”. If left nil, the type will be inferred from the keystore filename.

  • ssl[:ca_file] (String) — default: nil —

    OpenSSL-style path to an X.509 certificate to use to validate SSL certificates

  • ssl[:client_cert] (String) — default: nil —

    OpenSSL-style path to an X.509 certificate to use for client authentication

  • ssl[:client_key] (String) — default: nil —

    OpenSSL-style path to an RSA key to use for client authentication

  • ssl[:track_state] (boolean) — default: false —

    Turn on or off connection state tracking. This helps prevent SSL information from leaking across threads, but means that connections can’t be shared across those threads. This should generally be left off unless you know what you’re doing.

Yields:

  • (builder, request_config)

See Also:



152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/manticore/client.rb', line 152

def initialize(options = {})
  builder  = client_builder
  builder.set_user_agent options.fetch(:user_agent, "Manticore #{VERSION}")
  @options = options
  @use_cookies = options.fetch(:cookies, false)
  builder.disable_cookie_management unless @use_cookies
  builder.disable_content_compression if options.fetch(:compression, true) == false
  builder.set_proxy get_proxy_host(options[:proxy]) if options.key?(:proxy)

  builder.set_retry_handler do |exception, executionCount, context|
    if (executionCount > options.fetch(:automatic_retries, 3))
      false
    else
      case exception
      when Java::OrgApacheHttp::NoHttpResponseException, Java::JavaNet::SocketException
        context.setAttribute "retryCount", executionCount
        true
      else
        false
      end
    end
  end

  # http://hc.apache.org/httpcomponents-client-ga/tutorial/html/advanced.html#stateful_conn
  # By default this is used to prevent different contexts from accessing SSL data
  # Since we're running this for JRuby which does not have context separation within the JVM
  # We can disable this for connection reuse.
  builder.disable_connection_state unless options.fetch(:ssl, {}).fetch(:track_state, false)

  keepalive = options.fetch(:keepalive, true)
  if keepalive == false
    builder.set_connection_reuse_strategy {|response, context| false }
  else
    builder.set_connection_reuse_strategy DefaultConnectionReuseStrategy.new
  end

  socket_config_builder = SocketConfig.custom
  socket_config_builder.set_so_timeout( options.fetch(:socket_timeout, DEFAULT_SOCKET_TIMEOUT) * 1000 )
  socket_config_builder.set_tcp_no_delay( options.fetch(:tcp_no_delay, true) )
  builder.set_default_socket_config socket_config_builder.build

  builder.set_connection_manager pool(options)

  request_config = RequestConfig.custom
  request_config.set_connection_request_timeout     options.fetch(:request_timeout, DEFAULT_REQUEST_TIMEOUT) * 1000
  request_config.set_connect_timeout                options.fetch(:connect_timeout, DEFAULT_CONNECT_TIMEOUT) * 1000
  request_config.set_socket_timeout                 options.fetch(:socket_timeout, DEFAULT_SOCKET_TIMEOUT) * 1000
  request_config.set_max_redirects                  options.fetch(:max_redirects, DEFAULT_MAX_REDIRECTS)
  request_config.set_expect_continue_enabled        options.fetch(:expect_continue, DEFAULT_EXPECT_CONTINUE)
  request_config.set_stale_connection_check_enabled options.fetch(:stale_check, DEFAULT_STALE_CHECK)
  request_config.set_circular_redirects_allowed false

  yield builder, request_config if block_given?

  builder.set_default_request_config request_config.build
  @client = builder.build
  @options = options
  @async_requests = []
  @stubs = {}
end

Instance Method Details

#clear_pending ⇒ Object

Remove all pending asynchronous requests.

Returns:

  • nil



307
308
309
310
 
# File 'lib/manticore/client.rb', line 307

def clear_pending
  @async_requests.clear
  nil
end

#clear_stubs! ⇒ Object

Wipe all currently-set stubs.



300
301
302
 
# File 'lib/manticore/client.rb', line 300

def clear_stubs!
  @stubs.clear
end

#delete(url, options = {}, &block) ⇒ Response

Perform a HTTP DELETE request

Examples:

Simple usage

body = client.delete("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.delete("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.delete("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



252
253
254
 
# File 'lib/manticore/client.rb', line 252

def delete(url, options = {}, &block)
  request HttpDelete, url, options, &block
end

#execute! ⇒ Array

Execute all queued async requests

Returns:

  • (Array) —

    An array of the responses from the requests executed.



315
316
317
318
319
320
321
322
323
324
325
326
 
# File 'lib/manticore/client.rb', line 315

def execute!
  method = executor.java_method(:submit, [java.util.concurrent.Callable.java_class])
  result = @async_requests.map {|r| method.call r }
  @async_requests.clear
  result.map do |future|
    begin
      future.get
    rescue Java::JavaUtilConcurrent::ExecutionException => e
      # These exceptions should be handled in on_failure blocks.
    end
  end
end

#executor ⇒ Object

Get at the underlying ExecutorService used to invoke asynchronous calls.



329
330
331
332
 
# File 'lib/manticore/client.rb', line 329

def executor
  create_executor_if_needed
  @executor
end

#get(url, options = {}, &block) ⇒ Response

Perform a HTTP GET request

Examples:

Simple usage

body = client.get("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.get("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.get("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



228
229
230
 
# File 'lib/manticore/client.rb', line 228

def get(url, options = {}, &block)
  request HttpGetWithEntity, url, options, &block
end

#head(url, options = {}, &block) ⇒ Response

Perform a HTTP HEAD request

Examples:

Simple usage

body = client.head("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.head("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.head("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



240
241
242
 
# File 'lib/manticore/client.rb', line 240

def head(url, options = {}, &block)
  request HttpHead, url, options, &block
end

#http(method, url, options = {}, &block) ⇒ Response

Perform an HTTP request, passing the method as a parameter

Parameters:

  • method (String, Symbol) —

    Method to call (get put head post options patch)

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



278
279
280
281
282
283
284
285
 
# File 'lib/manticore/client.rb', line 278

def http(method, url, options = {}, &block)
  case method.to_s.downcase
  when *%w(get put head post delete options patch)
    send method, url, options, &block
  else
    raise "Invalid method: #{method}"
  end
end

#options(url, options = {}, &block) ⇒ Response

Perform a HTTP OPTIONS request

Examples:

Simple usage

body = client.options("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.options("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.options("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

Returns:

Raises:



258
259
260
 
# File 'lib/manticore/client.rb', line 258

def options(url, options = {}, &block)
  request HttpOptions, url, options, &block
end

#patch(url, options = {}, &block) ⇒ Response

Perform a HTTP PATCH request

Examples:

Simple usage

body = client.patch("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.patch("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.patch("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

  • body (String) —

    Body to pass with the request

Returns:

Raises:



264
265
266
 
# File 'lib/manticore/client.rb', line 264

def patch(url, options = {}, &block)
  request HttpPatch, url, options, &block
end

#pool_stats ⇒ Object

Return a hash of statistics about this client’s HTTP connection pool



214
215
216
217
218
219
220
221
222
 
# File 'lib/manticore/client.rb', line 214

def pool_stats
  stats = @pool.get_total_stats
  {
    max: stats.get_max,
    leased: stats.get_leased,
    pending: stats.get_pending,
    available: stats.get_available
  }
end

#post(url, options = {}, &block) ⇒ Response

Perform a HTTP POST request

Examples:

Simple usage

body = client.post("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.post("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.post("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

  • body (String) —

    Body to pass with the request

Returns:

Raises:



246
247
248
 
# File 'lib/manticore/client.rb', line 246

def post(url, options = {}, &block)
  request HttpPost, url, options, &block
end

#put(url, options = {}, &block) ⇒ Response

Perform a HTTP PUT request

Examples:

Simple usage

body = client.put("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).body

Passing a block as the success handler:

body = client.put("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}) {|response| response.body }

Explicit success handler:

body = client.put("http://example.com/some/resource", params: {foo: "bar"}, headers: {"X-Custom-Header" => "whee"}).
  on_success {|response| response.body }

Parameters:

  • url (String) —

    URL to request

  • options (Hash) (defaults to: {})

Options Hash (options):

  • params (Hash) —

    Hash of options to pass as request parameters

  • headers (Hash) —

    Hash of options to pass as additional request headers

  • proxy (String) —

    Proxy host in form: proxy.org:1234

  • proxy (Hash) —

    Proxy host in form: ‘proxy.org’[, port: 80[, scheme: ‘http’]]

  • proxy (URI) —

    Proxy host as a URI object

  • connect_timeout (Integer) —

    Request-specific connect timeout

  • socket_timeout (Integer) —

    Request-specific socket timeout

  • request_timeout (Integer) —

    Request-specific request timeout

  • max_redirects (Integer) —

    Request-specific maximum redirect limit

  • follow_redirects (Boolean) —

    Specify whether this request should follow redirects

  • body (String) —

    Body to pass with the request

Returns:

Raises:



234
235
236
 
# File 'lib/manticore/client.rb', line 234

def put(url, options = {}, &block)
  request HttpPut, url, options, &block
end

#stub(url, stubs) ⇒ Object

Cause this client to return a stubbed response for this URL

Parameters:

  • url (String) —

    URL to stub for

  • stubs (Hash) —

    Hash of options to return for the stubbed response



290
291
292
 
# File 'lib/manticore/client.rb', line 290

def stub(url, stubs)
  @stubs[url_as_regex(url)] = stubs
end

#unstub(url) ⇒ Object

Cause this client to unstubbed previously-stubbed URL



295
296
297
 
# File 'lib/manticore/client.rb', line 295

def unstub(url)
  @stubs.delete(url_as_regex(url))
end