Class: FTW::Agent

Inherits:
Object
  • Object
show all
Includes:
Configuration, Protocol
Defined in:
lib/ftw/agent.rb

Overview

This should act as a proper web agent.

  • Reuse connections.

  • SSL/TLS.

  • HTTP Upgrade support.

  • HTTP 1.1 (RFC2616).

  • WebSockets (RFC6455).

  • Support Cookies.

All standard HTTP methods defined by RFC2616 are available as methods on this agent: get, head, put, etc.

Example:

agent = FTW::Agent.new
request = agent.get("http://www.google.com/")
response = agent.execute(request)
puts response.body.read

For any standard http method (like ‘get’) you can invoke it with ‘!’ on the end and it will execute and return a FTW::Response object:

agent = FTW::Agent.new
response = agent.get!("http://www.google.com/")
puts response.body.head

TODO(sissel): TBD: implement cookies… delicious chocolate chip cookies.

Defined Under Namespace

Modules: Configuration Classes: TooManyRedirects

Constant Summary collapse

STANDARD_METHODS =

List of standard HTTP methods described in RFC2616

%w(options get head post put delete trace connect)

Constants included from Configuration

Configuration::REDIRECTION_LIMIT, Configuration::SSL_TRUST_STORE

Constants included from CRLF

CRLF::CRLF

Instance Method Summary collapse

Methods included from Configuration

#configuration

Methods included from Protocol

#encode_chunked, #read_body, #read_http_body, #read_http_body_chunked, #read_http_body_length, #read_http_message, #write_http_body, #write_http_body_chunked, #write_http_body_normal

Constructor Details

#initialize ⇒ Agent

Returns a new instance of Agent.



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/ftw/agent.rb', line 60

def initialize
  @pool = FTW::Pool.new
  @logger = Cabin::Channel.get

  configuration[REDIRECTION_LIMIT] = 20

  @certificate_store = OpenSSL::X509::Store.new
  if File.readable?(OpenSSL::X509::DEFAULT_CERT_FILE)
    @logger.debug("Adding default certificate file",
                  :path => OpenSSL::X509::DEFAULT_CERT_FILE)
    @certificate_store.add_file(OpenSSL::X509::DEFAULT_CERT_FILE)
  else
    # Use some better defaults from http://curl.haxx.se/docs/caextract.html
    @logger.info("Using upstream ssl certs, possibly untrusty.")
    default_ca = File.join(File.dirname(__FILE__), "cacert.pem")

    # JRUBY-6870 - strip 'jar:' prefix if it is present.
    if default_ca =~ /^jar:file.*!/
      default_ca.gsub!(/^jar:/, "")
    end
    @certificate_store.add_file(default_ca)
  end

  # Handle the local user/app trust store as well.
  if File.directory?(configuration[SSL_TRUST_STORE])
    # This is a directory, so use add_path
    @logger.debug("Adding SSL_TRUST_STORE",
                  :path => configuration[SSL_TRUST_STORE])
    @certificate_store.add_path(configuration[SSL_TRUST_STORE])
  end

  # TODO(sissel): Add custom paths for ssl certs
end

Instance Method Details

#execute(request) ⇒ FTW::Response

Execute a FTW::Request in this Agent.

If an existing, idle connection is already open to the target server of this Request, it will be reused. Otherwise, a new connection is opened.

Redirects are always followed.

Parameters:

Returns:



302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
 
# File 'lib/ftw/agent.rb', line 302

def execute(request)
  # TODO(sissel): Make redirection-following optional, but default.

  connection, error = connect(request.headers["Host"], request.port,
                              request.protocol == "https")
  if !error.nil?
    p :error => error
    raise error
  end
  response = request.execute(connection)

  redirects = 0
  # Follow redirects
  while response.redirect? and response.headers.include?("Location")
    # RFC2616 section 10.3.3 indicates HEAD redirects must not include a
    # body. Otherwise, the redirect response can have a body, so let's
    # throw it away.
    if request.method == "HEAD" 
      # Head requests have no body
      connection.release
    elsif response.content?
      # Throw away the body
      response.body = connection
      # read_body will consume the body and release this connection
      response.read_http_body { |chunk| }
    end

    # TODO(sissel): If this response has any cookies, store them in the
    # agent's cookie store

    redirects += 1
    if redirects > configuration[REDIRECTION_LIMIT]
      # TODO(sissel): include original a useful debugging information like
      # the trace of redirections, etc.
      raise TooManyRedirects.new("Redirect more than " \
          "#{configuration[REDIRECTION_LIMIT]} times, aborting.", response)
      # I don't like this api from FTW::Agent. I think 'get' and other methods
      # should return (object, error), and if there's an error 
    end

    @logger.debug("Redirecting", :location => response.headers["Location"])
    request.use_uri(response.headers["Location"])
    connection, error = connect(request.headers["Host"], request.port, request.protocol == "https")
    # TODO(sissel): Do better error handling than raising.
    if !error.nil?
      p :error => error
      raise error
    end
    response = request.execute(connection)
  end # while being redirected

  # RFC 2616 section 9.4, HEAD requests MUST NOT have a message body.
  if request.method != "HEAD"
    response.body = connection
  else
    connection.release
  end
 
  # TODO(sissel): If this response has any cookies, store them in the
  # agent's cookie store
  return response
end

#request(method, uri, options) ⇒ Object

Build a request. Returns a FTW::Request object.

Arguments:

  • method - the http method

  • uri - the URI to make the request to

  • options - a hash of options

uri can be a valid url or an Addressable::URI object. The uri will be used to choose the host/port to connect to. It also sets the protocol (https, etc). Further, it will set the ‘Host’ header.

The ‘options’ hash supports the following keys:

  • :headers => { string => string, … }. This allows you to set header values.



273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
 
# File 'lib/ftw/agent.rb', line 273

def request(method, uri, options)
  @logger.info("Creating new request", :method => method, :uri => uri, :options => options)
  request = FTW::Request.new(uri)
  request.method = method
  request.headers.add("Connection", "keep-alive")

  if options.include?(:headers)
    options[:headers].each do |key, value|
      request.headers.add(key, value)
    end
  end

  if options.include?(:body)
    request.body = options[:body]
  end

  return request
end

#shutdown ⇒ Object

shutdown this agent.

This will shutdown all active connections.



368
369
370
371
372
373
374
 
# File 'lib/ftw/agent.rb', line 368

def shutdown
  @pool.each do |identifier, list|
    list.each do |connection|
      connection.disconnect("stopping agent")
    end
  end
end

#upgrade!(uri, protocol, options = {}) ⇒ Object

Send the request as an HTTP upgrade.

Returns the response and the FTW::Connection for this connection. If the upgrade was denied, the connection returned will be nil.



213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/ftw/agent.rb', line 213

def upgrade!(uri, protocol, options={})
  req = request("GET", uri, options)
  req.headers["Connection"] = "Upgrade"
  req.headers["Upgrade"] = protocol
  response = execute(req)
  if response.status == 101
    # Success, return the response object and the connection to hand off.
    return response, response.body
  else
    return response, nil
  end
end

#websocket!(uri, options = {}) ⇒ Object

Make a new websocket connection.

This will send the http request. If the websocket handshake is successful, a FTW::WebSocket instance will be returned. Otherwise, a FTW::Response will be returned.

See #request for what the ‘uri’ and ‘options’ parameters should be.



233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
 
# File 'lib/ftw/agent.rb', line 233

def websocket!(uri, options={})
  # TODO(sissel): Use FTW::Agent#upgrade! ?
  req = request("GET", uri, options)
  ws = FTW::WebSocket.new(req)
  response = execute(req)
  if ws.handshake_ok?(response)
    # response.body is a FTW::Connection
    ws.connection = response.body

    # There seems to be a bug in http_parser.rb where websocket responses
    # lead with a newline for some reason.  It's like the header terminator
    # CRLF still has the LF character left in the buffer. Work around it.
    data = response.body.read
    if data[0] == "\n"
      response.body.pushback(data[1..-1])
    else
      response.body.pushback(data)
    end

    return ws
  else
    return response
  end
end