Class: FTW::Agent

Inherits:
Object
  • Object
show all
Includes:
Configuration, Protocol
Defined in:
lib/ftw/agent.rb

Overview

This should act as a proper web agent.

  • Reuse connections.

  • SSL/TLS.

  • HTTP Upgrade support.

  • HTTP 1.1 (RFC2616).

  • WebSockets (RFC6455).

  • Support Cookies.

All standard HTTP methods defined by RFC2616 are available as methods on this agent: get, head, put, etc.

Example:

agent = FTW::Agent.new
request = agent.get("http://www.google.com/")
response = agent.execute(request)
puts response.body.read

For any standard http method (like ‘get’) you can invoke it with ‘!’ on the end and it will execute and return a FTW::Response object:

agent = FTW::Agent.new
response = agent.get!("http://www.google.com/")
puts response.body.head

TODO(sissel): TBD: implement cookies… delicious chocolate chip cookies.

Defined Under Namespace

Modules: Configuration Classes: TooManyRedirects

Constant Summary collapse

STANDARD_METHODS =

List of standard HTTP methods described in RFC2616

%w(options get head post put delete trace connect)

Constants included from Configuration

Configuration::REDIRECTION_LIMIT, Configuration::SSL_CIPHERS, Configuration::SSL_CIPHER_DEFAULT, Configuration::SSL_CIPHER_MAP, Configuration::SSL_TRUST_STORE, Configuration::SSL_USE_DEFAULT_CERTS, Configuration::SSL_VERSION

Constants included from CRLF

CRLF::CRLF

Instance Method Summary collapse

Methods included from Configuration

#configuration

Methods included from Protocol

#discard_body, #encode_chunked, #read_body, #read_http_body, #read_http_body_chunked, #read_http_body_length, #read_http_message, #write_all, #write_http_body, #write_http_body_chunked, #write_http_body_normal

Constructor Details

#initialize ⇒ Agent

Returns a new instance of Agent.



60
61
62
63
64
65
66
 
# File 'lib/ftw/agent.rb', line 60

def initialize
  @pool = FTW::Pool.new
  @logger = Cabin::Channel.get

  configuration[REDIRECTION_LIMIT] = 20

end

Instance Method Details

#execute(request) ⇒ FTW::Response

Execute a FTW::Request in this Agent.

If an existing, idle connection is already open to the target server of this Request, it will be reused. Otherwise, a new connection is opened.

Redirects are always followed.

Parameters:

Returns:



278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
 
# File 'lib/ftw/agent.rb', line 278

def execute(request)
  # TODO(sissel): Make redirection-following optional, but default.

  tries = 3
  begin
    connection, error = connect(request.headers["Host"], request.port,
                                request.protocol == "https")
    if !error.nil?
      p :error => error
      raise error
    end
    response = request.execute(connection)
  rescue EOFError => e
    tries -= 1
    @logger.warn("Error while sending request, will retry.",
                 :tries_left => tries,
                 :exception => e)
    retry if tries > 0
  end

  redirects = 0
  # Follow redirects
  while response.redirect? and response.headers.include?("Location")
    # RFC2616 section 10.3.3 indicates HEAD redirects must not include a
    # body. Otherwise, the redirect response can have a body, so let's
    # throw it away.
    if request.method == "HEAD" 
      # Head requests have no body
      connection.release
    elsif response.content?
      # Throw away the body
      response.body = connection
      # read_body will consume the body and release this connection
      response.read_http_body { |chunk| }
    end

    # TODO(sissel): If this response has any cookies, store them in the
    # agent's cookie store

    redirects += 1
    if redirects > configuration[REDIRECTION_LIMIT]
      # TODO(sissel): include original a useful debugging information like
      # the trace of redirections, etc.
      raise TooManyRedirects.new("Redirect more than " \
          "#{configuration[REDIRECTION_LIMIT]} times, aborting.", response)
      # I don't like this api from FTW::Agent. I think 'get' and other methods
      # should return (object, error), and if there's an error 
    end

    @logger.debug("Redirecting", :location => response.headers["Location"])
    request.use_uri(response.headers["Location"])
    connection, error = connect(request.headers["Host"], request.port, request.protocol == "https")
    # TODO(sissel): Do better error handling than raising.
    if !error.nil?
      p :error => error
      raise error
    end
    response = request.execute(connection)
  end # while being redirected

  # RFC 2616 section 9.4, HEAD requests MUST NOT have a message body.
  if request.method != "HEAD"
    response.body = connection
  else
    connection.release
  end
 
  # TODO(sissel): If this response has any cookies, store them in the
  # agent's cookie store
  return response
end

#request(method, uri, options) ⇒ Object

Build a request. Returns a FTW::Request object.

Arguments:

  • method - the http method

  • uri - the URI to make the request to

  • options - a hash of options

uri can be a valid url or an Addressable::URI object. The uri will be used to choose the host/port to connect to. It also sets the protocol (https, etc). Further, it will set the ‘Host’ header.

The ‘options’ hash supports the following keys:

  • :headers => { string => string, … }. This allows you to set header values.



249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
 
# File 'lib/ftw/agent.rb', line 249

def request(method, uri, options)
  @logger.info("Creating new request", :method => method, :uri => uri, :options => options)
  request = FTW::Request.new(uri)
  request.method = method
  request.headers.add("Connection", "keep-alive")

  if options.include?(:headers)
    options[:headers].each do |key, value|
      request.headers.add(key, value)
    end
  end

  if options.include?(:body)
    request.body = options[:body]
  end

  return request
end

#shutdown ⇒ Object

shutdown this agent.

This will shutdown all active connections.



353
354
355
356
357
358
359
 
# File 'lib/ftw/agent.rb', line 353

def shutdown
  @pool.each do |identifier, list|
    list.each do |connection|
      connection.disconnect("stopping agent")
    end
  end
end

#upgrade!(uri, protocol, options = {}) ⇒ Object

Send the request as an HTTP upgrade.

Returns the response and the FTW::Connection for this connection. If the upgrade was denied, the connection returned will be nil.



189
190
191
192
193
194
195
196
197
198
199
200
 
# File 'lib/ftw/agent.rb', line 189

def upgrade!(uri, protocol, options={})
  req = request("GET", uri, options)
  req.headers["Connection"] = "Upgrade"
  req.headers["Upgrade"] = protocol
  response = execute(req)
  if response.status == 101
    # Success, return the response object and the connection to hand off.
    return response, response.body
  else
    return response, nil
  end
end

#websocket!(uri, options = {}) ⇒ Object

Make a new websocket connection.

This will send the http request. If the websocket handshake is successful, a FTW::WebSocket instance will be returned. Otherwise, a FTW::Response will be returned.

See #request for what the ‘uri’ and ‘options’ parameters should be.



209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
 
# File 'lib/ftw/agent.rb', line 209

def websocket!(uri, options={})
  # TODO(sissel): Use FTW::Agent#upgrade! ?
  req = request("GET", uri, options)
  ws = FTW::WebSocket.new(req)
  response = execute(req)
  if ws.handshake_ok?(response)
    # response.body is a FTW::Connection
    ws.connection = response.body

    # There seems to be a bug in http_parser.rb where websocket responses
    # lead with a newline for some reason.  It's like the header terminator
    # CRLF still has the LF character left in the buffer. Work around it.
    data = response.body.read
    if data[0] == "\n"
      response.body.pushback(data[1..-1])
    else
      response.body.pushback(data)
    end

    return ws
  else
    return response
  end
end