TapHttp

A task library for submitting http requests using Tap.

Description

TapHttp provides modules to construct and submit HTTP requests from a hash that specifies the target url, headers, parameters, etc. TapHttp is designed to work with a Ubiquity command called redirect-http; together they allow the capture and resubmission of web forms.

Usage

TapHttp submits http requests using the Tap::Http::Dispatch module. Headers, parameters, and other configurations may be specified, but the only required field is :url.

include Tap::Http
res = Dispatch.submit(
  :params => {'q' => 'tap-http'},
  :url => 'http://www.google.com/search')
res.body[0,80]  # => "<!doctype html><head><title>tap-http - Google Search</title><style>body{backgrou"

Getting Http Configurations

More complicated http requests may be captured and resubmitted using a combination of tools that redirects web forms to a tap server and reformats the request as YAML. To do so:

Install Firefox
Install Ubiquity
Install redirect-http

Start a tap server from the command line (of course tap-http must be installed):

% tap server

Now in the browser, go to a web form like google and invoke the redirection.

Bring up Ubiquity in Firefox by pressing ‘option+space’
Enter the command: ‘redirect-http localhost:8080/http_to_yaml’

You should see a notice that the form is being redirected. Fill out the form and submit as normal; the redirect command will send the form to the tap server instead of performing the original action. The tap server returns a yaml file with the http configuration.

# Copy and paste into a configuration file. Multiple configs
# can be added to a single file to perform batch submission.  
- headers: 
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Accept-Encoding: gzip,deflate
    Accept-Language: en-us,en;q=0.5
    Connection: keep-alive
    Host: www.google.com
    Keep-Alive: "300"
    Referer: http://www.google.com/
    User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4
  params: 
    aq: f
    btnG: Google Search
    hl: en
    oq: ""
    q: tap-http
  request_method: GET
  url: http://www.google.com/search
  version: 1.1

Save the file as ‘request.yml’ and resubmit the form using the Tap::Http::Dispatch task.

% rap load requests.yml --:i dispatch --+ dump --no-audit
  I[10:51:40]               load request.yml
  I[10:51:40]                GET http://www.google.com/search
  I[10:51:41]                    OK
# date: 2008-11-25 10:51:41
---  
tap/http/request (2772040): 
- - !ruby/object:Net::HTTPOK 
    body: !binary |
      H4sIAAAAAAAC/6xabXPbNhL+3l/B0BeN1FAUJfktoihf07pppmkm06TXu0lz
      HZAEScQgQZOQZVfhf79dgBRJS4ndmRvPSAC42F3sPvsCyssnoQjkXU6NRKZ8
      tUwoCVdLySSnK0nycSJlboyNl0LEnBrvKCmCZDnRz5elvIMvX4R3W58EV3Eh
      1lm4OIqiyA0EF8XiyHEc...

Note the result is encoded as gzip, as per the parameters. As with all tasks, the request results could be passed into a workflow. Alternatively, the configuration could be used to submit the request using Dispatch.

Bugs/Known Issues

The Tap::Http::Utils#parse_cgi_request (used in parsing redirected requests into a YAML file) is currently untested because I can’t figure a way to setup the ENV variables in a standard way. Of course I could set them up myself, but I can’t be sure I’m setting up a realistic test environment.

The capture procedure seems to work in practice, but please report bugs and let me know if you know a way to setup a CGI environment for testing!

Installation

TapHttp is available as a gem on RubyForge. Use:

% gem install tap-http

Info

Developer: Simon Chiang, Biomolecular Structure Program, Hansen Lab
Support: CU Denver School of Medicine Deans Academic Enrichment Fund
Licence: MIT-Style