Mechanize examples
Note: Several examples show methods chained to the end of do/end blocks. do...end
is the same as curly braces ({...}
). For example, do ... end.submit
is the same as { ... }.submit
.
require 'rubygems'
require 'mechanize'
a = Mechanize.new { |agent|
agent.user_agent_alias = 'Mac Safari'
}
a.get('http://google.com/') do |page|
search_result = page.form_with(:id => 'gbqf') do |search|
search.q = 'Hello world'
end.submit
search_result.links.each do |link|
puts link.text
end
end
File Upload
Upload a file to flickr.
require 'rubygems'
require 'mechanize'
abort "#{$0} login passwd filename" if (ARGV.size != 3)
a = Mechanize.new { |agent|
# Flickr refreshes after login
agent. = true
}
a.get('http://flickr.com/') do |home_page|
signin_page = a.click(home_page.link_with(:text => /Sign In/))
my_page = signin_page.form_with(:name => 'login_form') do |form|
form.login = ARGV[0]
form.passwd = ARGV[1]
end.submit
# Click the upload link
upload_page = a.click(my_page.link_with(:text => /Upload/))
# We want the basic upload page.
upload_page = a.click(upload_page.link_with(:text => /basic Uploader/))
# Upload the file
upload_page.form_with(:method => 'POST') do |upload_form|
upload_form.file_uploads.first.file_name = ARGV[2]
end.submit
end
Pluggable Parsers
Let’s say you want HTML pages to automatically be parsed with Rubyful Soup. This example shows you how:
require 'rubygems'
require 'mechanize'
require 'rubyful_soup'
class SoupParser < Mechanize::Page
attr_reader :soup
def initialize(uri = nil, response = nil, body = nil, code = nil)
@soup = BeautifulSoup.new(body)
super(uri, response, body, code)
end
end
agent = Mechanize.new
agent.pluggable_parser.html = SoupParser
Now all HTML pages will be parsed with the SoupParser class, and automatically give you access to a method called ‘soup’ where you can get access to the Beautiful Soup for that page.
Using a proxy
require 'rubygems'
require 'mechanize'
agent = Mechanize.new
agent.set_proxy 'localhost', 8000
page = agent.get(ARGV[0])
puts page.body
The transact method
Mechanize#transact runs the given block and then resets the page history. I.e. after the block has been executed, you’re back at the original page; no need to count how many times to call the back method at the end of a loop (while accounting for possible exceptions).
This example also demonstrates subclassing Mechanize.
require 'rubygems'
require 'mechanize'
class TestMech < Mechanize
def process
get 'http://rubygems.org/'
search_form = page.forms.first
search_form.words = 'WWW'
submit search_form
page.links_with(:href => %r{/projects/} ).each do |link|
next if link.href =~ %r{/projects/support/}
puts 'Loading %-30s %s' % [link.href, link.text]
begin
transact do
click link
# Do stuff, maybe click more links.
end
# Now we're back at the original page.
rescue => e
$stderr.puts "#{e.class}: #{e.}"
end
end
end
end
TestMech.new.process
Client Certificate Authentication (Mutual Auth)
In most cases a client certificate is created as an additional layer of security for certain websites. The specific case that this was initially tested on was for automating the download of archived images from a banks (Wachovia) lockbox system. Once the certificate is installed into your browser you will have to export it and split the certificate and private key into separate files.
require 'rubygems'
require 'mechanize'
# create Mechanize instance
agent = Mechanize.new
# set the path of the certificate file
agent.cert = 'example.cer'
# set the path of the private key file
agent.key = 'example.key'
# get the login form & fill it out with the username/password
login_form = agent.get("http://example.com/login_page").form('Login')
login_form.Userid = 'TestUser'
login_form.Password = 'TestPassword'
# submit login form
agent.submit(login_form, login_form..first)
Exported files are usually in .p12 format (IE 7 & Firefox 2.0) which stands for PKCS #12. You can convert them from p12 to pem format by using the following commands:
openssl pkcs12 -in input_file.p12 -clcerts -out example.key -nocerts -nodes
openssl pkcs12 -in input_file.p12 -clcerts -out example.cer -nokeys