MetaInspector

MetaInspector is a gem for web scraping purposes. You give it an URL, and it returns you metadata from it.

Dependencies

MetaInspector uses the nokogiri gem to parse HTML. You can install it from github.

Run the following if you haven’t already:

gem sources -a http://gems.github.com

Then install the gem:

sudo gem install tenderlove-nokogiri

If you’re on Ubuntu, you might need to install these packages before installing nokogiri:

sudo aptitude install libxslt-dev libxml2 libxml2-dev

Installation

Run the following if you haven’t already:

gem sources -a http://gems.github.com

Then install the gem:

sudo gem install jaimeiniesta-metainspector

Usage

Initialize a MetaInspector instance with an URL like this:

page = MetaInspector.new('http://pagerankalert.com')

Once scraped, you can see the scraped data like this:

page.address       # URL of the page
page.title         # title of the page, as string
page.description   # meta description, as string
page.keywords      # meta keywords, as string
page.links         # array of strings, with every link found on the page

The full scraped document if accessible from:

page.document # Nokogiri doc that you can use it to get any element from the page

Examples

You can find some sample scripts on the samples folder, including a basic scraping and a spider that will follow external links using a queue. What follows is an example of use from irb:

$ irb
>> require 'metainspector'
=> true

>> page = MetaInspector.new('http://pagerankalert.com')
=> #<MetaInspector:0x11330c0 @document=nil, @links=nil, @address="http://pagerankalert.com", @description=nil, @keywords=nil, @title=nil>

>> page.title
=> "PageRankAlert.com :: Track your pagerank changes"

>> page.description
=> "Track your PageRank(TM) changes and receive alert by email"

>> page.keywords
=> "pagerank, seo, optimization, google"

>> page.links.size
=> 31

>> page.links[30]
=> "http://www.nuvio.cz/"

>> page.document.class
=> Nokogiri::HTML::Document

To Do

Get page.base_dir from the address
Distinguish between external and internal links, returning page.links for all of them as found, page.external_links and page.internal_links converted to absolute URLs
Return array of images in page as absolute URLs
Return contents of meta robots tag
Be able to set a timeout in seconds
Detect charset
If keywords seem to be separated by blank spaces, replace them with commas
Mocks
Check content type, process only HTML pages, don’t try to scrape TAR files like ftp.ruby-lang.org/pub/ruby/ruby-1.9.1-p129.tar.bz2 or video files like isabel.dit.upm.es/component/option,com_docman/task,doc_download/gid,831/Itemid,74/