Class: Metrocot

Inherits:

Object

• Object
• Metrocot

Defined in:

lib/metrocot.rb

Overview

Purpose

This class implements the main Metrocot HTML scanner and a number of handy input scanners (for grabbing time, numbers, or text from HTML). The purpose of the Metrocot is to scan a XML dom for the patterns specified in the Metrocot pattern language.

Pattern Language

The Metrocot pattern language allows for the following types of patterns:

...

matches anything

+“some string”

matches that string

+/(some|pattern)/ matches that regexp pattern

./HPRICOT_PATH

matches a certain type of dom subtree

SPACE

matches zero or more white spaces

+(PATTERN_A PATTERN_B)

matches PATTERN_A followed by PATTERN_B

PATTERN+

matches one or more occurrences of PATTERN

Usage

0) create a Metricot and define the types of fields you want to extract (and their names). 1) use Hpricot to get the doc’s dom 2) use descend(xpath) to create a NodeScraper rooted at the Hpricot node(s) matching the xpath 3) use collect(pattern) to collect all entries found in the HTML which match the Metricot pattern

Defined Under Namespace

Modules: Scanners Classes: AnythingPattern, BasePattern, CompositePattern, MatchRange, NodeScraper, OneOrMorePattern, OptSpacePattern, PathPattern, TextPattern

Constant Summary

VERSION =

'1.0.3'

Instance Attribute Summary

• #verbose ⇒ Object

  Returns the value of attribute verbose.

Instance Method Summary

• #compile_pattern(pattern_s, node_scraper) ⇒ Object
• #initialize(scanners) ⇒ Metrocot constructor

  A new instance of Metrocot.

• #log(s) ⇒ Object
• #scanner_by_name(name) ⇒ Object

  given a Symbol of a scanner (name), return a handle for that scanner declared when the metrocot was created.

• #scrape(doc) ⇒ Object

Constructor Details

#initialize(scanners) ⇒ Metrocot

Returns a new instance of Metrocot.

# File 'lib/metrocot.rb', line 1154

def initialize( scanners )

  @scanners = {}
  @compiled_patterns = {}
  
  scanners.each { |name, value|
    if value.is_a? Class
      @scanners[name] = value.new
    else
      @scanners[name] = value
    end
  }

  @verbose = false

  log("scanners: #{@scanners.inspect}")

end

Instance Attribute Details

#verbose ⇒ Object

Returns the value of attribute verbose.

# File 'lib/metrocot.rb', line 1151

def verbose
  @verbose
end

Instance Method Details

#compile_pattern(pattern_s, node_scraper) ⇒ Object

# File 'lib/metrocot.rb', line 1181

def compile_pattern( pattern_s, node_scraper )  

#   if @compiled_patterns.key? pattern_s 
#     return @compiled_patterns[ pattern_s ] 
#   end

  s = pattern_s
  patterns = []

  log("compiling: #{s}")
  
  while !s.empty? 

    log("left: #{s}")

    if s.index(")") == 0
      break
    end

    name = nil
    s.scan( /^(\w+)=/ ) { |match| 
      log "got name #{match.class} (#{match.size})"
      name = match[0]
    }
    s = s[(name.length + 1) .. -1] if name

    log("after name #{name}: #{s}")

    pattern = nil

    [PathPattern, TextPattern, AnythingPattern, OptSpacePattern].each { |pattern_class|
      pattern = pattern_class.parse(s)
      if pattern 
        pattern.metrocot = self
        pattern.node_scraper = node_scraper
        break 
      end
      log "not a #{pattern_class}"
    }

    if pattern 
      s = s[pattern.source.size .. -1]
      patterns << pattern
      log("found: #{pattern.description}")
      if name
        log("scanned as: #{name}")
        pattern.name = name.to_sym
      end
      next
    end
  
    if s[0..0] == "+" 
      raise "+ must follow pattern" unless patterns.size > 0
      raise "+ applied twice does not make sense" if patterns[-1].is_a? OneOrMorePattern
      pattern = OneOrMorePattern.new( patterns[-1] ) 
      pattern.metrocot = self
      pattern.node_scraper = node_scraper
      patterns[-1] = pattern
      log("now one or more: #{pattern.repeatee}")
      s = s[1 .. -1]
      next
    end

    if s[0..0] == "(" 
      pattern = compile_pattern( s[1 .. -1], node_scraper )
      close_par_index = pattern.source.size + 1
      raise "expected ')' found '#{s[0..0]}'" unless s[close_par_index..close_par_index] == ")"
      s = s[close_par_index + 1 .. -1]
      log("found nested: #{pattern.description} \"#{pattern.source}\"")
      patterns << pattern
      if name
        pattern.name = name.to_sym
      end
      next
    end

    raise "unrecognizable pattern: \"#{s[0..10]}...\""
      
  end
  
  pattern = if patterns.size > 1
    CompositePattern.new( patterns )
  elsif patterns.size == 1
    patterns[0]
  else 
    nil
  end

  if pattern
    pattern.metrocot = self
    pattern.node_scraper = node_scraper
    pattern.source = pattern_s[0 .. (0 - (1 + s.size))] 
  end

  return @compiled_patterns[ pattern_s ] = pattern
  
end

#log(s) ⇒ Object

# File 'lib/metrocot.rb', line 1137

def log( s )
  puts( s ) if @verbose
end

#scanner_by_name(name) ⇒ Object

given a Symbol of a scanner (name), return a handle for that scanner declared when the metrocot was created.

# File 'lib/metrocot.rb', line 1145

def scanner_by_name( name )
  @scanners[name]
end

#scrape(doc) ⇒ Object

# File 'lib/metrocot.rb', line 1176

def scrape(doc) 
  NodeScraper.new( self, nil, nil, doc )
end