Module: Crawlr::Parser

Defined in:
lib/crawlr/parser.rb

Overview

Document parsing and callback execution engine.

The Parser module provides the core document processing functionality for the Crawlr framework. It efficiently parses HTML and XML content using Nokogiri and executes registered callbacks on matching elements. The module optimizes performance by grouping callbacks by document format to minimize parsing overhead.

Examples:

Basic callback execution

content = '<html><body><h1>Title</h1><p>Content</p></body></html>'

callbacks = [
  {
    format: :html,
    selector_type: :css,
    selector: 'h1',
    block: ->(node, ctx) { ctx.titles << node.text }
  }
]

context = OpenStruct.new(titles: [])
Crawlr::Parser.apply_callbacks(
  content: content,
  callbacks: callbacks,
  context: context
)
puts context.titles #=> ["Title"]

Mixed HTML and XML parsing

callbacks = [
  {
    format: :html,
    selector_type: :css,
    selector: '.product',
    block: ->(node, ctx) { process_html_product(node, ctx) }
  },
  {
    format: :xml,
    selector_type: :xpath,
    selector: '//item[@type="product"]',
    block: ->(node, ctx) { process_xml_product(node, ctx) }
  }
]

Crawlr::Parser.apply_callbacks(
  content: xml_content,
  callbacks: callbacks,
  context: scraping_context
)

Performance optimization with format grouping

# Multiple callbacks for same format - document parsed only once
callbacks = [
  { format: :html, selector_type: :css, selector: 'title', block: title_proc },
  { format: :html, selector_type: :css, selector: 'meta', block: meta_proc },
  { format: :html, selector_type: :xpath, selector: '//a[@href]', block: link_proc }
]

# HTML content parsed once, all callbacks executed on same document
Crawlr::Parser.apply_callbacks(content: html, callbacks: callbacks, context: ctx)

Since:

  • 0.1.0

Class Method Summary collapse

Class Method Details

.apply_callbacks(content:, callbacks:, context:) ⇒ void

This method returns an undefined value.

Applies registered callbacks to parsed document content

This method is the main entry point for document processing. It efficiently handles multiple callbacks by grouping them by document format, ensuring that each piece of content is parsed only once per format regardless of how many callbacks are registered for that format.

The method performs the following operations:

  1. Groups callbacks by document format (:html or :xml)
  2. Parses content once per format using appropriate Nokogiri parser
  3. Executes all callbacks for each format on the parsed document
  4. Extracts matching nodes using CSS or XPath selectors
  5. Calls callback blocks with matched nodes and context

Examples:

Single callback execution

callbacks = [{
  format: :html,
  selector_type: :css,
  selector: '.article-title',
  block: ->(node, ctx) { ctx.titles << node.text.strip }
}]

Crawlr::Parser.apply_callbacks(
  content: html_content,
  callbacks: callbacks,
  context: context_object
)

Multiple callbacks with different selectors

callbacks = [
  {
    format: :html,
    selector_type: :css,
    selector: 'h1, h2, h3',
    block: ->(node, ctx) { ctx.headings << { text: node.text, level: node.name } }
  },
  {
    format: :html,
    selector_type: :xpath,
    selector: '//a[@href and text()]',
    block: ->(node, ctx) { ctx.links << { url: node['href'], text: node.text } }
  }
]

Crawlr::Parser.apply_callbacks(
  content: page_html,
  callbacks: callbacks,
  context: scraping_context
)

XML feed processing

callbacks = [{
  format: :xml,
  selector_type: :xpath,
  selector: '//item/title',
  block: ->(node, ctx) { ctx.feed_titles << node.text }
}]

Crawlr::Parser.apply_callbacks(
  content: rss_xml,
  callbacks: callbacks,
  context: feed_context
)

Complex data extraction

callbacks = [{
  format: :html,
  selector_type: :css,
  selector: '.product-card',
  block: ->(node, ctx) {
    product = {
      name: node.css('.product-name').text,
      price: node.css('.price').text,
      image: node.css('img')&.first&.[]('src')
    }
    ctx.products << product
  }
}]

Crawlr::Parser.apply_callbacks(
  content: product_page_html,
  callbacks: callbacks,
  context: product_context
)

Parameters:

  • content (String) —

    Raw HTML or XML content to parse

  • callbacks (Array<Hash>) —

    Array of callback configuration hashes

  • context (Object) —

    Context object passed to callback blocks

Options Hash (callbacks:):

  • :format (Symbol) —

    Document format (:html or :xml, defaults to :html)

  • :selector_type (Symbol) —

    Selector type (:css or :xpath)

  • :selector (String) —

    CSS or XPath selector string

  • :block (Proc) —

    Callback block to execute on matching nodes

Since:

  • 0.1.0



162
163
164
165
166
167
168
169
170
 
# File 'lib/crawlr/parser.rb', line 162

def self.apply_callbacks(content:, callbacks:, context:)
  # Group callbacks by format to minimize parsing
  callbacks_by_format = callbacks.group_by { |cb| cb[:format] || :html }

  callbacks_by_format.each do |format, format_callbacks|
    doc = parse_content(format, content)
    format_callbacks.each { |callback| apply_callback(doc, callback, context) }
  end
end