Class: Crawlr::Collector

Inherits:
Object
  • Object
show all
Defined in:
lib/crawlr/collector.rb

Overview

Main orchestrator class that manages scraping sessions.

The Collector is the central component of the Crawlr framework, responsible for:

  • Managing URL visits with configurable depth control
  • Handling concurrent requests with parallelism limits
  • Respecting robots.txt and implementing polite crawling delays
  • Executing registered callbacks on scraped content
  • Maintaining visit history and domain filtering
  • Providing hooks for custom behavior during scraping lifecycle

Examples:

Basic scraping setup

collector = Crawlr::Collector.new(max_depth: 3, max_parallelism: 5)

collector.on_html(:css, '.product-title') do |node, ctx|
  puts "Found: #{node.text} at #{ctx.page_url}"
end

collector.visit('https://example.com')

Paginated scraping

collector.paginated_visit(
  'https://api.example.com/items',
  batch_size: 10,
  start_page: 1
)

With hooks and configuration

collector = Crawlr::Collector.new(
  max_retries: 3,
  random_delay: 2.0,
  ignore_robots_txt: false
)

collector.hook(:before_visit) do |url, headers|
  puts "About to visit: #{url}"
end

collector.hook(:on_error) do |url, error|
  puts "Failed to scrape #{url}: #{error.message}"
end

Since:

  • 0.1.0

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ Collector

Initializes a new Collector instance with the given configuration

Examples:

collector = Crawlr::Collector.new(
  max_depth: 5,
  max_parallelism: 3,
  random_delay: 1.5
)

Parameters:

  • options (Hash) (defaults to: {}) —

    Configuration options for the collector

Options Hash (options):

  • :max_depth (Integer) —

    Maximum crawling depth (default: nil for unlimited)

  • :max_parallelism (Integer) —

    Maximum concurrent requests (default: 1)

  • :random_delay (Float) —

    Maximum random delay between requests in seconds

  • :ignore_robots_txt (Boolean) —

    Whether to ignore robots.txt (default: false)

  • :max_retries (Integer) —

    Maximum retry attempts for failed requests

  • :allow_url_revisit (Boolean) —

    Allow revisiting previously scraped URLs

Since:

  • 0.1.0



82
83
84
85
86
87
88
89
90
 
# File 'lib/crawlr/collector.rb', line 82

def initialize(options = {})
  @config = Crawlr::Config.new(options)
  @http = Crawlr::HTTPInterface.new(@config)
  @visits = Crawlr::Visits.new(@config)
  @domains = Crawlr::Domains.new(@config)
  @hooks = Crawlr::Hooks.new
  @callbacks = Crawlr::Callbacks.new
  @robots = Crawlr::Robots.new
end

Instance Attribute Details

#config ⇒ Crawlr::Config (readonly)

Returns The configuration object for this collector.

Returns:

Since:

  • 0.1.0



59
60
61
 
# File 'lib/crawlr/collector.rb', line 59

def config
  @config
end

#context ⇒ Crawlr::Context, ...

Returns:

Since:

  • 0.1.0



64
65
66
 
# File 'lib/crawlr/collector.rb', line 64

def context
  @context
end

#http ⇒ Crawlr::Context, ...

Returns:

Since:

  • 0.1.0



64
65
66
 
# File 'lib/crawlr/collector.rb', line 64

def http
  @http
end

#visits ⇒ Crawlr::Context, ...

Returns:

Since:

  • 0.1.0



64
65
66
 
# File 'lib/crawlr/collector.rb', line 64

def visits
  @visits
end

Instance Method Details

#clone ⇒ Crawlr::Collector

Creates a clone of the current collector with shared HTTP and visit state

This is useful for creating multiple collectors that share the same HTTP connection pool and visit history while having independent callback and hook configurations.

Examples:

main_collector = Crawlr::Collector.new(max_parallelism: 10)
product_collector = main_collector.clone

product_collector.on_html(:css, '.product') do |node, ctx|
  # Process products with shared visit history
end

Returns:

Since:

  • 0.1.0



267
268
269
270
271
272
273
 
# File 'lib/crawlr/collector.rb', line 267

def clone
  new_collector = self.class.new(@config.to_h)
  new_collector.http = @http
  new_collector.visits = @visits

  new_collector
end

#hook(event, &block) {|args| ... } ⇒ void

This method returns an undefined value.

Registers a hook for specific scraping lifecycle events

Hooks allow you to execute custom code at specific points during the scraping process, such as before/after visits or on errors.

Examples:

Hook before each visit

hook(:before_visit) do |url, headers|
  puts "About to visit: #{url}"
  headers['Custom-Header'] = 'value'
end

Hook after each visit

hook(:after_visit) do |url, response|
  puts "Visited #{url}, got status: #{response.status}"
end

Hook for error handling

hook(:on_error) do |url, error|
  logger.error "Failed to scrape #{url}: #{error.message}"
end

Parameters:

  • event (Symbol) —

    The event to hook into (:before_visit, :after_visit, :on_error)

  • block (Proc) —

    The block to execute when the event occurs

Yield Parameters:

  • args (Array) —

    Event-specific arguments passed to the block

Since:

  • 0.1.0



248
249
250
 
# File 'lib/crawlr/collector.rb', line 248

def hook(event, &block)
  @hooks.register(event, &block)
end

#on_html(selector_type, selector, &block) {|node, ctx| ... } ⇒ void

This method returns an undefined value.

Registers a callback for HTML content using CSS or XPath selectors

Examples:

Register CSS selector for HTML

on_html(:css, '.article-title') do |node, ctx|
  ctx.titles << node.text.strip
end

Register XPath selector for HTML

on_html(:xpath, '//a[@class="next-page"]') do |link, ctx|
  next_url = URI.join(ctx.base_url, link['href'])
  ctx.queue_url(next_url.to_s)
end

Parameters:

  • selector_type (Symbol) —

    The type of selector (:css or :xpath)

  • selector (String) —

    The selector string to match elements

  • block (Proc) —

    The callback block to execute when elements match

Yield Parameters:

  • node (Nokogiri::XML::Node) —

    The matched DOM node

  • ctx (Crawlr::Context) —

    The scraping context

Since:

  • 0.1.0



111
112
113
 
# File 'lib/crawlr/collector.rb', line 111

def on_html(selector_type, selector, &block)
  @callbacks.register(:html, selector_type, selector, &block)
end

#on_xml(selector_type, selector, &block) {|node, ctx| ... } ⇒ void

This method returns an undefined value.

Registers a callback for XML content using CSS or XPath selectors

Examples:

Register XPath selector for XML feeds

on_xml(:xpath, '//item/title') do |title_node, ctx|
  ctx.feed_titles << title_node.text
end

Register CSS selector for XML

on_xml(:css, 'product[price]') do |product, ctx|
  ctx.products << parse_product(product)
end

Parameters:

  • selector_type (Symbol) —

    The type of selector (:css or :xpath)

  • selector (String) —

    The selector string to match elements

  • block (Proc) —

    The callback block to execute when elements match

Yield Parameters:

  • node (Nokogiri::XML::Node) —

    The matched DOM node

  • ctx (Crawlr::Context) —

    The scraping context

Since:

  • 0.1.0



133
134
135
 
# File 'lib/crawlr/collector.rb', line 133

def on_xml(selector_type, selector, &block)
  @callbacks.register(:xml, selector_type, selector, &block)
end

#paginated_visit(url, current_depth: 0, query: "page", batch_size: 5, start_page: 1) {|collector| ... } ⇒ void

This method returns an undefined value.

Performs paginated scraping by automatically generating page URLs

This method is specifically designed for APIs or websites that use query parameter pagination (e.g., ?page=1, ?page=2, etc.). It automatically generates URLs and stops when pages return 404 or too many failures occur.

Examples:

Basic pagination

paginated_visit('https://api.example.com/items', batch_size: 10)

Custom query parameter and start page

paginated_visit(
  'https://example.com/products',
  query: 'p',
  start_page: 2,
  batch_size: 3
)

With configuration block

paginated_visit('https://api.site.com/data') do |collector|
  collector.on_xml(:css, 'item') do |node, ctx|
    process_item(node, ctx)
  end
end

Parameters:

  • url (String) —

    Base URL for pagination

  • current_depth (Integer) (defaults to: 0) —

    Starting depth for crawling limits

  • query (String) (defaults to: "page") —

    Query parameter name for pagination (default: "page")

  • batch_size (Integer) (defaults to: 5) —

    Number of pages to process in parallel batches (default: 5)

  • start_page (Integer) (defaults to: 1) —

    Starting page number (default: 1)

  • block (Proc) —

    Optional block to configure the collector before visiting

Yield Parameters:

Since:

  • 0.1.0



212
213
214
215
216
217
218
219
220
221
 
# File 'lib/crawlr/collector.rb', line 212

def paginated_visit(url, current_depth: 0, query: "page", batch_size: 5, start_page: 1)
  return unless valid_url?(url)

  yield self if block_given?
  fetch_robots_txt(url) unless @config.ignore_robots_txt
  return unless can_visit?(url, @config.headers)

  pages_to_visit = build_initial_pages(url, query, batch_size, start_page)
  process_page_batches(pages_to_visit, current_depth, batch_size, query)
end

#stats ⇒ Hash<Symbol, Object>

Returns comprehensive statistics about the collector's state and activity

Provides metrics about configuration, registered hooks/callbacks, visit history, and retry behavior for monitoring and debugging.

Examples:

stats = collector.stats
puts "Visited #{stats[:total_visits]} pages"
puts "Registered #{stats[:callbacks_count]} callbacks"

Parameters:

  • return (Hash) —

    a customizable set of options

Returns:

  • (Hash<Symbol, Object>) —

    Statistics hash containing various metrics

Since:

  • 0.1.0



295
296
297
298
299
300
301
302
303
304
305
306
 
# File 'lib/crawlr/collector.rb', line 295

def stats
  base = {
    max_depth: @config.max_depth,
    allow_url_revisit: @config.allow_url_revisit
  }

  base.merge!(@hooks.stats)
  base.merge!(@callbacks.stats)
  base.merge!(@visits.stats)
  base.merge!(retry_stats) if @config.max_retries
  base
end

#visit(input, current_depth = 0) {|collector| ... } ⇒ void

This method returns an undefined value.

Visits one or more URLs and processes them according to registered callbacks

This method handles the core scraping workflow including:

  • robots.txt checking (unless disabled)
  • URL validation and filtering
  • Concurrent processing with parallelism limits
  • Depth tracking and limits
  • Error handling and retry logic

Examples:

Visit a single URL

visit('https://example.com/products')

Visit multiple URLs

visit(['https://site1.com', 'https://site2.com'])

Visit with configuration block

visit('https://example.com') do |collector|
  collector.on_html(:css, '.product') do |node, ctx|
    # Process products
  end
end

Recursive crawling with depth control

visit('https://example.com', 0) # Start at depth 0

Parameters:

  • input (String, Array<String>) —

    Single URL or array of URLs to visit

  • current_depth (Integer) (defaults to: 0) —

    Current depth level for recursive crawling

  • block (Proc) —

    Optional block to configure the collector before visiting

Yield Parameters:

Since:

  • 0.1.0



167
168
169
170
171
172
173
174
175
176
177
178
 
# File 'lib/crawlr/collector.rb', line 167

def visit(input, current_depth = 0)
  yield self if block_given?

  urls = normalize_urls(input)
  return if exceeded_max_depth?(urls, current_depth)

  process_robots(urls) unless @config.ignore_robots_txt
  urls = filter_urls(urls)
  return if urls.empty?

  perform_visits(urls, current_depth)
end