Class: Crawlr::Robots

Inherits:
Object
  • Object
show all
Defined in:
lib/crawlr/robots.rb

Overview

Robots.txt parser and compliance checker for respectful web scraping.

The Robots class implements full robots.txt specification compliance, including user-agent matching, path pattern matching with wildcards, allow/disallow precedence rules, and crawl-delay directives. It helps ensure that scrapers respect website crawling policies and avoid making unwanted requests.

Examples:

Basic robots.txt compliance

robots = Crawlr::Robots.new

# Parse robots.txt content
robots_content = <<~ROBOTS
  User-agent: *
  Disallow: /private/
  Allow: /public/
  Crawl-delay: 1
ROBOTS

robots.parse('https://example.com', robots_content)

# Check URL permissions
robots.allowed?('https://example.com/public/page', 'MyBot/1.0')  #=> true
robots.allowed?('https://example.com/private/data', 'MyBot/1.0') #=> false

Complex user-agent matching

robots_content = <<~ROBOTS
  User-agent: Googlebot
  Disallow: /admin/

  User-agent: *
  Disallow: /
  Allow: /public/
ROBOTS

robots.parse('https://site.com', robots_content)

robots.allowed?('https://site.com/admin/', 'Googlebot/2.1')      #=> false
robots.allowed?('https://site.com/public/', 'Googlebot/2.1')     #=> true
robots.allowed?('https://site.com/anything/', 'OtherBot/1.0')    #=> false

Wildcard pattern matching

robots_content = <<~ROBOTS
  User-agent: *
  Disallow: /*.pdf$
  Disallow: /temp/*
  Allow: /temp/public/*
ROBOTS

robots.parse('https://example.com', robots_content)

robots.allowed?('https://example.com/document.pdf', 'Bot')        #=> false
robots.allowed?('https://example.com/temp/secret.txt', 'Bot')     #=> false
robots.allowed?('https://example.com/temp/public/file.txt', 'Bot') #=> true

Since:

  • 0.1.0

Defined Under Namespace

Classes: Rule

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Robots

Initializes a new Robots instance

Creates an empty store for caching parsed robots.txt files by domain. Each domain's robots.txt is parsed once and cached for subsequent permission checks.

Examples:

robots = Crawlr::Robots.new

Since:

  • 0.1.0



86
87
88
 
# File 'lib/crawlr/robots.rb', line 86

def initialize
  @store = {}
end

Instance Attribute Details

#store ⇒ Hash<String, Array<Rule>> (readonly)

Returns Internal store of parsed robots.txt rules by domain.

Returns:

  • (Hash<String, Array<Rule>>) —

    Internal store of parsed robots.txt rules by domain

Since:

  • 0.1.0



76
77
78
 
# File 'lib/crawlr/robots.rb', line 76

def store
  @store
end

Instance Method Details

#allowed?(url, user_agent) ⇒ Boolean

Determines if a URL is allowed to be crawled according to robots.txt rules

This method implements the full robots.txt specification including:

  • User-agent matching with prefix matching and wildcards
  • Path pattern matching with wildcards and end anchors
  • Allow/disallow precedence with longest match wins
  • Graceful fallback when no robots.txt exists

Examples:

Basic permission checking

robots.allowed?('https://example.com/page.html', 'MyBot/1.0')

With specific user-agent rules

# robots.txt contains specific rules for "MyBot"
robots.allowed?('https://site.com/admin/', 'MyBot/2.0')  #=> depends on rules
robots.allowed?('https://site.com/admin/', 'OtherBot')   #=> uses wildcard rules

Pattern matching examples

# robots.txt: Disallow: /*.pdf$
robots.allowed?('https://site.com/doc.pdf', 'Bot')     #=> false
robots.allowed?('https://site.com/doc.pdf.html', 'Bot') #=> true

# robots.txt: Disallow: /temp/*
robots.allowed?('https://site.com/temp/file.txt', 'Bot') #=> false
robots.allowed?('https://site.com/temporary/', 'Bot')    #=> true

Parameters:

  • url (String) —

    The full URL to check for crawling permission

  • user_agent (String) —

    The user-agent string to match against rules

Returns:

  • (Boolean) —

    true if the URL is allowed to be crawled

Since:

  • 0.1.0



131
132
133
134
135
136
137
138
139
140
 
# File 'lib/crawlr/robots.rb', line 131

def allowed?(url, user_agent)
  rule = get_rule(url, user_agent)
  return true unless rule

  path = URI.parse(url).path
  matched = matched_rules(rule, path)
  return true if matched.empty?

  longest_match_allows?(matched)
end

#exists?(origin) ⇒ Boolean

Checks if robots.txt has been parsed and cached for a given origin

Examples:

robots.exists?('https://example.com')  #=> false
robots.parse('https://example.com', robots_content)
robots.exists?('https://example.com')  #=> true

Parameters:

  • origin (String) —

    The origin URL (scheme + host + port)

Returns:

  • (Boolean) —

    true if robots.txt data exists for this origin

Since:

  • 0.1.0



99
100
101
 
# File 'lib/crawlr/robots.rb', line 99

def exists?(origin)
  @store.key?(origin)
end

#parse(url, content) ⇒ void

This method returns an undefined value.

Parses robots.txt content and stores rules for the given URL's domain

Extracts and processes all robots.txt directives including:

  • User-agent declarations
  • Allow and Disallow rules
  • Crawl-delay directives
  • Sitemap declarations
  • Comment and empty line handling

Examples:

Parse standard robots.txt

robots_content = <<~ROBOTS
  # This is a comment
  User-agent: *
  Disallow: /private/
  Allow: /public/
  Crawl-delay: 2

  User-agent: Googlebot
  Allow: /

  Sitemap: https://example.com/sitemap.xml
ROBOTS

robots.parse('https://example.com/robots.txt', robots_content)

Parse with wildcards and patterns

robots_content = <<~ROBOTS
  User-agent: *
  Disallow: /*.json$
  Disallow: /api/v*/private/
  Allow: /api/v*/public/
ROBOTS

robots.parse('https://api.example.com', robots_content)

Parameters:

  • url (String) —

    The URL where this robots.txt was fetched from

  • content (String) —

    Raw robots.txt file content

Since:

  • 0.1.0



180
181
182
183
184
185
186
187
188
189
190
191
 
# File 'lib/crawlr/robots.rb', line 180

def parse(url, content)
  uri = URI.parse(url)
  domain = uri.host.downcase
  hash = parse_to_hash(content)

  rules = []
  hash[:rules].each do |user_agent, rule|
    rules << Rule.new(user_agent, rule[:allow], rule[:disallow], rule[:crawl_delay])
  end

  @store[domain] ||= rules
end