Class: Crawlr::Domains

Inherits:
Object
  • Object
show all
Defined in:
lib/crawlr/domains.rb

Overview

Domain filtering and validation class for controlling scraping scope.

The Domains class manages which domains are allowed to be scraped by implementing both explicit domain allowlists and glob pattern matching. It provides flexible domain filtering to restrict scraping to specific sites or domain patterns while normalizing domain names for consistent comparison.

Examples:

Allow specific domains

config = Crawlr::Config.new(
  allowed_domains: ['example.com', 'api.example.com', 'subdomain.site.org']
)
domains = Crawlr::Domains.new(config)

domains.allowed?('https://example.com/page')      #=> true
domains.allowed?('https://www.example.com/page')  #=> true (www. stripped)
domains.allowed?('https://forbidden.com/page')    #=> false

Use glob patterns for flexible matching

config = Crawlr::Config.new(
  domain_glob: ['*.example.com', '*.api.*.com', 'site?.org']
)
domains = Crawlr::Domains.new(config)

domains.allowed?('https://sub.example.com/path')     #=> true
domains.allowed?('https://api.service.com/data')     #=> true
domains.allowed?('https://site1.org/content')        #=> true

No restrictions (allow all domains)

config = Crawlr::Config.new  # No domain restrictions
domains = Crawlr::Domains.new(config)

domains.allowed?('https://any-site.com')  #=> true

Since:

  • 0.1.0

Instance Method Summary collapse

Constructor Details

#initialize(config) ⇒ Domains

Initializes a new Domains instance with the given configuration

Examples:

config = Crawlr::Config.new(allowed_domains: ['site.com'])
domains = Crawlr::Domains.new(config)

Parameters:

  • config (Crawlr::Config) —

    Configuration object containing domain restrictions

Since:

  • 0.1.0



47
48
49
50
51
 
# File 'lib/crawlr/domains.rb', line 47

def initialize(config)
  @config = config
  @allowed_domains = extract_allowed_domains(@config.allowed_domains)
  @domain_glob = @config.domain_glob
end

Instance Method Details

#allowed?(url) ⇒ Boolean

Checks if a URL is allowed based on configured domain restrictions

The method performs the following checks in order:

  1. If no restrictions are configured, allows all URLs
  2. If glob patterns are configured, tests URL against each pattern
  3. If explicit domains are configured, checks normalized domain name
  4. Logs rejection for debugging purposes

Examples:

With explicit domain allowlist

domains.allowed?('https://example.com/page')        #=> true (if allowed)
domains.allowed?('https://www.example.com/page')    #=> true (www. stripped)
domains.allowed?('https://subdomain.example.com')   #=> false (unless explicitly allowed)

With glob patterns

# config.domain_glob = ['*.example.com']
domains.allowed?('https://api.example.com')         #=> true
domains.allowed?('https://cdn.example.com/asset')   #=> true
domains.allowed?('https://other.com')               #=> false

No restrictions

# config.allowed_domains = [], config.domain_glob = []
domains.allowed?('https://any-domain.com')          #=> true

Parameters:

  • url (String) —

    The URL to check for domain allowance

Returns:

  • (Boolean) —

    true if the URL's domain is allowed, false otherwise

Since:

  • 0.1.0



78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/crawlr/domains.rb', line 78

def allowed?(url)
  return true if @allowed_domains.empty? && @domain_glob.empty?
  return true if !@domain_glob.empty? && matches_domain_glob?(url)

  uri = URI(url)
  base_name = uri.host.sub("www.", "")
  allowed = @allowed_domains.include?(base_name)

  Crawlr.logger.info("URL not allowed: #{url}") unless allowed
  allowed
end

#domain_stats ⇒ Hash<Symbol, Integer>

Returns statistics about the configured domain restrictions

Provides metrics about the number of explicitly allowed domains and glob patterns configured for monitoring and debugging purposes.

Examples:

stats = domains.domain_stats
puts "Allowing #{stats[:allowed_domains]} explicit domains"
puts "Using #{stats[:domain_glob]} glob patterns"

Parameters:

  • return (Hash) —

    a customizable set of options

Returns:

  • (Hash<Symbol, Integer>) —

    Statistics hash containing domain counts

Since:

  • 0.1.0



103
104
105
106
107
108
 
# File 'lib/crawlr/domains.rb', line 103

def domain_stats
  {
    allowed_domains: @allowed_domains.size,
    domain_glob: @domain_glob.size
  }
end