Class: Crawlr::Config

Inherits:

Object

Object
Crawlr::Config

show all

Defined in:: lib/crawlr/config.rb

Overview

Configuration management class for Crawlr scraping sessions.

The Config class centralizes all configuration options for the Crawlr framework, providing sensible defaults while allowing extensive customization of scraping behavior, networking settings, error handling, and crawling policies.

Examples:

Basic configuration

config = Crawlr::Config.new(
  timeout: 15,
  max_depth: 3,
  max_parallelism: 5
)

Advanced configuration with domain filtering and retries

config = Crawlr::Config.new(
  allowed_domains: ['example.com', 'api.example.com'],
  max_retries: 3,
  retry_delay: 2.0,
  retry_backoff: 1.5,
  random_delay: 1.0,
  allow_cookies: true,
  ignore_robots_txt: false
)

Proxy configuration

config = Crawlr::Config.new(
  proxies: ['proxy1.com:8080', 'proxy2.com:8080'],
  proxy_strategy: :random,
  max_parallelism: 10
)

Since:

0.1.0

Instance Attribute Summary collapse

Instance Method Summary collapse

#initialize(options = {}) ⇒ Config constructor
Initializes a new Config instance with the provided options.
#to_h ⇒ Hash<Symbol, Object>
Converts the configuration to a hash representation.

Constructor Details

#initialize(options = {}) ⇒ `Config`

Initializes a new Config instance with the provided options

Examples:

Minimal configuration

config = Crawlr::Config.new

Timeout and parallelism configuration

config = Crawlr::Config.new(
  timeout: 30,
  max_parallelism: 8
)

Domain filtering with explicit domains

config = Crawlr::Config.new(
  allowed_domains: ['site1.com', 'api.site1.com']
)

Domain filtering with glob patterns

config = Crawlr::Config.new(
  domain_glob: ['*.example.com', '*.api.example.com']
)

Retry configuration with custom errors

config = Crawlr::Config.new(
  max_retries: 5,
  retry_delay: 0.5,
  retry_backoff: 1.5,
  retryable_errors: [Timeout::Error, Net::ReadTimeout]
)

Parameters:

options (Hash) (defaults to: {}) —
Configuration options hash

Options Hash (options):

:timeout (Integer) — default: 10 —
HTTP request timeout in seconds
:default_headers (Hash<String, String>) —
Default HTTP headers
:allowed_domains (Array<String>) — default: [] —
Explicit list of allowed domains
:domain_glob (Array<String>) — default: [] —
Glob patterns for domain filtering
:allow_cookies (Boolean) — default: false —
Enable cookie handling
:max_depth (Integer) — default: 0 —
Maximum crawling depth (0 = unlimited)
:random_delay (Float) — default: 0 —
Maximum random delay between requests
:max_parallelism (Integer) — default: 1 —
Maximum concurrent requests
:allow_url_revisit (Boolean) — default: false —
Allow revisiting URLs
:max_retries (Integer) — default: 0 —
Maximum retry attempts (0 = disabled)
:retry_delay (Float) — default: 1.0 —
Base retry delay in seconds
:retry_backoff (Float) — default: 2.0 —
Exponential backoff multiplier
:retryable_errors (Array<Class>) —
Custom list of retryable exceptions
:max_visited (Integer) — default: 10000 —
Maximum URLs to track in history
:proxies (Array<String>) — default: [] —
List of proxy servers
:proxy_strategy (Symbol) — default: :round_robin —
Proxy selection strategy
:ignore_robots_txt (Boolean) — default: false —
Ignore robots.txt restrictions

Raises:

(StandardError) —
When both :allowed_domains and :domain_glob are specified

Since:

0.1.0

# File 'lib/crawlr/config.rb', line 108

def initialize(options = {})
  initialize_domain_settings(options)
  initialize_parallelism_settings(options)
  initialize_throttle_settings(options)
  initialize_http_settings(options)
  initialize_retry_settings(options)
  initialize_visit_settings(options)
  initialize_proxy_settings(options)
  initialize_robots_settings(options)

  validate
end

Instance Attribute Details

#allow_cookies ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def allow_cookies
  @allow_cookies
end

#allow_url_revisit ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def allow_url_revisit
  @allow_url_revisit
end

#allowed_domains ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def allowed_domains
  @allowed_domains
end

#domain_glob ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def domain_glob
  @domain_glob
end

#headers ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def headers
  @headers
end

#ignore_robots_txt ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def ignore_robots_txt
  @ignore_robots_txt
end

#max_depth ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def max_depth
  @max_depth
end

#max_parallelism ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def max_parallelism
  @max_parallelism
end

#max_retries ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def max_retries
  @max_retries
end

#max_visited ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def max_visited
  @max_visited
end

#proxies ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def proxies
  @proxies
end

#proxy_strategy ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def proxy_strategy
  @proxy_strategy
end

#random_delay ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def random_delay
  @random_delay
end

#retry_backoff ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def retry_backoff
  @retry_backoff
end

#retry_delay ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def retry_delay
  @retry_delay
end

#retryable_errors ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def retryable_errors
  @retryable_errors
end

#timeout ⇒ `Integer`, ...

Returns:

(Integer) —
HTTP request timeout in seconds
(Hash<String, String>) —
Default HTTP headers for all requests
(Array<String>) —
Glob patterns for allowed domains
(Array<String>) —
Explicit list of allowed domains
(Boolean) —
Whether to enable cookie handling
(Integer) —
Maximum crawling depth (0 for unlimited)
(Float) —
Maximum random delay between requests in seconds
(Integer) —
Maximum number of concurrent requests
(Boolean) —
Whether to allow revisiting previously scraped URLs
(Integer, nil) —
Maximum number of retry attempts (nil to disable)
(Float) —
Base delay between retry attempts in seconds
(Float) —
Exponential backoff multiplier for retry delays
(Array<Class>) —
List of exception classes that trigger retries
(Integer) —
Maximum number of URLs to track in visit history
(Array<String>) —
List of proxy server addresses
(Symbol) —
Strategy for selecting proxies (:round_robin, :random)
(Boolean) —
Whether to ignore robots.txt restrictions

Since:

0.1.0



54
55
56

# File 'lib/crawlr/config.rb', line 54

def timeout
  @timeout
end

Instance Method Details

#to_h ⇒ `Hash<Symbol, Object>`

Converts the configuration to a hash representation

This method is useful for serialization, debugging, or creating new Config instances with the same settings.

Examples:

config = Crawlr::Config.new(timeout: 15, max_depth: 3)
hash = config.to_h
new_config = Crawlr::Config.new(hash)

Inspect configuration

puts config.to_h.inspect

Returns:

(Hash<Symbol, Object>) —
Hash containing all configuration values

Since:

0.1.0

# File 'lib/crawlr/config.rb', line 135

def to_h
  attrs = %i[
    timeout headers allowed_domains domain_glob allow_cookies max_depth
    random_delay max_parallelism allow_url_revisit max_retries retry_delay
    retry_backoff retryable_errors max_visited proxies proxy_strategy
    ignore_robots_txt
  ]

  attrs.each_with_object({}) { |name, hash| hash[name] = instance_variable_get("@#{name}") }
end

Class: Crawlr::Config

Overview

Examples:

Basic configuration

Advanced configuration with domain filtering and retries

Proxy configuration

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ Config

Examples:

Minimal configuration

Timeout and parallelism configuration

Domain filtering with explicit domains

Domain filtering with glob patterns

Retry configuration with custom errors

Instance Attribute Details

#allow_cookies ⇒ Integer, ...

#allow_url_revisit ⇒ Integer, ...

#allowed_domains ⇒ Integer, ...

#domain_glob ⇒ Integer, ...

#headers ⇒ Integer, ...

#ignore_robots_txt ⇒ Integer, ...

#max_depth ⇒ Integer, ...

#max_parallelism ⇒ Integer, ...

#max_retries ⇒ Integer, ...

#max_visited ⇒ Integer, ...

#proxies ⇒ Integer, ...

#proxy_strategy ⇒ Integer, ...

#random_delay ⇒ Integer, ...

#retry_backoff ⇒ Integer, ...

#retry_delay ⇒ Integer, ...

#retryable_errors ⇒ Integer, ...

#timeout ⇒ Integer, ...

Instance Method Details

#to_h ⇒ Hash<Symbol, Object>

Examples:

Inspect configuration

#initialize(options = {}) ⇒ `Config`

#allow_cookies ⇒ `Integer`, ...

#allow_url_revisit ⇒ `Integer`, ...

#allowed_domains ⇒ `Integer`, ...

#domain_glob ⇒ `Integer`, ...

#headers ⇒ `Integer`, ...

#ignore_robots_txt ⇒ `Integer`, ...

#max_depth ⇒ `Integer`, ...

#max_parallelism ⇒ `Integer`, ...

#max_retries ⇒ `Integer`, ...

#max_visited ⇒ `Integer`, ...

#proxies ⇒ `Integer`, ...

#proxy_strategy ⇒ `Integer`, ...

#random_delay ⇒ `Integer`, ...

#retry_backoff ⇒ `Integer`, ...

#retry_delay ⇒ `Integer`, ...

#retryable_errors ⇒ `Integer`, ...

#timeout ⇒ `Integer`, ...

#to_h ⇒ `Hash<Symbol, Object>`