Class: Crawlr::Robots::Rule

Inherits:

Struct

• Object
• Struct
• Crawlr::Robots::Rule

Defined in:

lib/crawlr/robots.rb

Overview

Represents a robots.txt rule for a specific user-agent

Since:

• 0.1.0

Instance Attribute Summary

• #allow ⇒ Array<String> readonly

  Array of allowed path patterns.

• #crawl_delay ⇒ String^? readonly

  Crawl delay in seconds for this user-agent.

• #disallow ⇒ Array<String> readonly

  Array of disallowed path patterns.

• #user_agent ⇒ String readonly

  User-agent pattern this rule applies to.

Instance Attribute Details

#allow ⇒ Array<String> (readonly)

Returns Array of allowed path patterns.

Returns:

• (Array<String>) —

  Array of allowed path patterns

# File 'lib/crawlr/robots.rb', line 73

Rule = Struct.new(:user_agent, :allow, :disallow, :crawl_delay)

#crawl_delay ⇒ String^? (readonly)

Returns Crawl delay in seconds for this user-agent.

Returns:

• (String, nil) —

  Crawl delay in seconds for this user-agent

# File 'lib/crawlr/robots.rb', line 73

Rule = Struct.new(:user_agent, :allow, :disallow, :crawl_delay)

#disallow ⇒ Array<String> (readonly)

Returns Array of disallowed path patterns.

Returns:

• (Array<String>) —

  Array of disallowed path patterns

# File 'lib/crawlr/robots.rb', line 73

Rule = Struct.new(:user_agent, :allow, :disallow, :crawl_delay)

#user_agent ⇒ String (readonly)

Returns User-agent pattern this rule applies to.

Returns:

• (String) —

  User-agent pattern this rule applies to

# File 'lib/crawlr/robots.rb', line 73

Rule = Struct.new(:user_agent, :allow, :disallow, :crawl_delay)