Class: CMSScanner::RobotsTxt

Inherits:
InterestingFinding show all
Defined in:
app/models/robots_txt.rb

Overview

Robots.txt

Constant Summary

Constants included from Finders::Finding

Finders::Finding::FINDING_OPTS

Instance Attribute Summary

Attributes inherited from InterestingFinding

#to_s, #url

Instance Method Summary collapse

Methods inherited from InterestingFinding

#==, #entries, #initialize

Methods included from Finders::Finding

#<=>, #confidence, #confidence=, #confirmed_by, #eql?, included, #parse_finding_options

Constructor Details

This class inherits a constructor from CMSScanner::InterestingFinding

Instance Method Details

#interesting_entries ⇒ Array<String>

TODO:

Better detection, currently everything not empty or / is returned

Returns The interesting Allow/Disallow rules detected.

Returns:

  • (Array<String>) —

    The interesting Allow/Disallow rules detected



7
8
9
10
11
12
13
14
15
16
17
18
19
 
# File 'app/models/robots_txt.rb', line 7

def interesting_entries
  results = []

  entries.each do |entry|
    next unless entry =~ /\A(?:dis)?allow:\s*(.+)\z/i
    match = Regexp.last_match(1)
    next if match == '/'

    results << match
  end

  results.uniq
end