Module: Wmap
- Defined in:
- lib/wmap/utils/utils.rb,
lib/wmap.rb,
lib/wmap/utils/logger.rb,
lib/wmap/utils/url_magic.rb,
lib/wmap/utils/domain_root.rb,
lib/wmap/url_crawler/adware_tag.rb,
lib/wmap/domain_tracker/sub_domain.rb,
lib/wmap/host_tracker/primary_host.rb,
lib/wmap/site_tracker/deactivated_site.rb
Overview
Class to trace de-activated site. This is need for basic state tracking for our sites.
Defined Under Namespace
Modules: Utils Classes: CidrTracker, DnsBruter, DomainTracker, GeoIPTracker, GoogleSearchScraper, HostTracker, NetworkProfiler, PortScanner, SiteTracker, UrlChecker, UrlCrawler, Whois, WpTracker
Constant Summary collapse
- NAME =
"Wmap"
- GEM =
"wmap"
- VERSION =
File.dirname(__FILE__) + "/../version.txt"
Class Attribute Summary collapse
-
.known_internet_domains ⇒ Object
Returns the value of attribute known_internet_domains.
-
.verbose ⇒ Object
writeonly
Sets the attribute verbose.
Class Method Summary collapse
-
.banner ⇒ Object
Project banner in ASCII Art ‘soft’ format, courtesy to patorjk.com/software/taag/.
-
.check(url) ⇒ Object
URL checker - check the status of the remote URL.
-
.crawl(url) ⇒ Object
Crawler to search url contents for new sites.
-
.dns_brute(domain) ⇒ Object
DNS Brute Forcer.
-
.domain_known?(domain) ⇒ Boolean
Domain Tracking - check with the trust domain seed file locally, to determine if it’s a new internet domain NOT to confuse with the Internet ‘whois’ lookup.
-
.domain_root(host) ⇒ Object
Retrieve root domain from a host.
-
.dump(file) ⇒ Object
Dump out the unique sites into a plain file.
-
.dump_xml(file) ⇒ Object
Dump out the unique sites into a XML file.
-
.geoip(host) ⇒ Object
GeoIP Tracking - check the host / IP against the GeoIP data repository, return the Geographic information if found.
-
.google ⇒ Object
Search the Google engines and sort out sites known by Google.
-
.host_known?(host) ⇒ Boolean
Host Tracking - check local hosts file to see if this is a hostname known from the host seed file NOT to confuse with a regular DNS lookup over the internet.
-
.ip_known?(ip) ⇒ Boolean
IP Tracking - check local hosts file to see if this is an IP known from the seed file NOT to confuse with a regular reverse DNS lookup over the internet.
-
.ip_trusted?(ip) ⇒ Boolean
Check if the IP is within the range of the known CIDR blocks.
-
.mutation(host) ⇒ Object
Host-name mutation for catch easily guessable hostname, i.e.
-
.print(site) ⇒ Object
Print a site’s full information from the repository.
-
.print_all ⇒ Object
Print a site’s full information from the repository.
-
.read_ver ⇒ Object
Simple parser for the project version file.
-
.refresh(site) ⇒ Object
Refresh the site information in the local data repository.
-
.refresh_all ⇒ Object
Refresh the site information in the local data repository.
-
.response_code(url) ⇒ Object
Check URL/Site response code.
-
.scan(host) ⇒ Object
Fast tcp port scanner on a single host or IP.
-
.scans(target_list) ⇒ Object
Fast multi-processes tcp port scanner on a list of targets.
-
.search(pattern) ⇒ Object
Search the site repository for all entries that match the pattern.
-
.sub_domain_known?(host) ⇒ Boolean
Sub-domain tracking - check local hosts file to see if the sub-domain is already known.
-
.track(host) ⇒ Object
CIDR Tracking - check the host against the local CIDR seed file, return the CIDR tracking path if found.
-
.whois(domain) ⇒ Object
whois query and sort the result into structured data.
-
.wlog(msg, agent, log_file) ⇒ Object
Log the information into file.
-
.wmap(seed) ⇒ Object
Explorer to discover and inventory web application / service automatically.
Class Attribute Details
.known_internet_domains ⇒ Object
Returns the value of attribute known_internet_domains.
37 38 39 |
# File 'lib/wmap.rb', line 37 def known_internet_domains @known_internet_domains end |
.verbose=(value) ⇒ Object (writeonly)
Sets the attribute verbose
38 39 40 |
# File 'lib/wmap.rb', line 38 def verbose=(value) @verbose = value end |
Class Method Details
.banner ⇒ Object
Project banner in ASCII Art ‘soft’ format, courtesy to patorjk.com/software/taag/
59 60 61 62 63 64 65 66 67 68 |
# File 'lib/wmap.rb', line 59 def ver=read_ver art=",--. ,--. ,--. ,--. ,--. | | | | ,---. | |-. | `.' | ,--,--. ,---. ,---. ,---. ,--.--. | |.'.| || .-. :| .-. ' | |'.'| |' ,-. || .-. || .-. || .-. :| .--' | ,'. |\ --.| `-' | | | | |\ '-' || '-' '| '-' '\ --.| | '--' '--' `----' `---' `--' `--' `--`--'| |-' | |-' `----'`--' `--' `--' " string = "-"*80 + "\n" + art + "\n" + "Version: " + ver["version"] + "\tRelease Date: " + ver["date"] + "\nDesigned and developed by: " + ver["author"] + "\nEmail: " + ver["email"] + "\tLinkedIn: " + ver["linkedin"] + "\n" + "-"*80 end |
.check(url) ⇒ Object
URL checker - check the status of the remote URL
113 114 115 116 |
# File 'lib/wmap.rb', line 113 def check(url) checker=Wmap::UrlChecker.new(:verbose=>false) checker.url_worker(url) end |
.crawl(url) ⇒ Object
Crawler to search url contents for new sites
77 78 79 80 |
# File 'lib/wmap.rb', line 77 def crawl(url) crawler=Wmap::UrlCrawler.new crawler.crawl(url) end |
.dns_brute(domain) ⇒ Object
DNS Brute Forcer
149 150 151 152 |
# File 'lib/wmap.rb', line 149 def dns_brute(domain) bruter=Wmap::DnsBruter.new bruter.query(domain) end |
.domain_known?(domain) ⇒ Boolean
Domain Tracking - check with the trust domain seed file locally, to determine if it’s a new internet domain NOT to confuse with the Internet ‘whois’ lookup
126 127 128 129 |
# File 'lib/wmap.rb', line 126 def domain_known?(domain) tracker=Wmap::DomainTracker.instance tracker.domain_known?(domain) end |
.domain_root(host) ⇒ Object
Retrieve root domain from a host
155 156 157 |
# File 'lib/wmap.rb', line 155 def domain_root(host) Wmap::Utils.get_domain_root(host) end |
.dump(file) ⇒ Object
Dump out the unique sites into a plain file
182 183 184 185 186 |
# File 'lib/wmap.rb', line 182 def dump(file) store=Wmap::SiteTracker.instance store.verbose=true store.save_uniq_sites(file) end |
.dump_xml(file) ⇒ Object
Dump out the unique sites into a XML file
189 190 191 192 |
# File 'lib/wmap.rb', line 189 def dump_xml(file) store=Wmap::SiteTracker.instance store.save_uniq_sites_xml(file) end |
.geoip(host) ⇒ Object
GeoIP Tracking - check the host / IP against the GeoIP data repository, return the Geographic information if found
107 108 109 110 |
# File 'lib/wmap.rb', line 107 def geoip(host) tracker=Wmap::GeoIPTracker.new tracker.query(host) end |
.google ⇒ Object
Search the Google engines and sort out sites known by Google
209 210 211 |
# File 'lib/wmap.rb', line 209 def google sites=Wmap::GoogleSearchScraper.new.workers.keys end |
.host_known?(host) ⇒ Boolean
Host Tracking - check local hosts file to see if this is a hostname known from the host seed file NOT to confuse with a regular DNS lookup over the internet
133 134 135 |
# File 'lib/wmap.rb', line 133 def host_known?(host) tracker=Wmap::HostTracker.instance.host_known?(host) end |
.ip_known?(ip) ⇒ Boolean
IP Tracking - check local hosts file to see if this is an IP known from the seed file NOT to confuse with a regular reverse DNS lookup over the internet
144 145 146 |
# File 'lib/wmap.rb', line 144 def ip_known?(ip) tracker=Wmap::HostTracker.instance.ip_known?(ip) end |
.ip_trusted?(ip) ⇒ Boolean
Check if the IP is within the range of the known CIDR blocks
119 120 121 122 |
# File 'lib/wmap.rb', line 119 def ip_trusted?(ip) tracker=Wmap::CidrTracker.new tracker.ip_trusted?(ip) end |
.mutation(host) ⇒ Object
Host-name mutation for catch easily guessable hostname, i.e. “ww1.example.com” => [“ww1,example.com”,“ww2.example.com”,…]
165 166 167 |
# File 'lib/wmap.rb', line 165 def mutation (host) Wmap::DnsBruter.new.hostname_mutation(host) end |
.print(site) ⇒ Object
Print a site’s full information from the repository
214 215 216 217 |
# File 'lib/wmap.rb', line 214 def print(site) searcher=Wmap::SiteTracker.instance searcher.print_site(site) end |
.print_all ⇒ Object
Print a site’s full information from the repository
220 221 222 223 |
# File 'lib/wmap.rb', line 220 def print_all searcher=Wmap::SiteTracker.instance searcher.print_all_sites end |
.read_ver ⇒ Object
Simple parser for the project version file
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
# File 'lib/wmap.rb', line 41 def read_ver ver=Hash.new f=File.open(VERSION,'r') f.each do |line| line.chomp! case line when /^(\s)*#/ next when /\=/ entry=line.split("=").map! {|x| x.strip} ver[entry[0]]=entry[1] end end f.close return ver end |
.refresh(site) ⇒ Object
Refresh the site information in the local data repository
195 196 197 198 199 |
# File 'lib/wmap.rb', line 195 def refresh(site) store=Wmap::SiteTracker.instance store.refresh(site) store.save! end |
.refresh_all ⇒ Object
Refresh the site information in the local data repository
202 203 204 205 206 |
# File 'lib/wmap.rb', line 202 def refresh_all store=Wmap::SiteTracker.instance store.refresh_all store.save! end |
.response_code(url) ⇒ Object
Check URL/Site response code
170 171 172 173 |
# File 'lib/wmap.rb', line 170 def response_code(url) checker=Wmap::UrlChecker.new checker.response_code(url) end |
.scan(host) ⇒ Object
Fast tcp port scanner on a single host or IP
89 90 91 92 |
# File 'lib/wmap.rb', line 89 def scan(host) scanner=Wmap::PortScanner.new scanner.scan(host) end |
.scans(target_list) ⇒ Object
Fast multi-processes tcp port scanner on a list of targets
95 96 97 98 |
# File 'lib/wmap.rb', line 95 def scans(target_list) scanner=Wmap::PortScanner.new scanner.scans(target_list) end |
.search(pattern) ⇒ Object
Search the site repository for all entries that match the pattern
176 177 178 179 |
# File 'lib/wmap.rb', line 176 def search(pattern) searcher=Wmap::SiteTracker.instance searcher.search(pattern) end |
.sub_domain_known?(host) ⇒ Boolean
Sub-domain tracking - check local hosts file to see if the sub-domain is already known
138 139 140 |
# File 'lib/wmap.rb', line 138 def sub_domain_known?(host) tracker=Wmap::HostTracker.instance.sub_domain_known?(host) end |
.track(host) ⇒ Object
CIDR Tracking - check the host against the local CIDR seed file, return the CIDR tracking path if found
101 102 103 104 |
# File 'lib/wmap.rb', line 101 def track(host) tracker=Wmap::CidrTracker.new tracker.cidr_worker(host) end |
.whois(domain) ⇒ Object
whois query and sort the result into structured data
83 84 85 86 |
# File 'lib/wmap.rb', line 83 def whois(domain) whois=Wmap::Whois.new(:verbose=>false) whois.query(domain) end |
.wlog(msg, agent, log_file) ⇒ Object
Log the information into file
160 161 162 |
# File 'lib/wmap.rb', line 160 def wlog(msg,agent,log_file) Wmap::Utils.wlog(msg,agent,log_file) end |
.wmap(seed) ⇒ Object
Explorer to discover and inventory web application / service automatically
71 72 73 74 |
# File 'lib/wmap.rb', line 71 def wmap(seed) cmd="bin/wmap" + " " + seed system(cmd) end |