Module: Wmap

Defined in:
lib/wmap/utils/utils.rb,
lib/wmap.rb,
lib/wmap/utils/logger.rb,
lib/wmap/utils/url_magic.rb,
lib/wmap/utils/wp_detect.rb,
lib/wmap/utils/domain_root.rb,
lib/wmap/url_crawler/adware_tag.rb,
lib/wmap/site_tracker/wp_tracker.rb,
lib/wmap/domain_tracker/sub_domain.rb,
lib/wmap/host_tracker/primary_host.rb,
lib/wmap/site_tracker/deactivated_site.rb

Overview

Class to trace de-activated site. This is need for basic state tracking for our sites.

Defined Under Namespace

Modules: Utils Classes: CidrTracker, DnsBruter, DomainTracker, GeoIPTracker, GoogleSearchScraper, HostTracker, NetworkProfiler, PortScanner, SiteTracker, UrlChecker, UrlCrawler, Whois

Constant Summary collapse

NAME = 
"Wmap"
GEM = 
"wmap"
VERSION = 
File.dirname(__FILE__) + "/../version.txt"

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.known_internet_domains ⇒ Object

Returns the value of attribute known_internet_domains.



38
39
40
 
# File 'lib/wmap.rb', line 38

def known_internet_domains
  @known_internet_domains
end

.verbose=(value) ⇒ Object (writeonly)

Sets the attribute verbose

Parameters:

  • value —

    the value to set the attribute verbose to.



39
40
41
 
# File 'lib/wmap.rb', line 39

def verbose=(value)
  @verbose = value
end

Class Method Details

Project banner in ASCII Art ‘soft’ format, courtesy to patorjk.com/software/taag/



60
61
62
63
64
65
66
67
68
69
 
# File 'lib/wmap.rb', line 60

def banner
	ver=read_ver
	art=",--.   ,--.       ,--.       ,--.   ,--.
|  |   |  | ,---. |  |-.     |   `.'   | ,--,--. ,---.  ,---.  ,---. ,--.--.
|  |.'.|  || .-. :| .-. '    |  |'.'|  |' ,-.  || .-. || .-. || .-. :|  .--'
|   ,'.   |\   --.| `-' |    |  |   |  |\ '-'  || '-' '| '-' '\   --.|  |
'--'   '--' `----' `---'     `--'   `--' `--`--'|  |-' |  |-'  `----'`--'
                                               `--'   `--'                  "
	string = "-"*80 + "\n" + art + "\n" + "Version: " + ver["version"] + "\tRelease Date: " + ver["date"] + "\nDesigned and developed by: " + ver["author"] + "\nEmail: " + ver["email"] + "\tLinkedIn: " + ver["linkedin"] + "\n" + "-"*80
end

.check(url) ⇒ Object

URL checker - check the status of the remote URL



122
123
124
125
 
# File 'lib/wmap.rb', line 122

def check(url)
	checker=Wmap::UrlChecker.new(:verbose=>false)
	checker.url_worker(url)
end

.crawl(url) ⇒ Object

Crawler to search url contents for new sites



86
87
88
89
 
# File 'lib/wmap.rb', line 86

def crawl(url)
	crawler=Wmap::UrlCrawler.new
	crawler.crawl(url)
end

.data_dir(data_path) ⇒ Object 



71
72
73
 
# File 'lib/wmap.rb', line 71

def data_dir(data_path)
  @data_dir=data_path.to_s
end

.dns_brute(domain) ⇒ Object

DNS Brute Forcer



186
187
188
189
 
# File 'lib/wmap.rb', line 186

def dns_brute(domain)
	bruter=Wmap::DnsBruter.new
	bruter.query(domain)
end

.domain_known?(domain) ⇒ Boolean

Domain Tracking - check with the trust domain seed file locally, to determine if it’s a new internet domain NOT to confuse with the Internet ‘whois’ lookup

Returns:

  • (Boolean) 


140
141
142
143
144
145
146
147
148
 
# File 'lib/wmap.rb', line 140

def domain_known?(domain)
	tracker=Wmap::DomainTracker.instance
   if @data_dir
     tracker.data_dir=@data_dir
     tracker.domains_file=tracker.data_dir + "/" + "domains"
     tracker.load_domains_from_file(tracker.domains_file)
   end
	tracker.domain_known?(domain)
end

.domain_root(host) ⇒ Object

Retrieve root domain from a host



192
193
194
 
# File 'lib/wmap.rb', line 192

def domain_root(host)
	Wmap::Utils.get_domain_root(host)
end

.dump(file) ⇒ Object

Dump out the unique sites into a plain file



224
225
226
227
228
229
230
231
232
 
# File 'lib/wmap.rb', line 224

def dump(file)
		store=Wmap::SiteTracker.instance
     if @data_dir
       store.data_dir = @data_dir
       store.sites_file = searcher.data_dir + "/" + "sites"
       store.load_site_stores_from_file(searcher.sites_file)
     end
		store.save_uniq_sites(file)
end

.dump_xml(file) ⇒ Object

Dump out the unique sites into a XML file



235
236
237
238
239
240
241
242
243
 
# File 'lib/wmap.rb', line 235

def dump_xml(file)
		store=Wmap::SiteTracker.instance
     if @data_dir
       store.data_dir = @data_dir
       store.sites_file = searcher.data_dir + "/" + "sites"
       store.load_site_stores_from_file(searcher.sites_file)
     end
     store.save_uniq_sites_xml(file)
end

.geoip(host) ⇒ Object

GeoIP Tracking - check the host / IP against the GeoIP data repository, return the Geographic information if found



116
117
118
119
 
# File 'lib/wmap.rb', line 116

def geoip(host)
	tracker=Wmap::GeoIPTracker.new
	tracker.query(host)
end

.google ⇒ Object

Search the Google engines and sort out sites known by Google



270
271
272
 
# File 'lib/wmap.rb', line 270

def google
	sites=Wmap::GoogleSearchScraper.new.workers.keys
end

.host_known?(host) ⇒ Boolean

Host Tracking - check local hosts file to see if this is a hostname known from the host seed file NOT to confuse with a regular DNS lookup over the internet

Returns:

  • (Boolean) 


152
153
154
155
156
157
158
159
160
 
# File 'lib/wmap.rb', line 152

def host_known?(host)
	tracker=Wmap::HostTracker.instance
   if @data_dir
     tracker.data_dir = data_dir
     tracker.hosts_file = tracker.data_dir + "/" + "hosts"
     tracker.load_known_hosts_from_file(tracker.hosts_file)
   end
   tracker.host_known?(host)
end

.ip_known?(ip) ⇒ Boolean

IP Tracking - check local hosts file to see if this is an IP known from the seed file NOT to confuse with a regular reverse DNS lookup over the internet

Returns:

  • (Boolean) 


175
176
177
178
179
180
181
182
183
 
# File 'lib/wmap.rb', line 175

def ip_known?(ip)
	tracker=Wmap::HostTracker.instance
   if @data_dir
     tracker.data_dir = data_dir
     tracker.hosts_file = tracker.data_dir + "/" + "hosts"
     tracker.load_known_hosts_from_file(tracker.hosts_file)
   end
   tracker.ip_known?(ip)
end

.ip_trusted?(ip) ⇒ Boolean

Check if the IP is within the range of the known CIDR blocks

Returns:

  • (Boolean) 


128
129
130
131
132
133
134
135
136
 
# File 'lib/wmap.rb', line 128

def ip_trusted?(ip)
	tracker=Wmap::CidrTracker.new
   if @data_dir
     tracker.data_dir=@data_dir
     tracker.cidr_seeds=tracker.data_dir + "/" + "cidrs"
     tracker.load_cidr_blks_from_file(tracker.cidr_seeds)
   end
	tracker.ip_trusted?(ip)
end

.mutation(host) ⇒ Object

Host-name mutation for catch easily guessable hostname, i.e. “ww1.example.com” => [“ww1,example.com”,“ww2.example.com”,…]



202
203
204
 
# File 'lib/wmap.rb', line 202

def mutation (host)
	Wmap::DnsBruter.new.hostname_mutation(host)
end

.print(site) ⇒ Object

Print a site’s full information from the repository



275
276
277
278
 
# File 'lib/wmap.rb', line 275

def print(site)
	searcher=Wmap::SiteTracker.instance
	searcher.print_site(site)
end

.print_all ⇒ Object

Print a site’s full information from the repository



281
282
283
284
285
286
287
288
289
 
# File 'lib/wmap.rb', line 281

def print_all
	searcher=Wmap::SiteTracker.instance
   if @data_dir
     searcher.data_dir = @data_dir
     searcher.sites_file = searcher.data_dir + "/" + "sites"
     searcher.load_site_stores_from_file(searcher.sites_file)
   end
   searcher.print_all_sites
end

.read_ver ⇒ Object

Simple parser for the project version file



42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/wmap.rb', line 42

def read_ver
	ver=Hash.new
	f=File.open(VERSION,'r')
	f.each do |line|
		line.chomp!
		case line
		when /^(\s)*#/
			next
		when /\=/
			entry=line.split("=").map! {|x| x.strip}
			ver[entry[0]]=entry[1]
		end
	end
	f.close
	return ver
end

.refresh(site) ⇒ Object

Refresh the site information in the local data repository



246
247
248
249
250
251
252
253
254
255
 
# File 'lib/wmap.rb', line 246

def refresh(site)
		store=Wmap::SiteTracker.instance
     if @data_dir
       store.data_dir = @data_dir
       store.sites_file = searcher.data_dir + "/" + "sites"
       store.load_site_stores_from_file(searcher.sites_file)
     end
		store.refresh(site)
		store.save!
end

.refresh_all ⇒ Object

Refresh the site information in the local data repository



258
259
260
261
262
263
264
265
266
267
 
# File 'lib/wmap.rb', line 258

def refresh_all
		store=Wmap::SiteTracker.instance
     if @data_dir
       store.data_dir = @data_dir
       store.sites_file = searcher.data_dir + "/" + "sites"
       store.load_site_stores_from_file(searcher.sites_file)
     end
		store.refresh_all
		store.save!
end

.response_code(url) ⇒ Object

Check URL/Site response code



207
208
209
210
 
# File 'lib/wmap.rb', line 207

def response_code(url)
	checker=Wmap::UrlChecker.new
	checker.response_code(url)
end

.scan(host) ⇒ Object

Fast tcp port scanner on a single host or IP



98
99
100
101
 
# File 'lib/wmap.rb', line 98

def scan(host)
	scanner=Wmap::PortScanner.new
	scanner.scan(host)
end

.scans(target_list) ⇒ Object

Fast multi-processes tcp port scanner on a list of targets



104
105
106
107
 
# File 'lib/wmap.rb', line 104

def scans(target_list)
	scanner=Wmap::PortScanner.new
	scanner.scans(target_list)
end

.search(pattern) ⇒ Object

Search the site repository for all entries that match the pattern



213
214
215
216
217
218
219
220
221
 
# File 'lib/wmap.rb', line 213

def search(pattern)
	searcher=Wmap::SiteTracker.instance
   if @data_dir
     searcher.data_dir = @data_dir
     searcher.sites_file = searcher.data_dir + "/" + "sites"
     searcher.load_site_stores_from_file(searcher.sites_file)
   end
	searcher.search(pattern)
end

.sub_domain_known?(host) ⇒ Boolean

Sub-domain tracking - check local hosts file to see if the sub-domain is already known

Returns:

  • (Boolean) 


163
164
165
166
167
168
169
170
171
 
# File 'lib/wmap.rb', line 163

def sub_domain_known?(host)
	tracker=Wmap::HostTracker.instance
   if @data_dir
     tracker.data_dir = data_dir
     tracker.hosts_file = tracker.data_dir + "/" + "hosts"
     tracker.load_known_hosts_from_file(tracker.hosts_file)
   end
   tracker.sub_domain_known?(host)
end

.track(host) ⇒ Object

CIDR Tracking - check the host against the local CIDR seed file, return the CIDR tracking path if found



110
111
112
113
 
# File 'lib/wmap.rb', line 110

def track(host)
	tracker=Wmap::CidrTracker.new
	tracker.cidr_worker(host)
end

.whois(domain) ⇒ Object

whois query and sort the result into structured data



92
93
94
95
 
# File 'lib/wmap.rb', line 92

def whois(domain)
	whois=Wmap::Whois.new(:verbose=>false)
	whois.query(domain)
end

.wlog(msg, agent, log_file) ⇒ Object

Log the information into file



197
198
199
 
# File 'lib/wmap.rb', line 197

def wlog(msg,agent,log_file)
	Wmap::Utils.wlog(msg,agent,log_file)
end

.wmap(seed) ⇒ Object

Explorer to discover and inventory web application / service automatically



76
77
78
79
80
81
82
83
 
# File 'lib/wmap.rb', line 76

def wmap(seed)
   if @data_dir
     cmd = "bin/wmap" + " -d " + @data_dir + " -t " + seed
   else
	    cmd="bin/wmap" + " -t " + seed
   end
	system(cmd)
end