Class: Wmap::HostTracker
- Inherits:
-
Object
- Object
- Wmap::HostTracker
- Includes:
- Singleton, Utils
- Defined in:
- lib/wmap/host_tracker.rb,
lib/wmap/host_tracker/primary_host.rb
Overview
Class to handle the local host data repository file where lists of known hosts from discovery and past assessment efforts are stored
Direct Known Subclasses
Defined Under Namespace
Classes: PrimaryHost
Constant Summary
Constants included from Utils::UrlMagic
Utils::UrlMagic::Max_http_timeout
Constants included from Utils::DomainRoot
Utils::DomainRoot::File_ccsld, Utils::DomainRoot::File_cctld, Utils::DomainRoot::File_gtld, Utils::DomainRoot::File_tld
Instance Attribute Summary collapse
-
#alias ⇒ Object
readonly
Returns the value of attribute alias.
-
#data_dir ⇒ Object
Returns the value of attribute data_dir.
-
#hosts_file ⇒ Object
Returns the value of attribute hosts_file.
-
#known_hosts ⇒ Object
readonly
Returns the value of attribute known_hosts.
-
#max_parallel ⇒ Object
Returns the value of attribute max_parallel.
-
#verbose ⇒ Object
Returns the value of attribute verbose.
Instance Method Summary collapse
-
#add(host) ⇒ Object
Setter to add host entry to the cache once at a time.
-
#bulk_add(list, num = @max_parallel) ⇒ Object
(also: #adds)
Setter to add host entry to the local hosts in batch (from an array).
-
#bulk_delete(list) ⇒ Object
(also: #dels)
‘setter’ to delete host entry to the cache in batch (from an array).
-
#count ⇒ Object
Count numbers of entries in the local host repository.
-
#delete(host) ⇒ Object
‘setter’ to remove entry from the local hosts one at a time.
-
#dump_sub_domains ⇒ Object
(also: #get_sub_domains)
Extract a list of sub-domains from the local host repository @known_hosts.
-
#file_add(file) ⇒ Object
‘setter’ to add host entry to the local hosts in batch (from a file).
-
#file_delete(file) ⇒ Object
Setter to delete host entries in the local hosts in batch (from a file).
-
#get_a_records ⇒ Object
(also: #dump_a_records)
Extract hostname without the root domain part from the @known_hosts.
-
#get_root_domains ⇒ Object
(also: #dump_root_domains)
Extract known root domains from the local host repository @known_hosts.
-
#host_aliases(host) ⇒ Object
(also: #aliases)
Search local host repository and return a list of aliases for the host.
-
#host_known?(host) ⇒ Boolean
(also: #is_known?)
Check if the specific host within @known_hosts table.
-
#initialize(params = {}) ⇒ HostTracker
constructor
Instance default variables.
-
#ip_known?(ip) ⇒ Boolean
(also: #has_a_record?)
Check if the specific IP within @known_hosts table.
-
#local_host_2_ip(host) ⇒ Object
Perform DNS lookup on the local host repository.
-
#local_ip_2_host(ip) ⇒ Object
Perform reverse DNS lookup on the local host repository.
-
#print_host(host) ⇒ Object
(also: #print)
Print summary report on the cache.
-
#print_known_hosts ⇒ Object
(also: #print_all)
Print summary report on the cache.
-
#refresh(host) ⇒ Object
Setter to refresh the entry from the cache one at a time.
-
#refresh_all ⇒ Object
Refresh all the entries in the local hosts by querying the Internet.
-
#save_known_hosts_to_file!(f_hosts = @file_hosts) ⇒ Object
(also: #save!)
Save the current local hosts hash table into a (random) data repository file.
-
#search(pattern) ⇒ Object
(also: #find)
Search potential matching sites from the host store by using simple regular expression.
-
#sub_domain_known?(domain) ⇒ Boolean
Based on the current host store, to determine if an entry is a known sub-domain.
-
#top_hostname(num) ⇒ Object
Top hostname - sort out most common host-name in the host store in descendant order.
Methods included from Utils
#cidr_2_ips, #file_2_hash, #file_2_list, #get_nameserver, #get_nameservers, #host_2_ip, #host_2_ips, #is_cidr?, #is_fqdn?, #is_ip?, #list_2_file, #reverse_dns_lookup, #sort_ips, #valid_dns_record?, #zone_transferable?
Methods included from Utils::Logger
Methods included from Utils::UrlMagic
#create_absolute_url_from_base, #create_absolute_url_from_context, #host_2_url, #is_site?, #is_ssl?, #is_url?, #landing_location, #make_absolute, #normalize_url, #open_page, #redirect_location, #response_code, #url_2_host, #url_2_path, #url_2_port, #url_2_site, #urls_on_same_domain?
Methods included from Utils::DomainRoot
#get_domain_root, #get_domain_root_by_ccsld, #get_domain_root_by_cctld, #get_domain_root_by_tlds, #get_sub_domain, #is_domain_root?, #print_ccsld, #print_cctld, #print_gtld
Constructor Details
#initialize(params = {}) ⇒ HostTracker
Instance default variables
21 22 23 24 25 26 27 28 29 30 31 32 |
# File 'lib/wmap/host_tracker.rb', line 21 def initialize (params = {}) @verbose=params.fetch(:verbose, false) @data_dir=params.fetch(:data_dir, File.dirname(__FILE__)+'/../../data/') Dir.mkdir(@data_dir) unless Dir.exist?(@data_dir) # Set default instance variables @file_hosts=@data_dir + 'hosts' file=params.fetch(:hosts_file, @file_hosts) @max_parallel=params.fetch(:max_parallel, 40) # Initialize the instance variables File.write(@file_hosts, "") unless File.exist?(@file_hosts) @known_hosts=load_known_hosts_from_file(file) end |
Instance Attribute Details
#alias ⇒ Object (readonly)
Returns the value of attribute alias.
18 19 20 |
# File 'lib/wmap/host_tracker.rb', line 18 def alias @alias end |
#data_dir ⇒ Object
Returns the value of attribute data_dir.
17 18 19 |
# File 'lib/wmap/host_tracker.rb', line 17 def data_dir @data_dir end |
#hosts_file ⇒ Object
Returns the value of attribute hosts_file.
17 18 19 |
# File 'lib/wmap/host_tracker.rb', line 17 def hosts_file @hosts_file end |
#known_hosts ⇒ Object (readonly)
Returns the value of attribute known_hosts.
18 19 20 |
# File 'lib/wmap/host_tracker.rb', line 18 def known_hosts @known_hosts end |
#max_parallel ⇒ Object
Returns the value of attribute max_parallel.
17 18 19 |
# File 'lib/wmap/host_tracker.rb', line 17 def max_parallel @max_parallel end |
#verbose ⇒ Object
Returns the value of attribute verbose.
17 18 19 |
# File 'lib/wmap/host_tracker.rb', line 17 def verbose @verbose end |
Instance Method Details
#add(host) ⇒ Object
Setter to add host entry to the cache once at a time
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 |
# File 'lib/wmap/host_tracker.rb', line 106 def add(host) begin puts "Add entry to the local host repository: #{host}" host=host.strip.downcase unless host.nil? unless @known_hosts.key?(host) ip=host_2_ip(host) record=Hash.new if is_ip?(ip) # filter host to known domains only root=get_domain_root(host) puts "Domain root: #{root}" if @verbose domain_tracker=Wmap::DomainTracker.instance domain_tracker.data_dir=@data_dir if domain_tracker.domain_known?(root) domain_tracker=nil record[host]=ip record[ip]=host puts "Host data repository entry loaded: #{host} <=> #{ip}" # Replace instance with the class variable to avoid potential race condition under parallel engine # add additional logic to update the sub-domain table as well, 02/10/2014 sub=get_sub_domain(host) if sub!=root tracker=Wmap::DomainTracker::SubDomain.instance tracker.data_dir=@data_dir unless tracker.domain_known?(sub) tracker.add(sub) tracker.save! end tracker=nil end @known_hosts.merge!(record) return record else domain_tracker=nil puts "Error - host #{host} has an untrusted internet root domain: #{root}\nPlease update the trusted domain seeds file first if necessary." end else puts "Problem resolve host #{host} - unknown IP: #{ip}" end else puts "Host is already exist. Skip: #{host}" end rescue => ee puts "Exception on method #{__method__}: #{ee}" if @verbose end end |
#bulk_add(list, num = @max_parallel) ⇒ Object Also known as: adds
Setter to add host entry to the local hosts in batch (from an array)
154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 |
# File 'lib/wmap/host_tracker.rb', line 154 def bulk_add(list, num=@max_parallel) begin puts "Add entries to the local host repository: #{list}" results=Hash.new if list.size > 0 puts "Start parallel host update processing on:\n #{list}" if @verbose Parallel.map(list, :in_processes => num) { |target| add(target) }.each do |process| if process.nil? next elsif process.empty? #do nothing else results.merge!(process) end end @known_hosts.merge!(results) puts "Done loading entries." return results else puts "Error: empty list - no entry is loaded. Please check your input list and try again." end return results rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#bulk_delete(list) ⇒ Object Also known as: dels
‘setter’ to delete host entry to the cache in batch (from an array)
215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 |
# File 'lib/wmap/host_tracker.rb', line 215 def bulk_delete(list) puts "Delete entries to the local host repository from:\n #{list}" begin hosts=list changes=Array.new if hosts.size > 0 hosts.map do |x| host=delete(x) changes.push(host) unless host.nil? end puts "Done deleting hosts." return changes else puts "Error: empty list - no entry is loaded. Please check your list and try again." end rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#count ⇒ Object
Count numbers of entries in the local host repository
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
# File 'lib/wmap/host_tracker.rb', line 89 def count puts "Counting number of entries in the local host repository ..." begin cnt=0 @known_hosts.keys.map do |key| unless is_ip?(key) cnt=cnt+1 end end puts "Current number of entries: #{cnt}" return cnt rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#delete(host) ⇒ Object
‘setter’ to remove entry from the local hosts one at a time
198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 |
# File 'lib/wmap/host_tracker.rb', line 198 def delete(host) puts "Remove entry from the local host repository: #{host} " begin host=host.strip.downcase if @known_hosts.key?(host) @known_hosts.delete(host) puts "Entry cleared." return host else puts "Entry not fund. Skip: #{host}" end rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#dump_sub_domains ⇒ Object Also known as: get_sub_domains
Extract a list of sub-domains from the local host repository @known_hosts
439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 |
# File 'lib/wmap/host_tracker.rb', line 439 def dump_sub_domains puts "Dump out all active sub domains from the local hosts." if @verbose begin subs=Array.new @known_hosts.keys.each do |hostname| next if is_ip?(hostname) hostname = hostname.strip sub = get_subdomain(hostname) subs.push(sub) unless sub.nil? end subs.uniq!.sort! unless subs.empty? puts "Found sub domains: #{subs}" if @verbose return subs rescue Exception => ee puts "Exception on method #{__method__}: #{ee}" return subs end end |
#file_add(file) ⇒ Object
‘setter’ to add host entry to the local hosts in batch (from a file)
185 186 187 188 189 190 191 192 193 194 195 |
# File 'lib/wmap/host_tracker.rb', line 185 def file_add(file) begin puts "Add entries to the local host repository from file: #{file}" raise "File non-exist. Please check your file path and name again: #{file}" unless File.exist?(file) hosts=file_2_list(file) changes=bulk_add(hosts) return changes rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#file_delete(file) ⇒ Object
Setter to delete host entries in the local hosts in batch (from a file)
237 238 239 240 241 242 243 244 245 246 247 248 |
# File 'lib/wmap/host_tracker.rb', line 237 def file_delete(file) begin puts "Delete the local host repository entries from file: #{file}" raise "File non-exist. Please check your file path and name again: #{file}" unless File.exist?(file) hosts=file_2_list(file) changes=bulk_delete(hosts) puts "Delete done." return changes rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#get_a_records ⇒ Object Also known as: dump_a_records
Extract hostname without the root domain part from the @known_hosts. Data can be used for statistics study.
325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 |
# File 'lib/wmap/host_tracker.rb', line 325 def get_a_records puts "Dump out all known A records from the local hosts." begin records=Array.new (@known_hosts.keys-["",nil]).map do |hostname| next if is_ip?(hostname) hostname = hostname.strip root = get_domain_root(hostname) record = hostname.sub('.'+root,'') records.push(record) unless record.nil? end records.sort! return records rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#get_root_domains ⇒ Object Also known as: dump_root_domains
Extract known root domains from the local host repository @known_hosts
306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 |
# File 'lib/wmap/host_tracker.rb', line 306 def get_root_domains begin puts "Dump out all active root domains from the cache." zones=Array.new (@known_hosts.keys-["",nil]).map do |hostname| next if is_ip?(hostname) hostname = hostname.strip zone = get_domain_root(hostname) zones.push(zone) unless zone.nil? end zones.uniq!.sort! return zones rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#host_aliases(host) ⇒ Object Also known as: aliases
Search local host repository and return a list of aliases for the host
491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 |
# File 'lib/wmap/host_tracker.rb', line 491 def host_aliases (host) puts "Search aliases in the local hosts data repository for host: #{host}" if @verbose begin host.strip! raise "Unknown method input: #{host} We expect a FQDN host-name string from you. " unless is_fqdn?(host) aliases=Array.new if @known_hosts.key?(host) ip=local_host_2_ip(host) @known_hosts.keys.map do |key| my_ip=local_host_2_ip(key) if ip == my_ip aliases.push(key) end end else raise "Unknown host-name in the local hosts data repository: #{host}" end return aliases-[host] rescue Exception => ee puts "Exception on method #{__method__}: #{ee}" return nil end end |
#host_known?(host) ⇒ Boolean Also known as: is_known?
Check if the specific host within @known_hosts table
392 393 394 395 396 397 398 399 400 401 402 403 |
# File 'lib/wmap/host_tracker.rb', line 392 def host_known? (host) begin host=host.strip.downcase unless host.nil? return false if @known_hosts==nil return @known_hosts.key?(host.strip) rescue => ee if @verbose puts "Host Lookup Error: #{ee}" end return false end end |
#ip_known?(ip) ⇒ Boolean Also known as: has_a_record?
Check if the specific IP within @known_hosts table
375 376 377 378 379 380 381 382 383 384 385 386 387 388 |
# File 'lib/wmap/host_tracker.rb', line 375 def ip_known? (ip) known = false begin ip=ip.strip unless ip.nil? return false if @known_hosts==nil return @known_hosts.key?(ip.strip) rescue => ee if @verbose puts "IP Lookup Error: #{ee}" end return false end return known end |
#local_host_2_ip(host) ⇒ Object
Perform DNS lookup on the local host repository. Not to confuse with the DNS lookup from the Internet
423 424 425 426 427 428 429 430 431 432 433 434 435 436 |
# File 'lib/wmap/host_tracker.rb', line 423 def local_host_2_ip (host) puts "DNS lookup from the local host repository" if @verbose begin host=host.strip unless host.nil? if @known_hosts.key?(host) return @known_hosts[host] else return nil end rescue => ee puts "Exception on method #{__method__}: #{ee}" return nil end end |
#local_ip_2_host(ip) ⇒ Object
Perform reverse DNS lookup on the local host repository. Not to confuse with the reverse DNS lookup from the Internet
407 408 409 410 411 412 413 414 415 416 417 418 419 420 |
# File 'lib/wmap/host_tracker.rb', line 407 def local_ip_2_host (ip) puts "Reverse DNS lookup from the local host repository" if @verbose begin ip=ip.strip unless ip.nil? if @known_hosts.key?(ip) return @known_hosts[ip] else return nil end rescue => ee puts "Exception on method #{__method__}: #{ee}" end return nil end |
#print_host(host) ⇒ Object Also known as: print
Print summary report on the cache
357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 |
# File 'lib/wmap/host_tracker.rb', line 357 def print_host(host) puts "Local host store entry for #{host}" begin host.strip! raise "Invalid input: #{host}" unless is_fqdn?(host) if @known_hosts.key?(host) value=@known_hosts[host] puts "#{host}\t#{value}" else puts "Unknown host in the local store: #{host}" end rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#print_known_hosts ⇒ Object Also known as: print_all
Print summary report on the cache
345 346 347 348 349 350 351 352 353 |
# File 'lib/wmap/host_tracker.rb', line 345 def print_known_hosts puts "\nSummary of local hosts Table:" puts "Total entries: #{@known_hosts.size}" (@known_hosts.keys.sort-["",nil]).each do |key| value=@known_hosts[key] puts "#{key}\t#{value}" if is_fqdn?(key) end puts "End of the summary" end |
#refresh(host) ⇒ Object
Setter to refresh the entry from the cache one at a time
251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 |
# File 'lib/wmap/host_tracker.rb', line 251 def refresh(host) begin puts "Refresh the local host repository for host: #{host} " host=host.strip.downcase if @known_hosts.key?(host) old_ip=@known_hosts[host] new_ip=host_2_ip(host) if is_ip?(new_ip) if old_ip==new_ip puts "No change for the host entry: #{host}\t#{old_ip}" return nil else @known_hosts[host]=new_ip @known_hosts[new_ip]=host puts "Entry refreshed: #{host}\t#{@known_hosts[host]}" return host end else puts "Host can no longer be resolved in the Internet. Entry removed: #{host}\t#{@known_hosts[host]}" @known_hosts.delete(host) return host end else puts "Error entry non exist: #{host}" end rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#refresh_all ⇒ Object
Refresh all the entries in the local hosts by querying the Internet
282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 |
# File 'lib/wmap/host_tracker.rb', line 282 def refresh_all begin puts "Refresh all the entries in the local host repository in one shot." changes=Hash.new hosts=@known_hosts.keys @known_hosts=Hash.new changes=bulk_add(hosts) @known_hosts.merge!(changes) #@known_hosts.keys.map do |key| # unless is_ip?(key) # host=refresh(key) # changes.push(host) unless host.nil? # end #end puts "\n#{changes.size} Entries Refreshed:" if changes.size>0 #changes.map { |x| puts x } puts "Done refreshing the local hosts." return changes rescue => ee puts "Exception on method #{__method__}: #{ee}" end end |
#save_known_hosts_to_file!(f_hosts = @file_hosts) ⇒ Object Also known as: save!
Save the current local hosts hash table into a (random) data repository file
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
# File 'lib/wmap/host_tracker.rb', line 68 def save_known_hosts_to_file!(f_hosts=@file_hosts) #begin puts "Saving the local host repository from memory to file: #{f_hosts} ..." =Time.now f=File.open(f_hosts, 'w') f.write "# local hosts file created by the #{self.class} class #{__method__} method at: #{}" @known_hosts.keys.sort.map do |key| unless key =~ /\d+\.\d+\.\d+\.\d+/ f.write "\n#{key}\t#{@known_hosts[key]}" end end f.write "\n" f.close puts "local host repository is successfully saved to: #{f_hosts}" #rescue => ee # puts "Exception on method #{__method__}: #{ee}" #end end |
#search(pattern) ⇒ Object Also known as: find
Search potential matching sites from the host store by using simple regular expression. Note that any upper-case char in the search string will be automatically converted into lower case
472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 |
# File 'lib/wmap/host_tracker.rb', line 472 def search (pattern) puts "Search host store based on the regular expression: #{pattern}" if @verbose begin pattern=pattern.strip.downcase results=Array.new @known_hosts.keys.map do |key| if key =~ /#{pattern}/i results.push(key) end end return results rescue Exception => ee puts "Exception on method #{__method__}: #{ee}" return nil end end |
#sub_domain_known?(domain) ⇒ Boolean
Based on the current host store, to determine if an entry is a known sub-domain
460 461 462 463 464 465 466 467 468 469 |
# File 'lib/wmap/host_tracker.rb', line 460 def sub_domain_known?(domain) puts "Validate sub-domain: #{domain}" if @verbose begin domain=domain.strip.downcase subs=dump_sub_domains return subs.include?(domain) rescue Exception => ee puts "Exception on method #{__method__}: #{ee}" end end |
#top_hostname(num) ⇒ Object
Top hostname - sort out most common host-name in the host store in descendant order
517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 |
# File 'lib/wmap/host_tracker.rb', line 517 def top_hostname (num) puts "Sort the host store for the most common hostname. " if @verbose h=Hash.new host_store=Hash.new top=Array.new begin # Build a host table from the host file f=File.open(@file_hosts, 'r') f.each do |line| next unless line =~ /\d+\.\d+\.\d+\.\d+/ # skip the domain roots in the host list next if is_domain_root?(line.chomp) entry=line.chomp.split(%r{\t+|\s+|\,}) key=entry[0].downcase value=entry[1] puts "Loading key value pair: #{key} - #{value}" if @verbose host_store[key] = Hash.new unless known_hosts.key?(key) host_store[key]= value end f.close host_store.keys.map do |key| host=key.split('.') if h.key?(host[0]) h[host[0]]+=1 else h[host[0]]=1 end end result = h.keys.sort { |a,b| h[b] <=> h[a] } # Sort by value descendantly num = result.size if result.size < num for i in 0...num top.push(result[i]) end return top rescue Exception => ee puts "Exception on method #{__method__}: #{ee}" return nil end end |