Module: Mdm::Host::OperatingSystemNormalization

Included in:
Mdm::Host
Defined in:
lib/mdm/host/operating_system_normalization.rb

Overview

TODO:

Handle OS icon incompatiblities with new fingerprint names Note that VMWare ESX(i) was special cased before as well, make sure it still works 1) Cisco IOS -> IOS breaks the icon mapping in MSP/MSCE of /cisco/ 2) Ubuntu Linux -> Linux breaks the distro selection The real solution is to add os_vendor and take this into account for icons

TODO:

Implement rspec coverage for normalize_os()

TODO:

Implement smb.generic fingerprint database (replace #parse_windows_os_str?)

TODO:

Implement Samba version matching for specific distributions and OS versions

TODO:

Implement DD-WRT and various embedded device signatures currently missing

TODO:

Correct inconsistencies in os_name use by removing the vendor string (Microsoft Windows -> Windows) This applies to MSF core and a handful of modules, not to mention some Recog fingerprints.

TODO:

Rename host.os_sp to host.os_version

TODO:

Add host.os_vendor

TODO:

Add host.os_confidence

TODO:

Add host.domain

Rules for operating system fingerprinting in Metasploit

The os.product key identifies the common-name of a specific operating system Examples include: Linux, Windows XP, Mac OS X, IOS, AIX, HP-UX, VxWorks

The os.version key identifies the service pack or version of the operating system Sometimes this means a kernel or firmware version when the distribution or OS version is not available. Examples include: SP2, 10.04, 2.6.47, 10.6.1

The os.vendor key identifies the manufacturer of the operating system Examples include: Microsoft, Ubuntu, Cisco, HP, IBM, Wind River

The os.family key identifies the group of the operating system. This is often a duplicate of os.product, unless a more specific product name is available. Examples include: Windows, Linux, IOS, HP-UX, AIX

The os.edition key identifies the specific variant of the operating system Examples include: Enterprise, Professional, Starter, Evaluation, Home, Datacenter

An example breakdown of a common operating system is shown below

  • Microsoft Windows XP Professional Service Pack 3 English (x86)
    • os.product = 'Windows XP'
    • os.edition = 'Professional'
    • os.vendor = 'Microsoft'
    • os.version = 'SP3'
    • os.language = 'English'
    • os.arch = 'x86'

These rules are then mapped to the Mdm::Host attributes below:

  • os_name - Maps to a normalized os.product key
  • os_flavor - Maps to a normalized os.edition key
  • os_sp - Maps to a normalized os.version key (soon os_version)
  • os_lang - Maps to a normalized os.language key
  • arch - Maps to a normalized os.arch key

Additional rules include the following mappings:

  • name - Maps to the host.name key
  • mac - Maps to the host.mac key

The following keys are not mapped to Mdm::Host at this time (but should be):

  • os.vendor

In order to execute these rules, this module is responsible for mapping various fingerprint sources to Mdm::Host values. This requires some ugly glue code to account for differences between each supported input (external scanners), the Recog gem and associated databases, and how Metasploit itself likes to handle these values. Getting a mapping wrong is often harmless, but can impact the automatic targetting capabilities of certain exploit modules.

In other words, this is a best-effort attempt to rationalize multiple competing sources of information about a host and come up with the values representing a normalized assessment of the system. The use of Recog and multiple scanner fingerprints can result in a comprehensive (and confident) identification of the remote operating system and associated services.

Historically, there are direct conflicts between certain Metasploit modules, certain scanners, and external fingerprint databases in terms of how a particular OS and patch level is represented. This module attempts to fix what it can and serve as documentation and live workarounds for the rest.

Examples of known conflicts that are still in progress:

  • Metasploit defines an OS constant of 'win'/'windows' as Microsoft Windows

    • Scanner modules report a mix of 'Microsoft Windows' and 'Windows'
    • Nearly all exploit modules reference 'Windows SP'
    • Nmap (and other scanners) also prefix the vendor before Windows

  • Windows service packs represented as 'Service Pack X' or 'SPX'

    • The preferred form is to set os.version to 'SPX'
    • Many external scanners & Recog prefer 'Service Pack X'

  • Apple Mac OS X, Cisco IOS, IBM AIX, Ubuntu Linux, all reported with vendor prefix

    • The preferred form is to remove the vendor from os.product
    • Mdm::Host currently has no vendor field, so this information is lost today
    • Many scanners report leading vendor strings and require normalization
    • The os_flavor field is used in contradictory ways across Metasploit
    • The preferred form is to be a 'display only' field
    • Some Recog fingerprints still append the edition to os.product
    • Many scanners report the edition as a trailing suffix to os.product

Maintenance:

  1. Ensure that the latest Recog gem is present and installed
  2. For new operating system releases, update relevant sections a) Windows releases will require updates to a few methods 1) parse_windows_os_str() 2) normalize_nmap_fingerprint() 3) normalize_nexpose_fingerprint() 4) Other scanner normalizers b) Mobile operating systems are minimally recognized

Constant Summary collapse

MAX_NMAP_CERTAINTY =

Cap nmap certainty at 0.84 until we update it more frequently XXX: Without this, Nmap will beat the default certainty of recog matches and its less-confident guesses will take precedence over service-based fingerprints.

0.84

Instance Method Summary collapse

Instance Method Details

#apply_match_to_host(match) ⇒ Object

Examine the assertations of the merged best match and map these back to fields of Mdm::Host. Take particular care not to leave related fields (os_*) in a conflicting state, leverage existing values where possible, and use the most confident values we have.



395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 395

def apply_match_to_host(match)
  host = self

  # These values in a match always override the current value unless
  # the host attribute has been explicitly locked by the user

  if match['host.mac'] && !host.attribute_locked?(:mac)
    host.mac = sanitize(match['host.mac'])
  end

  if match['host.name'] && !host.attribute_locked?(:name)
    host.name = sanitize(match['host.name'])
  end

  # Select the os architecture if available
  if match['os.arch'] && !host.attribute_locked?(:arch)
    host.arch = sanitize(match['os.arch'])
  end

  # Guess the purpose using some basic heuristics
  if ! host.attribute_locked?(:purpose)
    host.purpose = guess_purpose_from_match(match)
  end

  #
  # Map match fields from Recog fingerprint style to Metasploit style
  #

  # os.build:                 Examples: 9001, 2600, 7602
  # os.device:                Examples: General, ADSL Modem, Broadband router, Cable Modem, Camera, Copier, CSU/DSU
  # os.edition:               Examples: Web, Storage, HPC, MultiPoint, Enterprise, Home, Starter, Professional
  # os.family:                Examples: Windows, Linux, Solaris, NetWare, ProCurve, Mac OS X, HP-UX, AIX
  # os.product:               Examples: Windows, Linux, Windows Server 2008 R2, Windows XP, Enterprise Linux, NEO Tape Library
  # os.vendor:                Examples: Microsoft, HP, IBM, Sun, 3Com, Ricoh, Novell, Ubuntu, Apple, Cisco, Xerox
  # os.version:               Examples: SP1, SP2, 6.5 SP3 CPR, 10.04, 8.04, 12.10, 4.0, 6.1, 8.5
  # os.language:              Examples: English, Arabic, German
  # linux.kernel.version:     Examples: 2.6.32

  # Metasploit currently ignores os.build, os.device, and os.vendor as separate fields.

  # Select the OS name from os.name, fall back to os.family
  if ! host.attribute_locked?(:os_name)
    # Try to fill this value from os.product first if it exists
    if match.has_key?('os.product')
      host.os_name = sanitize(match['os.product'])
    else
      # Fall back to os.family otherwise, if available
      if match.has_key?('os.family')
        host.os_name = sanitize(match['os.family'])
      end
    end
  end

  if match.has_key?('os.family')
    host.os_family = sanitize(match['os.family'])
  end

  # Select the flavor from os.edition if available
  if match.has_key?('os.edition') and ! host.attribute_locked?(:os_flavor)
    host.os_flavor = sanitize(match['os.edition'])
  end

  # Select an OS version as os.version, fall back to linux.kernel.version
  if ! host.attribute_locked?(:os_sp)
    if match['os.version']
      host.os_sp = sanitize(match['os.version'])
    else
      if match['linux.kernel.version']
        host.os_sp = sanitize(match['linux.kernel.version'])
      end
    end
  end

  # Select the os language if available
  if match.has_key?('os.language') and ! host.attribute_locked?(:os_lang)
    host.os_lang = sanitize(match['os.language'])
  end

  # Normalize MAC addresses to lower-case colon-delimited format
  if host.mac and ! host.attribute_locked?(:mac)
    host.mac = host.mac.downcase
    if host.mac =~ /^[a-f0-9]{12}$/
      host.mac = host.mac.scan(/../).join(':')
    end
  end
end

#get_arch_from_string(str) ⇒ Object

Return a normalized architecture based on patterns in the input string. This will identify things like sparc, powerpc, x86_x64, and i686



937
938
939
940
941
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 937

def get_arch_from_string(str)
  res = Recog::Nizer.match("architecture", str)
  return unless (res and res['os.arch'])
  res['os.arch']
end

#guess_purpose_from_match(match) ⇒ Object

Loosely guess the purpose of a device based on available match values. In the future, also take into account the exposed services and rename to guess_purpose_with_match()



487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 487

def guess_purpose_from_match(match)
  # some data that is sent to this is numeric; we do not want that
  pstr = ""
  # Go through each character of each value and make sure it is all
  # UTF-8
  match.values.each do |i|
    if i.respond_to?(:encoding)
      i.each_char do |j|
        begin
          pstr << j.downcase.encode("UTF-8")
        rescue Encoding::UndefinedConversionError
        # rescue Encoding::UndefinedConversionError => e
          # this works in Framework, but causes a Travis CI error
          # elog("Found incompatible (non-ANSI) character in guess_purpose_from_match")
        end
      end
    end
  end
  # Loosely map keywords to specific purposes
  case pstr
  when /windows server|windows (nt|20)/
    'server'
  when /windows (xp|vista|[78]|10)/
    'client'
  when /printer|print server/
    'printer'
  when /router/
    'router'
  when /firewall/
    'firewall'
  when /linux/
    'server'
  else
    'device'
  end
end

#normalize_fusionvm_fingerprint(data) ⇒ Object

Normalize data from FusionVM fingerprints



801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 801

def normalize_fusionvm_fingerprint(data)
  ret = {}
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))
    when /Linux ([^[:space:]]*) ([^[:space:]]*) .* (\(.*\))/
      ret['os.product'] = "Linux"
      ret['host.name']  = $1
      ret['os.version'] = $2
      ret['os.arch']    = get_arch_from_string($3)
    else
      ret['os.product'] = data[:os]
  end
  ret['os.arch'] = data[:arch] if data[:arch]
  ret['host.name'] = data[:name] if data[:name]
  [ ret ]
end

#normalize_generic_fingerprint(data) ⇒ Object

Normalize data from generic fingerprints



822
823
824
825
826
827
828
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 822

def normalize_generic_fingerprint(data)
  ret = {}
  ret['os.product'] = data[:os_name] || data[:os] || data[:os_fingerprint] || "Unknown"
  ret['os.arch'] = data[:os_arch] if data[:os_arch]
  ret['os.certainty'] = data[:os_certainty] || 0.5
  [ ret ]
end

#normalize_match(m) ⇒ Object

Normalize matches in order to handle inconsistencies between fingerprint sources and our desired usage in Metasploit. This amounts to yet more duct tape, but the situation should improve as the fingerprint sources are updated and enhanced. In the future, this method will no longer be needed (or at least, doing less and less work)



316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 316

def normalize_match(m)
  # Normalize os.version strings containing 'Service Pack X' to just 'SPX'
  if m['os.version'] and m['os.version'].index('Service Pack ') == 0
    m['os.version'] = m['os.version'].gsub(/Service Pack /, 'SP')
  end

  if m['os.product']

    # Normalize Apple Mac OS X to just Mac OS X
    if m['os.product'] =~ /^Apple Mac/
      m['os.product']  = m['os.product'].gsub(/Apple Mac/, 'Mac')
      m['os.vendor'] ||= 'Apple'
    end

    # Normalize Sun Solaris/Sun SunOS to just Solaris/SunOS
    if m['os.product'] =~ /^Sun (Solaris|SunOS)/
      m['os.product']  = m['os.product'].gsub(/^Sun /, '')
      m['os.vendor'] ||= 'Oracle'
    end

    # Normalize Microsoft Windows to just Windows to catch any stragglers
    if m['os.product'] =~ /^Microsoft Windows/
      m['os.product']  = m['os.product'].gsub(/Microsoft Windows/, 'Windows')
      m['os.vendor'] ||= 'Microsoft'
    end

    # Normalize Windows Server to just Windows to match Metasploit target names
    if m['os.product'] =~ /^Windows Server/
      m['os.product'] = m['os.product'].gsub(/Windows Server/, 'Windows')
    end

    # Normalize OS Family
    m = normalize_match_family(m)
  end

  m
end

#normalize_match_family(m) ⇒ Object

Normalize matches in order to ensure that an os.family entry exists if we have enough data to put one together.



356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 356

def normalize_match_family(m)
  # If the os.family already exists, we don't need to do anything
  return m if m['os.family'].present?
  case m['os.product']
    when /Windows/
      m['os.family'] = 'Windows'
    when /Linux/
      m['os.family'] = 'Linux'
    when /Solaris/
      m['os.family'] = 'Solaris'
    when /SunOS/
      m['os.family'] = 'SunOS'
    when /AIX/
      m['os.family'] = 'AIX'
    when /HP-UX/
      m['os.family'] = 'HP-UX'
    when /OS X/
      m['os.family'] = 'OS X'
  end
  m
end

#normalize_mbsa_fingerprint(data) ⇒ Object

Normalize data from MBSA fingerprints



596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 596

def normalize_mbsa_fingerprint(data)
  ret = {}
  # :os_match=>"Microsoft Windows Vista SP0 or SP1, Server 2008, or Windows 7 Ultimate (build 7000)"
  #    :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"7" :os_accuracy=>"100"
  ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy]
  ret['os.family']    = data[:os_family] if data[:os_family]
  ret['os.vendor']    = data[:os_vendor] if data[:os_vendor]

  if data[:os_family] and data[:os_version]
    ret['os.product'] = data[:os_family] + " " + data[:os_version]
  end

  ret['host.name'] = data[:hostname] if data[:hostname]

  [ ret ]
end

#normalize_nessus_fingerprint(data) ⇒ Object

Normalize data from Nessus fingerprints



701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 701

def normalize_nessus_fingerprint(data)
  ret = {}
  # :os=>"Microsoft Windows 2000 Advanced Server (English)"
  # :os=>"Microsoft Windows 2000\nMicrosoft Windows XP"
  # :os=>"Linux Kernel 2.6"
  # :os=>"Sun Solaris 8"
  # :os=>"IRIX 6.5"

  # Nessus sometimes jams multiple OS names together with a newline.
  oses = data[:os].split(/\n/)
  if oses.length > 1
    # Multiple fingerprints means Nessus wasn't really sure, reduce
    # the certainty accordingly
    ret['os.certainty'] = 0.5
  else
    ret['os.certainty'] = 0.8
  end

  # Since there is no confidence associated with them, the best we
  # can do is just take the first one.
  case oses.first
    when /^(Microsoft |)Windows/
      ret.update(parse_windows_os_str(data[:os]))

    when /(2\.[46]\.\d+[-a-zA-Z0-9]+)/
      # Look for older Linux kernel versions
      ret['os.product'] = "Linux"
      ret['os.version'] = $1

    when /^Linux Kernel ([\d\.]+)(.*)/
      # Look for strings like "Linux Kernel 2.6 on Ubuntu 9.10 (karmic)"
      # Ex: Linux Kernel 2.2 on Red Hat Linux release 6.2 (Zoot)
      # Ex: Linux Kernel 2.6 on Ubuntu Linux 8.04 (hardy)
      ret['os.product'] = "Linux"
      ret['os.version'] = $1

      vendor = $2.to_s

      # Try to snag the vendor name as well
      if vendor =~ /on (\w+|\w+ \w+|\w+ \w+ \w+) (Linux|\d)/
        ret['os.vendor'] = $1
      end

    when /(.*) ([0-9\.]+)$/
      # Then we don't necessarily know what the os is, but this fingerprint has
      # some version information at the end, pull it off, treat the first part
      # as the OS, and the rest as the version.
      ret['os.product'] = $1.gsub("Kernel", '').strip
      ret['os.version'] = $2
    else
      # TODO: Return each OS guess as a separate match
      ret['os.product'] = oses.first
  end

  ret['host.name'] = data[:hname] if data[:hname]
  [ ret ]
end

#normalize_nexpose_fingerprint(data) ⇒ Object

Normalize data from Nexpose fingerprints



617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 617

def normalize_nexpose_fingerprint(data)
  ret = {}
  # :family=>"Windows" :certainty=>"0.85" :vendor=>"Microsoft" :product=>"Windows 7 Ultimate Edition"
  # :family=>"Windows" :certainty=>"0.67" :vendor=>"Microsoft" :arch=>"x86" :product=>'Windows 7' :version=>'SP1'
  # :family=>"Linux" :certainty=>"0.64" :vendor=>"Linux" :product=>"Linux"
  # :family=>"Linux" :certainty=>"0.80" :vendor=>"Ubuntu" :product=>"Linux"
  # :family=>"IOS" :certainty=>"0.80" :vendor=>"Cisco" :product=>"IOS"
  # :family=>"embedded" :certainty=>"0.61" :vendor=>"Linksys" :product=>"embedded"

  ret['os.certainty'] = data[:certainty] if data[:certainty]
  ret['os.family']    = data[:family]    if data[:family]
  ret['os.vendor']    = data[:vendor]    if data[:vendor]

  case data[:product]
  when /^Windows/

    # TODO: Verify Windows CE and Windows 8 RT fingerprints
    # Translate the version into the representation we want

    case data[:version].to_s

    # These variants are normalized to just 'Windows <Version>'
    when "NT", "2000", "95", "ME", "XP", "Vista", "7", "8", "8.1"
      ret['os.product'] = "Windows #{data[:version]}"

    # Service pack in the version field should be recognized
    when /^SP\d+/, /^Service Pack \d+/
      ret['os.product'] = data[:product]
      ret['os.version'] = data[:version]

    # No version means the version is part of the product already
    when nil, ''
      # Trim any 'Server' suffix and use as it is
      ret['os.product'] = data[:product].sub(/ Server$/, '')

    # Otherwise, we assume a Server version of Windows
    else
      ret['os.product'] = "Windows Server #{data[:version]}"
    end

    # Extract the edition string if it is present
    if data[:product] =~ /(XP|Vista|\d+(?:\.\d+)) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/
      ret['os.edition'] = $2
    end

  when nil, 'embedded'
    # Use the family or vendor name when the product is empty or 'embedded'
    ret['os.product']   = data[:family] unless data[:family] == 'embedded'
    ret['os.product'] ||= data[:vendor]
    ret['os.version']   = data[:version] if data[:version]
  else
    # Default to using the product name reported by Nexpose
    ret['os.product'] = data[:product] if data[:product]
  end

  ret['os.arch'] = get_arch_from_string(data[:arch]) if data[:arch]
  ret['os.arch'] ||= get_arch_from_string(data[:desc]) if data[:desc]

  [ ret ]
end

#normalize_nmap_fingerprint(data) ⇒ Object

Normalize data from Nmap fingerprints



565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 565

def normalize_nmap_fingerprint(data)
  ret = {}

  # :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"2000" :os_accuracy=>"94"
  ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy]
  if (data[:os_vendor] == data[:os_family])
    ret['os.product'] = data[:os_family]
  else
    ret['os.product'] = data[:os_family]
    ret['os.vendor'] = data[:os_vendor]
  end

  # Nmap places the type of Windows (XP, 7, etc) into the version field
  if ret['os.product'] == 'Windows' and data[:os_version]
    ret['os.product'] = ret['os.product'] + ' ' + data[:os_version].to_s
  else
    ret['os.version'] = data[:os_version]
  end

  ret['host.name'] = data[:hostname] if data[:hostname]

  if ret['os.certainty']
    ret['os.certainty'] = [ ret['os.certainty'].to_f, MAX_NMAP_CERTAINTY ].min.to_s
  end

  [ ret ]
end

#normalize_os ⇒ Object

Normalize the operating system fingerprints provided by various scanners (nmap, nexpose, retina, nessus, metasploit modules, and more!)

These are stored as notes (instead of directly in the os_* fields) specifically for this purpose.

The goal is to infer as much as we can about the OS of the device and the various services offered using the Recog gem and some glue logic to determine the best weights. This method can result in changes to the recorded Mdm::Host#os_name, Mdm::Host#os_flavor, Mdm::Host#os_sp, Mdm::Host#os_lang, Mdm::Host#purpose, Mdm::Host#name, Mdm::Host#arch, and the service details.



151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 151

def normalize_os
  host   = self
  matches = []

  # Note that we're already restricting the query to this host by using
  # host.notes instead of Note, so don't need a host_id in the
  # conditions.
  fingerprintable_notes = self.notes.where("ntype like '%%fingerprint'")
  fingerprintable_notes.each do |fp_note|
    matches += recog_matches_for_note(fp_note)
  end

  # XXX: This hack solves the memory leak generated by self.services.each {}
  fingerprintable_services = self.services.where("name is not null and name != '' and info is not null and info != ''")
  fingerprintable_services.each do |s|
    matches += recog_matches_for_service(s)
  end

  #
  # Look for generic fingerprint.match notes that generate a match hash from modules
  # This handles ad-hoc os.language, host.name, etc identifications
  #
  generated_matches = self.notes.where(ntype: 'fingerprint.match')
  generated_matches.each do |m|
    next unless (m.data and m.data.kind_of?(::Hash))
    matches << m.data.dup
  end

  # Normalize matches for consistency during the ranking phase
  matches = matches.map{ |m| normalize_match(m) }

  # Calculate the best OS match based on fingerprint hits
  match = Recog::Nizer.best_os_match(matches)

  # Merge and normalize the best match to the host object
  apply_match_to_host(match) if match

  # Set some sane defaults if needed
  host.os_name ||= 'Unknown'
  host.purpose ||= 'device'

  host.save if host.changed?
end

#normalize_qualys_fingerprint(data) ⇒ Object

Normalize data from Qualys fingerprints



762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 762

def normalize_qualys_fingerprint(data)
  ret = {}
  # :os=>"Microsoft Windows 2000"
  # :os=>"Windows 2003"
  # :os=>"Microsoft Windows XP Professional SP3"
  # :os=>"Ubuntu Linux"
  # :os=>"Cisco IOS 12.0(3)T3"
  # :os=>"Red-Hat Linux 6.0"
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))

    when /^(Cisco) (IOS) (\d+[^\s]+)/
      ret['os.product'] = $2
      ret['os.vendor']  = $1
      ret['os.version'] = $3

    when /^([^\s]+) (Linux)(.*)/
      ret['os.product'] = $2
      ret['os.vendor'] = $1

      ver = $3.to_s.strip.split(/\s+/).first
      if ver =~ /^\d+\./
        ret['os.version'] = ver
      end

    else
      parts = data[:os].split(/\s+/, 3)
      ret['os.product'] = "Unknown"
      ret['os.product'] = parts[0] if parts[0]
      ret['os.product'] << " " + parts[1] if parts[1]
      ret['os.version'] = parts[2] if parts[2]
  end
  [ ret ]
end

#normalize_retina_fingerprint(data) ⇒ Object

Normalize data from Retina fingerprints



682
683
684
685
686
687
688
689
690
691
692
693
694
695
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 682

def normalize_retina_fingerprint(data)
  ret = {}
  # :os=>"Windows Server 2003 (X64), Service Pack 2"
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))
    else
      # No idea what this looks like if it isn't windows.  Just store
      # the whole thing and hope for the best.
      # TODO: Add examples of non-Windows results
      ret['os.product'] = data[:os] if data[:os]
  end
  [ ret ]
end

#normalize_scanner_fp(fp) ⇒ Object

Convert a host.os._fingerprint Note into a hash containing 'os.' and 'host.*' fields

Also includes a os.certainty which is a float from 0 - 1.00 indicating the scanner's confidence in its fingerprint. If the particular scanner does not provide such information, default to 0.80.



837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 837

def normalize_scanner_fp(fp)
  hits = []

  return hits if not validate_fingerprint_data(fp)

  case fp.ntype
  when /^host\.os\.(.*_fingerprint)$/
    pname = $1
    pmeth = 'normalize_' + pname
    if self.respond_to?(pmeth)
      hits = self.send(pmeth, fp.data)
    else
      hits = normalize_generic_fingerprint(fp.data)
    end
  end
  hits.each {|hit| hit['os.certainty'] ||= 0.80}
  hits
end

#normalize_session_fingerprint(data) ⇒ Object

Normalize data from Meterpreter's client.sys.config.sysinfo()



533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 533

def normalize_session_fingerprint(data)
  ret = {}
  case data[:os]
    when /Windows/
      ret.update(parse_windows_os_str(data[:os]))
  # Switch to this code block once the multi-meterpreter code review is complete
=begin

    when /^(Windows \w+)\s*\(Build (\d+)(.*)\)/
      ret['os.product'] = $1
      ret['os.build'] = $2
      ret['os.vendor'] = 'Microsoft'
      possible_sp = $3
      if possible_sp =~ /Service Pack (\d+)/
        ret['os.version'] = 'SP' + $1
      end
=end
    when /Linux (\d+\.\d+\.\d+\S*)\s* \((\w*)\)/
      ret['os.product'] = "Linux"
      ret['os.version'] = $1
      ret['os.arch']    = get_arch_from_string($2)
    else
      ret['os.product'] = data[:os]
  end
  ret['os.arch'] = data[:arch] if data[:arch]
  ret['host.name'] = data[:name] if data[:name]
  [ ret ]
end

#parse_windows_os_str(str) ⇒ Object

Take a windows version string and return a hash with fields suitable for Host this object's version fields. This is used as a fall-back to parse external fingerprints and should eventually be replaced by per-source mappings.

A few example strings that this will have to parse: sessions Windows XP (Build 2600, Service Pack 3). Windows .NET Server (Build 3790). Windows 2008 (Build 6001, Service Pack 1). retina Windows Server 2003 (X64), Service Pack 2 nessus Microsoft Windows 2000 Advanced Server (English) qualys Microsoft Windows XP Professional SP3 Windows 2003

Note that this list doesn't include nexpose or nmap, since they are both kind enough to give us the various strings in seperate pieces that we don't have to parse out manually.



879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 879

def parse_windows_os_str(str)
  ret = {}

  # Set some reasonable defaults for Windows
  ret['os.vendor']  = 'Microsoft'
  ret['os.product'] = 'Windows'

  # Determine the actual Windows product name
  case str
    when /\.NET Server/
      ret['os.product'] << ' Server 2003'
    when / (2000|2003|2008|2012)/
      ret['os.product'] << ' Server ' + $1
    when / (NT (?:3\.51|4\.0))/
      ret['os.product'] << ' ' + $1
    when /Windows (95|98|ME|XP|Vista|[\d\.]+)/
      ret['os.product'] << ' ' + $1
    else
      # If we couldn't pull out anything specific for the flavor, just cut
      # off the stuff we know for sure isn't it and hope for the best
      ret['os.product'] = (ret['os.product'] + ' ' + str.gsub(/(Microsoft )|(Windows )|(Service Pack|SP) ?(\d+)/i, '').strip).strip

      # Make sure the product name doesn't include any non-alphanumeric stuff
      # This fixes cases where the above code leaves 'Windows XX (Build 3333,)...'
      ret['os.product'] = ret['os.product'].split(/[^a-zA-Z0-9 ]/).first.strip

  end

  # Take a guess at the architecture
  arch = get_arch_from_string(str)
  ret['os.arch'] = arch if arch

  # Extract any service pack value in the string
  if str =~ /(Service Pack|SP) ?(\d+)/i
    ret['os.version']  = "SP#{$2}"
  end

  # Extract any build ID found in the string
  if str =~ /build (\d+)/i
    ret['os.build'] = $1
  end

  # Extract the OS edition if available
  if str =~ /(\d+|\d+\.\d+) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/
    ret['os.edition'] = $2
  else
    if str =~ /(Professional|Enterprise|Pro|Home|Start|Datacenter|Web|Storage|MultiPoint)/
      ret['os.edition'] = $1
    end
  end

  ret
end

#recog_matches_for_note(note) ⇒ Array<Hash>

Recog matches for the fingerprint in note.

Returns:

  • (Array<Hash>) —

    Keys will be host, service, and os attributes



244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 244

def recog_matches_for_note(note)
  # Skip notes that are missing the correct structure or have been blacklisted
  return [] if not validate_fingerprint_data(note)

  #
  # These rules define the relationship between fingerprint note keys
  # and specific Recog databases for detailed matching. Notes that do
  # not match a rule are passed to the generic matcher.
  #
  fingerprint_note_match_keys = {
    'smb.fingerprint'  => {
      :native_os               => [ 'smb.native_os' ],
      :native_lm               => [ 'smb.native_lm' ],
    },
    'http.fingerprint' => {
      :header_server           => [ 'http_header.server', 'apache_os' ],
      :header_set_cookie       => [ 'http_header.cookie' ],
      :header_www_authenticate => [ 'http_header.wwwauth' ],
    # TODO: Candidates for future Recog support
    # :content                 => 'http_body'
    # :code                    => 'http_response_code'
    # :message                 => 'http_response_message'
    }
  }

  matches = []

  # Look for a specific Recog database for this type and data key
  if fingerprint_note_match_keys.has_key?( note.ntype )
    fingerprint_note_match_keys[ note.ntype ].each_pair do |k,rdbs|
      if note.data.has_key?(k)
        rdbs.each do |rdb|
          res = Recog::Nizer.match(rdb, note.data[k])
          matches << res if res
        end
      end
    end
  else
    # Add all generic match results to the overall match array
    normalize_scanner_fp(note).each do |m|
      next unless m
      matches << m
    end
  end

  matches
end

#recog_matches_for_service(s) ⇒ Array<Hash>

Recog matches for the s service.

Parameters:

Returns:

  • (Array<Hash>) —

    Keys will be host, service, and os attributes



199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 199

def recog_matches_for_service(s)
  #
  # We assume that the service.info field contains certain types of probe
  # replies and associate these with one or more Recog databases. The mapping
  # of service.name to a specific database only fits into so many places and
  # Mdm currently serves that role.
  #

  service_match_keys = {
    # TODO: Implement smb.generic fingerprint database
    # 'smb'     => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match
    # 'netbios' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match

    'ssh'     => [ 'ssh.banner' ], # Recog expects just the vendor string, not the protocol version
    'http'    => [ 'http_header.server', 'apache_os'], # The 'Apache' fingerprints try to infer OS/distribution from the extra information in the Server header
    'https'   => [ 'http_header.server', 'apache_os'], # XXX: verify vmware esx(i) case on https (TODO: normalize https to http, track SSL elsewhere, such as a new set of fields)
    'snmp'    => [ 'snmp.sys_description' ],
    'telnet'  => [ 'telnet.banner' ],
    'smtp'    => [ 'smtp.banner' ],
    'imap'    => [ 'imap4.banner' ],  # Metasploit reports 143/993 as imap (TODO: normalize imap to imap4)
    'pop3'    => [ 'pop3.banner' ],   # Metasploit reports 110/995 as pop3
    'nntp'    => [ 'nntp.banner' ],
    'ftp'     => [ 'ftp.banner' ],
    'ssdp'    => [ 'http_header.server' ]
  }

  matches = []

  return matches unless service_match_keys.has_key?(s.name)

  service_match_keys[s.name].each do |rdb|
    banner = s.info
    if self.respond_to?("service_banner_recog_filter_#{s.name}")
      banner = self.send("service_banner_recog_filter_#{s.name}", banner)
    end
    res = Recog::Nizer.match(rdb, banner)
    matches << res if res
  end

  matches
end

#sanitize(text) ⇒ Object

Ensure that the host attribute is using ascii safe text and escapes any other byte value.



526
527
528
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 526

def sanitize(text)
  Rex::Text.ascii_safe_hex(text)
end

#service_banner_recog_filter_ssh(banner) ⇒ Object

Recog assumes that the protocol version of the SSH banner has been removed



381
382
383
384
385
386
387
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 381

def service_banner_recog_filter_ssh(banner)
  if banner =~ /^SSH-\d+\.\d+-(.*)/
    $1
  else
    banner
  end
end

#validate_fingerprint_data(fp) ⇒ Object

Determine if the fingerprint data is readable. If not, it nearly always means that there was a problem with the YAML or the Marshal'ed data, so let's log that for later investigation.



295
296
297
298
299
300
301
302
303
304
305
306
307
 
# File 'lib/mdm/host/operating_system_normalization.rb', line 295

def validate_fingerprint_data(fp)
  if fp.data.kind_of?(Hash) and !fp.data.empty?
    return true
  elsif fp.ntype == "postgresql.fingerprint"
    # Special case postgresql.fingerprint; it's always a string,
    # and should not be used for OS fingerprinting (yet), so
    # don't bother logging it. TODO: fix os fingerprint finding, this
    # name collision seems silly.
    return false
  else
    return false
  end
end