Class: Recog::Fingerprint

Inherits:
Object
  • Object
show all
Defined in:
lib/recog/fingerprint.rb,
lib/recog/fingerprint/regexp_factory.rb

Overview

A fingerprint that can be matched against a particular kind of fingerprintable data, e.g. an HTTP Server header

Defined Under Namespace

Modules: RegexpFactory Classes: Test

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(xml, match_key = nil, protocol = nil, example_path = nil) ⇒ Fingerprint

Returns a new instance of Fingerprint.

Parameters:

  • xml (Nokogiri::XML::Element)
  • match_key (String) (defaults to: nil) —

    See Recog::DB

  • protocol (String) (defaults to: nil) —

    Protocol such as ftp, mssql, http, etc.

  • example_path (String) (defaults to: nil) —

    Directory path for fingerprint example files



43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/recog/fingerprint.rb', line 43

def initialize(xml, match_key = nil, protocol = nil, example_path = nil)
  @match_key = match_key
  @protocol = protocol&.downcase
  @name   = parse_description(xml)
  @regex  = create_regexp(xml)
  @line   = xml.line
  @params = {}
  @tests = []

  parse_examples(xml, example_path)
  parse_params(xml)
end

Instance Attribute Details

#line ⇒ Integer (readonly)

The line number of the XML entity in the source file for this fingerprint.

Returns:

  • (Integer) —

    The line number of this entity.



37
38
39
 
# File 'lib/recog/fingerprint.rb', line 37

def line
  @line
end

#name ⇒ String (readonly)

A human readable name describing this fingerprint

Returns:

  • (String) —

    Contents of the source XML's description tag



15
16
17
 
# File 'lib/recog/fingerprint.rb', line 15

def name
  @name
end

#params ⇒ Hash<String,Array> (readonly)

Collection of indexes for capture groups created by #match

Returns:

  • (Hash<String,Array>) —

    Keys are things like "os.name", values are a two element Array. The first element is an index for the capture group that returns that thing. If the index is 0, the second element is a static value for that thing; otherwise it is undefined.



26
27
28
 
# File 'lib/recog/fingerprint.rb', line 26

def params
  @params
end

#regex ⇒ Regexp (readonly)

Regular expression pulled from the DB xml file.

Returns:

  • (Regexp) —

    the Regexp to try when calling #match

See Also:

  • #create_regexp


21
22
23
 
# File 'lib/recog/fingerprint.rb', line 21

def regex
  @regex
end

#tests ⇒ void (readonly)

This method returns an undefined value.

Collection of example strings that should #match our #regex



31
32
33
 
# File 'lib/recog/fingerprint.rb', line 31

def tests
  @tests
end

Instance Method Details

#match(match_string) ⇒ Hash?

Attempt to match the given string.

Parameters:

  • match_string (String)

Returns:

  • (Hash, nil) —

    Keys will be host, service, and os attributes



69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
 
# File 'lib/recog/fingerprint.rb', line 69

def match(match_string)
  # match_string.force_encoding('BINARY') if match_string
  begin
    match_data = @regex.match(match_string)
  rescue Encoding::CompatibilityError
    begin
      # Replace invalid UTF-8 characters with spaces, just as DAP does.
      encoded_str = match_string.encode('UTF-8', invalid: :replace, undef: :replace, replace: '')
      match_data = @regex.match(encoded_str)
    rescue Exception => e
      output_diag_data('Exception while re-encoding match_string to UTF-8', match_string, e)
    end
  rescue Exception => e
    output_diag_data('Exception while running regex against match_string', match_string, e)
  end
  return if match_data.nil?

  result = { 'matched' => @name }
  replacements = {}
  @params.each_pair do |k, v|
    pos = v[0]
    if pos == 0
      # A match offset of 0 means this param has a hardcoded value
      result[k] = v[1]
      # if this value uses interpolation, note it for handling later
      v[1].scan(/\{([^\s{}]+)\}/).flatten.each do |replacement|
        replacements[k] ||= Set[]
        replacements[k] << replacement
      end
    else
      # A match offset other than 0 means the value should come from
      # the corresponding match result index
      result[k] = match_data[pos]
    end
  end

  # Use the protocol specified in the XML database if there isn't one
  # provided as part of this fingerprint.
  result['service.protocol'] = @protocol if @protocol && !(result['service.protocol'])

  result['fingerprint_db'] = @match_key if @match_key

  # for everything identified as using interpolation, do so
  replacements.each_pair do |replacement_k, replacement_vs|
    replacement_vs.each do |replacement|
      if result[replacement]
        result[replacement_k] = result[replacement_k].gsub(/\{#{replacement}\}/, result[replacement])
      else
        # if the value uses an interpolated value that does not exist, in general this could be
        # very bad, but over time we have allowed the use of regexes with
        # optional captures that are then used for parts of the asserted
        # fingerprints.  This is frequently done for optional version
        # strings.  If the key in question is cpe23 and the interpolated
        # value we are trying to replace is version related, use the CPE
        # standard of '-' for the version, otherwise raise and exception as
        # this code currently does not handle interpolation of undefined
        # values in other cases.
        raise "Invalid use of nil interpolated non-version value #{replacement} in non-cpe23 fingerprint param #{replacement_k}" unless replacement_k =~ (/\.cpe23$/) && replacement =~ (/\.version$/)

        result[replacement_k] = result[replacement_k].gsub(/\{#{replacement}\}/, '-')

      end
    end
  end

  # After performing interpolation, remove temporary keys from results
  result.each_pair do |k, _|
    result.delete(k) if k.start_with?('_tmp.')
  end

  result
end

#output_diag_data(message, data, exception) ⇒ Object 



56
57
58
59
60
61
62
63
 
# File 'lib/recog/fingerprint.rb', line 56

def output_diag_data(message, data, exception)
  $stderr.puts message
  $stderr.puts exception.inspect
  $stderr.puts "Length:   #{data.length}"
  $stderr.puts "Encoding: #{data.encoding}"
  $stderr.puts "Problematic data:\n#{data}"
  $stderr.puts "Raw bytes:\n#{data.pretty_inspect}\n"
end

#verify_params {|status, message| ... } ⇒ Object

Ensure all the #params are valid

Yield Parameters:

  • status (Symbol) —

    One of :warn, :fail, or :success to indicate whether a param is valid

  • message (String) —

    A human-readable string explaining the status



148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/recog/fingerprint.rb', line 148

def verify_params
  return if params.empty?

  params.each do |param_name, pos_value|
    pos, value = pos_value
    if pos > 0 && !value.to_s.empty?
      yield :fail, "'#{@name}'s #{param_name} is a non-zero pos but specifies a value of '#{value}'"
    elsif pos == 0 && value.to_s.empty?
      yield :fail, "'#{@name}'s #{param_name} is not a capture (pos=0) but doesn't specify a value"
    end
  end
end

#verify_tests {|status, message| ... } ⇒ Object

Ensure all the #tests actually match the fingerprint and return the expected capture groups.

Yield Parameters:

  • status (Symbol) —

    One of :warn, :fail, or :success to indicate whether a test worked

  • message (String) —

    A human-readable string explaining the status



168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
 
# File 'lib/recog/fingerprint.rb', line 168

def verify_tests(&block)
  # look for the presence of test cases
  if tests.size == 0
    yield :warn, "'#{@name}' has no test cases"
    return
  end

  # make sure each test case passes
  tests.each do |test|
    result = match(test.content)
    if result.nil?
      yield :fail, "'#{@name}' failed to match #{test.content.inspect} with #{@regex}'"
      next
    end

    message = test
    status = :success
    # Ensure that all the attributes as provided by the example were parsed
    # out correctly and match the capture group values we expect.
    test.attributes.each do |k, v|
      next if k == '_encoding'
      next if k == '_filename'

      next unless !result.key?(k) || result[k] != v

      message = "'#{@name}' failed to find expected capture group #{k} '#{v}'. Result was #{result[k]}"
      status = :fail
      break
    end
    yield status, message
  end

  # make sure there are capture groups for all params that use them
  verify_tests_have_capture_groups(&block)
end

#verify_tests_have_capture_groups {|status, message| ... } ⇒ Object

For fingerprints that specify parameters that are defined by capture groups, ensure that each parameter has at least one test that defines an attribute to test for the correct capture of that parameter.

Yield Parameters:

  • status (Symbol) —

    One of :warn, :fail, or :success to indicate whether a test worked

  • message (String) —

    A human-readable string explaining the status



213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
 
# File 'lib/recog/fingerprint.rb', line 213

def verify_tests_have_capture_groups
  capture_group_used = {}
  unless params.empty?
    # get a list of parameters that are defined by capture groups
    params.each do |param_name, pos_value|
      pos, value = pos_value
      capture_group_used[param_name] = false if pos > 0 && value.to_s.empty?
    end
  end

  # match up the fingerprint parameters with test attributes
  tests.each do |test|
    test.attributes.each_key do |k|
      capture_group_used[k] = true if capture_group_used.key?(k)
    end
  end

  # alert on untested parameters unless they are temporary
  capture_group_used.each do |param_name, param_used|
    next unless !param_used && !param_name.start_with?('_tmp.')

    message = "'#{@name}' is missing an example that checks for parameter '#{param_name}' " \
              'which is derived from a capture group'
    yield :fail, message
  end
end