Class: Recog::Fingerprint

Inherits:
Object
  • Object
show all
Defined in:
lib/recog/fingerprint.rb,
lib/recog/fingerprint/regexp_factory.rb

Overview

A fingerprint that can be matched against a particular kind of fingerprintable data, e.g. an HTTP Server header

Defined Under Namespace

Modules: RegexpFactory Classes: Test

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(xml, match_key = nil, protocol = nil, example_path = nil) ⇒ Fingerprint

Returns a new instance of Fingerprint.

Parameters:

  • xml (Nokogiri::XML::Element)
  • match_key (String) (defaults to: nil) —

    See Recog::DB

  • protocol (String) (defaults to: nil) —

    Protocol such as ftp, mssql, http, etc.

  • example_path (String) (defaults to: nil) —

    Directory path for fingerprint example files



42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/recog/fingerprint.rb', line 42

def initialize(xml, match_key=nil, protocol=nil, example_path=nil)
  @match_key = match_key
  @protocol = protocol
  @name   = parse_description(xml)
  @regex  = create_regexp(xml)
  @line   = xml.line
  @params = {}
  @tests = []

  @protocol.downcase! if @protocol
  parse_examples(xml, example_path)
  parse_params(xml)
end

Instance Attribute Details

#line ⇒ Integer (readonly)

The line number of the XML entity in the source file for this fingerprint.

Returns:

  • (Integer) —

    The line number of this entity.



36
37
38
 
# File 'lib/recog/fingerprint.rb', line 36

def line
  @line
end

#name ⇒ String (readonly)

A human readable name describing this fingerprint

Returns:

  • (String) —

    Contents of the source XML's description tag



14
15
16
 
# File 'lib/recog/fingerprint.rb', line 14

def name
  @name
end

#params ⇒ Hash<String,Array> (readonly)

Collection of indexes for capture groups created by #match

Returns:

  • (Hash<String,Array>) —

    Keys are things like "os.name", values are a two element Array. The first element is an index for the capture group that returns that thing. If the index is 0, the second element is a static value for that thing; otherwise it is undefined.



25
26
27
 
# File 'lib/recog/fingerprint.rb', line 25

def params
  @params
end

#regex ⇒ Regexp (readonly)

Regular expression pulled from the DB xml file.

Returns:

  • (Regexp) —

    the Regexp to try when calling #match

See Also:

  • #create_regexp


20
21
22
 
# File 'lib/recog/fingerprint.rb', line 20

def regex
  @regex
end

#tests ⇒ void (readonly)

This method returns an undefined value.

Collection of example strings that should #match our #regex



30
31
32
 
# File 'lib/recog/fingerprint.rb', line 30

def tests
  @tests
end

Instance Method Details

#match(match_string) ⇒ Hash?

Attempt to match the given string.

Parameters:

  • match_string (String)

Returns:

  • (Hash, nil) —

    Keys will be host, service, and os attributes



69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
 
# File 'lib/recog/fingerprint.rb', line 69

def match(match_string)
  # match_string.force_encoding('BINARY') if match_string
  begin
    match_data = @regex.match(match_string)
  rescue Encoding::CompatibilityError => e
    begin
      # Replace invalid UTF-8 characters with spaces, just as DAP does.
      encoded_str = match_string.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => '')
      match_data = @regex.match(encoded_str)
    rescue Exception => e
      output_diag_data('Exception while re-encoding match_string to UTF-8', match_string, e)
    end
  rescue Exception => e
    output_diag_data('Exception while running regex against match_string', match_string, e)
  end
  return if match_data.nil?

  result = { 'matched' => @name }
  replacements = {}
  @params.each_pair do |k,v|
    pos = v[0]
    if pos == 0
      # A match offset of 0 means this param has a hardcoded value
      result[k] = v[1]
      # if this value uses interpolation, note it for handling later
      v[1].scan(/\{([^\s{}]+)\}/).flatten.each do |replacement|
        replacements[k] ||= Set[]
        replacements[k] << replacement
      end
    else
      # A match offset other than 0 means the value should come from
      # the corresponding match result index
      result[k] = match_data[ pos ]
    end
  end

  # Use the protocol specified in the XML database if there isn't one
  # provided as part of this fingerprint.
  if @protocol
    unless result['service.protocol']
      result['service.protocol'] = @protocol
    end
  end

  result['fingerprint_db'] = @match_key if @match_key

  # for everything identified as using interpolation, do so
  replacements.each_pair do |replacement_k, replacement_vs|
    replacement_vs.each do |replacement|
      if result[replacement]
        result[replacement_k] = result[replacement_k].gsub(/\{#{replacement}\}/, result[replacement])
      else
        # if the value uses an interpolated value that does not exist, in general this could be
        # very bad, but over time we have allowed the use of regexes with
        # optional captures that are then used for parts of the asserted
        # fingerprints.  This is frequently done for optional version
        # strings.  If the key in question is cpe23 and the interpolated
        # value we are trying to replace is version related, use the CPE
        # standard of '-' for the version, otherwise raise and exception as
        # this code currently does not handle interpolation of undefined
        # values in other cases.
        if replacement_k =~ /\.cpe23$/ and replacement =~ /\.version$/
          result[replacement_k] = result[replacement_k].gsub(/\{#{replacement}\}/, '-')
        else
          raise "Invalid use of nil interpolated non-version value #{replacement} in non-cpe23 fingerprint param #{replacement_k}"
        end
      end
    end
  end

  # After performing interpolation, remove temporary keys from results
  result.each_pair do |k, _|
    if k.start_with?('_tmp.')
      result.delete(k)
    end
  end

  return result
end

#output_diag_data(message, data, exception) ⇒ Object 



56
57
58
59
60
61
62
63
 
# File 'lib/recog/fingerprint.rb', line 56

def output_diag_data(message, data, exception)
  STDERR.puts message
  STDERR.puts exception.inspect
  STDERR.puts "Length:   #{data.length}"
  STDERR.puts "Encoding: #{data.encoding}"
  STDERR.puts "Problematic data:\n#{data}"
  STDERR.puts "Raw bytes:\n#{data.pretty_inspect}\n"
end

#verify_params {|status, message| ... } ⇒ Object

Ensure all the #params are valid

Yield Parameters:

  • status (Symbol) —

    One of :warn, :fail, or :success to indicate whether a param is valid

  • message (String) —

    A human-readable string explaining the status



155
156
157
158
159
160
161
162
163
164
165
 
# File 'lib/recog/fingerprint.rb', line 155

def verify_params(&block)
  return if params.empty?
  params.each do |param_name, pos_value|
    pos, value = pos_value
    if pos > 0 && !value.to_s.empty?
      yield :fail, "'#{@name}'s #{param_name} is a non-zero pos but specifies a value of '#{value}'"
    elsif pos == 0 && value.to_s.empty?
      yield :fail, "'#{@name}'s #{param_name} is not a capture (pos=0) but doesn't specify a value"
    end
  end
end

#verify_tests {|status, message| ... } ⇒ Object

Ensure all the #tests actually match the fingerprint and return the expected capture groups.

Yield Parameters:

  • status (Symbol) —

    One of :warn, :fail, or :success to indicate whether a test worked

  • message (String) —

    A human-readable string explaining the status



174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
 
# File 'lib/recog/fingerprint.rb', line 174

def verify_tests(&block)

  # look for the presence of test cases
  if tests.size == 0
    yield :warn, "'#{@name}' has no test cases"
    return
  end

  # make sure each test case passes
  tests.each do |test|
    result = match(test.content)
    if result.nil?
      yield :fail, "'#{@name}' failed to match #{test.content.inspect} with #{@regex}'"
      next
    end

    message = test
    status = :success
    # Ensure that all the attributes as provided by the example were parsed
    # out correctly and match the capture group values we expect.
    test.attributes.each do |k, v|
      next if k == '_encoding'
      next if k == '_filename'
      if !result.has_key?(k) || result[k] != v
        message = "'#{@name}' failed to find expected capture group #{k} '#{v}'. Result was #{result[k]}"
        status = :fail
        break
      end
    end
    yield status, message
  end

  # make sure there are capture groups for all params that use them
  verify_tests_have_capture_groups(&block)
end

#verify_tests_have_capture_groups {|status, message| ... } ⇒ Object

For fingerprints that specify parameters that are defined by capture groups, ensure that each parameter has at least one test that defines an attribute to test for the correct capture of that parameter.

Yield Parameters:

  • status (Symbol) —

    One of :warn, :fail, or :success to indicate whether a test worked

  • message (String) —

    A human-readable string explaining the status



219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
 
# File 'lib/recog/fingerprint.rb', line 219

def verify_tests_have_capture_groups(&block)
  capture_group_used = {}
  if !params.empty?
    # get a list of parameters that are defined by capture groups
    params.each do |param_name, pos_value|
      pos, value = pos_value
      if pos > 0 && value.to_s.empty?
        capture_group_used[param_name] = false
      end
    end
  end

  # match up the fingerprint parameters with test attributes
  tests.each do |test|
    test.attributes.each do |k,v|
      if capture_group_used.has_key?(k)
        capture_group_used[k] = true
      end
    end
  end

  # alert on untested parameters unless they are temporary
  capture_group_used.each do |param_name, param_used|
    if !param_used && !param_name.start_with?('_tmp.')
      message = "'#{@name}' is missing an example that checks for parameter '#{param_name}' " +
                "which is derived from a capture group"
      yield :fail, message
    end
  end
end