Class: NVDFeedScraper::Feed

Inherits:
Object
  • Object
show all
Defined in:
lib/nvd_feed_api.rb

Overview

Feed object.

Class Attribute Summary collapse

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name, updated, meta_url, gz_url, zip_url) ⇒ Feed

A new instance of Feed.

Parameters:



94
95
96
97
98
99
100
101
102
103
 
# File 'lib/nvd_feed_api.rb', line 94

def initialize(name, updated, meta_url, gz_url, zip_url)
  @name = name
  @updated = updated
  @meta_url = meta_url
  @gz_url = gz_url
  @zip_url = zip_url
  # do not pull meta and json automatically for speed and memory footprint
  @meta = nil
  @json_file = nil
end

Class Attribute Details

.default_storage_location ⇒ String

Get / set default feed storage location, where will be stored JSON feeds and archives by default.

Examples:

NVDFeedScraper::Feed.default_storage_location = '/srv/downloads/'

Returns:

  • (String) —

    default feed storage location. Default to /tmp/.



35
36
37
 
# File 'lib/nvd_feed_api.rb', line 35

def default_storage_location
  @default_storage_location
end

Instance Attribute Details

#gz_url ⇒ String

Returns the URL of the gz archive of the feed.

Examples:

'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.json.gz'

Returns:

  • (String) —

    the URL of the gz archive of the feed.



57
58
59
 
# File 'lib/nvd_feed_api.rb', line 57

def gz_url
  @gz_url
end

#json_file ⇒ String (readonly)

Note:

Return nil if not previously loaded by #json_pull.

Returns the path of the saved JSON file.

Examples:

s = NVDFeedScraper.new
s.scrap
f = s.feeds("CVE-2014")
f.json_file # => nil
f.json_pull
f.json_file # => "/tmp/nvdcve-1.0-2014.json"

Returns:

  • (String) —

    the path of the saved JSON file.



86
87
88
 
# File 'lib/nvd_feed_api.rb', line 86

def json_file
  @json_file
end

#meta ⇒ Meta (readonly)

Note:

Return nil if not previously loaded by #meta_pull. Note that #json_pull also calls #meta_pull.

Returns the Meta object of the feed.

Examples:

s = NVDFeedScraper.new
s.scrap
f = s.feeds("CVE-2014")
f.meta # => nil
f.meta_pull
f.meta # => #<NVDFeedScraper::Meta:0x00555b53027570 ... >

Returns:

  • (Meta) —

    the Meta object of the feed.



75
76
77
 
# File 'lib/nvd_feed_api.rb', line 75

def meta
  @meta
end

#meta_url ⇒ String

Returns the URL of the metadata file of the feed.

Examples:

'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.meta'

Returns:

  • (String) —

    the URL of the metadata file of the feed.



52
53
54
 
# File 'lib/nvd_feed_api.rb', line 52

def meta_url
  @meta_url
end

#name ⇒ String

Returns the name of the feed.

Examples:

'CVE-2007'

Returns:

  • (String) —

    the name of the feed.



42
43
44
 
# File 'lib/nvd_feed_api.rb', line 42

def name
  @name
end

#updated ⇒ String

Returns the last update date of the feed information on the NVD website.

Examples:

'10/19/2017 3:27:02 AM -04:00'

Returns:

  • (String) —

    the last update date of the feed information on the NVD website.



47
48
49
 
# File 'lib/nvd_feed_api.rb', line 47

def updated
  @updated
end

#zip_url ⇒ String

Returns the URL of the zip archive of the feed.

Examples:

'https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.json.zip'

Returns:

  • (String) —

    the URL of the zip archive of the feed.



62
63
64
 
# File 'lib/nvd_feed_api.rb', line 62

def zip_url
  @zip_url
end

Instance Method Details

#available_cves ⇒ Array<String>

Return a list with the name of all available CVEs in the feed. Can only be called after #json_pull.

Returns:

  • (Array<String>) —

    List with the name of all available CVEs. May return thousands CVEs.



246
247
248
249
250
251
252
253
254
255
256
257
258
259
 
# File 'lib/nvd_feed_api.rb', line 246

def available_cves
  raise 'json_file is nil, it needs to be populated with json_pull' if @json_file.nil?
  raise "json_file (#{@json_file}) doesn't exist" unless File.file?(@json_file)
  doc = Oj::Doc.open(File.read(@json_file))
  # Quicker than doc.fetch('/CVE_Items').size
  doc_size = doc.fetch('/CVE_data_numberOfCVEs').to_i
  cve_names = []
  (1..doc_size).each do |i|
    doc.move("/CVE_Items/#{i}")
    cve_names.push(doc.fetch('cve/CVE_data_meta/ID'))
  end
  doc.close
  return cve_names
end

#cve(cve) ⇒ Hash #cve(cve_arr) ⇒ Array #cve(cve, *) ⇒ Array

TODO:

implement a CVE Class instead of returning a Hash.

Note:

#json_pull is needed before using this method. Remember you’re searching only in the current feed.

Search for CVE in the feed.

Examples:

s = NVDFeedScraper.new
s.scrap
f = s.feeds("CVE-2014")
f.json_pull
f.cve("CVE-2014-0002", "cve-2014-0001")

Overloads:

  • #cve(cve) ⇒ Hash

    One CVE.

    Parameters:

    • cve (String) —

      CVE ID, case insensitive.

    Returns:

    • (Hash) —

      a Ruby Hash corresponding to the CVE.

  • #cve(cve_arr) ⇒ Array

    An array of CVEs.

    Parameters:

    • cve_arr (Array<String>) —

      Array of CVE ID, case insensitive.

    Returns:

    • (Array) —

      an Array of CVE, each CVE is a Ruby Hash. May not be in the same order as provided.

  • #cve(cve, *) ⇒ Array

    Multiple CVEs.

    Parameters:

    • cve (String) —

      CVE ID, case insensitive.

    • * (String) —

      As many CVE ID as you want.

    Returns:

    • (Array) —

      an Array of CVE, each CVE is a Ruby Hash. May not be in the same order as provided.

See Also:



194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
 
# File 'lib/nvd_feed_api.rb', line 194

def cve(*arg_cve)
  raise 'json_file is nil, it needs to be populated with json_pull' if @json_file.nil?
  raise "json_file (#{@json_file}) doesn't exist" unless File.file?(@json_file)
  return_value = nil
  raise 'no argument provided, 1 or more expected' if arg_cve.empty?
  if arg_cve.length == 1
    if arg_cve[0].is_a?(String)
      raise "bad CVE name (#{arg_cve[0]})" unless /^CVE-[0-9]{4}-[0-9]{4,}$/i.match?(arg_cve[0])
      doc = Oj::Doc.open(File.read(@json_file))
      # Quicker than doc.fetch('/CVE_Items').size
      doc_size = doc.fetch('/CVE_data_numberOfCVEs').to_i
      (1..doc_size).each do |i|
        if arg_cve[0].upcase == doc.fetch("/CVE_Items/#{i}/cve/CVE_data_meta/ID")
          return_value = doc.fetch("/CVE_Items/#{i}")
          break
        end
      end
      doc.close
    elsif arg_cve[0].is_a?(Array)
      return_value = []
      # Sorting CVE can allow us to parse quicker
      # Upcase to be sure include? works
      cves_to_find = arg_cve[0].map(&:upcase).sort
      raise 'one of the provided arguments is not a String' unless cves_to_find.all? { |x| x.is_a?(String) }
      raise 'bad CVE name' unless cves_to_find.all? { |x| /^CVE-[0-9]{4}-[0-9]{4,}$/i.match?(x) }
      doc = Oj::Doc.open(File.read(@json_file))
      # Quicker than doc.fetch('/CVE_Items').size
      doc_size = doc.fetch('/CVE_data_numberOfCVEs').to_i
      (1..doc_size).each do |i|
        doc.move("/CVE_Items/#{i}")
        cve_id = doc.fetch('cve/CVE_data_meta/ID')
        if cves_to_find.include?(cve_id)
          return_value.push(doc.fetch)
          cves_to_find.delete(cve_id)
        elsif cves_to_find.empty?
          break
        end
      end
      raise "#{cves_to_find.join(', ')} are unexisting CVEs in this feed" unless cves_to_find.empty?
    else
      raise "the provided argument (#{arg_cve[0]}) is nor a String or an Array"
    end
  else
    # Overloading a list of arguments as one array argument
    return_value = cve(arg_cve)
  end
  return return_value
end

#download_file(file_url, opts = {}) ⇒ String (protected)

Download a file.

Examples:

download_file('https://example.org/example.zip') # => '/tmp/example.zip'
download_file('https://example.org/example.zip', destination_path: '/srv/save/') # => '/srv/save/example.zip'
download_file('https://example.org/example.zip', {destination_path: '/srv/save/', sha256: '70d6ea136d5036b6ce771921a949357216866c6442f44cea8497f0528c54642d'}) # => '/srv/save/example.zip'

Parameters:

  • file_url (String) —

    the URL of the file.

  • opts (Hash) (defaults to: {}) —

    the optional downlaod parameters.

Options Hash (opts):

  • :destination_path (String) —

    the destination path (may overwrite existing file). Default use default_storage_location.

  • :sha256 (String) —

    the SHA256 hash to check, if the file already exist and the hash matches then the download will be skipped.

Returns:

  • (String) —

    the saved file path.



321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
 
# File 'lib/nvd_feed_api.rb', line 321

def download_file(file_url, opts = {})
  opts[:destination_path] ||= Feed.default_storage_location
  opts[:sha256] ||= nil

  destination_path = opts[:destination_path]
  destination_path += '/' unless destination_path[-1] == '/'
  skip_download = false
  uri = URI(file_url)
  filename = uri.path.split('/').last
  destination_file = destination_path + filename
  unless opts[:sha256].nil?
    if File.file?(destination_file)
      # Verify hash to see if it is the latest
      computed_h = Digest::SHA256.file(destination_file)
      skip_download = true if opts[:sha256].casecmp(computed_h.hexdigest).zero?
    end
  end
  unless skip_download
    res = Net::HTTP.get_response(uri)
    raise "#{file_url} ended with #{res.code} #{res.message}" unless res.is_a?(Net::HTTPSuccess)
    open(destination_file, 'wb') do |file|
      file.write(res.body)
    end
  end
  return destination_file
end

#download_gz(opts = {}) ⇒ String

Download the gz archive of the feed.

Examples:

afeed.download_gz
afeed.download_gz(destination_path: '/srv/save/')

Parameters:

Returns:

  • (String) —

    the saved gz file path.



121
122
123
 
# File 'lib/nvd_feed_api.rb', line 121

def download_gz(opts = {})
  download_file(@gz_url, opts)
end

#download_zip(opts = {}) ⇒ String

Download the zip archive of the feed.

Examples:

afeed.download_zip
afeed.download_zip(destination_path: '/srv/save/')

Parameters:

Returns:

  • (String) —

    the saved zip file path.



131
132
133
 
# File 'lib/nvd_feed_api.rb', line 131

def download_zip(opts = {})
  download_file(@zip_url, opts)
end

#json_pull(opts = {}) ⇒ String

Note:

Will downlaod and save the zip of the JSON file, unzip and save it. This massively consume time.

Download the JSON feed and fill the attribute.

Parameters:

Returns:

See Also:



140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/nvd_feed_api.rb', line 140

def json_pull(opts = {})
  opts[:destination_path] ||= Feed.default_storage_location

  skip_download = false
  destination_path = opts[:destination_path]
  destination_path += '/' unless destination_path[-1] == '/'
  filename = URI(@zip_url).path.split('/').last.chomp('.zip')
  # do not use @json_file for destination_file because of offline loading
  destination_file = destination_path + filename
  meta_pull
  if File.file?(destination_file)
    # Verify hash to see if it is the latest
    computed_h = Digest::SHA256.file(destination_file)
    skip_download = true if meta.sha256.casecmp(computed_h.hexdigest).zero?
  end
  if skip_download
    @json_file = destination_file
  else
    zip_path = download_zip(opts)
    Archive::Zip.open(zip_path) do |z|
      z.extract(destination_path, flatten: true)
    end
    @json_file = zip_path.chomp('.zip')
    # Verify hash integrity
    computed_h = Digest::SHA256.file(@json_file)
    raise "File corruption: #{@json_file}" unless meta.sha256.casecmp(computed_h.hexdigest).zero?
  end
  return @json_file
end

#meta_pull ⇒ Meta

Create or update the Meta object (fill the attribute).

Returns:

  • (Meta) —

    the updated Meta object of the feed.

See Also:



108
109
110
111
112
113
 
# File 'lib/nvd_feed_api.rb', line 108

def meta_pull
  meta_content = NVDFeedScraper::Meta.new(@meta_url)
  meta_content.parse
  # update @meta
  @meta = meta_content
end