Class: DaimonSkycrawlers::Crawler::Default

Inherits:
Base
  • Object
show all
Defined in:
lib/daimon_skycrawlers/crawler/default.rb

Overview

The default crawler

This crawler can GET given URL and store response to storage

Instance Attribute Summary

Attributes inherited from Base

#n_processed_urls, #storage

Instance Method Summary collapse

Methods inherited from Base

#connection, #get, #initialize, #post, #prepare, #process, #setup_connection, #skipped?

Methods included from DaimonSkycrawlers::ConfigMixin

#initialize

Methods included from LoggerMixin

included, #initialize

Constructor Details

This class inherits a constructor from DaimonSkycrawlers::Crawler::Base

Instance Method Details

#fetch(url, message) {|data| ... } ⇒ Object

Yields:

  • (data) 


11
12
13
14
15
16
17
18
19
 
# File 'lib/daimon_skycrawlers/crawler/default.rb', line 11

def fetch(url, message)
  response = get(url)
  data = [url.to_s, response.headers, response.body]

  yield(*data) if block_given?

  storage.save(*data)
  schedule_to_process(url.to_s, message)
end