Class: DaimonSkycrawlers::Consumer::URL

Inherits:
Base
  • Object
show all
Includes:
SongkickQueue::Consumer
Defined in:
lib/daimon_skycrawlers/consumer/url.rb

Overview

URL consumer class

Class Method Summary collapse

Instance Method Summary collapse

Methods included from DaimonSkycrawlers::ConfigMixin

#initialize

Methods included from LoggerMixin

included, #initialize

Class Method Details

.crawlers ⇒ Array<Crawler>

Returns registered crawlers

Returns:



29
30
31
 
# File 'lib/daimon_skycrawlers/consumer/url.rb', line 29

def crawlers
  @crawlers ||= []
end

.queue_name ⇒ Object 



36
37
38
 
# File 'lib/daimon_skycrawlers/consumer/url.rb', line 36

def queue_name
  "#{DaimonSkycrawlers.configuration.queue_name_prefix}.url"
end

.register(crawler) ⇒ void

This method returns an undefined value.

Register a given crawler

Parameters:

  • crawler (Crawler) —

    instance which implements ‘fetch` method



20
21
22
 
# File 'lib/daimon_skycrawlers/consumer/url.rb', line 20

def register(crawler)
  crawlers << crawler
end

Instance Method Details

#process(message) ⇒ Object 



46
47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/daimon_skycrawlers/consumer/url.rb', line 46

def process(message)
  crawler_interval = DaimonSkycrawlers.configuration.crawler_interval

  # XXX When several crawlers are registered, how should they behave?
  self.class.crawlers.each do |crawler|
    crawler.process(message)
    if crawler.skipped?
      sleep(crawler_interval) if crawler.n_processed_urls % 50 == 0
    else
      sleep(crawler_interval)
    end
  end
end