Module: DaimonSkycrawlers::Crawler

Defined in:
lib/daimon_skycrawlers/crawler.rb,
lib/daimon_skycrawlers/crawler/base.rb,
lib/daimon_skycrawlers/crawler/default.rb

Overview

Name space for crawler related classes

Defined Under Namespace

Classes: Base, Default

Class Method Summary collapse

Class Method Details

.config ⇒ DaimonSkycrawlers::Configuration

Shortcut of DaimonSkycrawlers.configuration

Returns:



41
42
43
 
# File 'lib/daimon_skycrawlers/crawler.rb', line 41

def config
  DaimonSkycrawlers.configuration
end

.default_process_name ⇒ String

Default process name

Returns:

  • (String) —

    Default process name



59
60
61
 
# File 'lib/daimon_skycrawlers/crawler.rb', line 59

def default_process_name
  "#{config.queue_name_prefix}:url"
end

.enqueue_url(url, message = {}) ⇒ void

This method returns an undefined value.

Enqueue a URL to crawler queue

Parameters:

  • url (String) —

    Specify absolute URL

  • message (Hash) (defaults to: {}) —

    Extra parameters for crawler



30
31
32
33
34
 
# File 'lib/daimon_skycrawlers/crawler.rb', line 30

def enqueue_url(url, message = {})
  message[:url] = url
  config.logger.debug("#{queue_name}: #{url}")
  SongkickQueue.publish(queue_name, message)
end

.queue_name ⇒ String

Queue name for crawler

Returns:

  • (String) —

    Queue name



50
51
52
 
# File 'lib/daimon_skycrawlers/crawler.rb', line 50

def queue_name
  "#{config.queue_name_prefix}.url"
end

.run(process_name: default_process_name) ⇒ Object

Run registered crawlers

Parameters:

  • process_name (String) (defaults to: default_process_name) —

    Process name



17
18
19
20
21
22
 
# File 'lib/daimon_skycrawlers/crawler.rb', line 17

def run(process_name: default_process_name)
  if config.shutdown_interval > 0
    DaimonSkycrawlers::Timer.setup_shutdown_timer(config.queue_name_prefix, interval: config.shutdown_interval)
  end
  SongkickQueue::Worker.new(process_name, [DaimonSkycrawlers::Consumer::URL]).run
end