Module: DaimonSkycrawlers::Processor

Defined in:
lib/daimon_skycrawlers/processor.rb,
lib/daimon_skycrawlers/processor/base.rb,
lib/daimon_skycrawlers/processor/proc.rb,
lib/daimon_skycrawlers/processor/spider.rb,
lib/daimon_skycrawlers/processor/default.rb

Defined Under Namespace

Classes: Base, Default, Proc, Spider

Class Method Summary collapse

Class Method Details

.config ⇒ DaimonSkycrawlers::Configuration

Shortcut of DaimonSkycrawlers.configuration

Returns:



38
39
40
 
# File 'lib/daimon_skycrawlers/processor.rb', line 38

def config
  DaimonSkycrawlers.configuration
end

.default_process_name ⇒ String

Default process name

Returns:

  • (String) —

    Default process name



56
57
58
 
# File 'lib/daimon_skycrawlers/processor.rb', line 56

def default_process_name
  "#{config.queue_name_prefix}:http-response"
end

.enqueue_http_response(url, message = {}) ⇒ void

This method returns an undefined value.

Enqueue a URL to processor queue

Parameters:

  • Specify (String) —

    absolute URL

  • Extra (Hash) —

    parameters for crawler



27
28
29
30
31
 
# File 'lib/daimon_skycrawlers/processor.rb', line 27

def enqueue_http_response(url, message = {})
  message[:url] = url
  config.logger.debug("#{queue_name}: #{url}")
  SongkickQueue.publish(queue_name, message)
end

.queue_name ⇒ String

Queue name for processor

Returns:

  • (String) —

    Queue name



47
48
49
 
# File 'lib/daimon_skycrawlers/processor.rb', line 47

def queue_name
  "#{config.queue_name_prefix}.http-response"
end

.run(process_name: default_process_name) ⇒ Object

Run registered processors

Parameters:

  • process_name (String) (defaults to: default_process_name) —

    Process name



14
15
16
17
18
19
 
# File 'lib/daimon_skycrawlers/processor.rb', line 14

def run(process_name: default_process_name)
  if config.shutdown_interval > 0
    DaimonSkycrawlers::Timer.setup_shutdown_timer(config.queue_name_prefix, interval: config.shutdown_interval)
  end
  SongkickQueue::Worker.new(process_name, [DaimonSkycrawlers::Consumer::HTTPResponse]).run
end