Class: Blat::ConsumingQueue

Inherits:
Queue
  • Object
show all
Defined in:
lib/blat/queue.rb

Overview

Similar to a queue, except that it explicitly calls a block in order to acquire new URLs.

This makes it suitable for use in producer/consumer patterns.

Direct Known Subclasses

ListConsumingQueue

Instance Attribute Summary

Attributes inherited from Queue

#max_connections, #pipeline

Instance Method Summary collapse

Methods inherited from Queue

#active?, #add, #cancel, #idle?, #initialize, #perform, #perform_nonblock, #remove, #request_count, #requests

Constructor Details

This class inherits a constructor from Blat::Queue

Instance Method Details

#consume(connections = @max_connects, &block) ⇒ Object

Executes the given block in order to keep the curl pool working at its maximum capacity.

consume blocks as long as links are being downloaded, as it relies on Curl::Multi#perform

Note that blocks providing links must also perform their own configuration, e.g.:

q.consume do
  url = get_url
  if(url)
    c = Curl::Easy.new(url)
    c.follow_location = true
    c.on_complete{ |c| puts "Retrieved: #{c.body_str}" }
    c
  else
    nil
  end
end



153
154
155
156
157
158
159
 
# File 'lib/blat/queue.rb', line 153

def consume(connections = @max_connects, &block)
  perform do
    while request_count < connections && new_link = yield
      add(new_link) if new_link
    end
  end
end