Class: Curl::ThreadPool

Inherits:
Object
  • Object
show all
Defined in:
lib/curb_threadpool.rb

Overview

A multi-threaded worker pool for Curb

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(size = 4) ⇒ ThreadPool

Returns a new instance of ThreadPool.



7
8
9
10
 
# File 'lib/curb_threadpool.rb', line 7

def initialize(size=4)
  @size = size
  reset()
end

Instance Attribute Details

#results ⇒ Object (readonly)

Returns the value of attribute results.



5
6
7
 
# File 'lib/curb_threadpool.rb', line 5

def results
  @results
end

Instance Method Details

#[]=(key, url) ⇒ Object Also known as: add

Add a URL to be fetched

Parameters:

  • key (Object) —

    to use for request

  • url (String) —

    URL to fetch



16
17
18
 
# File 'lib/curb_threadpool.rb', line 16

def []=(key, url)
  @reqs[key] = url
end

#close ⇒ Object

Close all active Curl connections



28
29
30
 
# File 'lib/curb_threadpool.rb', line 28

def close
  @clients.each { |c| c.reset(); c.close() } if @clients
end

#get(urls) ⇒ Array

Utility method for retrieving a list of URLs

Parameters:

  • urls (Array<String>) —

    list of URLs

Returns:

  • (Array) —

    array of response bodies



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/curb_threadpool.rb', line 46

def get(urls)
  if urls.nil? or urls.empty? then
    return {}
  end

  urls = [urls] if not urls.kind_of? Array
  urls.each_with_index do |url, i|
    @reqs[i] = url.to_s
  end

  results = exec()

  ret = []
  (0..results.size-1).each do |i|
    ret << results[i]
  end

  return ret
end

#join ⇒ Object

Wait for all threads to complete



22
23
24
25
 
# File 'lib/curb_threadpool.rb', line 22

def join
  @threads.each { |t| t.join }
  @threads.clear
end

#perform(no_block = false, &block) ⇒ Hash<Key, String>

Execute requests. By default, will block until complete and return results.

Parameters:

  • no_block (Boolean) (defaults to: false) —

    If true, will not wait for requests to finish. (Default=false)

  • block (Block) —

    If passed, responses will be passed into the callback instead of being returned directly

Returns:

  • (Hash<Key, String>) —

    Hash of responses, if no block given



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
# File 'lib/curb_threadpool.rb', line 75

def perform(no_block=false, &block)

  @results = {} if not @results.empty?

  @clients.each do |client|
    @threads << Thread.new do

      loop do
        break if @reqs.empty?
        (key, url) = @reqs.shift
        client.url = url
        client.http_get
        if block then
          yield(key, client.body_str)
        else
          @results[key] = client.body_str
        end
      end

    end
  end

  return {} if no_block

  join()
  return true if block

  ret = @results
  @results = {}
  return ret
end

#reset ⇒ Object

Reset the ThreadPool



33
34
35
36
37
38
39
40
 
# File 'lib/curb_threadpool.rb', line 33

def reset
  close()
  @reqs = {}
  @results = {}
  @clients = []
  @threads = []
  @size.times{ @clients << Curl::Easy.new }
end