Class: Arachni::Support::Database::Queue
- Defined in:
- lib/arachni/support/database/queue.rb
Overview
Flat-file Queue implementation
Behaves pretty much like a Ruby Queue however it transparently serializes and saves its entries to the file-system under the OS’s temp directory after a specified #max_buffer_size (for in-memory entries) has been exceeded.
It’s pretty useful when you want to reduce memory footprint without having to refactor any code since it behaves just like a Ruby Queue implementation.
Constant Summary collapse
- DEFAULT_MAX_BUFFER_SIZE =
Default #max_buffer_size.
100
Instance Attribute Summary collapse
-
#buffer ⇒ Array<Object>
readonly
Objects stored in the memory buffer.
-
#disk ⇒ Array<String>
readonly
Paths to files stored to disk.
-
#max_buffer_size ⇒ Integer
How many entries to keep in memory before starting to off-load to disk.
Instance Method Summary collapse
- #<<(obj) ⇒ Object (also: #push, #enq)
- #buffer_size ⇒ Object
-
#clear ⇒ Object
Removes all objects from the queue.
- #disk_size ⇒ Object
-
#empty? ⇒ Bool
‘true` if the queue if empty, `false` otherwise.
-
#initialize(*args) ⇒ Queue
constructor
A new instance of Queue.
- #num_waiting ⇒ Object
-
#pop(non_block = false) ⇒ Object
(also: #deq, #shift)
Removes an object from the queue and returns it.
-
#size ⇒ Integer
(also: #length)
Size of the queue, the number of objects it currently holds.
Constructor Details
#initialize(*args) ⇒ Queue
Returns a new instance of Queue.
41 42 43 44 45 46 47 |
# File 'lib/arachni/support/database/queue.rb', line 41 def initialize( *args ) super( *args ) @disk = [] @buffer = [] @waiting = [] @mutex = Mutex.new end |
Instance Attribute Details
#buffer ⇒ Array<Object> (readonly)
Returns Objects stored in the memory buffer.
34 35 36 |
# File 'lib/arachni/support/database/queue.rb', line 34 def buffer @buffer end |
#disk ⇒ Array<String> (readonly)
Returns Paths to files stored to disk.
38 39 40 |
# File 'lib/arachni/support/database/queue.rb', line 38 def disk @disk end |
#max_buffer_size ⇒ Integer
Defaults to DEFAULT_MAX_BUFFER_SIZE.
Returns How many entries to keep in memory before starting to off-load to disk.
30 31 32 |
# File 'lib/arachni/support/database/queue.rb', line 30 def max_buffer_size @max_buffer_size end |
Instance Method Details
#<<(obj) ⇒ Object Also known as: push, enq
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
# File 'lib/arachni/support/database/queue.rb', line 59 def <<( obj ) synchronize do if @buffer.size < max_buffer_size @buffer << obj else @disk << dump( obj ) end begin t = @waiting.shift t.wakeup if t rescue ThreadError retry end end end |
#buffer_size ⇒ Object
103 104 105 |
# File 'lib/arachni/support/database/queue.rb', line 103 def buffer_size @buffer.size end |
#clear ⇒ Object
Removes all objects from the queue.
118 119 120 121 122 123 124 125 126 |
# File 'lib/arachni/support/database/queue.rb', line 118 def clear @buffer.clear while !@disk.empty? path = @disk.pop next if !path delete_file path end end |
#disk_size ⇒ Object
107 108 109 |
# File 'lib/arachni/support/database/queue.rb', line 107 def disk_size @disk.size end |
#empty? ⇒ Bool
Returns ‘true` if the queue if empty, `false` otherwise.
113 114 115 |
# File 'lib/arachni/support/database/queue.rb', line 113 def empty? @buffer.empty? && @disk.empty? end |
#num_waiting ⇒ Object
128 129 130 |
# File 'lib/arachni/support/database/queue.rb', line 128 def num_waiting @waiting.size end |
#pop(non_block = false) ⇒ Object Also known as: deq, shift
Returns Removes an object from the queue and returns it.
80 81 82 83 84 85 86 87 88 89 90 91 92 |
# File 'lib/arachni/support/database/queue.rb', line 80 def pop( non_block = false ) synchronize do loop do if empty? raise ThreadError, 'queue empty' if non_block @waiting.push Thread.current @mutex.sleep else return @buffer.shift || load_and_delete_file( @disk.shift ) end end end end |
#size ⇒ Integer Also known as: length
Returns Size of the queue, the number of objects it currently holds.
98 99 100 |
# File 'lib/arachni/support/database/queue.rb', line 98 def size buffer_size + disk_size end |