Class: Arachni::Support::Database::Queue

Inherits:
Base show all
Defined in:
lib/arachni/support/database/queue.rb

Overview

Flat-file Queue implementation

Behaves pretty much like a Ruby Queue however it transparently serializes and saves its entries to the file-system under the OS’s temp directory after a specified #max_buffer_size (for in-memory entries) has been exceeded.

It’s pretty useful when you want to reduce memory footprint without having to refactor any code since it behaves just like a Ruby Queue implementation.

Author:

Constant Summary collapse

DEFAULT_MAX_BUFFER_SIZE =

Default #max_buffer_size.

100

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(*args) ⇒ Queue

Returns a new instance of Queue.

See Also:

  • Database::Base#initialize


41
42
43
44
45
46
47
 
# File 'lib/arachni/support/database/queue.rb', line 41

def initialize( *args )
    super( *args )
    @disk    = []
    @buffer  = []
    @waiting = []
    @mutex   = Mutex.new
end

Instance Attribute Details

#buffer ⇒ Array<Object> (readonly)

Returns Objects stored in the memory buffer.

Returns:

  • (Array<Object>) —

    Objects stored in the memory buffer.



34
35
36
 
# File 'lib/arachni/support/database/queue.rb', line 34

def buffer
  @buffer
end

#disk ⇒ Array<String> (readonly)

Returns Paths to files stored to disk.

Returns:



38
39
40
 
# File 'lib/arachni/support/database/queue.rb', line 38

def disk
  @disk
end

#max_buffer_size ⇒ Integer

Note:

Defaults to DEFAULT_MAX_BUFFER_SIZE.

Returns How many entries to keep in memory before starting to off-load to disk.

Returns:

  • (Integer) —

    How many entries to keep in memory before starting to off-load to disk.



30
31
32
 
# File 'lib/arachni/support/database/queue.rb', line 30

def max_buffer_size
  @max_buffer_size
end

Instance Method Details

#<<(obj) ⇒ Object Also known as: push, enq

Parameters:

  • obj (Object) —

    Object to add to the queue.



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/arachni/support/database/queue.rb', line 59

def <<( obj )
    synchronize do
        if @buffer.size < max_buffer_size
            @buffer << obj
        else
            @disk << dump( obj )
        end

        begin
            t = @waiting.shift
            t.wakeup if t
        rescue ThreadError
            retry
        end
    end
end

#buffer_size ⇒ Object 



103
104
105
 
# File 'lib/arachni/support/database/queue.rb', line 103

def buffer_size
    @buffer.size
end

#clear ⇒ Object

Removes all objects from the queue.



118
119
120
121
122
123
124
125
126
 
# File 'lib/arachni/support/database/queue.rb', line 118

def clear
    @buffer.clear

    while !@disk.empty?
        path = @disk.pop
        next if !path
        delete_file path
    end
end

#disk_size ⇒ Object 



107
108
109
 
# File 'lib/arachni/support/database/queue.rb', line 107

def disk_size
    @disk.size
end

#empty? ⇒ Bool

Returns ‘true` if the queue if empty, `false` otherwise.

Returns:

  • (Bool) —

    ‘true` if the queue if empty, `false` otherwise.



113
114
115
 
# File 'lib/arachni/support/database/queue.rb', line 113

def empty?
    @buffer.empty? && @disk.empty?
end

#num_waiting ⇒ Object 



128
129
130
 
# File 'lib/arachni/support/database/queue.rb', line 128

def num_waiting
    @waiting.size
end

#pop(non_block = false) ⇒ Object Also known as: deq, shift

Returns Removes an object from the queue and returns it.

Returns:

  • (Object) —

    Removes an object from the queue and returns it.



80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/arachni/support/database/queue.rb', line 80

def pop( non_block = false )
    synchronize do
        loop do
            if empty?
                raise ThreadError, 'queue empty' if non_block
                @waiting.push Thread.current
                @mutex.sleep
            else
                return @buffer.shift || load_and_delete_file( @disk.shift )
            end
        end
    end
end

#size ⇒ Integer Also known as: length

Returns Size of the queue, the number of objects it currently holds.

Returns:

  • (Integer) —

    Size of the queue, the number of objects it currently holds.



98
99
100
 
# File 'lib/arachni/support/database/queue.rb', line 98

def size
    buffer_size + disk_size
end