Class: Spider::Model::Storage::BaseStorage Abstract

Inherits:
Object
  • Object
show all
Includes:
Logger
Defined in:
lib/spiderfw/model/storage/base_storage.rb

Overview

This class is abstract.

This class is subclassed by classes that interact with different storage backends. See also Db::DbStorage, Document::DocumentStorage.

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Logger

add, check_request_level, close, close_all, datetime_format, datetime_format=, #debug, debug, debug?, #debug?, enquire_loggers, #error, error, #error?, error?, fatal, #fatal, fatal?, #fatal?, info, #info, info?, #info?, #log, log, method_missing, open, reopen, request_level, send_to_loggers, set_request_level, unknown, #unknown, warn, #warn, warn?, #warn?

Constructor Details

#initialize(url) ⇒ BaseStorage

Creates a new storage instance.

Parameters:

  • url (String) —

    The backend-specific url for the connection



116
117
118
119
120
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 116

def initialize(url)
    @url = url
    @configuration = {}
    parse_url(url)
end

Class Attribute Details

.capabilities ⇒ Hash (readonly)

An Hash of storage capabilities. The default for db storages is => false, :sequences => true, :transactions => true (The BaseStorage class provides file sequences in case the subclass does not support them.)

Returns:

  • (Hash) 


22
23
24
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 22

def capabilities
  @capabilities
end

Instance Attribute Details

#instance_name ⇒ Object

Returns the value of attribute instance_name.



11
12
13
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 11

def instance_name
  @instance_name
end

#url ⇒ Object (readonly)

Returns the value of attribute url.



10
11
12
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 10

def url
  @url
end

Class Method Details

.base_types ⇒ Array

Returns Base types supported by the backend.

Returns:

  • (Array) —

    Base types supported by the backend.



35
36
37
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 35

def base_types
    Model.base_types
end

.connection_alive?(conn) ⇒ void

This method is abstract.

This method returns an undefined value.

Checks whether a connection is still alive. Must be implemented by subclasses.

Parameters:

  • conn (Object) —

    The native connection



101
102
103
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 101

def connection_alive?(conn)
    raise "Virtual"
end

.connection_attributes ⇒ Hash

Returns current connection attributes.

Returns:

  • (Hash) —

    current connection attributes



200
201
202
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 200

def self.connection_attributes
    @connection_attributes ||= {}
end

.connection_pools ⇒ Hash

Returns An Hash of connection pools for each backend.

Returns:

  • (Hash) —

    An Hash of connection pools for each backend.



57
58
59
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 57

def connection_pools
    @pools ||= {}
end

.disconnect(conn) ⇒ void

This method is abstract.

This method returns an undefined value.

Closes the native connection to the backend.

Parameters:

  • conn (Object) —

    The native connection



93
94
95
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 93

def disconnect(conn)
    raise "Virtual"
end

.get_connection(*args) ⇒ Object

Returns Retrieves a native connection to the backend from the ConnectionPool.

Parameters:

  • Storage (*args) —

    specific arguments

Returns:



63
64
65
66
67
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 63

def get_connection(*args)
    @pools ||= {}
    @pools[args] ||= ConnectionPool.new(args, self)
    @pools[args].get_connection
end

.inherited(subclass) ⇒ void

This method returns an undefined value.

Copies capabilities on subclasses

Parameters:



108
109
110
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 108

def inherited(subclass)
    subclass.instance_variable_set("@capabilities", @capabilities)
end

.max_connections ⇒ Fixnum|nil

This method is abstract.

Returns Maximum number of connections possible for this backend (or nil if unlimited).

Returns:

  • (Fixnum|nil) —

    Maximum number of connections possible for this backend (or nil if unlimited)



52
53
54
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 52

def max_connections
    nil
end

.new_connection(*args) ⇒ Object

This method is abstract.

Returns a new connection. Must be implemented by the subclasses; args are implementation specific.

Returns:

  • (Object) —

    Returns a new connection. Must be implemented by the subclasses; args are implementation specific.



46
47
48
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 46

def new_connection(*args)
    raise "Unimplemented"
end

.release_connection(conn, conn_params) ⇒ void

This method returns an undefined value.

Frees a connection, relasing it to the pool

Parameters:

  • conn (Object) —

    The connection

  • conn_params (Array) —

    An array of connection params that were used to create the connection.



73
74
75
76
77
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 73

def release_connection(conn, conn_params)
    return unless conn
    return unless @pools && @pools[conn_params]
    @pools[conn_params].release(conn)
end

.remove_connection(conn, conn_params) ⇒ void

This method returns an undefined value.

Removes a connection from the pool.

Parameters:

  • conn (Object) —

    The connection

  • conn_params (Array) —

    An array of connection params that were used to create the connection.



83
84
85
86
87
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 83

def remove_connection(conn, conn_params)
    return unless conn
    return unless @pools && @pools[conn_params]
    @pools[conn_params].remove(conn)
end

.sequence_sync ⇒ Sync

Returns A Sync object to use for sequences.

Returns:

  • (Sync) —

    A Sync object to use for sequences



30
31
32
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 30

def sequence_sync
    @sequence_sync ||= ::Sync.new
end

.storage_type ⇒ Symbol

Returns A label for the storage’s class.

Returns:

  • (Symbol) —

    A label for the storage’s class.



25
26
27
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 25

def storage_type
    :none
end

.supports?(capability) ⇒ bool

Returns True if given named capability is supported by the backend.

Returns:

  • (bool) —

    True if given named capability is supported by the backend.



40
41
42
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 40

def supports?(capability)
    @capabilities[capability]
end

Instance Method Details

#==(storage) ⇒ bool

Returns True if the other storage is of the same class, and has the same connection url.

Returns:

  • (bool) —

    True if the other storage is of the same class, and has the same connection url



257
258
259
260
261
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 257

def ==(storage)
    return false unless self.class == storage.class
    return false unless self.url == storage.url
    return true
end

#commit ⇒ bool

Commits the current transaction

Returns:

  • (bool) —

    True if the transaction was successfully committed, false if transactions are not enabled (Raises a StorageException if transactions are supported but were not started)

Raises:



312
313
314
315
316
317
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 312

def commit
    return false unless transactions_enabled?
    raise StorageException, "Commit without a transaction" unless in_transaction?
    return curr[:savepoints].pop unless curr[:savepoints].empty?
    commit!
end

#commit! ⇒ void

This method returns an undefined value.

Commits current transaction, resets transaction nesting, and releases the connection.



336
337
338
339
340
341
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 336

def commit!
    Spider.logger.debug("#{self.class.name} commit connection #{curr[:conn].object_id}")
    curr[:transaction_nesting] = 0
    do_commit
    release
end

#commit_or_continue ⇒ bool

Commits the current transaction, or decreases transaction nesting.

Returns:

  • (bool) —

    True if the transaction was successfully committed, false if transactions are not enabled (Raises a StorageException if transactions are supported but were not started)

Raises:



322
323
324
325
326
327
328
329
330
331
332
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 322

def commit_or_continue
    return false unless transactions_enabled?
    raise StorageException, "Commit without a transaction" unless in_transaction?
    if curr[:transaction_nesting] == 1
        commit
        curr[:transaction_nesting] = 0
        return true
    else
        curr[:transaction_nesting] -= 1
    end
end

#configure(conf) ⇒ void

This method returns an undefined value.

Sets configuration for the Storage

Parameters:

  • conf (Hash) —

    The configuration



125
126
127
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 125

def configure(conf)
    @configuration.merge!(conf.to_hash)
end

#connect ⇒ void

This method returns an undefined value.

Instantiates a new connection with current connection params.



173
174
175
176
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 173

def connect
    return self.class.get_connection(*@connection_params)
    #Spider::Logger.debug("#{self.class.name} in thread #{Thread.current} acquired connection #{@conn}")
end

#connected? ⇒ bool

Returns True if currently connected.

Returns:

  • (bool) —

    True if currently connected.



179
180
181
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 179

def connected?
    curr[:conn] != nil
end

#connection {|Object| ... } ⇒ Object

Returns the current connection, or creates a new one. If a block is given, will release the connection after yielding.

Yields:

  • (Object) —

    If a block is given, it is passed the connection, which is released after the block ends.

Returns:



188
189
190
191
192
193
194
195
196
197
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 188

def connection
    curr[:conn] = connect
    if block_given?
        yield curr[:conn]
        release # unless is_connected
        return true
    else
        return curr[:conn]
    end
end

#connection_attributes ⇒ Hash

Returns current connection attributes.

Returns:

  • (Hash) —

    current connection attributes



205
206
207
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 205

def connection_attributes
    self.class.connection_attributes[connection] ||= {}
end

#connection_pool ⇒ ConnectionPool|nil

Returns The ConnectionPool managing the current connection params.

Returns:

  • (ConnectionPool|nil) —

    The ConnectionPool managing the current connection params



167
168
169
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 167

def connection_pool
    self.class.connection_pools[@connection_params]
end

#create_sequence(name, start = 1, increment = 1) ⇒ void

This method returns an undefined value.

Creates a new sequence

Parameters:

  • name (String) —

    Sequence name

  • start (Fixnum) (defaults to: 1)
  • increment (Fixnum) (defaults to: 1) 


422
423
424
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 422

def create_sequence(name, start=1, increment=1)
    sequence_next(name, start-1, increment)
end

#curr ⇒ Hash

Returns An hash of thread-local values for this connection.

Returns:

  • (Hash) —

    An hash of thread-local values for this connection



151
152
153
154
155
156
157
158
159
160
161
162
163
164
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 151

def curr
    var = nil
    if Spider.conf.get('storage.shared_connection')
        $STORAGES ||= {}
        var = $STORAGES
    else
        var = Thread.current
    end
    var[:storages] ||= {}
    var[:storages][self.class.storage_type] ||= {}
    var[:storages][self.class.storage_type][@connection_params] ||= {
        :transaction_nesting => 0, :savepoints => []
    }
end

#do_commit ⇒ void

This method is abstract.

This method returns an undefined value.

Implemented by subclasses to interact with the backend

Raises:



346
347
348
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 346

def do_commit
    raise StorageException, "The current storage does not support transactions" 
end

#do_rollback ⇒ void

This method is abstract.

This method returns an undefined value.

Implemented by subclasses to interact with the backend

Raises:



379
380
381
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 379

def do_rollback
    raise StorageException, "The current storage does not support transactions" 
end

#do_start_transaction ⇒ Object

This method is abstract.

Implemented by subclasses to interact with the backend

Raises:



287
288
289
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 287

def do_start_transaction
   raise StorageException, "The current storage does not support transactions" 
end

#generate_uuid ⇒ String

Returns A new UUID.

Returns:



427
428
429
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 427

def generate_uuid
    Spider::DataTypes::UUID.generate
end

#get_mapper(model) ⇒ Mapper

Returns the instance of a mapper for the storage and the given model

Parameters:

Returns:

  • (Mapper) —

    Returns the instance of a mapper for the storage and the given model

Raises:



140
141
142
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 140

def get_mapper(model)
    raise StorageException, "Unimplemented"
end

#in_transaction ⇒ bool

Starts a transaction, or increases transaction nesting.

Returns:

  • (bool) —

    True if a transaction was already active, false otherwise



293
294
295
296
297
298
299
300
301
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 293

def in_transaction
    if in_transaction?
        curr[:transaction_nesting] += 1
        return true
    else
        start_transaction
        return false
    end
end

#in_transaction? ⇒ bool

Returns True if a transaction is currently active.

Returns:

  • (bool) —

    True if a transaction is currently active



304
305
306
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 304

def in_transaction?
    return false
end

#parse_url(url) ⇒ Array

This method is abstract.

Splits a backend-specific connection url into parts

Parameters:

Returns:

  • (Array)

Raises:



133
134
135
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 133

def parse_url(url)
    raise StorageException, "Unimplemented"
end

#prepare_value(type, value) ⇒ Object

Prepares a value that will be used by the backend (see also #value_for_save and #value_for_condition, which by default call this method, but can be override to do more specific processiong).

Parameters:

Returns:

  • (Object) —

    The prepared value



252
253
254
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 252

def prepare_value(type, value)
    return value
end

#release ⇒ void

This method returns an undefined value.

Releases the current connection to the pool.



211
212
213
214
215
216
217
218
219
220
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 211

def release
    # The subclass should check if the connection is alive, and if it is not call remove_connection instead
    c = curr[:conn]
    #Spider.logger.debug("#{self} in thread #{Thread.current} releasing #{curr[:conn]}")
    curr[:conn] = nil
    self.class.release_connection(c, @connection_params)
    #Spider.logger.debug("#{self} in thread #{Thread.current} released #{curr[:conn]}")
    return nil
    #@conn = nil
end

#rollback ⇒ void

This method returns an undefined value.

Rolls back the current transaction. Raises an error if in a nested transaction.



352
353
354
355
356
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 352

def rollback
    raise "Can't rollback in a nested transaction" if curr[:transaction_nesting] > 1
    return rollback_savepoint(curr[:savepoints].last) unless curr[:savepoints].empty?
    rollback!
end

#rollback! ⇒ void

This method returns an undefined value.

Rolls back the current transaction, regardless of transaction nesting, and releases the connection



360
361
362
363
364
365
366
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 360

def rollback!
    curr[:transaction_nesting] = 0
    Spider.logger.debug("#{self.class.name} rollback")
    do_rollback
    curr[:savepoints] = []
    release
end

#rollback_or_continue ⇒ Object 



368
369
370
371
372
373
374
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 368

def rollback_or_continue
    if curr[:transaction_nesting] == 1
        rollback
    else
        curr[:transaction_nesting] -= 1 if curr[:transaction_nesting] > 1
    end
end

#rollback_savepoint(name = nil) ⇒ void

This method returns an undefined value.

Rolls back a savepoint

Parameters:

  • name (String) (defaults to: nil) 


393
394
395
396
397
398
399
400
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 393

def rollback_savepoint(name=nil)
    if name
        curr[:savepoints] = curr[:savepoints][0,(curr[:savepoints].index(name))]
        name
    else
        curr[:savepoints].pop
    end
end

#savepoint(name) ⇒ void

This method returns an undefined value.

Creates a new savepoint

Parameters:



386
387
388
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 386

def savepoint(name)
    curr[:savepoints] << name
end

#sequence_exists?(name) ⇒ bool

Returns True if the sequence file exists.

Parameters:

  • name (String) —

    Sequence name

Returns:

  • (bool) —

    True if the sequence file exists



413
414
415
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 413

def sequence_exists?(name)
    File.exist?(sequence_file_path(name))
end

#sequence_file_path(name) ⇒ String

Returns Path to the sequence file.

Parameters:

  • name (String) —

    Sequence name

Returns:

  • (String) —

    Path to the sequence file



406
407
408
409
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 406

def sequence_file_path(name)
    path = File.join(Spider.paths[:var], 'sequences', name)
    return path
end

#sequence_next(name, newval = nil, increment = 1) ⇒ Fixnum

Increments a named sequence and returns the new value

Parameters:

  • name (String) —

    Sequence name

  • newval (Fixnum) (defaults to: nil) —

    New value for the sequence

  • increment (Fixnum) (defaults to: 1)

Returns:

  • (Fixnum) —

    New value for the sequence



445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 445

def sequence_next(name, newval=nil, increment=1)
    path = sequence_file_path(name)
    FileUtils.mkpath(File.dirname(path))
    self.class.sequence_sync.lock(::Sync::EX)
    if newval
        seq = newval
    else
        seq = 0
        File.open(path, 'a+') do |f|
            f.rewind
            f.flock File::LOCK_EX
            cur = f.gets
            if (cur)
                seq, increment_str = cur.split('|')
            else
                seq, increment_str = 0, 1
            end
            seq = seq.to_i
            increment = increment_str.to_i if increment_str
            f.close
        end
        seq += increment
    end
    File.open(path, 'w+') do |f|
        f.print(seq)
        f.print("|#{increment}") if (increment != 1)
        f.flock File::LOCK_UN
        f.close
    end
    self.class.sequence_sync.lock(::Sync::UN)
    return seq
end

#start_transaction ⇒ bool

Starts a new transaction on the backend

Returns:

  • (bool) —

    True if a new transaction was started, false otherwise



275
276
277
278
279
280
281
282
283
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 275

def start_transaction
    return unless transactions_enabled?
    curr[:transaction_nesting] += 1
    return savepoint("point#{curr[:savepoints].length}") if in_transaction?

    Spider.logger.debug("#{self.class.name} starting transaction for connection #{connection.object_id}")
    do_start_transaction
    return true
end

#supports?(capability) ⇒ bool

Returns True if the backend supports the given capability.

Parameters:

Returns:

  • (bool) —

    True if the backend supports the given capability



146
147
148
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 146

def supports?(capability)
    self.class.supports?(capability)
end

#supports_transactions? ⇒ bool

Returns True if the backend support stransaction.

Returns:

  • (bool) —

    True if the backend support stransaction



264
265
266
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 264

def supports_transactions?
    return self.class.supports?(:transactions)
end

#transactions_enabled? ⇒ bool

Returns True if transactions are supported by the backend and enabled in the storage’s configuration.

Returns:

  • (bool) —

    True if transactions are supported by the backend and enabled in the storage’s configuration.



269
270
271
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 269

def transactions_enabled?
    @configuration['enable_transactions'] && supports_transactions?
end

#update_sequence(name, val) ⇒ Fixnum

Updates a sequence

Parameters:

  • name (String) —

    Sequence name

  • val (Fixnum) —

    New value for the sequence

Returns:

  • (Fixnum) —

    New value for the sequence



435
436
437
438
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 435

def update_sequence(name, val)
    # not an alias because the set value behaviour of next_sequence isn't expected in subclasses
    sequence_next(name, val)
end

#value_for_condition(type, value) ⇒ Object

Prepares a value that will be used in a condition on the backend.

Parameters:

Returns:

  • (Object) —

    The prepared value



235
236
237
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 235

def value_for_condition(type, value)
    return prepare_value(type, value)
end

#value_for_save(type, value, save_mode) ⇒ Object

Prepares a value which will be saved into the backend.

Parameters:

  • type (Class)
  • value (Object)
  • save_mode (Symbol) —

    :insert or :update or generic :save

Returns:

  • (Object) —

    The prepared value



227
228
229
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 227

def value_for_save(type, value, save_mode)
    return prepare_value(type, value)
end

#value_to_mapper(type, value) ⇒ Object

Prepares a value coming from the backend for the mapper

Parameters:

Returns:

  • (Object) —

    The prepared value



243
244
245
 
# File 'lib/spiderfw/model/storage/base_storage.rb', line 243

def value_to_mapper(type, value)
    value
end