Class: Pupa::Processor::DocumentStore::RedisStore

Inherits:
Object
  • Object
show all
Defined in:
lib/pupa/processor/document_store/redis_store.rb

Overview

Note:

Redis support depends on the redis-store gem. You may optionally use the hiredis gem to further improve performance.

Stores JSON documents in Redis.

Pupa flushes the JSON document store before scraping. If you use Redis, DO NOT share a Redis database with Pupa and other applications. You can select a different database than the default 0 for use with Pupa by passing an argument like redis://localhost:6379/0.

Instance Method Summary collapse

Constructor Details

#initialize(address, pipelined: false) ⇒ RedisStore

Returns a new instance of RedisStore.

Parameters:

  • address (String) —

    the address (e.g. redis://localhost:6379/0) in which to dump JSON documents

  • pipelined (Boolean) (defaults to: false) —

    whether to enable pipelining



18
19
20
21
22
23
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 18

def initialize(address, pipelined: false)
  @pipelined = pipelined
  options = {marshalling: false}
  options.update(driver: :hiredis) if defined?(Hiredis)
  @redis = Redis::Store::Factory.create(address, options)
end

Instance Method Details

#clear ⇒ Object

Deletes all keys in the database.



93
94
95
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 93

def clear
  @redis.flushdb
end

#delete(name) ⇒ Object

Delete a key.

Parameters:



88
89
90
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 88

def delete(name)
  @redis.del(name)
end

#entries ⇒ Array<String>

Returns all keys in the database.

Returns:

  • (Array<String>) —

    all keys in the store



36
37
38
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 36

def entries
  @redis.keys('*')
end

#exist?(name) ⇒ Boolean

Returns whether database contains an entry for the given key.

Parameters:

Returns:

  • (Boolean) —

    whether the store contains an entry for the given key



29
30
31
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 29

def exist?(name)
  @redis.exists(name)
end

#pipelined ⇒ Object

Collects commands to run all at once.



98
99
100
101
102
103
104
105
106
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 98

def pipelined
  if @pipelined
    @redis.pipelined do
      yield
    end
  else
    yield
  end
end

#read(name) ⇒ Hash

Returns, as JSON, the value of the given key.

Parameters:

Returns:

  • (Hash) —

    the value of the given key



44
45
46
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 44

def read(name)
  Oj.load(@redis.get(name))
end

#read_multi(names) ⇒ Array<Hash>

Returns, as JSON, the values of the given keys.

Parameters:

Returns:

  • (Array<Hash>) —

    the values of the given keys



52
53
54
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 52

def read_multi(names)
  @redis.mget(*names).map{|value| Oj.load(value)}
end

#write(name, value) ⇒ Object

Writes, as JSON, the value to a key.

Parameters:

  • name (String) —

    a key

  • value (Hash) —

    a value



60
61
62
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 60

def write(name, value)
  @redis.set(name, Oj.dump(value, mode: :compat, time_format: :ruby))
end

#write_multi(pairs) ⇒ Object

Writes, as JSON, the values to keys.

Parameters:

  • pairs (Hash) —

    key-value pairs



76
77
78
79
80
81
82
83
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 76

def write_multi(pairs)
  args = []
  pairs.each do |key,value|
    args << key
    args << Oj.dump(value, mode: :compat, time_format: :ruby)
  end
  @redis.mset(*args)
end

#write_unless_exists(name, value) ⇒ Boolean

Writes, as JSON, the value to a key, unless the key exists.

Parameters:

  • name (String) —

    a key

  • value (Hash) —

    a value

Returns:

  • (Boolean) —

    whether the key was set



69
70
71
 
# File 'lib/pupa/processor/document_store/redis_store.rb', line 69

def write_unless_exists(name, value)
  @redis.setnx(name, Oj.dump(value, mode: :compat, time_format: :ruby))
end