Class: Dis::Storage

Inherits:
Object
  • Object
show all
Defined in:
lib/dis/storage.rb

Overview

Dis Storage

This is the interface for interacting with the storage layers.

All queries are scoped by object type, which will default to the table name of the model. Take care to use your own scope if you interact with the store directly, as models will purge expired content when they change.

Files are stored with a SHA1 digest of the file contents as the key. This ensures data is deduplicated per scope. Hash collisions will be silently ignored.

Layers should be added to Dis::Storage.layers. At least one writeable, non-delayed layer must exist.

Class Method Summary collapse

Class Method Details

.change_type(prev_type, new_type, key) ⇒ Object

Changes the type of an object. Kicks off a Dis::Jobs::ChangeType job if any delayed layers are defined.

Dis::Storage.change_type("old_things", "new_things", key)



45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/dis/storage.rb', line 45

def change_type(prev_type, new_type, key)
  require_writeable_layers!
  file = get(prev_type, key)
  store_immediately!(new_type, file)
  layers.immediate.writeable.each do |layer|
    layer.delete(prev_type, key)
  end
  if layers.delayed.writeable.any?
    Dis::Jobs::ChangeType.perform_later(prev_type, new_type, key)
  end
  key
end

.delayed_delete(type, key) ⇒ Object

Deletes content from all delayed layers.

Dis::Storage.delayed_delete("things", hash)



140
141
142
143
144
 
# File 'lib/dis/storage.rb', line 140

def delayed_delete(type, key)
  layers.delayed.writeable.each do |layer|
    layer.delete(type, key)
  end
end

.delayed_store(type, hash) ⇒ Object

Transfers files from immediate layers to all delayed layers.

Dis::Storage.delayed_store("things", hash)



75
76
77
78
79
80
 
# File 'lib/dis/storage.rb', line 75

def delayed_store(type, hash)
  file = get(type, hash)
  layers.delayed.writeable.each do |layer|
    layer.store(type, hash, file)
  end
end

.delete(type, key) ⇒ Object

Deletes a file from all layers. Kicks off a Dis::Jobs::Delete job if any delayed layers are defined. Returns true if the file existed in any immediate layers, or false if not.

Dis::Storage.delete("things", key)
# => true
Dis::Storage.delete("things", key)
# => false



125
126
127
128
129
130
131
132
133
134
135
 
# File 'lib/dis/storage.rb', line 125

def delete(type, key)
  require_writeable_layers!
  deleted = false
  layers.immediate.writeable.each do |layer|
    deleted = true if layer.delete(type, key)
  end
  if layers.delayed.writeable.any?
    Dis::Jobs::Delete.perform_later(type, key)
  end
  deleted
end

.exists?(type, key) ⇒ Boolean

Returns true if the file exists in any layer.

Dis::Storage.exists?("things", key) # => true

Returns:

  • (Boolean) 


85
86
87
88
89
90
91
 
# File 'lib/dis/storage.rb', line 85

def exists?(type, key)
  require_layers!
  layers.each do |layer|
    return true if layer.exists?(type, key)
  end
  false
end

.file_digest(file) {|hash| ... } ⇒ Object

Returns a hex digest for a given binary. Accepts files, strings and Fog models.

Yields:

  • (hash) 


22
23
24
25
26
27
28
29
30
31
32
33
 
# File 'lib/dis/storage.rb', line 22

def file_digest(file)
  hash = case file
         when Fog::Model
           digest.hexdigest(file.body)
         when String
           digest.hexdigest(file)
         else
           digest.file(file.path).hexdigest
         end
  yield hash if block_given?
  hash
end

.get(type, key) ⇒ Object

Retrieves a file from the store.

stuff = Dis::Storage.get("things", hash)

If any misses are detected, it will try to fetch the file from the first available layer, then store it in all immediate layer.

Returns an instance of Fog::Model.



101
102
103
104
105
106
107
108
109
110
111
112
113
114
 
# File 'lib/dis/storage.rb', line 101

def get(type, key)
  require_layers!

  fetch_count = 0
  result = layers.inject(nil) do |res, layer|
    res || lambda do
      fetch_count += 1
      layer.get(type, key)
    end.call
  end || raise(Dis::Errors::NotFoundError)

  store_immediately!(type, result) if fetch_count > 1
  result
end

.layers ⇒ Object

Exposes the layer set, which is an instance of Dis::Layers.



37
38
39
 
# File 'lib/dis/storage.rb', line 37

def layers
  @layers ||= Dis::Layers.new
end

.store(type, file) ⇒ Object

Stores a file and returns a digest. Kicks off a Dis::Jobs::Store job if any delayed layers are defined.

hash = Dis::Storage.store("things", File.open('foo.bin'))
# => "8843d7f92416211de9ebb963ff4ce28125932878"



63
64
65
66
67
68
69
70
 
# File 'lib/dis/storage.rb', line 63

def store(type, file)
  require_writeable_layers!
  hash = store_immediately!(type, file)
  if layers.delayed.writeable.any?
    Dis::Jobs::Store.perform_later(type, hash)
  end
  hash
end