Class: Dis::Storage

Inherits:
Object
  • Object
show all
Defined in:
lib/dis/storage.rb

Overview

Dis Storage

This is the interface for interacting with the storage layers.

All queries are scoped by object type, which will default to the table name of the model. Take care to use your own scope if you interact with the store directly, as models will purge expired content when they change.

Files are stored with a SHA1 digest of the file contents as the key. This ensures data is deduplicated per scope. Hash collisions will be silently ignored.

Layers should be added to Dis::Storage.layers. At least one writeable, non-delayed layer must exist.

Class Method Summary collapse

Class Method Details

.delayed_delete(type, hash) ⇒ Object

Deletes content from all delayed layers.

Dis::Storage.delayed_delete("things", hash)



125
126
127
128
129
 
# File 'lib/dis/storage.rb', line 125

def delayed_delete(type, hash)
  layers.delayed.writeable.each do |layer|
    layer.delete(type, hash)
  end
end

.delayed_store(type, hash) ⇒ Object

Transfers files from immediate layers to all delayed layers.

Dis::Storage.delayed_store("things", hash)



58
59
60
61
62
63
 
# File 'lib/dis/storage.rb', line 58

def delayed_store(type, hash)
  file = get(type, hash)
  layers.delayed.writeable.each do |layer|
    layer.store(type, hash, file)
  end
end

.delete(type, hash) ⇒ Object

Deletes a file from all layers. Kicks off a Dis::Jobs::Delete job if any delayed layers are defined. Returns true if the file existed in any immediate layers, or false if not.

Dis::Storage.delete("things", hash)
# => true
Dis::Storage.delete("things", hash)
# => false



110
111
112
113
114
115
116
117
118
119
120
 
# File 'lib/dis/storage.rb', line 110

def delete(type, hash)
  require_writeable_layers!
  deleted = false
  layers.immediate.writeable.each do |layer|
    deleted = true if layer.delete(type, hash)
  end
  if layers.delayed.writeable.any?
    Dis::Jobs::Delete.perform_later(type, hash)
  end
  deleted
end

.exists?(type, hash) ⇒ Boolean

Returns true if the file exists in any layer.

Dis::Storage.exists?("things", hash) # => true

Returns:

  • (Boolean) 


68
69
70
71
72
73
74
 
# File 'lib/dis/storage.rb', line 68

def exists?(type, hash)
  require_layers!
  layers.each do |layer|
    return true if layer.exists?(type, hash)
  end
  false
end

.file_digest(file) {|hash| ... } ⇒ Object

Returns a hex digest for a given binary. Accepts files, strings and Fog models.

Yields:

  • (hash) 


22
23
24
25
26
27
28
29
30
31
32
33
 
# File 'lib/dis/storage.rb', line 22

def file_digest(file)
  hash = case file
         when Fog::Model
           digest.hexdigest(file.body)
         when String
           digest.hexdigest(file)
         else
           digest.file(file.path).hexdigest
         end
  yield hash if block_given?
  hash
end

.get(type, hash) ⇒ Object

Retrieves a file from the store.

stuff = Dis::Storage.get("things", hash)

If any misses are detected, it will try to fetch the file from the first available layer, then store it in all immediate layer.

Returns an instance of Fog::Model.

Raises:



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/dis/storage.rb', line 84

def get(type, hash)
  require_layers!

  fetch_count = 0
  result = layers.inject(nil) do |res, layer|
    res || lambda do
      fetch_count += 1
      layer.get(type, hash)
    end.call
  end

  raise Dis::Errors::NotFoundError unless result

  store_immediately!(type, result) if fetch_count > 1
  result
end

.layers ⇒ Object

Exposes the layer set, which is an instance of Dis::Layers.



37
38
39
 
# File 'lib/dis/storage.rb', line 37

def layers
  @layers ||= Dis::Layers.new
end

.store(type, file) ⇒ Object

Stores a file and returns a digest. Kicks off a Dis::Jobs::Store job if any delayed layers are defined.

hash = Dis::Storage.store("things", File.open('foo.bin'))
# => "8843d7f92416211de9ebb963ff4ce28125932878"



46
47
48
49
50
51
52
53
 
# File 'lib/dis/storage.rb', line 46

def store(type, file)
  require_writeable_layers!
  hash = store_immediately!(type, file)
  if layers.delayed.writeable.any?
    Dis::Jobs::Store.perform_later(type, hash)
  end
  hash
end