Class: Neofiles::File

Inherits:
Object
  • Object
show all
Includes:
Mongoid::Document, Mongoid::Timestamps
Defined in:
app/models/neofiles/file.rb

Overview

This model stores file metadata like name, size, md5 hash etc. A model ID is essentially what is a “file” in the rest of an application. In some way Neofiles::File may be seen as remote filesystem, where you drop in files and keep their generated IDs to fetch them later, or setup web frontend via Neofiles::Files/ImagesController and request file bytes by ID from there.

When persisting new file to the MongoDB database one must initialize new instance with metadata and set field #file= to IO-like object that holds real bytes. When #save method is called, file metadata are saved into Neofiles::File and file content is read and saved into collection of Neofiles::FileChunk, each of maximum length of #chunk_size bytes:

logo = Neofiles::File.new
logo.description = 'ACME inc logo'
logo.file = '~/my-first-try.png' # or some opened file handle, or IO stream
logo.filename = 'acme.png'
logo.save
logo.chunks.to_a # return an array of Neofiles::FileChunk in order
logo.data # byte string of file contents

# in view.html.slim
- logo = Neofiles::File.find 'xxx'
= neofiles_file_url logo          # 'http://doma.in/neofiles/serve-file/#{logo.id}'
= neofiles_link logo, 'Our logo'  # '<a href="...#{logo.id}">Our logo</a>'

This file/chunks concept is called Mongo GridFS (Grid File System) and is described as a standard way of storing files in MongoDB.

MongoDB collection & client (session) can be changed via Rails.application.config.neofiles.mongo_files_collection and Rails.application.config.neofiles.mongo_client

Model fields:

filename      - real name of file, is guessed when setting #file= but can be changed manually later
content_type  - MIME content type, is guessed when setting #file= but can be changed manually later
length        - file size in bytes
chunk_size    - max Neofiles::FileChunk size in bytes
md5           - md5 hash of file (to find duplicates for example)
description   - arbitrary description
owner_type/id - as in Mongoid polymorphic belongs_to relation, a class name & ID of object this file belongs to
is_deleted    - flag that file was once marked as deleted (just a flag for future use, affects nothing)

There is no sense in deleting a file since space it used to hold is not reallocated by MongoDB, so files are considered forever lasting. But technically it is possible to delete model instance and it’s chunks will be deleted as well.

Direct Known Subclasses

Image, Swf

Constant Summary collapse

DEFAULT_CHUNK_SIZE = 
Rails.application.config.neofiles.mongo_default_chunk_size

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#file ⇒ Object

Returns the value of attribute file.



137
138
139
 
# File 'app/models/neofiles/file.rb', line 137

def file
  @file
end

Class Method Details

.binary_for(*buf) ⇒ Object

Construct Mongoid binary object from string of bytes.



232
233
234
 
# File 'app/models/neofiles/file.rb', line 232

def self.binary_for(*buf)
  BSON::Binary.new(buf.join, :generic)
end

.chunking(io, chunk_size, &block) ⇒ Object

Split IO stream by chunks chunk_size bytes each and yield each chunk in block.



210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
 
# File 'app/models/neofiles/file.rb', line 210

def self.chunking(io, chunk_size, &block)
  if io.method(:read).arity == 0
    data = io.read
    i = 0
    loop do
      offset = i * chunk_size
      length = i + chunk_size < data.size ? chunk_size : data.size - offset

      break if offset >= data.size

      buf = data[offset, length]
      block.call(buf)
      i += 1
    end
  else
    while buf = io.read(chunk_size)
      block.call(buf)
    end
  end
end

.class_by_content_type(content_type) ⇒ Object

Guess descendant class of Neofiles::File by MIME content type to use special purpose class for different file types:

Neofiles::File.file_class_by_content_type('image/jpeg') # -> Neofiles::Image
Neofiles::File.file_class_by_content_type('some/unknown') # -> Neofiles::File

Can be used when persisting new files or loading from database.



280
281
282
283
284
285
286
287
288
289
290
291
 
# File 'app/models/neofiles/file.rb', line 280

def self.class_by_content_type(content_type)
  case content_type
  when 'image/svg+xml'
    ::Neofiles::File
  when /\Aimage\//
    ::Neofiles::Image
  when 'application/x-shockwave-flash'
    ::Neofiles::Swf
  else
    self
  end
end

.class_by_file_name(file_name) ⇒ Object

Same as file_class_by_content_type but for file name string.



294
295
296
 
# File 'app/models/neofiles/file.rb', line 294

def self.class_by_file_name(file_name)
  class_by_content_type(extract_content_type(file_name))
end

.class_by_file_object(file_object) ⇒ Object

Same as file_class_by_content_type but for file-like object.



299
300
301
 
# File 'app/models/neofiles/file.rb', line 299

def self.class_by_file_object(file_object)
  class_by_file_name(extract_basename(file_object))
end

.cleanname(pathname) ⇒ Object

Extract only file name partion from path.



269
270
271
 
# File 'app/models/neofiles/file.rb', line 269

def self.cleanname(pathname)
  ::File.basename(pathname.to_s)
end

.extract_basename(object) ⇒ Object

Try different methods to extract file name or path from argument object.



237
238
239
240
241
242
243
244
245
246
 
# File 'app/models/neofiles/file.rb', line 237

def self.extract_basename(object)
  filename = nil
  %i{ original_path original_filename path filename pathname }.each do |msg|
    if object.respond_to?(msg)
      filename = object.send(msg)
      break
    end
  end
  filename ? cleanname(filename) : nil
end

.extract_content_type(basename) ⇒ Object

Try different methods to extract MIME content type from file name, e.g. jpeg -> image/jpeg



249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
 
# File 'app/models/neofiles/file.rb', line 249

def self.extract_content_type(basename)
  if defined?(MIME)
    content_type = MIME::Types.type_for(basename.to_s).first
  else
    ext = ::File.extname(basename.to_s).downcase.sub(/[.]/, '')
    if ext.in? %w{ jpeg jpg gif png }
      content_type = 'image/' + ext.sub(/jpg/, 'jpeg')
    elsif ext == 'swf'
      content_type = 'application/x-shockwave-flash'
    elsif ext == 'svg'
      content_type = 'image/svg+xml'
    else
      content_type = nil
    end
  end

  content_type.to_s if content_type
end

.reading(arg, &block) ⇒ Object

Yield block with IO stream made from input arg, which can be file name or other IO readable object.



197
198
199
200
201
202
203
204
205
206
207
 
# File 'app/models/neofiles/file.rb', line 197

def self.reading(arg, &block)
  if arg.respond_to?(:read)
    self.rewind(arg) do |io|
      block.call(io)
    end
  else
    open(arg.to_s) do |io|
      block.call(io)
    end
  end
end

.rewind(io, &block) ⇒ Object

Yield IO-like argument to block rewinding it first, if possible.



304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
 
# File 'app/models/neofiles/file.rb', line 304

def self.rewind(io, &block)
  begin
    pos = io.pos
    io.flush
    io.rewind
  rescue
    nil
  end

  begin
    block.call(io)
  ensure
    begin
      io.pos = pos
    rescue
      nil
    end
  end
end

Instance Method Details

#admin_compact_view(template) ⇒ Object

Representation of file in admin “compact” mode, @see Neofiles::AdminController#file_compact. To be redefined by descendants.



192
193
194
 
# File 'app/models/neofiles/file.rb', line 192

def admin_compact_view(template)
  template.neofiles_link self, nil, target: '_blank'
end

#base64 ⇒ Object

Encode bytes in base64.



113
114
115
 
# File 'app/models/neofiles/file.rb', line 113

def base64
  Array(to_s).pack('m')
end

#bytes(&block) ⇒ Object

Bytes as chunks array, if block is given — yield it.



124
125
126
127
128
129
130
131
132
133
 
# File 'app/models/neofiles/file.rb', line 124

def bytes(&block)
  if block
    each { |data| block.call(data) }
    length
  else
    bytes = []
    each { |data| bytes.push(*data) }
    bytes
  end
end

#data ⇒ Object

Chunks bytes concatenated, that is the whole file content.



106
107
108
109
110
 
# File 'app/models/neofiles/file.rb', line 106

def data
  data = ''
  each { |chunk| data << chunk }
  data
end

#data_uri(options = {}) ⇒ Object

Encode bytes id data uri.



118
119
120
121
 
# File 'app/models/neofiles/file.rb', line 118

def data_uri(options = {})
  data = base64.chomp
  "data:#{content_type};base64,#{data}"
end

#each(&block) ⇒ Object

Yield block for each chunk.



72
73
74
75
76
 
# File 'app/models/neofiles/file.rb', line 72

def each(&block)
  chunks.all.order_by([:n, :asc]).each do |chunk|
    block.call(chunk.to_s)
  end
end

#nullify_unpersisted_file ⇒ Object

Reset @file after save.



186
187
188
 
# File 'app/models/neofiles/file.rb', line 186

def nullify_unpersisted_file
  @file = nil
end

#save_file ⇒ Object

Real file saving goes here. File length and md5 hash are computed automatically.



160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
 
# File 'app/models/neofiles/file.rb', line 160

def save_file
  if @file
    self.chunks.delete_all

    md5 = Digest::MD5.new
    length, n = 0, 0

    self.class.reading(@file) do |io|
      self.class.chunking(io, chunk_size) do |buf|
        md5 << buf
        length += buf.size
        chunk = self.chunks.build
        chunk.data = self.class.binary_for(buf)
        chunk.n = n
        n += 1
        chunk.save!
        self.chunks.push(chunk)
      end
    end

    self.length = length
    self.md5    = md5.hexdigest
  end
end

#slice(*args) ⇒ Object

Get a portion of chunks, either via Range of Fixnum (length).



79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
 
# File 'app/models/neofiles/file.rb', line 79

def slice(*args)
  case args.first
    when Range
      range = args.first
      first_chunk = (range.min / chunk_size).floor
      last_chunk = (range.max / chunk_size).ceil
      offset = range.min % chunk_size
      length = range.max - range.min + 1
    when Fixnum
      start = args.first
      start = self.length + start if start < 0
      length = args.size == 2 ? args.last : 1
      first_chunk = (start / chunk_size).floor
      last_chunk = ((start + length) / chunk_size).ceil
      offset = start % chunk_size
  end

  data = ''

  chunks.where(n: first_chunk..last_chunk).order_by(n: :asc).each do |chunk|
    data << chunk
  end

  data[offset, length]
end

#unpersisted_file? ⇒ Boolean

Are we going to save file bytes on next #save?

Returns:

  • (Boolean) 


154
155
156
 
# File 'app/models/neofiles/file.rb', line 154

def unpersisted_file?
  not @file.nil?
end