Class: FSDB::Database

Inherits:
Object
  • Object
show all
Includes:
DirectoryIterators, Formats, PathUtilities
Defined in:
lib/fsdb/database.rb,
lib/fsdb/util.rb

Overview

A thread-safe, process-safe object database class which uses the native file system as its back end and allows multiple file formats.

Defined Under Namespace

Classes: AbortedTransaction, CacheEntry, CreateFileError, DirIsImmutableError, DirNotEmptyError, FormatError, MissingFileError, MissingObjectError, NotDirError, PathComponentError

Constant Summary collapse

MTIME_RESOLUTION =

Even when linux mounts FAT, the mtime granularity is 1 sec.

1.1
CLOCK_SKEW =

in seconds, adjust as needed for stability on NFS

0.0
DEFAULT_META_PREFIX =

Subclasses can change the defaults.

'..fsdb.meta.'
DEFAULT_LOCK_TYPE =

else

:flock
LOCK_TYPES =

These must be methods of File.

[:flock, :fcntl_lock]
FORMATS =

Subclasses can define their own list of formats, with specified search order

[TEXT_FORMAT, MARSHAL_FORMAT].freeze

Constants included from Formats

Formats::BINARY_FORMAT, Formats::DIR_FORMAT, Formats::DIR_LOAD, Formats::DIR_LOAD_FROM_PATH, Formats::DIR_PAT, Formats::HIDDEN_FILE_PAT, Formats::MARSHAL_FORMAT, Formats::TEXT_FORMAT, Formats::YAML_FORMAT

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from DirectoryIterators

#browse_dir, #browse_each_child, #delete_each_child, #edit_dir, #edit_each_child, #replace_each_child

Methods included from PathUtilities

#canonical, #canonical?, #directory?, #glob, #valid?, #validate

Constructor Details

#initialize(dir, opts = {}) ⇒ Database

Create a new database object that accesses dir. Makes sure that the directory exists on disk, but doesn’t create or open any other files. The opts hash can include:

:lock_type

:flock by default, or :fcntl_lock

:meta_prefix

'..fsdb.meta.' by default

:formats

nil by default, so the class’s FORMATS is used



147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
 
# File 'lib/fsdb/database.rb', line 147

def initialize dir, opts = {}
  @dir = File.expand_path(dir)

  @lock_type = opts[:lock_type] || DEFAULT_LOCK_TYPE
  unless LOCK_TYPES.include? @lock_type
    raise "Unknown lock type: #{lock_type}"
  end
  
  if @lock_type == :fcntl_lock
    require 'fcntl_lock' ## hack.
  end

  @meta_prefix = opts[:meta_prefix] || DEFAULT_META_PREFIX

  @formats = opts[:formats]
  
  FileUtils.makedirs(@dir)
end

Class Attribute Details

.cache ⇒ Object (readonly)

:nodoc:



124
125
126
 
# File 'lib/fsdb/database.rb', line 124

def cache
  @cache
end

.cache_mutex ⇒ Object (readonly)

:nodoc:



124
125
126
 
# File 'lib/fsdb/database.rb', line 124

def cache_mutex
  @cache_mutex
end

Instance Attribute Details

#dir ⇒ Object (readonly)

The root directory of the db, to which paths are relative.



131
132
133
 
# File 'lib/fsdb/database.rb', line 131

def dir
  @dir
end

#lock_type ⇒ Object (readonly)

The lock type of the db, by default :flock, optionally :fcntl_lock.



135
136
137
 
# File 'lib/fsdb/database.rb', line 135

def lock_type
  @lock_type
end

Class Method Details

.[](path) ⇒ Object

Shortcut to create a new database at path.



167
168
169
 
# File 'lib/fsdb/database.rb', line 167

def Database.[](path)
  new(path)
end

.abort ⇒ Object

Same as #abort.

Raises:



446
 
# File 'lib/fsdb/database.rb', line 446

def self.abort; raise AbortedTransaction; end

Instance Method Details

#_get_file_id(abs_path) ⇒ Object

:nodoc:



200
201
202
203
 
# File 'lib/fsdb/database.rb', line 200

def _get_file_id(abs_path) # :nodoc:
  File.stat(abs_path) # just to generate the right exceptions
  abs_path # might not be unique, due to links, etc.
end

#abort ⇒ Object

Abort the current transaction (#browse, #edit, #replace, or #delete, roll back the state of the object, and return nil from the transaction.

In the #browse case, the only effect is to end the transaction.

Note that any exception that breaks out of the transaction will also abort the transaction, and be re-raised.

Raises:



443
 
# File 'lib/fsdb/database.rb', line 443

def abort;      raise AbortedTransaction; end

#absolute(path) ⇒ Object Also known as: absolute_path_to

Convert a relative path (relative to the db dir) to an absolute path.



187
188
189
190
191
192
193
 
# File 'lib/fsdb/database.rb', line 187

def absolute(path)
  abs_path = File.expand_path(File.join(@dir, path))
  if File.directory?(abs_path)
    abs_path << ?/ # prevent Errno::EINVAL on UFS
  end
  abs_path
end

#browse(path = "/") ⇒ Object

Browse the object. Yields the object to the caller’s block, and returns the value of the block.

Changes to the object are not persistent, but should be avoided (they will be seen by other threads, but only in the current process, and only until the cache is cleared). If you return the object from the block, or keep a reference to it in some other way, the object will no longer be protected from concurrent writers.



486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
 
# File 'lib/fsdb/database.rb', line 486

def browse(path = "/")                # :yields: object
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  
  ## put these outside method, and pass in params?
  do_when_first = proc do |cache_entry|
    raise if cache_entry.file_handle

    begin
      if PLATFORM_IS_WINDOWS_ME
        abs_path.sub!(/\/+$/, "")
      end
      f = File.open(abs_path, "r")
    rescue Errno::ENOENT
      raise MissingFileError
    rescue Errno::EINTR
      retry
    end

    cache_entry.file_handle = f
    f.lock_shared(@lock_type)
    identify_file_type(f, path, abs_path)
      ## could avoid if cache_object says so
    object = cache_object(f, cache_entry)
  end
  
  do_when_last = proc do |cache_entry|
    # last one out closes the file
    f = cache_entry.file_handle
    if f
      f.close
      cache_entry.file_handle = nil
    end
  end
  
  object_shared(file_id, do_when_first, do_when_last) do |cache_entry|
    object = cache_entry.just_gimme_the_damn_object!
    yield object if block_given?
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    raise if File::CAN_OPEN_DIR
    raise unless File.directory?(abs_path) ### redundant!
    yield Formats::DIR_LOAD_FROM_PATH[abs_path] if block_given?
  end
  clear_entry(file_id)
  default_browse(path) {|x| yield x if block_given?}
rescue AbortedTransaction
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  yield Formats::DIR_LOAD_FROM_PATH[abs_path] if block_given?
end

#cache ⇒ Object 



127
 
# File 'lib/fsdb/database.rb', line 127

def cache; Database.cache; end

#cache_mutex ⇒ Object 



128
 
# File 'lib/fsdb/database.rb', line 128

def cache_mutex; Database.cache_mutex; end

#clear_cache ⇒ Object

Can be called occasionally to reduce memory footprint, esp. if cached objects are large and infrequently used.



280
281
282
283
284
285
286
 
# File 'lib/fsdb/database.rb', line 280

def clear_cache
  cache_mutex.synchronize do
    cache.delete_if do |file_id, cache_entry|
      cache_entry.unused?
    end
  end
end

#clear_entry(file_id) ⇒ Object

For housekeeping, so that stale entries don’t result in unused, but uncollectable, CacheEntry objects.



269
270
271
272
273
274
275
276
 
# File 'lib/fsdb/database.rb', line 269

def clear_entry(file_id)
  if file_id
    cache_mutex.synchronize do
      cache_entry = cache[file_id]
      cache.delete(file_id) if cache_entry and cache_entry.unused?
    end
  end
end

#default_browse(path) ⇒ Object

Called when #browse doesn’t find anything at the path. The original caller’s block is available to be yielded to.



453
454
455
 
# File 'lib/fsdb/database.rb', line 453

def default_browse(path)
  object_missing(path) {|x| yield x}
end

#default_edit(path) ⇒ Object

Called when #edit doesn’t find anything at the path. The original caller’s block is available to be yielded to.



459
460
461
 
# File 'lib/fsdb/database.rb', line 459

def default_edit(path)
  object_missing(path) {|x| yield x}
end

#default_fetch(path) ⇒ Object

Called when #fetch doesn’t find anything at the path. Default definition just returns nil.



471
 
# File 'lib/fsdb/database.rb', line 471

def default_fetch(path); nil; end

#delete(path, load = true) ⇒ Object

Delete the object from the db. If a block is given, yields the object (or nil if none) before deleting it from the db (but before releasing the lock on the path), and returns the value of the block. Otherwise, just returns the object (or nil, if none). Raises DirNotEmptyError if path refers to a non-empty dir. If the dir is empty, it is deleted, and the returned value is true. The block is not yielded to. If the load argument is false, delete the object from the db without loading it or yielding, returning true.



675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
 
# File 'lib/fsdb/database.rb', line 675

def delete(path, load=true)                  # :yields: object
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  delete_later = false
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      if load
        object = cache_object(f, cache_entry)
        result = block_given? ? (yield object) : object
      else
        result = true
      end
      if File::CAN_DELETE_OPEN_FILE
        File.delete(abs_path)
      else
        delete_later = true
      end
      cache_entry.stale!
      del_version_of(f)
      result
    end
  end
rescue DirIsImmutableError
  begin
    Dir.delete(abs_path)
  rescue Errno::ENOENT
    # Someone else got it first.
  end
  true
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if File.symlink?(abs_path) # get_file_id fails if target deleted
    File.delete(abs_path) rescue nil
  end
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    Dir.delete(abs_path)
  end
  nil
rescue Errno::ENOTEMPTY
  raise DirNotEmptyError, "Directory not empty - #{path} in #{inspect}"
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  Dir.delete(abs_path)
  true
rescue AbortedTransaction
ensure
  if delete_later
    begin
      File.delete(abs_path) rescue Dir.delete(abs_path)
    rescue Errno::ENOENT
    end
  end
  clear_entry(file_id)
end

#dump(object, f) ⇒ Object

Writes object to f (must be open for writing).



774
775
776
 
# File 'lib/fsdb/database.rb', line 774

def dump(object, f)
  f.format.dump(object, f)
end

#edit(path = "/") ⇒ Object

Edit the object in place. Changes to the yielded object made within the caller’s block become persistent. Returns the value of the block. Note that assigning to the block argument variable does not change the state of the object. Use destructive methods on the object.



547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
 
# File 'lib/fsdb/database.rb', line 547

def edit(path = "/")
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      object = cache_object(f, cache_entry)
      result = yield object if block_given?
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      result
    end
  end
rescue DirIsImmutableError
  raise DirIsImmutableError, "Cannot edit dir #{path} in #{inspect}"
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  raise DirIsImmutableError if PLATFORM_IS_WINDOWS_ME and
          File.directory?(abs_path)
  clear_entry(file_id)
  default_edit(path) {|x| yield x if block_given?}
rescue AbortedTransaction
  clear_entry(file_id) # The cached object may have edits which are not valid.
  nil
rescue Exception
  clear_entry(file_id)
  raise
end

#fetch(path = "/") ⇒ Object Also known as: []

Fetch a copy of the object at the path for private use by the current thread/process. (The copy is a deep copy.)

Note that this is inherently less efficient than #browse, because #browse leaves the object in the cache, but, for safety, #fetch can only return a copy and wipe the cache, since the copy is going to be used outside of any transaction. Subsequent transactions will have to read the object again.



740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
 
# File 'lib/fsdb/database.rb', line 740

def fetch(path = "/")
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_read_lock(path) do |f|
      object = cache_object(f, cache_entry)
      cache_entry.stale!
      object
    end
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    return Formats::DIR_LOAD_FROM_PATH[abs_path]
  end
  clear_entry(file_id)
  default_fetch(path)
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  return Formats::DIR_LOAD_FROM_PATH[abs_path]
end

#find_format(path, abs_path = absolute(path)) ⇒ Object 



802
803
804
805
806
807
808
809
 
# File 'lib/fsdb/database.rb', line 802

def find_format(path, abs_path = absolute(path))
  if DIR_FORMAT === abs_path
    DIR_FORMAT
  else
    path = path.sub(/^\//, "") # So that db['foo'] and db['/foo'] are same
    formats.find {|fmt| fmt === path}
  end
end

#formats ⇒ Object 



781
782
783
 
# File 'lib/fsdb/database.rb', line 781

def formats
  @formats || self.class::FORMATS
end

#formats=(fmts) ⇒ Object 



785
786
787
 
# File 'lib/fsdb/database.rb', line 785

def formats=(fmts)
  @formats = fmts
end

#get_file_id(abs_path) ⇒ Object

Convert an absolute path to a unique key for the cache, raising MissingFileError if the file does not exist.



213
214
215
216
217
218
219
220
221
222
 
# File 'lib/fsdb/database.rb', line 213

def get_file_id(abs_path)
  _get_file_id(abs_path)
rescue Errno::ENOTDIR
  # db['x'] = 0; db.edit 'x/' do end
  raise NotDirError
rescue Errno::ENOENT
  raise MissingFileError, "Cannot find file at #{abs_path}"
rescue Errno::EINTR
  retry
end

#identify_file_type(f, path, abs_path = absolute(path)) ⇒ Object

path is relative to the database, and initial ‘/’ is ignored



793
794
795
796
797
798
799
800
 
# File 'lib/fsdb/database.rb', line 793

def identify_file_type(f, path, abs_path = absolute(path))
  format = find_format(path, abs_path)
  unless format
    raise FormatError, "No format found for path #{path.inspect}"
  end
  f.binmode if format.binary?
  f.format = format
end

#insert(path, object) ⇒ Object Also known as: []=

Insert the object, replacing anything at the path. Returns the object. (The object remains a local copy, distinct from the one which will be returned when accessing the path through database transactions.)

If path ends in “/”, then object is treated as a collection of key-value pairs, and each value is inserted at the corresponding key under path. (You can omit the “/” if the dir already exists.) is this still true?



633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
 
# File 'lib/fsdb/database.rb', line 633

def insert(path, object)
  abs_path = absolute(path)
  file_id = make_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      object
    end
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue FormatError
  File.delete(abs_path)
  raise
rescue PathComponentError
  raise PathComponentError, "Some component of #{path} in #{inspect} " +
      "already exists and is not a directory"
rescue CreateFileError
  if PLATFORM_IS_WINDOWS_ME and /\/$/ =~ path
    raise DirIsImmutableError
  else
    raise CreateFileError, "Cannot create file at #{path} in #{inspect}"
  end
rescue MissingFileError
  raise DirIsImmutableError if PLATFORM_IS_WINDOWS_ME
ensure
  clear_entry(file_id) # no one else can get this copy of object
end

#inspect ⇒ Object 



184
 
# File 'lib/fsdb/database.rb', line 184

def inspect; "#<#{self.class}:#{dir}>"; end

#link(old_name, new_name) ⇒ Object

Create a hard link, using File.link. The names are relative to the database’s path.



130
131
132
 
# File 'lib/fsdb/util.rb', line 130

def link(old_name, new_name)
  File.link(absolute(old_name), absolute(new_name))
end

#load(f) ⇒ Object

Returns object read from f (must be open for reading).



769
770
771
 
# File 'lib/fsdb/database.rb', line 769

def load(f)
  f.format.load(f)
end

#make_file_id(abs_path) ⇒ Object

Convert an absolute path to a unique key for the cache, creating the file if it does not exist. Raises CreateFileError if it can’t be created.



238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
 
# File 'lib/fsdb/database.rb', line 238

def make_file_id(abs_path)
  dirname = File.dirname(abs_path)
  begin
    FileUtils.makedirs(dirname)
  rescue Errno::EEXIST
    raise PathComponentError
  end
  begin
    _get_file_id(abs_path)
  rescue Errno::EINTR
    retry
  end
rescue Errno::ENOTDIR
  # db['x'] = 0; db.replace 'x/' do end
  raise NotDirError
rescue Errno::ENOENT
  begin
    File.open(abs_path, "w") do |f|
      _get_file_id(abs_path)
    end
  rescue Errno::EISDIR
    raise DirIsImmutableError
  rescue Errno::EINVAL
    raise DirIsImmutableError # for windows
  rescue StandardError
    raise CreateFileError
  end
end

#object_missing(path) ⇒ Object

The default behavior of both #default_edit and #default_browse. Raises MissingObjectError by default, but it can yield to the original block.

Raises:



465
466
467
 
# File 'lib/fsdb/database.rb', line 465

def object_missing(path)
  raise MissingObjectError, "No object at #{path} in #{inspect}"
end

#replace(path) ⇒ Object

Replace the yielded object (or nil) with the return value of the block. Returns the object that was replaced. No object need exist at path.

Use replace instead of edit when accessing db over a drb connection. Use replace instead of insert if the path needs to be protected while the object is prepared for insertion.

Note that (unlike #edit) destructive methods on the object do not persistently change the state of the object, unless the object is the return value of the block.



586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
 
# File 'lib/fsdb/database.rb', line 586

def replace(path)
  abs_path = absolute(path)
  file_id = make_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      old_object = f.stat.zero? ? nil : cache_object(f, cache_entry)
      object = yield old_object if block_given?
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      old_object
    end
  end
rescue DirIsImmutableError
  raise DirIsImmutableError, "Cannot replace dir #{path} in #{inspect}"
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue AbortedTransaction
  clear_entry(file_id) # The cached object may have edits which are not valid.
  nil
rescue FormatError
  clear_entry(file_id)
  File.delete(abs_path)
  raise
rescue PathComponentError
  raise PathComponentError, "Some component of #{path} in #{inspect} " +
      "already exists and is not a directory"
rescue CreateFileError
  raise CreateFileError, "Cannot create file at #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    raise DirIsImmutableError
  else
    raise NotDirError
  end
rescue Exception
  clear_entry(file_id)
  raise
end

#subdb(path) ⇒ Object

Create a new database object that accesses path relative to the database directory. A process can have any number of dbs accessing overlapping dirs. The cost of creating an additional db is very low; its state is just the dir and some options. Caching is done in structures owned by the Database class itself.



176
177
178
179
180
181
182
 
# File 'lib/fsdb/database.rb', line 176

def subdb path
  self.class.new(File.join(@dir, path),
    :lock_type => @lock_type,
    :meta_prefix => @meta_prefix,
    :formats => @formats && @formats.dup
  )
end

#symlink(old_name, new_name) ⇒ Object

Create a symbolic link, using File.symlink. The names are relative to the database’s path.



136
137
138
 
# File 'lib/fsdb/util.rb', line 136

def symlink(old_name, new_name)
  File.symlink(absolute(old_name), absolute(new_name))
end