Simple DBM-style key-value database using SQLite3

Description

dbmlite3 is a simple key-value store built on top of SQLite3 that provides a Hash-like interface. It is a drop-in replacement for DBM or YAML::DBM that uses SQLite3 to do the underlying storage.

Why?

Because DBM is really simple and SQLite3 is solid, reliable, ubiquitous, and file-format-compatible across all platforms. This gem gives you the best of both worlds.

Synopsis

require 'dbmlite3'

# Open a table in a database
settings = Lite3::DBM.new("config.sqlite3", "settings")

# You use it like a hash
settings["speed"] = 88
settings["date"] = Date.new(1955, 11, 5)  # Normal Ruby values are allowed
settings["power_threshold"] = 2.2

puts settings['power_threshold']

settings.each{|k,v| puts "setting: #{k} = #{v}" }

# But you also have transactions
settings.transaction{
  settings["speed"] = settings["speed"] * 2
}

# You can open other tables in the same database if you want, as above
# or with a block
Lite3::DBM.open("config.sqlite3", "stats") { |stats|
  stats["max"] = 42

  # You can even open multiple handles to the same table if you need to
  Lite3::DBM.open("config.sqlite3", "stats") { |stats2|
    stats2["max"] += 1
  }

  puts "stats=#{stats["max"]}"
}

settings.close

Complete documentation is available in the accompanying rdoc.

Installation

dbmlite3 is available as a gem:

$ [sudo] gem install dbmlite3

Alternately, you can fetch the source code from GitLab and build it yourself:

$ git clone https://gitlab.com/suetanvil/dbmlite3
$ cd dbmlite3
$ rake

Obviously, it depends on the gem sqlite3.

Quirks and Hints

Remember that a DBM is a (potentially) shared file

It is important to keep in mind that while Lite3::DBM objects look like Hashes, they are accessing files on disk that other processes could modify at any time.

For example, an innocuous-looking expression like

db['foo'] = db['foo'] + 1

or its shorter equivalent

db['foo'] += 1

contains a race condition. If (e.g.) two copies of this script are running at the same time, it is possible for both to perform the read before one of them writes, losing the others' result.

There are two ways to deal with this. You can wrap the read-modify-write cycle in a transaction:

db.transaction { db['foo'] += 1 }

Or, of course, you could just design your script or program so that only one program accesses the table at a time.

Transactions and performance

If you need to do a large number of accesses in a short amount of time (e.g. loading data from a file), it is significantly faster to do these in batches in one or more transactions.

Serialization Safety

Lite3::DBM stores Ruby data by first serializing values using the Marshal or Psych modules. This can pose a security risk if an untrusted third party has direct access to the underlying SQLite3 database. This tends to be pretty rare for most use-cases but if it is a concern, you can always configure Lite3::DBM to store its values as plain strings.

Forking safely

It is a documented limitation of SQLite3 that database objects cannot be carried across a process fork. Either the parent or the child process will keep the open handle and the other one must forget it completely.

For this reason, if you need both the parent and child process to be able to use Lite3::DBM after a fork, you must first call Lite3::SQL.close_all. Not only will this make it safe but it also lets the child and parent use the same Lite3::DBM objects.

Lite3::DBM objects act like file handles but are not

While it is generally safe to treat Lite3::DBM as a wrapper around file handle (i.e. open and close work as expected), you should be aware that this is not precisely the way things actually work. Instead, the gem maintains a pool of database handles, one per file, and associates them with Lite3::DBM instances as needed. This is necessary for transactions to work correctly.

See the reference doc for Lite3::SQL for more details.

Mostly, you don't need to worry about this but certain types of bugs could behave in unexpected ways and knowing this may help you make sense of them.