Class: ScoutAgent::Database

Inherits:
Object
  • Object
show all
Defined in:
lib/scout_agent/database.rb,
lib/scout_agent/database/queue.rb,
lib/scout_agent/database/statuses.rb,
lib/scout_agent/database/snapshots.rb,
lib/scout_agent/database/mission_log.rb

Overview

This wrapper over Amalgalite::Database adds database loading by name, schema migrations when they are loaded, installation of standardized busy handlers and taps for the agent as a connection is prepared, and helper methods for managing database locks and building schemas.

Direct Known Subclasses

MissionLog, Queue, Snapshots, Statuses

Defined Under Namespace

Classes: MissionLog, Queue, Snapshots, Statuses

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(log = WireTap.new(nil)) ⇒ Database

Builds a new database instance, optionally tied to a log.

This is very low-level, bypassing the standard preparation and migration process. Thus databases should usually be created with load() instead.



51
52
53
54
55
56
 
# File 'lib/scout_agent/database.rb', line 51

def initialize(log = WireTap.new(nil))
  @log          = log
  @sqlite       = Amalgalite::Database.new(path.to_s)
  @read_locked  = false
  @write_locked = false
end

Instance Attribute Details

#log ⇒ Object (readonly)

The log file (or bit buckect by default) this database reports to.



59
60
61
 
# File 'lib/scout_agent/database.rb', line 59

def log
  @log
end

Class Method Details

.load(name, log = WireTap.new(nil)) ⇒ Object

Loads a database by name, optionally preparing log to receive any error messages.

The process loads the proper code wrapper for the database by name, installs the handlers and taps used for all connections by the agent, migrates the schema up to the latest code, and returns the connection handle. If anything goes wrong in this process, nil is returned instead.



30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/scout_agent/database.rb', line 30

def self.load(name, log = WireTap.new(nil))
  require LIB_DIR + "database/#{name}"
  
  begin
    db = ScoutAgent::Database.const_get(name.to_s.CamelCase).new(log)
    db.prepare_connection
    db.migrate
  rescue Amalgalite::SQLite3::Error => error  # failed to migrate database
    log.error("Database migration error:  #{error.message}.")
    return nil  # cannot load database
  end
  
  db
end

.path(name) ⇒ Object

Returns the path to a database file based on the configured data storage location and the passed name of the database.



16
17
18
 
# File 'lib/scout_agent/database.rb', line 16

def self.path(name)
  Plan.db_dir + "#{name}.sqlite"
end

Instance Method Details

#locked? ⇒ Boolean

Returns true if this connection is currently read or write locked.

Returns:

  • (Boolean) 


198
199
200
 
# File 'lib/scout_agent/database.rb', line 198

def locked?
  read_locked? or write_locked?
end

#maintain ⇒ Object

This method should be called periodically to VACUUM databases and reclaim the space they have consumed on the hard disk. Returns true if maintenance was successful, false if it wasn’t needed, and nil if errors prevented the process from completing. The next scheduled run for maintenance is stored in a table of the database added during the migration process.

This method uses some external synchronization to ensure that only one process can be in here at once, thus processes may block for a time when they call it.



147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/scout_agent/database.rb', line 147

def maintain
  # 
  # This cannot be in a transaction (+VACUUM+ fails).  Given that, we use an
  # external locking mechanism on the database code file to prevent doubled
  # +VACUUM+ race conditions.
  # 
  Pathname.new(path.to_s.sub(/\.sqlite\z/, ".lock")).open("a") do |db|
    begin
      db.chmod(0777)  # make sure this file is shared by all
    rescue Exception  # we didn't create the file
      # do nothing:  the creator already switched the permissions
    end
    db.flock(File::LOCK_EX)
    begin
      if @sqlite.first_value_from(
           "SELECT ROWID FROM maintenance WHERE next_run_at <= ?",
           Time.now.to_db_s
         )
        @sqlite.execute("VACUUM")
        @sqlite.execute(<<-END_UPDATE_MAINTENANCE_TIME.trim)
        INSERT OR REPLACE INTO
          maintenance( ROWID,                            next_run_at )
          VALUES(          1, datetime('now', 'localtime', '+1 day') );
        END_UPDATE_MAINTENANCE_TIME
        true  # maintenance successful
      else
        false  # maintenance not needed
      end
    ensure
      db.flock(File::LOCK_UN)
    end
  end
rescue Amalgalite::SQLite3::Error => error  # failed to +VACUUM+ database
  log.error("Database maintenance error:  #{error.message}.")
  nil  # maintenance failed, we will try again later
rescue Exception => error  # file locking error
  log.error("Database maintenance locking error:  #{error.message}.")
  nil  # maintenance failed, we will try again later
end

#migrate ⇒ Object

This method updates a database schema. It does that by grabbing a write lock and then pulling the current schema_version() and feeding that to update_schema() for SQL to upgrade the database with. SQL is passed through filters for type and trigger expansions, then batch processed by SQLite. This process will be repeated until update_schema() returns nil to indicate that the schema is now up-to-date.

Some maintenance SQL is also inserted with the first migration to initialize database maintenance tracking.



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/scout_agent/database.rb', line 102

def migrate
  write_to_sqlite do |sqlite|
    loop do
      version = schema_version
      sql     = update_schema(version) or break
      sql     = "#{add_maintenance_sql(version)}#{sql}"
      sql.gsub!(/^([ \t]*)(\w+)[ \t]+(\w+_TYPE)\b/) {           # types
        begin
          send($3.downcase, $2).gsub(/^/, $1.to_s)
        rescue NoMethodError  # unsupported type
          $&
        end
      }
      sql.gsub!(/^[ \t]*(\w+_TRIGGER)[ \t]+(\S.*?)[ \t]*$/) {  # triggers
        begin
          send($1.downcase, *$2.split)
        rescue NoMethodError  # unsupported trigger
          $&
        end
      }
      sqlite.execute_batch(sql)
    end
  end
end

#path ⇒ Object

This is a shortcut for the class method of the same name, passing a snake_case version of this Class name as the database name.



65
66
67
 
# File 'lib/scout_agent/database.rb', line 65

def path
  self.class.path(self.class.short_name.snake_case)
end

#prepare_connection ⇒ Object

This method is invoked after a database is openned, but before a connection is used. This provides a chance to install handles, taps, and functions before SQL is executed through the connection.



74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/scout_agent/database.rb', line 74

def prepare_connection
  # 
  # Wait up to 60 seconds for a database lock, attempting to grab it every
  # 100 milliseconds.
  # 
  @sqlite.busy_handler(Amalgalite::BusyTimeout.new(600, 100))
  # Install a trace tap for SQL when debugging.
  if log.debug?
    @sqlite.trace_tap = Amalgalite::TraceTap.new(log, :debug)
  end
end

#query(sql, *params, &transform) ⇒ Object

A convenience for running sql, with any params, in a read lock and then applying transform to each row before the result set is returned.



253
254
255
256
257
258
259
 
# File 'lib/scout_agent/database.rb', line 253

def query(sql, *params, &transform)
  read_from_sqlite { |sqlite|
    results = sqlite.execute(sql, *params)
    results.each(&transform) unless transform.nil?
    results
  }
end

#read_from_sqlite ⇒ Object

This method is used to wrap some code in a read lock transaction. The block you pass is ensured to be run inside of a transaction. If one is already is active, the code will just be run normally. Otherwise, a new transaction is started and it will be completed after your block runs. This ensures that multiple calls to this method nest properly.



209
210
211
212
213
214
215
216
217
218
219
220
221
222
 
# File 'lib/scout_agent/database.rb', line 209

def read_from_sqlite
  if locked?
    yield @sqlite
  else
    begin
      @sqlite.transaction("IMMEDIATE") {
        @read_locked = true
        yield @sqlite
      }
    ensure
      @read_locked = false
    end
  end
end

#read_locked? ⇒ Boolean

Returns true if this connection is currently read locked.

Returns:

  • (Boolean) 


188
189
190
 
# File 'lib/scout_agent/database.rb', line 188

def read_locked?
  @read_locked
end

#schema_version ⇒ Object

Returns the current schema version number tracked by SQLite.



87
88
89
 
# File 'lib/scout_agent/database.rb', line 87

def schema_version
  read_from_sqlite { |sqlite| sqlite.pragma(:schema_version).first.first }
end

#update_schema(version = schema_version) ⇒ Object

Subclasses override this method to setup schema migrations as described in migrate().



131
132
133
 
# File 'lib/scout_agent/database.rb', line 131

def update_schema(version = schema_version)
  nil
end

#write_locked? ⇒ Boolean

Returns true if this connection is currently write locked.

Returns:

  • (Boolean) 


193
194
195
 
# File 'lib/scout_agent/database.rb', line 193

def write_locked?
  @write_locked
end

#write_to_sqlite ⇒ Object

Works just like read_from_sqlite(), but with a write lock.

This method will raise a RuntimeError if called inside a read_from_sqlite() block. It’s too late to upgrade to a write lock at that point as a deadlock condition could be introduced. Such code must be reworked to aquire the write lock first.



232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
 
# File 'lib/scout_agent/database.rb', line 232

def write_to_sqlite
  if read_locked?
    raise "Cannot upgrade a read lock to a write lock"
  elsif write_locked?
    yield @sqlite
  else
    begin
      @sqlite.transaction("EXCLUSIVE") {
        @write_locked = true
        yield @sqlite
      }
    ensure
      @write_locked = false
    end
  end
end