Module: Iriq::Storage

Defined in:
lib/iriq/storage.rb,
lib/iriq/storage/json.rb,
lib/iriq/storage/memory.rb,
lib/iriq/storage/sqlite.rb

Overview

Storage is the persistence layer for a Corpus. It owns every counter and per-(host, prefix) frequency map; the Corpus class delegates state to it.

Three concrete backends ship:

Storage::Memory   — in-memory only; matches the original behavior.
Storage::Json     — Memory backend wrapped with load/save against a JSON file.
Storage::Sqlite   — incremental UPSERTs against a SQLite database.

File-extension dispatch keeps callers simple: .json (or anything else) picks Json, .db/.sqlite/.sqlite3 picks Sqlite.

Defined Under Namespace

Classes: Json, Memory, Sqlite

Constant Summary collapse

SQLITE_EXTS = 
%w[.db .sqlite .sqlite3].freeze
TEMP_SEQ_LOCK = 
Mutex.new

Class Method Summary collapse

Class Method Details

.next_temp_seq ⇒ Object 



18
 
# File 'lib/iriq/storage.rb', line 18

def self.next_temp_seq = TEMP_SEQ_LOCK.synchronize { @temp_seq += 1 }

.open(path, classifier: SegmentClassifier::DEFAULT, max_values_per_position: PositionStats::DEFAULT_MAX_VALUES) ⇒ Object

Opens (or creates) a storage at path, picking the backend by extension. If path is nil, returns a Memory backend.



24
25
26
27
28
29
30
31
32
33
34
35
 
# File 'lib/iriq/storage.rb', line 24

def open(path, classifier: SegmentClassifier::DEFAULT,
               max_values_per_position: PositionStats::DEFAULT_MAX_VALUES)
  return Memory.new(classifier: classifier, max_values_per_position: max_values_per_position) if path.nil?

  if SQLITE_EXTS.include?(File.extname(path).downcase)
    require "iriq/storage/sqlite"
    Sqlite.open(path, classifier: classifier, max_values_per_position: max_values_per_position)
  else
    require "iriq/storage/json"
    Json.open(path, classifier: classifier, max_values_per_position: max_values_per_position)
  end
end

.write_atomically(path, contents) ⇒ Object

Replace path atomically via a per-write temp file, PATH...tmp (Rust's writer uses the same shape; --reset sweeps exactly it), + rename. A shared PATH.tmp let concurrent writers rename each other's file away (ENOENT). Last writer still wins: a JSON corpus is single-writer.



41
42
43
44
45
46
47
48
49
 
# File 'lib/iriq/storage.rb', line 41

def write_atomically(path, contents)
  tmp = "#{path}.#{Process.pid}.#{Storage.next_temp_seq}.tmp"
  File.write(tmp, contents)
  File.rename(tmp, path)
rescue SystemCallError => e
  raise CorpusError, "corpus #{path}: #{Iriq.os_error_message(e)}"
ensure
  File.delete(tmp) if tmp && File.exist?(tmp)
end