Class: Wp2txt::FtsIndex

Inherits:
Object
  • Object
show all
Defined in:
lib/wp2txt/fts_index.rb

Overview

Tier 2: contentless FTS5 full-text index over cleaned section text. Stores only the inverted index plus a rowid -> (page_id, heading, ord) mapping; the dump itself remains the single source of truth for text (snippets are re-rendered on demand via multistream random access). Queries ATTACH the Tier 1 metadata DB so category/section/redirect filters compose with MATCH in plain SQL.

Defined Under Namespace

Classes: ShortQueryError

Constant Summary collapse

SCHEMA_VERSION = 
2
CACHE_SUFFIX = 
"_fts.sqlite3"
TRIGRAM_LANGS =

Languages without word delimiters: use character-trigram tokenization

%w[ja zh ko yue wuu th km lo my bo].freeze
TOKENIZERS = 
%w[unicode61 trigram porter].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(db_path, meta_db_path) ⇒ FtsIndex

Returns a new instance of FtsIndex.



37
38
39
40
41
 
# File 'lib/wp2txt/fts_index.rb', line 37

def initialize(db_path, meta_db_path)
  @db_path = db_path
  @meta_db_path = meta_db_path
  @db = nil
end

Instance Attribute Details

#db_path ⇒ Object (readonly)

Returns the value of attribute db_path.



35
36
37
 
# File 'lib/wp2txt/fts_index.rb', line 35

def db_path
  @db_path
end

#meta_db_path ⇒ Object (readonly)

Returns the value of attribute meta_db_path.



35
36
37
 
# File 'lib/wp2txt/fts_index.rb', line 35

def meta_db_path
  @meta_db_path
end

Class Method Details

.current_render_digest ⇒ Object

Digest of everything that determines how section text is rendered. Snippets re-render sections at query time and must reproduce what was indexed; a digest mismatch means the index was built by code whose rendering differs from the current code.

Parameters:

  • optimize (Boolean) —

    merge FTS segments into one (single-threaded and slow on large indexes; skip and run #optimize! later if build time matters)



152
153
154
 
# File 'lib/wp2txt/fts_index.rb', line 152

def self.current_render_digest
  Digest::MD5.hexdigest(SectionRenderer::RENDER_CONFIG.inspect)[0, 12]
end

.default_tokenizer(multistream_path) ⇒ Object

Pick a tokenizer from the dump's language (e.g. "jawiki-..." -> trigram)



51
52
53
54
 
# File 'lib/wp2txt/fts_index.rb', line 51

def self.default_tokenizer(multistream_path)
  lang = File.basename(multistream_path)[/\A([a-z_-]+?)wiki/, 1]
  TRIGRAM_LANGS.include?(lang) ? "trigram" : "unicode61"
end

.path_for(multistream_path, cache_dir: nil) ⇒ Object 



43
44
45
46
47
48
 
# File 'lib/wp2txt/fts_index.rb', line 43

def self.path_for(multistream_path, cache_dir: nil)
  dir = cache_dir || File.expand_path("~/.wp2txt/cache")
  basename = File.basename(multistream_path, ".*").sub(/\.xml\z/, "")
  path_hash = Digest::MD5.hexdigest(multistream_path)[0, 8]
  File.join(dir, "#{basename}_#{path_hash}#{CACHE_SUFFIX}")
end

Instance Method Details

#built? ⇒ Boolean

Status

Returns:

  • (Boolean) 


60
61
62
63
64
65
66
67
 
# File 'lib/wp2txt/fts_index.rb', line 60

def built?
  return false unless File.exist?(@db_path)

  meta = read_metadata
  !meta.nil? && meta[:schema_version].to_i == SCHEMA_VERSION && !meta[:built_at].nil?
rescue SQLite3::Exception
  false
end

#close ⇒ Object 



94
95
96
97
 
# File 'lib/wp2txt/fts_index.rb', line 94

def close
  @db&.close
  @db = nil
end

#finalize_build!(source_path, optimize: true) ⇒ Object 



156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
 
# File 'lib/wp2txt/fts_index.rb', line 156

def finalize_build!(source_path, optimize: true)
  db = open_db
  db.execute("CREATE INDEX IF NOT EXISTS idx_fts_map_page ON fts_map(page_id)")
  db.execute("INSERT INTO fts_sections(fts_sections) VALUES('optimize')") if optimize
  stat = File.stat(source_path)
  save_metadata(
    schema_version: SCHEMA_VERSION,
    wp2txt_version: Wp2txt::VERSION,
    render_digest: self.class.current_render_digest,
    tokenizer: @pending_tokenizer,
    source_path: source_path,
    source_size: stat.size,
    source_mtime: stat.mtime.to_i,
    optimized: optimize,
    built_at: Time.now.utc.iso8601
  )
  close
  if @build_path
    File.rename(@build_path, @db_path)
    FileUtils.rm_f(["#{@db_path}-wal", "#{@db_path}-shm"])
    @build_path = nil
  end
end

#insert_batch(rows) ⇒ Object

rows: [[page_id, heading, ord, text], ...]



131
132
133
134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/wp2txt/fts_index.rb', line 131

def insert_batch(rows)
  db = open_db
  db.transaction do
    fts_stmt = db.prepare("INSERT INTO fts_sections (rowid, text) VALUES (?, ?)")
    map_stmt = db.prepare("INSERT INTO fts_map (rowid, page_id, heading, ord) VALUES (?, ?, ?, ?)")
    rows.each do |page_id, heading, ord, text|
      @next_rowid += 1
      fts_stmt.execute([@next_rowid, text])
      map_stmt.execute([@next_rowid, page_id, heading, ord])
    end
    fts_stmt.close
    map_stmt.close
  end
end

#optimize! ⇒ Object

Merge all FTS segments of an existing index (idempotent). Queries work without this, just somewhat slower; run once, any time after a --no-fts-optimize build.

Raises:

  • (ArgumentError) 


189
190
191
192
193
194
195
196
197
 
# File 'lib/wp2txt/fts_index.rb', line 189

def optimize!
  raise ArgumentError, "index not built" unless built?

  db = open_db
  db.execute("INSERT INTO fts_sections(fts_sections) VALUES('optimize')")
  db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
  save_metadata(optimized: true)
  true
end

#optimized? ⇒ Boolean

Returns:

  • (Boolean) 


199
200
201
 
# File 'lib/wp2txt/fts_index.rb', line 199

def optimized?
  read_metadata&.dig(:optimized) == "true"
end

#prepare_build!(tokenizer:) ⇒ Object

Build into a sidecar file and atomically rename in finalize_build!, so a failed multi-hour rebuild never destroys a working index

Raises:

  • (ArgumentError) 


105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
# File 'lib/wp2txt/fts_index.rb', line 105

def prepare_build!(tokenizer:)
  raise ArgumentError, "unknown tokenizer: #{tokenizer}" unless TOKENIZERS.include?(tokenizer)

  FileUtils.mkdir_p(File.dirname(@db_path))
  close
  @build_path = "#{@db_path}.building"
  FileUtils.rm_f([@build_path, "#{@build_path}-wal", "#{@build_path}-shm"])
  db = open_db
  db.execute("CREATE TABLE metadata (key TEXT PRIMARY KEY, value TEXT)")
  db.execute("CREATE VIRTUAL TABLE fts_sections USING fts5(text, content='', tokenize='#{tokenizer}')")
  db.execute(<<~SQL)
    CREATE TABLE fts_map (
      rowid INTEGER PRIMARY KEY,
      page_id INTEGER,
      -- heading: normalized like page_sections.heading; '' for the lead text
      heading TEXT,
      -- ord: section position in the article; 0 = lead text (stored here,
      -- unlike page_sections which starts at 1). Same numbering otherwise.
      ord INTEGER
    )
  SQL
  @pending_tokenizer = tokenizer
  @next_rowid = 0
end

#render_current? ⇒ Boolean

True when the current code's rendering configuration matches what this index was built with (snippet fidelity guarantee)

Returns:

  • (Boolean) 


182
183
184
 
# File 'lib/wp2txt/fts_index.rb', line 182

def render_current?
  read_metadata&.dig(:render_digest) == self.class.current_render_digest
end

#search(query, mode: "phrase", sections: nil, category: nil, depth: 0, limit: 20, offset: 0, count: "capped", count_cap: 1000) ⇒ Hash

Returns { total:, total_is_capped:, hits: [title:, heading:, ord:] }.

Parameters:

  • query (String) —

    search string

  • mode (String) (defaults to: "phrase") —

    "phrase" (literal, default) or "query" (raw FTS5 syntax)

  • sections (Array<String>, nil) (defaults to: nil) —

    restrict to these headings

  • category (String, nil) (defaults to: nil) —

    category scope (recursion via depth)

  • count (String) (defaults to: "capped") —

    "capped" (default; stops at count_cap) or "exact"

Returns:

  • (Hash) —

    { total:, total_is_capped:, hits: [title:, heading:, ord:] }



213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
 
# File 'lib/wp2txt/fts_index.rb', line 213

def search(query, mode: "phrase", sections: nil, category: nil, depth: 0,
           limit: 20, offset: 0, count: "capped", count_cap: 1000)
  if mode == "phrase" && tokenizer == "trigram" && query.length < 3
    raise ShortQueryError, "trigram phrase searches require at least 3 Unicode characters"
  end
  match_expr = mode == "query" ? query : phrase_query(query)

  conds = ["fts_sections MATCH ?", "p.namespace = 0", "p.redirect_to IS NULL"]
  params = [match_expr]
  cte = nil

  if category
    cte, cond, cat_params = category_condition(category, depth)
    conds << cond
    params = cat_params + params if cte     # CTE placeholders precede MATCH in SQL order
    params += cat_params unless cte
  end

  if sections && !sections.empty?
    placeholders = sections.map { "?" }.join(",")
    conds << "fm.heading COLLATE NOCASE IN (#{placeholders})"
    params += sections
  end

  base = "FROM fts_sections f JOIN fts_map fm ON fm.rowid = f.rowid " \
         "JOIN meta.pages p ON p.page_id = fm.page_id WHERE #{conds.join(' AND ')}"
  with = cte ? "WITH RECURSIVE #{cte} " : ""

  db = attached_db
  hits = db.execute(
    "#{with}SELECT fm.page_id, p.title, fm.heading, fm.ord #{base} ORDER BY rank LIMIT ? OFFSET ?",
    params + [limit, offset]
  ).map { |page_id, title, heading, ord| { page_id: page_id, title: title, heading: heading, ord: ord } }

  if count == "exact"
    total = db.get_first_value("#{with}SELECT COUNT(*) #{base}", params).to_i
    { total: total, total_is_capped: false, hits: hits }
  else
    capped = db.get_first_value(
      "#{with}SELECT COUNT(*) FROM (SELECT 1 #{base} LIMIT #{count_cap.to_i + 1})", params
    ).to_i
    { total: [capped, count_cap].min, total_is_capped: capped > count_cap, hits: hits }
  end
end

#stats ⇒ Object 



82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/wp2txt/fts_index.rb', line 82

def stats
  return nil unless File.exist?(@db_path)

  meta = read_metadata || {}
  { db_path: @db_path, db_size: File.size(@db_path),
    tokenizer: meta[:tokenizer], built_at: meta[:built_at],
    built_with: meta[:wp2txt_version],
    render_current: meta[:render_digest] == self.class.current_render_digest,
    optimized: meta[:optimized] == "true",
    section_count: (open_db.get_first_value("SELECT COUNT(*) FROM fts_map") rescue 0) }
end

#tokenizer ⇒ Object 



78
79
80
 
# File 'lib/wp2txt/fts_index.rb', line 78

def tokenizer
  read_metadata&.dig(:tokenizer)
end

#valid_for?(multistream_path) ⇒ Boolean

Returns:

  • (Boolean) 


69
70
71
72
73
74
75
76
 
# File 'lib/wp2txt/fts_index.rb', line 69

def valid_for?(multistream_path)
  return false unless built?
  return false unless File.exist?(multistream_path)

  meta = read_metadata
  stat = File.stat(multistream_path)
  meta[:source_size].to_i == stat.size && meta[:source_mtime].to_i == stat.mtime.to_i
end