Module: OpenLoam::CustomFieldIndex

Defined in:
lib/open_loam/custom_field_index.rb

Overview

The read-model index for custom fields (OpenLoam::CustomFieldValue) — a typed-EAV projection that makes custom-field filter/sort/search index-backed.

OpenLoam::CustomFieldIndex.filter(DamageReport, "severity", "eq", "critical")  # a relation
OpenLoam::CustomFieldIndex.order(DamageReport, "severity", :asc)
OpenLoam::CustomFieldIndex.reindex(DamageReport)                               # rebuild

SAFETY: field_key must name a real OpenLoam::FieldDefinition (no arbitrary keys); values are cast to the declared type; queries are tenant-scoped throughout (index rows carry tenant_id, the model relation is default-scoped), so a filter can't reach across tenants. Encrypted data lives in real columns, never in a custom field — nothing encrypted is projected here.

Constant Summary collapse

OPS = 
%w[eq neq gt gte lt lte contains present blank].freeze
NUMERIC_PREDICATES =

Numeric comparisons hit the indexed value_number column. Built through Arel rather than an interpolated fragment: op reaches here from params, and a comparison operator cannot be a bound parameter, so the only safe form is one that never becomes a string.

{ "gt" => :gt, "gte" => :gteq, "lt" => :lt, "lte" => :lteq }.freeze

Class Method Summary collapse

Class Method Details

.base(model) ⇒ Object

--- internals ---



190
191
192
 
# File 'lib/open_loam/custom_field_index.rb', line 190

def base(model)
  OpenLoam::CustomFieldValue.where(indexable_type: index_type(model)) # tenant-scoped by default_scope
end

.clear_pending(tenant_id, model_name) ⇒ Object 



153
154
155
 
# File 'lib/open_loam/custom_field_index.rb', line 153

def clear_pending(tenant_id, model_name)
  pending_reindex.delete([ tenant_id, model_name ])
end

.coverage(model, field_key) ⇒ Object

How complete the index is for a field: how many live records HAVE a value (the authoritative JSON column) vs how many index rows exist. The trust signal — "9,980 of 10,000 indexed" tells an operator whether the index is complete or drifting (legacy data, a bulk write that bypassed the hook, a field def added before a backfill). Counting expected does a JSON pass, so it's a periodic READOUT, never a per-row query cost.



109
110
111
112
113
114
115
 
# File 'lib/open_loam/custom_field_index.rb', line 109

def coverage(model, field_key)
  field_key = field_key.to_s
  exp = expected(model, field_key)
  got = indexed(model, field_key)
  { field_key: field_key, expected: exp, indexed: got, complete: got >= exp,
    ratio: exp.zero? ? 1.0 : (got.to_f / exp) }
end

.covered?(model, field_key) ⇒ Boolean

Returns:

  • (Boolean) 


117
118
119
 
# File 'lib/open_loam/custom_field_index.rb', line 117

def covered?(model, field_key)
  coverage(model, field_key)[:complete]
end

.expected(model, field_key) ⇒ Object 



125
126
127
128
129
130
 
# File 'lib/open_loam/custom_field_index.rb', line 125

def expected(model, field_key)
  return 0 unless model.column_names.include?("custom_fields")

  key = field_key.to_s
  model.pluck(:custom_fields).count { |cf| cf.is_a?(Hash) && cf[key].to_s.strip != "" }
end

.filter(model, field_key, op, value = nil) ⇒ Object

--- query API ---

Raises:



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
 
# File 'lib/open_loam/custom_field_index.rb', line 46

def filter(model, field_key, op, value = nil)
  field_key = field_key.to_s
  refuse_unknown_field!(model, field_key)
  refuse_unreadable_field!(model, field_key)
  raise OpenLoam::Error, "unknown custom-field op #{op.inspect}" unless OPS.include?(op.to_s)

  if covered?(model, field_key)
    self.partial = false
    index_filter(model, field_key, op, value)
  else
    # CORRECTNESS OVER SPEED: the index is known-incomplete for this field
    # (coverage gap), so serve an authoritative JSON-scan result — never a
    # silently-partial index set — AND enqueue a background reindex (deduped)
    # to heal the gap so subsequent queries are fast again. `partial?` lets
    # the caller surface an honest "results may be incomplete, reindexing…".
    self.partial = true
    schedule_reindex(model)
    json_filter(model, field_key, op, value)
  end
end

.index_filter(model, field_key, op, value) ⇒ Object

The index-backed relation (used when coverage is complete).



68
69
70
71
72
73
74
75
76
77
 
# File 'lib/open_loam/custom_field_index.rb', line 68

def index_filter(model, field_key, op, value)
  rows = base(model).where(field_key: field_key)

  if op.to_s == "blank"
    present_ids = rows.where.not(value_text: [ nil, "" ]).select(:indexable_id)
    return model.where.not(id: present_ids)
  end

  model.where(id: predicate(rows, op.to_s, value).select(:indexable_id))
end

.index_type(model_or_class) ⇒ Object 



194
195
196
197
 
# File 'lib/open_loam/custom_field_index.rb', line 194

def index_type(model_or_class)
  klass = model_or_class.is_a?(Class) ? model_or_class : model_or_class.class
  klass.base_class.name
end

.indexed(model, field_key) ⇒ Object 



121
122
123
 
# File 'lib/open_loam/custom_field_index.rb', line 121

def indexed(model, field_key)
  base(model).where(field_key: field_key.to_s).where.not(value_text: [ nil, "" ]).count
end

.json_filter(model, field_key, op, value) ⇒ Object

Authoritative correct filter over the JSON column — DB-agnostic (reads each record's cast custom_field), used only as the fallback when the index is incomplete. Slow (a scan), which is exactly why the heal runs.



168
169
170
171
 
# File 'lib/open_loam/custom_field_index.rb', line 168

def json_filter(model, field_key, op, value)
  ids = model.find_each.select { |record| json_match?(record, field_key, op, value) }.map(&:id)
  model.where(id: ids)
end

.json_match?(record, field_key, op, value) ⇒ Boolean

Returns:

  • (Boolean) 


173
174
175
176
177
178
179
180
181
182
183
184
185
186
 
# File 'lib/open_loam/custom_field_index.rb', line 173

def json_match?(record, field_key, op, value)
  actual = (record.custom_field(field_key) rescue nil)
  case op.to_s
  when "eq"       then actual.to_s == value.to_s
  when "neq"      then actual.to_s != value.to_s
  when "present"  then actual.to_s.strip != ""
  when "blank"    then actual.to_s.strip == ""
  when "contains" then actual.to_s.include?(value.to_s)
  when "gt"       then actual.to_f > value.to_f
  when "gte"      then actual.to_f >= value.to_f
  when "lt"       then actual.to_f < value.to_f
  when "lte"      then actual.to_f <= value.to_f
  end
end

.numeric(rows, value, op) ⇒ Object 



215
216
217
218
 
# File 'lib/open_loam/custom_field_index.rb', line 215

def numeric(rows, value, op)
  column = OpenLoam::CustomFieldValue.arel_table[:value_number]
  rows.where(column.public_send(NUMERIC_PREDICATES.fetch(op), value.to_f))
end

.order(model, field_key, dir = :asc) ⇒ Object

Order a model's records by an indexed custom field (value_text — good for dictionary/string fields, the common case; numeric-aware ordering is a follow-up). Records with no value sort last (LEFT JOIN + NULLs).



82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/open_loam/custom_field_index.rb', line 82

def order(model, field_key, dir = :asc)
  refuse_unknown_field!(model, field_key.to_s)
  refuse_unreadable_field!(model, field_key.to_s)
  # The LEFT JOIN keeps EVERY record (un-indexed ones sort as NULL), so an
  # incomplete index mis-orders the gap but never drops a row. Flag it and
  # heal in the background.
  unless covered?(model, field_key.to_s)
    self.partial = true
    schedule_reindex(model)
  end
  table = OpenLoam::CustomFieldValue.table_name
  conn = model.connection
  join = "LEFT JOIN #{table} ON #{table}.indexable_type = #{conn.quote(index_type(model))} " \
         "AND #{table}.indexable_id = #{model.table_name}.id " \
         "AND #{table}.field_key = #{conn.quote(field_key.to_s)} " \
         "AND #{table}.tenant_id = #{conn.quote(OpenLoam.tenant!.id)}"
  model.joins(join).order(Arel.sql("#{table}.value_text #{dir.to_s.casecmp('desc').zero? ? 'DESC' : 'ASC'}"))
end

.partial=(value) ⇒ Object 



138
139
140
 
# File 'lib/open_loam/custom_field_index.rb', line 138

def partial=(value)
  Thread.current[:open_loam_index_partial] = value
end

.partial? ⇒ Boolean

Was the last filter/order served over an incomplete index? The admin surfaces this as an honest "results may be incomplete, reindexing…".

Returns:

  • (Boolean) 


134
135
136
 
# File 'lib/open_loam/custom_field_index.rb', line 134

def partial?
  Thread.current[:open_loam_index_partial] == true
end

.pending_reindex ⇒ Object 



161
162
163
 
# File 'lib/open_loam/custom_field_index.rb', line 161

def pending_reindex
  @pending_reindex ||= Set.new
end

.predicate(rows, op, value) ⇒ Object 



199
200
201
202
203
204
205
206
207
 
# File 'lib/open_loam/custom_field_index.rb', line 199

def predicate(rows, op, value)
  case op
  when "eq"       then rows.where(value_text: value.to_s)
  when "neq"      then rows.where.not(value_text: value.to_s)
  when "contains" then rows.where("value_text LIKE ?", "%#{sanitize_like(value)}%")
  when "present"  then rows.where.not(value_text: [ nil, "" ])
  when "gt", "gte", "lt", "lte" then numeric(rows, value, op)
  end
end

.project(record) ⇒ Object

Re-project one record's custom fields (delete + insert). Soft-delete keeps the rows (the filter's base scope hides the record anyway — the same decision as the L-912 search token driver); a hard destroy removes them.



24
25
26
27
28
29
30
31
32
33
 
# File 'lib/open_loam/custom_field_index.rb', line 24

def project(record)
  model = record.class
  return unless model.respond_to?(:custom_field_definitions)

  type = index_type(model)
  OpenLoam::CustomFieldValue.where(indexable_type: type, indexable_id: record.id).delete_all

  rows = model.custom_field_definitions.filter_map { |definition| row_for(record, definition, type) }
  OpenLoam::CustomFieldValue.insert_all(rows) if rows.any?
end

.refuse_unknown_field!(model, field_key) ⇒ Object

Raises:



239
240
241
242
 
# File 'lib/open_loam/custom_field_index.rb', line 239

def refuse_unknown_field!(model, field_key)
  known = model.respond_to?(:custom_field_definitions) && model.custom_field_definitions.exists?(name: field_key)
  raise OpenLoam::Error, "no custom field #{field_key.inspect} on #{model.name}" unless known
end

.refuse_unreadable_field!(model, field_key) ⇒ Object

THE oracle guard: filtering or sorting on a custom field the current role may not read would let a user infer restricted values by observing which records match. Enforced against the current actor's role; a system/background context (no actor — reindex, a business rule) is trusted and not gated.

Raises:



248
249
250
251
252
253
254
255
256
257
258
259
 
# File 'lib/open_loam/custom_field_index.rb', line 248

def refuse_unreadable_field!(model, field_key)
  actor = OpenLoam::Current.actor
  return if actor.nil?

  definition = model.custom_field_definitions.find_by(name: field_key.to_s)
  return if definition.nil? # unknown field is refuse_unknown_field!'s job

  role = OpenLoam::Membership.find_by(user_id: actor.id)&.role
  return if definition.readable_by?(role)

  raise OpenLoam::FieldAccessError, "custom field #{field_key.inspect} is not readable by #{role.inspect}"
end

.reindex(model) ⇒ Object 



39
40
41
42
 
# File 'lib/open_loam/custom_field_index.rb', line 39

def reindex(model)
  OpenLoam::CustomFieldValue.where(indexable_type: index_type(model)).delete_all
  model.find_each { |record| project(record) }
end

.remove(record) ⇒ Object 



35
36
37
 
# File 'lib/open_loam/custom_field_index.rb', line 35

def remove(record)
  OpenLoam::CustomFieldValue.where(indexable_type: index_type(record.class), indexable_id: record.id).delete_all
end

.reset_pending! ⇒ Object 



157
158
159
 
# File 'lib/open_loam/custom_field_index.rb', line 157

def reset_pending!
  @pending_reindex = Set.new
end

.row_for(record, definition, type) ⇒ Object 



220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
 
# File 'lib/open_loam/custom_field_index.rb', line 220

def row_for(record, definition, type)
  value = (record.custom_field(definition.name) rescue nil)
  return nil if value.nil?

  # ALL value columns are present (nil when unused) so insert_all sees a
  # uniform key set across every row.
  row = {
    tenant_id: record.tenant_id, indexable_type: type, indexable_id: record.id,
    field_key: definition.name, value_text: value.to_s,
    value_number: nil, value_boolean: nil, value_datetime: nil
  }
  case definition.field_type
  when "integer", "decimal" then row[:value_number] = value.to_f
  when "boolean"            then row[:value_boolean] = ActiveModel::Type::Boolean.new.cast(value)
  when "date", "datetime"   then row[:value_datetime] = (Time.zone.parse(value.to_s) rescue nil)
  end
  row
end

.sanitize_like(value) ⇒ Object 



261
262
263
 
# File 'lib/open_loam/custom_field_index.rb', line 261

def sanitize_like(value)
  value.to_s.gsub(/[\\%_]/) { |c| "\\#{c}" }
end

.schedule_reindex(model) ⇒ Object

Enqueue a background reindex to heal a gap — DEDUPED so a hot gappy field doesn't enqueue one per request. In-process dedup here (single-process prototype); a DB/cache marker is the multi-process path.



145
146
147
148
149
150
151
 
# File 'lib/open_loam/custom_field_index.rb', line 145

def schedule_reindex(model)
  key = [ OpenLoam.tenant!.id, model.base_class.name ]
  return if pending_reindex.include?(key)

  pending_reindex << key
  OpenLoam::CustomFieldReindexJob.perform_later(OpenLoam.tenant!.id, model.base_class.name)
end