Class: Iriq::Corpus

Inherits:
Object
  • Object
show all
Defined in:
lib/iriq/corpus.rb

Overview

Streaming-friendly observer over a (potentially unbounded) corpus of IRIs. Maintains rolling aggregates and per-(host, prefix) frequency stats so that classification can improve as more data flows in.

The deterministic, single-IRI API (Iriq.normalize/explain) is unchanged — Corpus#normalize and Corpus#explain are the corpus-informed variants.

State lives in a Storage backend (Memory by default; Json or Sqlite when opened against a file). The classification logic on top is identical regardless of where the counters live.

Defined Under Namespace

Classes: StaleRebuild

Constant Summary collapse

VARIABLE_DOMINANCE_THRESHOLD =

Type-based: position is "mostly variable" (UUIDs/integers/etc.).

0.8
LITERAL_UNIQUENESS_THRESHOLD =

Cardinality-based: position has mostly distinct literal values, so the literal "type" is misleading — it's really a variable slot. We trigger on either:

- very high cardinality fraction (most observations are singletons), OR
- moderate cardinality fraction AND high absolute distinct count

The second branch catches realistic streams where popular outliers bring the frac down but the long tail is clearly variable.

0.8
LITERAL_UNIQUENESS_MODERATE_THRESHOLD = 
0.5
MIN_CARDINALITY_FOR_INFERENCE = 
20
MIN_OBSERVATIONS_FOR_INFERENCE =

Don't apply corpus heuristics until we have at least this many observations at a position — too easy to be wrong with tiny samples.

5
STABLE_LITERAL_THRESHOLD =

Value-fraction at or above which a literal is considered the stable occupant of its position.

0.5

5

3
HOST_STRATEGIES = 
i[full registrable none].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(classifier: SegmentClassifier::DEFAULT, max_values_per_position: PositionStats::DEFAULT_MAX_VALUES, host_strategy: :full, storage: nil) ⇒ Corpus

Returns a new instance of Corpus.

Raises:

  • (ArgumentError) 


49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/iriq/corpus.rb', line 49

def initialize(classifier: SegmentClassifier::DEFAULT,
               max_values_per_position: PositionStats::DEFAULT_MAX_VALUES,
               host_strategy: :full,
               storage: nil)
  raise ArgumentError, "host_strategy must be one of #{HOST_STRATEGIES.inspect}" \
    unless HOST_STRATEGIES.include?(host_strategy)

  @classifier    = classifier
  # Stored activations layer onto the base; @activations is the set
  # @classifier was built from.
  @base_classifier = classifier
  @activations     = []
  @host_strategy = host_strategy
  @storage       = storage || Storage::Memory.new(
    classifier: classifier,
    max_values_per_position: max_values_per_position,
  )
end

Instance Attribute Details

#classifier ⇒ Object (readonly)

Returns the value of attribute classifier.



47
48
49
 
# File 'lib/iriq/corpus.rb', line 47

def classifier
  @classifier
end

#host_strategy ⇒ Object (readonly)

Returns the value of attribute host_strategy.



47
48
49
 
# File 'lib/iriq/corpus.rb', line 47

def host_strategy
  @host_strategy
end

#storage ⇒ Object (readonly)

Returns the value of attribute storage.



47
48
49
 
# File 'lib/iriq/corpus.rb', line 47

def storage
  @storage
end

Class Method Details

.from_dump(h, classifier: SegmentClassifier::DEFAULT) ⇒ Object 



683
684
685
686
687
688
 
# File 'lib/iriq/corpus.rb', line 683

def self.from_dump(h, classifier: SegmentClassifier::DEFAULT)
  max_values = h.fetch("max_values_per_position", PositionStats::DEFAULT_MAX_VALUES)
  storage = Storage::Memory.new(classifier: classifier, max_values_per_position: max_values)
  storage.load_dump!(h)
  new(classifier: classifier, storage: storage)
end

.load(path, classifier: SegmentClassifier::DEFAULT) ⇒ Object 



690
691
692
 
# File 'lib/iriq/corpus.rb', line 690

def self.load(path, classifier: SegmentClassifier::DEFAULT)
  open(path, classifier: classifier)
end

.open(path, classifier: SegmentClassifier::DEFAULT, max_values_per_position: PositionStats::DEFAULT_MAX_VALUES, host_strategy: :full) ⇒ Object

Open a corpus against path. File extension picks the backend: .db/.sqlite/.sqlite3 use SQLite (incremental writes); anything else uses JSON.



71
72
73
74
75
76
77
78
79
80
 
# File 'lib/iriq/corpus.rb', line 71

def self.open(path, classifier: SegmentClassifier::DEFAULT,
                    max_values_per_position: PositionStats::DEFAULT_MAX_VALUES,
                    host_strategy: :full)
  storage = Storage.open(path,
                         classifier: classifier,
                         max_values_per_position: max_values_per_position)
  corpus = new(classifier: classifier, storage: storage, host_strategy: host_strategy)
  corpus.send(:reapply_activated_recognizers!) if storage.respond_to?(:each_activated_recognizer)
  corpus
end

Instance Method Details

#activate_proposal(proposal) ⇒ Object

Promote a RecognizerProposal into a live Recognizer for this corpus: store the activation, then reinfer existing observations through it. Both commit in one transaction — a failure leaves neither behind, in storage or in this corpus's classifier. Activating a recognizer the corpus already holds changes nothing.

Returns the synthesized Recognizer.



176
177
178
179
180
 
# File 'lib/iriq/corpus.rb', line 176

def activate_proposal(proposal)
  recognizer = SynthesizedRecognizer.from_proposal(proposal)
  activate(recognizer)
  recognizer
end

#activate_proposals_above(confidence_threshold, **propose_opts) ⇒ Object

Convenience: activate every proposal whose confidence clears the given threshold. Returns the Recognizers this call newly activated. Confidence incorporates both per-position coverage AND cross-host corroboration — see RecognizerProposal#compute_confidence.



186
187
188
189
190
191
 
# File 'lib/iriq/corpus.rb', line 186

def activate_proposals_above(confidence_threshold, **propose_opts)
  propose_recognizers(**propose_opts)
    .select { |p| p.confidence >= confidence_threshold }
    .map { |p| SynthesizedRecognizer.from_proposal(p) }
    .select { |r| activate(r) }
end

#activated_recognizer_count ⇒ Object

Number of activated recognizers persisted with this corpus.



194
195
196
197
 
# File 'lib/iriq/corpus.rb', line 194

def activated_recognizer_count
  return @storage.activated_recognizer_count if @storage.respond_to?(:activated_recognizer_count)
  0
end

#batch ⇒ Object

Wrap many observations in a single backend transaction. For SQLite this turns thousands of fsyncs into one; for in-memory backends it's a no-op. Use when ingesting a batch.

Every write runs in one, and classifies with exactly the activations stored as of its start — including any another connection committed.



388
389
390
391
392
393
 
# File 'lib/iriq/corpus.rb', line 388

def batch
  @storage.batch do |changed|
    reapply_activated_recognizers! if changed
    yield
  end
end

#close ⇒ Object 



378
379
380
 
# File 'lib/iriq/corpus.rb', line 378

def close
  @storage.close
end

#clusters ⇒ Object 



346
347
348
 
# File 'lib/iriq/corpus.rb', line 346

def clusters
  @storage.clusters
end

#cross_host_shapes(min_hosts: 2) ⇒ Object

Route shapes that recur across min_hosts or more distinct hosts. Returns CrossHostShape records sorted by host_count desc, then by observation_count desc, then by shape (stable, deterministic).

Cross-host recurrence is independent evidence of a real semantic pattern — two unrelated hosts inventing the same /users/{integer} structure by accident is unlikely. A natural follow-up is feeding this signal back into RecognizerProposal confidence: a proposal supported by N hosts is much stronger than one seen on a single host with the same per-position coverage.



209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
 
# File 'lib/iriq/corpus.rb', line 209

def cross_host_shapes(min_hosts: 2)
  by_shape = Hash.new { |h, k| h[k] = { hosts: Set.new, count: 0 } }
  @storage.clusters.each do |cluster|
    # Skip non-URL clusters (URN clusters have no host).
    next if cluster.host.nil? || cluster.host.empty?

    agg = by_shape[cluster.shape]
    agg[:hosts] << cluster.host
    agg[:count] += cluster.count
  end

  by_shape.filter_map do |shape, data|
    next nil if data[:hosts].size < min_hosts

    CrossHostShape.new(
      shape:             shape,
      hosts:             data[:hosts],
      observation_count: data[:count],
    )
  end.sort_by { |s| [-s.host_count, -s.observation_count, s.shape] }
end

#dump ⇒ Object

--- Legacy dump/load (JSON shape) ------------------------------------

The pre-Storage release exposed Corpus#dump, Corpus#save(path), and Corpus.load(path) for JSON-backed persistence. Those names still work but are now thin wrappers around the appropriate Storage backend.



679
680
681
 
# File 'lib/iriq/corpus.rb', line 679

def dump
  memory_view.to_dump
end

#each_position_stats(&block) ⇒ Object

Iterates Position → PositionStats over all observed positions. Used by inspection tooling; not part of the hot path.



342
343
344
 
# File 'lib/iriq/corpus.rb', line 342

def each_position_stats(&block)
  @storage.each_position_stats(&block)
end

#effective_host(host) ⇒ Object

Normalize the host for keying purposes. :full keeps the original host; :registrable collapses subdomains via the inline-PSL heuristic (api.foo.com + app.foo.com → foo.com); :none ignores host entirely so clusters group across all hosts by shape alone.



86
87
88
89
90
91
92
 
# File 'lib/iriq/corpus.rb', line 86

def effective_host(host)
  case @host_strategy
  when :registrable then RegistrableDomain.for(host)
  when :none        then ""
  else                   host
  end
end

#events_for(input) ⇒ Object

Build the ordered Event list for input without applying it. Useful for inspection, tests, and future event-log persistence. Each call is pure — no storage side-effects.



234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
 
# File 'lib/iriq/corpus.rb', line 234

def events_for(input)
  iri = coerce(input)
  hinted_entries = SegmentHints.derive(iri.path_segments, @classifier)
  shape        = Shape.from_entries(hinted_entries)
  raw_shape    = shape.render(hints: false)
  hinted_shape = shape.render(hints: true)
  keying_host  = effective_host(iri.host)

  events = [
    Event::HostSeen.new(keying_host),
    Event::PathLengthSeen.new(iri.path_segments.size),
    Event::RawShapeSeen.new(raw_shape),
    Event::FingerprintSeen.new(hinted_shape),
  ]

  prefix = ""
  hinted_entries.each do |entry|
    events << Event::PositionSeen.new(
      Position.path(host: keying_host, prefix: prefix),
      entry[:value], entry[:type],
    )
    prefix = "#{prefix}/#{placeholder(entry)}"
  end

  key, host, scheme, shape = Cluster.key_for(iri, classifier: @classifier, shape: hinted_shape, host: keying_host)
  events << Event::ClusterAddition.new(key, host, scheme, shape, iri)

  events
end

#explain(input) ⇒ Object

Per-segment explanation with corpus-informed classification. Returns an array of entries shaped like the Explanation rows plus classification: ∈ :stable_literal, :variable_identifier, :rare_literal, :ambiguous, :corpus_inferred_variable.



328
329
330
331
332
333
 
# File 'lib/iriq/corpus.rb', line 328

def explain(input)
  iri = coerce(input)
  annotate_segments(iri).map do |entry|
    entry.reject { |k, _| k == :prefix }
  end
end

#fingerprint_counts ⇒ Object 



338
 
# File 'lib/iriq/corpus.rb', line 338

def fingerprint_counts; @storage.fingerprint_counts; end

#host_counts ⇒ Object 



335
 
# File 'lib/iriq/corpus.rb', line 335

def host_counts;        @storage.host_counts;        end

#normalize(input, hints: true) ⇒ Object

Corpus-informed normalization. Falls back to mechanical normalization when the corpus has no signal for a position. Implemented as a thin call into Normalizer with evidence: self; the corpus-informed path and query rendering live in #render_path / #render_query below (the evidence-source interface).



269
270
271
272
 
# File 'lib/iriq/corpus.rb', line 269

def normalize(input, hints: true)
  iri = coerce(input)
  Normalizer.normalize_identifier(iri, classifier: @classifier, hints: hints, evidence: self)
end

#observe(input) ⇒ Object

Observe a single IRI. Returns an Observation.

Internally: builds an Event list for the IRI, then applies each event through the Reducer registry inside a single storage transaction. The event list is transient today — a future commit can persist it and replay against alternate reducers / thresholds for re-runnable inference. See lib/iriq/event.rb and lib/iriq/reducer.rb.



101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/iriq/corpus.rb', line 101

def observe(input)
  iri = coerce(input)

  addition = batch do
    # Derived inside the batch, which may have just adopted activations.
    events = events_for(iri)
    events.each { |e| Reducer.apply(e, @storage) }
    @storage.record_observation(iri.canonical) if @storage.respond_to?(:record_observation)
    events.find { |e| e.is_a?(Event::ClusterAddition) }
  end

  Observation.new(corpus: self, identifier: iri, cluster_key: addition.key)
end

#observe_all(iris) ⇒ Object

Observe every IRI. On SQLite they commit a turn of about a second at a time, so other processes writing the corpus get the lock in between; a failure keeps the turns already committed. Inside #batch they all join its transaction.



119
120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/iriq/corpus.rb', line 119

def observe_all(iris)
  done = 0
  while done < iris.size
    batch do
      while done < iris.size
        observe(iris[done])
        done += 1
        break if @storage.turn_over?
      end
    end
  end
  nil
end

#observed_iri_count ⇒ Object

Number of IRIs in the source-IRI log. The materialized views are derived from this log; reinfer replays it.



151
152
153
154
 
# File 'lib/iriq/corpus.rb', line 151

def observed_iri_count
  return @storage.observed_iri_count if @storage.respond_to?(:observed_iri_count)
  0
end

#params_for(input) ⇒ Object

Inferred params for the cluster input would fall into. Returns the same shape as Cluster#param_summary — useful for "what query params might this URL accept?" tooling. Empty array if no cluster has been observed for this shape yet.



314
315
316
317
318
319
320
321
322
 
# File 'lib/iriq/corpus.rb', line 314

def params_for(input)
  iri = coerce(input)
  hinted_shape = PathShape.new(classifier: @classifier, hints: true)
                          .from_entries(SegmentHints.derive(iri.path_segments, @classifier))
  key, * = Cluster.key_for(iri, classifier: @classifier, shape: hinted_shape,
                           host: effective_host(iri.host))
  cluster = @storage.cluster_for(key)
  cluster ? cluster.param_summary : []
end

#path_length_counts ⇒ Object 



336
 
# File 'lib/iriq/corpus.rb', line 336

def path_length_counts; @storage.path_length_counts; end

#propose_recognizers(strategies: ProposalStrategy::DEFAULTS, **opts) ⇒ Object

Scan observed values for shape patterns that recur frequently enough to suggest a new Recognizer. Returns RecognizerProposal records; nothing is automatically applied — the proposal carries enough evidence for a human to decide whether to bake the Recognizer in.

Strategies are pluggable; the default set lives in Iriq::ProposalStrategy::DEFAULTS. Pass strategies: to limit / extend. Pass min_observations: / min_coverage: / min_hosts: to tune what passes the noise floor.



165
166
167
 
# File 'lib/iriq/corpus.rb', line 165

def propose_recognizers(strategies: ProposalStrategy::DEFAULTS, **opts)
  strategies.flat_map { |s| s.propose(@storage, **opts) }
end

#raw_shape_counts ⇒ Object 



337
 
# File 'lib/iriq/corpus.rb', line 337

def raw_shape_counts;   @storage.raw_shape_counts;   end

#reinfer ⇒ Object

Rebuild every materialized view (host counts, position stats, clusters, …) by replaying the source-IRI log through the current events + reducers pipeline. Useful for:

- Tuning thresholds (swap a Corpus constant, call reinfer)
- Swapping the classifier (open the Corpus with a different
classifier, call reinfer

The views change all at once, keeping what other connections observe meanwhile; a failure leaves the prior views intact.



144
145
146
147
 
# File 'lib/iriq/corpus.rb', line 144

def reinfer
  rebuild(nil)
  nil
end

#render_path(iri, _classifier, hints) ⇒ Object

Evidence-source interface — called by Normalizer when this Corpus is passed as evidence:. Renders the path using corpus-informed classifications (variability promotion, popular-outlier preservation). Always emits a leading "/" — empty path collapses to "/" to match mechanical output and anchor any trailing query.



279
280
281
282
 
# File 'lib/iriq/corpus.rb', line 279

def render_path(iri, _classifier, hints)
  tokens = annotate_segments(iri).map { |entry| corpus_token(entry, hints) }
  "/" + tokens.join("/")
end

#render_query(iri, _classifier = @classifier) ⇒ Object

Evidence-source interface — render the query string. A param the cluster has seen at least MIN_OBSERVATIONS_FOR_INFERENCE times renders with the cluster's type; below that the corpus has no opinion and the param renders exactly as mechanical normalize would.



288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
 
# File 'lib/iriq/corpus.rb', line 288

def render_query(iri, _classifier = @classifier)
  return "" if iri.query_params.nil? || iri.query_params.empty?

  hinted_shape = PathShape.new(classifier: @classifier, hints: true)
                          .from_entries(SegmentHints.derive(iri.path_segments, @classifier))
  key, * = Cluster.key_for(iri, classifier: @classifier, shape: hinted_shape,
                           host: effective_host(iri.host))
  mechanical = NullEvidenceSource.new

  iri.query_params.keys.sort.map do |k|
    v     = iri.query_params[k].to_s
    stats = @storage.param_stats(key, k)
    shaped =
      if stats && stats.total >= MIN_OBSERVATIONS_FOR_INFERENCE
        render_param_value(v, Cluster.param_type_for(k, stats) || @classifier.classify(v))
      else
        mechanical.render_param(k, v, @classifier)
      end
    "#{k}=#{shaped}"
  end.join("&")
end

#save(path = nil) ⇒ Object

Persist the corpus.

save()           


369
370
371
372
373
374
375
376
 
# File 'lib/iriq/corpus.rb', line 369

def save(path = nil)
  backend_path = @storage.respond_to?(:path) ? @storage.path : nil
  if path.nil? || path == backend_path
    @storage.save
  else
    write_json_dump(path)
  end
end

#size ⇒ Object 



350
351
352
 
# File 'lib/iriq/corpus.rb', line 350

def size
  @storage.cluster_size
end

#stats_for(host_or_position, prefix = nil) ⇒ Object

Stats for a given (host, path-prefix) — useful for tests and debugging. Returns nil if nothing has been observed there. Accepts either a Position or (host, prefix) for ergonomics.



357
358
359
360
 
# File 'lib/iriq/corpus.rb', line 357

def stats_for(host_or_position, prefix = nil)
  position = host_or_position.is_a?(Position) ? host_or_position : Position.path(host: host_or_position, prefix: prefix)
  @storage.position_stats(position)
end