Class: Iriq::Storage::Memory

Inherits:
Object
  • Object
show all
Defined in:
lib/iriq/storage/memory.rb

Overview

Memory is the canonical backend — every other backend either wraps it (Json) or implements the same surface against an external store (Sqlite).

The contract is small enough to enumerate up top:

increment_host(host)
increment_path_length(length)
increment_raw_shape(shape)
increment_fingerprint(shape)
observe_position(position, value, type)        # position is Iriq::Position
add_to_cluster(key, host, scheme, shape, identifier)
record_observation(canonical)                  # append to source-IRI log

host_counts / path_length_counts / raw_shape_counts / fingerprint_counts
position_stats(position)
position_evidence(position, value)             # the narrow read normalize uses
param_stats(cluster_key, name)                 # one param, without the cluster
each_position_stats { |position, stats| ... }
each_observed_iri { |canonical| ... }
each_observed_iri_since(mark) { |canonical| ... } # → mark of the last
clear_materialized_views
begin_rebuild / install_rebuild / discard_rebuild # for reinfer
clusters / cluster_size

transaction { ... }    # backends may batch within
turn_over?             # a long batch should commit and let others in
flush                  # commit pending writes (no-op for Memory)
close                  # release resources

Direct Known Subclasses

Json

Constant Summary collapse

DUMP_KEYS =

Top-level keys of the dump. A JSON file with none of them isn't a corpus.

%w[host_counts path_length_counts raw_shape_counts fingerprint_counts
max_values_per_position position_stats clusterer observed_iris
activated_recognizers].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(classifier: SegmentClassifier::DEFAULT, max_values_per_position: PositionStats::DEFAULT_MAX_VALUES) ⇒ Memory

Returns a new instance of Memory.



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/iriq/storage/memory.rb', line 38

def initialize(classifier: SegmentClassifier::DEFAULT,
               max_values_per_position: PositionStats::DEFAULT_MAX_VALUES)
  @classifier              = classifier
  @max_values_per_position = max_values_per_position
  @host_counts             = Hash.new(0)
  @path_length_counts      = Hash.new(0)
  @raw_shape_counts        = Hash.new(0)
  @fingerprint_counts      = Hash.new(0)
  @position_stats          = {}
  @clusters                = {}
  # The source-IRI log. Persisted alongside materialized views; the
  # log is the source of truth, the views are derived. Corpus#reinfer
  # drops the views and replays the log through events + reducers.
  @observed_iris           = []
  # Recognizers promoted from RecognizerProposal via
  # Corpus#activate_proposal. Stored as {prefix, type, specificity}
  # hashes so reopens can re-synthesize them onto the corpus's
  # classifier.
  @activated_recognizers   = []
end

Instance Attribute Details

#max_values_per_position ⇒ Object (readonly)

Returns the value of attribute max_values_per_position.



32
33
34
 
# File 'lib/iriq/storage/memory.rb', line 32

def max_values_per_position
  @max_values_per_position
end

Instance Method Details

#activated_recognizer_count ⇒ Object 



143
144
145
 
# File 'lib/iriq/storage/memory.rb', line 143

def activated_recognizer_count
  @activated_recognizers.size
end

#add_to_cluster(key, host, scheme, shape, identifier) ⇒ Object 



102
103
104
105
106
107
108
109
 
# File 'lib/iriq/storage/memory.rb', line 102

def add_to_cluster(key, host, scheme, shape, identifier)
  cluster = @clusters[key] ||= Cluster.new(
    key: key, host: host, scheme: scheme, shape: shape,
    max_values: @max_values_per_position,
  )
  cluster.add(identifier, classifier: @classifier)
  cluster
end

#batch {|false| ... } ⇒ Object

Yields false: nothing else writes an in-memory corpus.

Yields:

  • (false) 


64
65
66
 
# File 'lib/iriq/storage/memory.rb', line 64

def batch
  yield false
end

#begin_rebuild ⇒ Object

Nothing else reads an in-memory corpus mid-rebuild: rebuild in place.



160
161
162
 
# File 'lib/iriq/storage/memory.rb', line 160

def begin_rebuild
  clear_materialized_views
end

#clear_materialized_views ⇒ Object

Drop every materialized view (host_counts, position_stats, clusters, …) without touching the source-IRI log. Corpus#reinfer calls this before replaying the log so views rebuild from scratch.



150
151
152
153
154
155
156
157
 
# File 'lib/iriq/storage/memory.rb', line 150

def clear_materialized_views
  @host_counts        = Hash.new(0)
  @path_length_counts = Hash.new(0)
  @raw_shape_counts   = Hash.new(0)
  @fingerprint_counts = Hash.new(0)
  @position_stats     = {}
  @clusters           = {}
end

#close ⇒ Object 



74
 
# File 'lib/iriq/storage/memory.rb', line 74

def close;  end

#cluster_for(key) ⇒ Object

O(1) lookup by cluster key. nil if no cluster has been observed under this key yet.



198
199
200
 
# File 'lib/iriq/storage/memory.rb', line 198

def cluster_for(key)
  @clusters[key]
end

#cluster_size ⇒ Object 



192
193
194
 
# File 'lib/iriq/storage/memory.rb', line 192

def cluster_size
  @clusters.size
end

#clusters ⇒ Object 



188
189
190
 
# File 'lib/iriq/storage/memory.rb', line 188

def clusters
  @clusters.values
end

#discard_rebuild ⇒ Object 



165
 
# File 'lib/iriq/storage/memory.rb', line 165

def discard_rebuild; end

#each_activated_recognizer(&block) ⇒ Object 



139
140
141
 
# File 'lib/iriq/storage/memory.rb', line 139

def each_activated_recognizer(&block)
  @activated_recognizers.each(&block)
end

#each_observed_iri(&block) ⇒ Object 



118
119
120
 
# File 'lib/iriq/storage/memory.rb', line 118

def each_observed_iri(&block)
  @observed_iris.each(&block)
end

#each_observed_iri_since(mark, &block) ⇒ Object

The observations logged after mark (0 for all), in order; returns the mark of the last one.



124
125
126
127
 
# File 'lib/iriq/storage/memory.rb', line 124

def each_observed_iri_since(mark, &block)
  @observed_iris.drop(mark).each(&block)
  [mark, @observed_iris.size].max
end

#each_position_stats(&block) ⇒ Object 



184
185
186
 
# File 'lib/iriq/storage/memory.rb', line 184

def each_position_stats(&block)
  @position_stats.each(&block)
end

#fingerprint_counts ⇒ Object 



172
 
# File 'lib/iriq/storage/memory.rb', line 172

def fingerprint_counts; @fingerprint_counts; end

#flush ⇒ Object 



73
 
# File 'lib/iriq/storage/memory.rb', line 73

def flush;  end

#host_counts ⇒ Object

--- Reads ------------------------------------------------------------



169
 
# File 'lib/iriq/storage/memory.rb', line 169

def host_counts;        @host_counts;        end

#increment_fingerprint(shape) ⇒ Object 



93
94
95
 
# File 'lib/iriq/storage/memory.rb', line 93

def increment_fingerprint(shape)
  @fingerprint_counts[shape] += 1
end

#increment_host(host) ⇒ Object

--- Increments -------------------------------------------------------



81
82
83
 
# File 'lib/iriq/storage/memory.rb', line 81

def increment_host(host)
  @host_counts[host] += 1 if host
end

#increment_path_length(length) ⇒ Object 



85
86
87
 
# File 'lib/iriq/storage/memory.rb', line 85

def increment_path_length(length)
  @path_length_counts[length] += 1
end

#increment_raw_shape(shape) ⇒ Object 



89
90
91
 
# File 'lib/iriq/storage/memory.rb', line 89

def increment_raw_shape(shape)
  @raw_shape_counts[shape] += 1
end

#install_rebuild ⇒ Object 



164
 
# File 'lib/iriq/storage/memory.rb', line 164

def install_rebuild; end

#load_dump!(h) ⇒ Object

Missing keys load as empty, so {} is a valid (empty) corpus.



215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
 
# File 'lib/iriq/storage/memory.rb', line 215

def load_dump!(h)
  @host_counts        = Hash.new(0).merge(h.fetch("host_counts", {}))
  @path_length_counts = Hash.new(0).merge(h.fetch("path_length_counts", {}).transform_keys(&:to_i))
  @raw_shape_counts   = Hash.new(0).merge(h.fetch("raw_shape_counts", {}))
  @fingerprint_counts = Hash.new(0).merge(h.fetch("fingerprint_counts", {}))
  @max_values_per_position = h.fetch("max_values_per_position", PositionStats::DEFAULT_MAX_VALUES)
  @position_stats = h.fetch("position_stats", []).each_with_object({}) do |entry, acc|
    position = Position.from_dump(entry["position"])
    acc[position] = PositionStats.from_dump(entry["stats"])
  end
  cdump = h.fetch("clusterer", { "clusters" => {} })
  @clusters = cdump["clusters"].transform_values { |c| Cluster.from_dump(c, max_values: @max_values_per_position) }
  @observed_iris         = h.fetch("observed_iris", [])
  @activated_recognizers = h.fetch("activated_recognizers", [])
  self
end

#observe_position(position, value, type) ⇒ Object 



97
98
99
100
 
# File 'lib/iriq/storage/memory.rb', line 97

def observe_position(position, value, type)
  stats = @position_stats[position] ||= PositionStats.new(max_values: @max_values_per_position)
  stats.observe(value, type)
end

#observed_iri_count ⇒ Object 



129
130
131
 
# File 'lib/iriq/storage/memory.rb', line 129

def observed_iri_count
  @observed_iris.size
end

#param_stats(key, name) ⇒ Object 



202
203
204
205
 
# File 'lib/iriq/storage/memory.rb', line 202

def param_stats(key, name)
  cluster = @clusters[key]
  cluster && cluster.param_stats[name]
end

#path ⇒ Object

Path of the underlying file, if any. Memory backends are unpathed; Json/Sqlite override.



36
 
# File 'lib/iriq/storage/memory.rb', line 36

def path; nil; end

#path_length_counts ⇒ Object 



170
 
# File 'lib/iriq/storage/memory.rb', line 170

def path_length_counts; @path_length_counts; end

#position_evidence(position, value) ⇒ Object

Built over the live stats, so nothing is copied.



179
180
181
182
 
# File 'lib/iriq/storage/memory.rb', line 179

def position_evidence(position, value)
  stats = @position_stats[position]
  stats && PositionEvidence.from_stats(stats, value)
end

#position_stats(position) ⇒ Object 



174
175
176
 
# File 'lib/iriq/storage/memory.rb', line 174

def position_stats(position)
  @position_stats[position]
end

#raw_shape_counts ⇒ Object 



171
 
# File 'lib/iriq/storage/memory.rb', line 171

def raw_shape_counts;   @raw_shape_counts;   end

#record_activated_recognizer(dump) ⇒ Object

--- Activated recognizers (Corpus#activate_proposal) -----------------



135
136
137
 
# File 'lib/iriq/storage/memory.rb', line 135

def record_activated_recognizer(dump)
  @activated_recognizers << dump
end

#record_observation(canonical) ⇒ Object

Append a canonical IRI to the source-IRI log. Called by Corpus#observe after the event reducers have applied; the log is the source of truth that Corpus#reinfer replays.



114
115
116
 
# File 'lib/iriq/storage/memory.rb', line 114

def record_observation(canonical)
  @observed_iris << canonical
end

#save(path = nil) ⇒ Object

No-op for in-memory; subclasses override.



77
 
# File 'lib/iriq/storage/memory.rb', line 77

def save(path = nil); end

#to_dump ⇒ Object 



232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
 
# File 'lib/iriq/storage/memory.rb', line 232

def to_dump
  {
    "host_counts"             => @host_counts,
    "path_length_counts"      => @path_length_counts.transform_keys(&:to_s),
    "raw_shape_counts"        => @raw_shape_counts,
    "fingerprint_counts"      => @fingerprint_counts,
    "max_values_per_position" => @max_values_per_position,
    "position_stats"          => @position_stats.map { |pos, s|
      { "position" => pos.to_dump, "stats" => s.dump }
    },
    "clusterer"               => {
      "clusters" => @clusters.transform_values(&:dump),
    },
    "observed_iris"           => @observed_iris,
    "activated_recognizers"   => @activated_recognizers,
  }
end

#transaction {|_self| ... } ⇒ Object

Yields:

  • (_self)

Yield Parameters:



59
60
61
 
# File 'lib/iriq/storage/memory.rb', line 59

def transaction
  yield self
end

#turn_over? ⇒ Boolean

Nothing else waits on an in-memory corpus.

Returns:

  • (Boolean) 


69
70
71
 
# File 'lib/iriq/storage/memory.rb', line 69

def turn_over?
  false
end