Class: Picky::Category

Inherits:
Object show all
Includes:
API::Tokenizer, Helpers::Indexing
Defined in:
lib/picky/category_indexed.rb,
lib/picky/category.rb,
lib/picky/category/location.rb,
lib/picky/category_indexing.rb,
lib/picky/category_realtime.rb,
lib/picky/category_convenience.rb

Defined Under Namespace

Modules: Location

Constant Summary collapse

@@known_keys =

Since the options hash might contain options that do not exist, we should warn people if they use the wrong options. (Problem is that if the option is not found, then Picky will use the default)

TODO Rewrite it such that this does not need to be maintained separately (and gets available options automatically).

[
  :hints,
  :indexing,
  :partial,
  :qualifier,
  :qualifiers,
  :ranging,
  :similarity,
  :source,
  :tokenize,
  :tokenizer,
  :weight,
]

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Helpers::Indexing

#index, #timed_indexing

Methods included from Helpers::Measuring

#timed

Constructor Details

#initialize(name, index, options = {}) ⇒ Category

Parameters:

  • name: Category name to use as identifier and file names.
  • index: Index to which this category is attached to.

Options:

  • partial: Partial::None.new, Partial::Substring.new(from:start_char, to:up_to_char) (defaults from:-3, to:-1)
  • similarity: Similarity::None.new (default), Similarity::DoubleMetaphone.new(amount_of_similarly_linked_words)
  • from: The source category identifier to take the data from.
  • key_format: What this category's keys are formatted with (default is :to_i)
  • backend: The backend to use. Default is Backends::Memory.new. Other options are: Backends::Redis.new, Backends::SQLite.new, Backends::File.new.
  • qualifiers: Which qualifiers can be used to predefine the category. E.g. "title:bla".

Advanced Options:

  • source: Use if the category should use a different source.
  • tokenize: Whether to use the tokenizer (default is true).
  • tokenizer: Use a subclass of Tokenizers::Base that implements #tokens_for and #empty_tokens.
  • weight: Weights::Logarithmic.new, Weights::Constant.new(int = 0), Weights::Dynamic.new(&block) or an object that responds to #weight_for(amount_of_ids_for_token) and returns a float.


36
37
38
39
40
41
42
43
44
45
46
 
# File 'lib/picky/category.rb', line 36

def initialize name, index, options = {}
  @name  = name
  @index = index
  
  # TODO Move.
  #
  options[:hints] = index.hints

  configure_from options
  configure_indexes_from options
end

Instance Attribute Details

#backend ⇒ Object (readonly)

Returns the backend.

If no specific backend has been defined for this



159
160
161
 
# File 'lib/picky/category.rb', line 159

def backend
  @backend
end

#exact ⇒ Object

Returns the value of attribute exact.



7
8
9
 
# File 'lib/picky/category.rb', line 7

def exact
  @exact
end

#name ⇒ Object (readonly)

Returns the value of attribute name.



9
10
11
 
# File 'lib/picky/category.rb', line 9

def name
  @name
end

#partial ⇒ Object

Returns the value of attribute partial.



7
8
9
 
# File 'lib/picky/category.rb', line 7

def partial
  @partial
end

#source ⇒ Object

Return an appropriate source.

If we have no explicit source, we'll check the index for one.



105
106
107
 
# File 'lib/picky/category_indexing.rb', line 105

def source
  @source || @index.source
end

Instance Method Details

#<<(thing) ⇒ Object

Add at the end.



76
77
78
 
# File 'lib/picky/category_realtime.rb', line 76

def << thing
  add thing, method: __method__
end

#==(other) ⇒ Object

Uniquely identified by index name and name.



216
217
218
219
220
 
# File 'lib/picky/category.rb', line 216

def == other
  return false unless other
  index_name == other.index_name &&
  name       == other.name
end

#add(object, method: :unshift, force_update: false) ⇒ Object

Adds and indexes this category of the given object.

Parameters:

  • object (Object) —

    The thing to index.

  • method (Symbol) (defaults to: :unshift) —

    The method name to use on the id array.

  • force_update (Boolean) (defaults to: false) —

    Whether to force update.



14
15
16
17
18
19
20
21
 
# File 'lib/picky/category_realtime.rb', line 14

def add object, method: :unshift, force_update: false
  data = if from.respond_to? :call
    from.call(object)
  else
    object.send(from)
  end
  add_text object.send(id), data, method: method, force_update: force_update
end

#add_text(id, text_or_tokens, method: :unshift, force_update: false) ⇒ Object

For the given id, adds the list of strings to the index for the given id.



89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
# File 'lib/picky/category_realtime.rb', line 89

def add_text id, text_or_tokens, method: :unshift, force_update: false
  # text_or_tokens = text_or_tokens.to_sym if @symbol_keys # SYMBOLS.
  tokens = nil
  if tokenizer
    tokens, _ = tokenizer.tokenize text_or_tokens
  else
    tokens = text_or_tokens
  end

  format = key_format?
  static = static?
  tokens.each do |text|
    add_tokenized_token id, text, method: method, format: format, static: static, force_update: force_update
  end
rescue NoMethodError => e
  show_informative_add_text_error_message_for e
end

#add_tokenized_token(id, text, method: :unshift, format: true, static: false, force_update: false) ⇒ Object 



117
118
119
120
121
122
123
124
125
126
127
128
129
 
# File 'lib/picky/category_realtime.rb', line 117

def add_tokenized_token id, text, method: :unshift, format: true, static: false, force_update: false
  return unless text

  id = id.send key_format if format
  text = text.to_sym if @symbol_keys # SYMBOLS.
  id.freeze

  exact.add id, text, method: method, static: static, force_update: force_update
  partial.add_partialized id, text, method: method, static: static, force_update: force_update
rescue NoMethodError => e
  puts e.message
  raise %Q{The object id with text "#{text}" does not respond to method #{key_format}.}
end

#build_realtime_mapping ⇒ Object

Builds the realtime mapping.



140
141
142
143
 
# File 'lib/picky/category_realtime.rb', line 140

def build_realtime_mapping
  exact.build_realtime @symbol_keys
  partial.build_realtime @symbol_keys
end

#bundle_for(token) ⇒ Object

Returns the right index bundle for this token.



78
79
80
 
# File 'lib/picky/category_indexed.rb', line 78

def bundle_for token
  token.select_bundle exact, partial
end

#cache(scheduler = Scheduler.new) ⇒ Object

Generates all caches for this category.



27
28
29
30
31
32
33
34
 
# File 'lib/picky/category_indexing.rb', line 27

def cache scheduler = Scheduler.new
  scheduler.schedule do
    empty
    retrieve
    dump
    nil # Note: Needed so procrastinate is happy.
  end
end

#clear ⇒ Object

Clears the caches.

THINK about the semantics of clear. Is a delete even needed or is it clear+dump?



130
131
132
133
 
# File 'lib/picky/category_indexing.rb', line 130

def clear
  exact.clear
  partial.clear
end

#clear_realtime ⇒ Object

Clears the realtime mapping.



133
134
135
136
 
# File 'lib/picky/category_realtime.rb', line 133

def clear_realtime
  exact.clear_realtime
  partial.clear_realtime
end

#configure_from(options) ⇒ Object 



48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/picky/category.rb', line 48

def configure_from options
  @from       = options.delete :from
  
  # Instantly extracted to raise an error instantly.
  #
  @source     = Source.from options[:source], true, @index.name
  @tokenize   = options[:tokenize] != false
  @tokenizer  = Tokenizer.from options[:indexing], @index.name, name
  @ranger     = options[:ranging] || Range

  @key_format  = options.delete :key_format
  @backend     = options.delete :backend

  @qualifiers  = extract_qualifiers_from options

  @symbol_keys = options[:symbol_keys] || @index.symbol_keys # SYMBOLS.
end

#configure_indexes_from(options) ⇒ Object

TODO I do a lot of helper method calls here. Refactor?



72
73
74
75
76
77
78
79
80
81
 
# File 'lib/picky/category.rb', line 72

def configure_indexes_from options
  warn_if_unknown options
  
  weights    = weights_from options
  partial    = partial_from options
  similarity = similarity_from options
  
  @exact     = exact_for weights, similarity, options
  @partial   = partial_for @exact, partial, weights, options
end

#dump ⇒ Object

Dumps both bundles.



149
150
151
152
153
 
# File 'lib/picky/category.rb', line 149

def dump
  exact.dump
  partial.dump
  Picky.logger.dump self
end

#each_bundle(&block) ⇒ Object 



5
6
7
8
9
10
11
12
 
# File 'lib/picky/category_convenience.rb', line 5

def each_bundle &block
  if block
    yield exact
    yield partial
  else
    [exact, partial]
  end
end

#each_category {|_self| ... } ⇒ Object

The category itself just yields itself.

Yields:

  • (_self)

Yield Parameters:



183
184
185
 
# File 'lib/picky/category.rb', line 183

def each_category
  yield self
end

#empty ⇒ Object

Empty all the indexes.

TODO Call clear.



40
41
42
43
 
# File 'lib/picky/category_indexing.rb', line 40

def empty
  exact.empty
  partial.empty
end

#exact_for(weights, similarity, options) ⇒ Object 



119
120
121
 
# File 'lib/picky/category.rb', line 119

def exact_for weights, similarity, options
  Bundle.new :exact, self, weights, Generators::Partial::None.new, similarity, options
end

#extract_qualifiers_from(options) ⇒ Object

Extract qualifiers from the options.



177
178
179
 
# File 'lib/picky/category.rb', line 177

def extract_qualifiers_from options
  options[:qualifiers] || options[:qualifier] && [options[:qualifier]]
end

#from ⇒ Object

Where the data is taken from.



97
98
99
 
# File 'lib/picky/category_indexing.rb', line 97

def from
  @from || name
end

#id ⇒ Object

Return the key format.

If no key_format is defined on the category and the source has no key format, ask the index for one.



87
88
89
 
# File 'lib/picky/category_indexing.rb', line 87

def id
  @id ||= @index.id
end

#identifier ⇒ Object

Identifier for technical output.



210
211
212
 
# File 'lib/picky/category.rb', line 210

def identifier
  :"#{@index.identifier}:#{name}"
end

#ids(token) ⇒ Object

Gets the ids for this token's text.



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/picky/category_indexed.rb', line 38

def ids token
  bundle = bundle_for token
  if range = token.range
    # Adding all to an array, then flattening
    # is faster than using ary + ary.
    #
    @ranger.new(*range).inject([]) do |result, text|
      # It is 30% faster using the empty check
      # than just << [].
      #
      ids = bundle.ids str_or_sym(text)
      ids.empty? ? result : result << ids
    end.flatten
  else
    # Optimization
    if tokenizer && tokenizer.stemmer?
      bundle.ids str_or_sym(token.stem(tokenizer))
    else
      bundle.ids str_or_sym(token.text)
    end
  end
end

#index_directory ⇒ Object 



201
202
203
 
# File 'lib/picky/category.rb', line 201

def index_directory
  @index.directory
end

#index_name ⇒ Object 



204
205
206
 
# File 'lib/picky/category.rb', line 204

def index_name
  @index.name
end

#indexer ⇒ Object

The indexer is lazily generated and cached.



111
112
113
 
# File 'lib/picky/category_indexing.rb', line 111

def indexer
  @indexer ||= source.respond_to?(:each) ? Indexers::Parallel.new(self) : Indexers::Serial.new(self)
end

#key_format ⇒ Object

Return the key format.

If no key_format is defined on the category and the source has no key format, ask the index for one.



74
75
76
 
# File 'lib/picky/category_indexing.rb', line 74

def key_format
  @key_format ||= @index.key_format
end

#key_format? ⇒ Boolean

Returns:

  • (Boolean) 


77
78
79
 
# File 'lib/picky/category_indexing.rb', line 77

def key_format?
  key_format
end

#load ⇒ Object

Loads the index from cache.



9
10
11
12
13
14
 
# File 'lib/picky/category_indexed.rb', line 9

def load
  Picky.logger.load self
  clear_realtime # THINK Should we really explicitly clear the realtime? Or should it just be loaded?
  exact.load @symbol_keys
  partial.load @symbol_keys
end

#partial_for(exact, partial_options, weights, options) ⇒ Object 



122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/picky/category.rb', line 122

def partial_for exact, partial_options, weights, options
  # TODO Also partial.extend Bundle::Exact like in the category.
  #
  # Instead of exact for partial, use respond_to? :exact= on eg. Partial::None, then set it on the instance? 
  #
  if partial_options.respond_to?(:use_exact_for_partial?) && partial_options.use_exact_for_partial?
    Wrappers::Bundle::ExactPartial.new exact
  else
    Bundle.new :partial, self, weights, partial_options, Generators::Similarity::None.new, options
  end
end

#partial_from(options) ⇒ Object 



113
114
115
 
# File 'lib/picky/category.rb', line 113

def partial_from options
  Generators::Partial.from options[:partial], index_name, name
end

#prepare(scheduler = Scheduler.new) ⇒ Object

Indexes, creates the "prepared_..." file.



14
15
16
17
18
19
20
21
22
23
 
# File 'lib/picky/category_indexing.rb', line 14

def prepare scheduler = Scheduler.new
  categories = Categories.new
  categories << self
  with_data_snapshot do
    scheduler.schedule do
      indexer.prepare categories, scheduler
      nil # Note: Needed so procrastinate is happy.
    end
  end
end

#prepared ⇒ Object

Lazily create a prepared index proxy.



136
137
138
 
# File 'lib/picky/category.rb', line 136

def prepared
  @prepared ||= Backends::Prepared::Text.new prepared_index_path
end

#prepared_index_file(&block) ⇒ Object

Get an opened index file.

Note: If you don't use it with the block, do not forget to close it.



196
197
198
199
 
# File 'lib/picky/category.rb', line 196

def prepared_index_file &block
  @prepared_index_file ||= Backends::Prepared::Text.new prepared_index_path
  @prepared_index_file.open &block
end

#prepared_index_path ⇒ Object

Path and partial filename of the prepared index on this category.



189
190
191
 
# File 'lib/picky/category.rb', line 189

def prepared_index_path
  @prepared_index_path ||= ::File.join(index_directory, name.to_s)
end

#qualifiers ⇒ Object

Returns the qualifiers if set or just the name if not.



172
173
174
 
# File 'lib/picky/category.rb', line 172

def qualifiers
  @qualifiers || [name]
end

#reindex ⇒ Object

Indexes and loads the category.



142
143
144
145
 
# File 'lib/picky/category.rb', line 142

def reindex
  index
  load
end

#remove(id) ⇒ Object

Removes an indexed object with the given id.

Parameters:

  • id (Object) —

    The id of the object.



28
29
30
31
32
 
# File 'lib/picky/category_realtime.rb', line 28

def remove id
  id = id.send key_format if key_format?
  exact.remove id
  partial.remove id
end

#replace(object, method: :unshift) ⇒ Object

Replaces an object. Will first check if each category of the object is in the index it would insert, and if it is, will not insert. Otherwise will delete and add.

Parameters:

  • object (Object) —

    The object to replace.

  • method (Symbol) (defaults to: :unshift) —

    The method name to use on the id array.



41
42
43
44
 
# File 'lib/picky/category_realtime.rb', line 41

def replace object, method: :unshift
  remove object.send id
  add object, method: method
end

#replace!(object, method: :unshift) ⇒ Object

Always removes the object's id, and then adds the object again.

Note: This puts a bit of a strain on Ruby's memory management.

Parameters:

  • object (Object) —

    The object to replace.

  • method (Symbol) (defaults to: :unshift) —

    The method name to use on the id array.



55
56
57
58
 
# File 'lib/picky/category_realtime.rb', line 55

def replace! object, method: :unshift
  remove object.send id
  add object, method: method
end

#replace_from(hash) ⇒ Object

Replaces just part of the indexed data.

Note: Takes a hash as opposed to the add/replace method.

Raises:



64
65
66
67
68
69
70
71
72
 
# File 'lib/picky/category_realtime.rb', line 64

def replace_from hash #, id = (hash[:id] || hash['id'] || raise(IdNotGivenException.new)).send(key_format)
  return unless text = hash[from] || hash[from.to_s]

  raise IdNotGivenException.new unless id = hash[:id] || hash['id']
  id = id.send key_format if key_format?

  remove id
  add_text id, text
end

#reset_backend ⇒ Object

Resets backends in both bundles.



164
165
166
167
 
# File 'lib/picky/category.rb', line 164

def reset_backend
  exact.reset_backend
  partial.reset_backend
end

#retrieve ⇒ Object

Retrieves the prepared index data into the indexes and generates the necessary derived indexes.



60
61
62
63
64
65
66
 
# File 'lib/picky/category_indexing.rb', line 60

def retrieve
  format = key_format?
  static = static?
  prepared.retrieve do |id, token|
    add_tokenized_token id, token, method: :<<, format: format, static: static, force_update: false
  end
end

#show_informative_add_text_error_message_for(e) ⇒ Object 



107
108
109
110
111
112
113
 
# File 'lib/picky/category_realtime.rb', line 107

def show_informative_add_text_error_message_for e
  if e.name == :each
    raise %Q{#{e.message}. You probably set tokenize: false on category "#{name}". It will need an Enumerator of previously tokenized tokens.}
  else
    raise e
  end
end

#similar(token) ⇒ Object

Gets the similars for this token's text.



63
64
65
66
 
# File 'lib/picky/category_indexed.rb', line 63

def similar token
  bundle = bundle_for token
  bundle.similar str_or_sym(token.text)
end

#similarity_from(options) ⇒ Object 



116
117
118
 
# File 'lib/picky/category.rb', line 116

def similarity_from options
  Generators::Similarity.from options[:similarity], index_name, name
end

#static? ⇒ Boolean

Returns:

  • (Boolean) 


91
92
93
 
# File 'lib/picky/category_indexing.rb', line 91

def static?
  @index.static? # || @static
end

#str_or_sym(text) ⇒ Object 



68
69
70
71
72
73
74
 
# File 'lib/picky/category_indexed.rb', line 68

def str_or_sym text
  if @symbol_keys
    text.to_sym
  else
    text
  end
end

#symbol_keys? ⇒ Boolean

Returns:

  • (Boolean) 


66
67
68
 
# File 'lib/picky/category.rb', line 66

def symbol_keys?
  @symbol_keys
end

#to_s ⇒ Object 



224
225
226
 
# File 'lib/picky/category.rb', line 224

def to_s
  "#{self.class}(#{identifier})"
end

#to_tree_s(indent = 0) ⇒ Object 



228
229
230
231
232
233
234
235
 
# File 'lib/picky/category.rb', line 228

def to_tree_s indent = 0
  s = <<-TREE
#{' ' * indent}#{self.class.name.gsub('Picky::','')}(#{name})
#{' ' * indent}  #{exact.to_tree_s(4)}
#{' ' * indent}  #{partial.to_tree_s(4)}
TREE
  s.chomp
end

#tokenizer ⇒ Object

Returns an appropriate tokenizer. If one isn't set on this category, will try the index, and finally the default index tokenizer.

Will return nil if tokenize is set to false.



121
122
123
 
# File 'lib/picky/category_indexing.rb', line 121

def tokenizer
  @tokenizer || @index.tokenizer if @tokenize
end

#unshift(thing) ⇒ Object

Add at the beginning.



82
83
84
 
# File 'lib/picky/category_realtime.rb', line 82

def unshift thing
  add thing, method: __method__
end

#warn_if_unknown(options) ⇒ Object 



101
102
103
104
105
106
107
108
109
 
# File 'lib/picky/category.rb', line 101

def warn_if_unknown options
  if options && (options.keys - @@known_keys).size > 0
    warn <<-WARNING

Warning: Category options #{options} for category #{name} contain an unknown option.
     Working options are: #@@known_keys.
WARNING
  end
end

#weight(token) ⇒ Object

Gets the weight for this token's text.



18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
 
# File 'lib/picky/category_indexed.rb', line 18

def weight token
  bundle = bundle_for token
  if range = token.range
    # TODO We might be able to return early?
    #
    @ranger.new(*range).inject(nil) do |sum, text|
      weight = bundle.weight str_or_sym(text)
      weight && (weight + (sum || 0)) || sum
    end
  else
    if tokenizer && tokenizer.stemmer?
      bundle.weight str_or_sym(token.stem(tokenizer))
    else
      bundle.weight str_or_sym(token.text)
    end
  end
end

#weights_from(options) ⇒ Object 



110
111
112
 
# File 'lib/picky/category.rb', line 110

def weights_from options
  Generators::Weights.from options[:weight], index_name, name
end

#with_data_snapshot ⇒ Object

Take a data snapshot if the source offers it.



47
48
49
50
51
52
53
54
55
 
# File 'lib/picky/category_indexing.rb', line 47

def with_data_snapshot
  if source.respond_to? :with_snapshot
    source.with_snapshot(@index) do
      yield
    end
  else
    yield
  end
end