Class: ElasticGraph::DatastoreCore::IndexDefinition::RolloverIndexTemplate

Inherits:
Object
  • Object
show all
Includes:
Base
Defined in:
lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb

Instance Method Summary collapse

Methods included from Base

#accessible_cluster_names_to_index_into, #accessible_from_queries?, #all_accessible_cluster_names, #cluster_to_query, #clusters_to_index_into, #flattened_env_setting_overrides, #has_custom_routing?, #ignored_values_for_routing, #known_related_query_rollover_indices, #list_counts_field_paths_for_source, #routing_value_for_prepared_record, #searches_could_hit_incomplete_docs?, #to_s, #use_updates_for_indexing?

Instance Method Details

#delete_from_datastore(datastore_client) ⇒ Object

We need to delete both the template and the actual indices for rollover indices



45
46
47
48
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 45

def delete_from_datastore(datastore_client)
  datastore_client.delete_index_template(name)
  datastore_client.delete_indices(index_expression_for_search)
end

#index_expression_for_search ⇒ Object

Two underscores used to avoid collisions with other types (e.g. payments_2020 and payments_xyz_2020), though regardless shouldn’t happen if types follow naming conventions.



62
63
64
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 62

def index_expression_for_search
  index_name_with_suffix("*")
end

#index_name_for_writes(record, timestamp_field_path: nil) ⇒ Object

Returns an index name to use for write operations. The index_definition selection is a function of the index_definition’s rollover configuration and the record’s timestamp.



68
69
70
71
72
73
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 68

def index_name_for_writes(record, timestamp_field_path: nil)
  index_name_with_suffix(rollover_index_suffix_for_record(
    record,
    timestamp_field_path: timestamp_field_path || self.timestamp_field_path
  ))
end

#mappings_in_datastore(datastore_client) ⇒ Object 



38
39
40
41
42
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 38

def mappings_in_datastore(datastore_client)
  IndexConfigNormalizer.normalize_mappings(
    datastore_client.get_index_template(name).dig("template", "mappings") || {}
  )
end

#related_rollover_index_for_timestamp(timestamp, setting_overrides = {}) ⇒ Object

Gets a single related ‘RolloverIndex` for a given timestamp.



120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 120

def related_rollover_index_for_timestamp(timestamp, setting_overrides = {})
  # @type var record: ::Hash[::String, untyped]
  # We need to use `__skip__` here because `inner_value` has different types on different
  # block iterations: initially, it's a string, then it becomes a hash. Steep has trouble
  # with this but it works fine.
  __skip__ = record = timestamp_field_path.split(".").reverse.reduce(timestamp) do |inner_value, field_name|
    {field_name => inner_value}
  end

  concrete_rollover_index_for(index_name_for_writes(record), setting_overrides)
end

#related_rollover_indices(datastore_client, only_if_exists: false) ⇒ Object

Returns a list of indices related to this template. This includes both indices that are specified in our configuration settings (e.g. via ‘setting_overrides_by_timestamp` and `custom_time_sets`) and also indices that have been auto-created from the template.

Note that there can be discrepancies between the configuration settings and the indices in the database. Sometimes this is planned/expected (e.g. such as when invoking ‘elasticgraph-admin` to configure an index newly defined in configuration) and in other cases it’s not.

The ‘only_if_exists` argument controls how a discrepancy is treated.

  • When ‘false` (the default), indices that are defined in config but do not exist in the datastore are still returned. This is generally what we want for indexing and cluster administration.

  • When ‘true`, any indices in our configuration that do not exist are ignored, and not included in the returned list. This is appropriate for searching the datastore: if we attempt to exclude an index which is defined in config but does not exist (e.g. via `-[index_name]` in the search index expression), the datastore will return an error, but we can safely ignore the index. Likewise, if we have an index in the datastore which we cannot infer a timestamp range, we need to ignore it to avoid getting errors. Ignoring an index is safe when searching because our search logic uses a wildcard to match all indices with the same prefix, and then excludes certain known indices that it can safely exclude based on their timestamp range. Ignored indices which exist will still be searched.

In addition, any indices which exist, but which are not controlled by our current configuration, are ignored. Examples:

- An index with a custom suffix (e.g. `__before_2019`) which has no corresponding configuration. We have no way to guess
  what the timestamp range is for such an index, and we want to completely ignore it.
- An index with for a different rollover frequency than our current configuration. For example, a `__2019-03` index,
  which must rollover monthly, would be ignored if our current rollover frequency is yearly or daily.

These latter cases are quite rare but can happen when we are dealing with indices defined before an update to our configuration. Our searches will continue to search these indices so long as their name matches the pattern, and we otherwise want to ignore these indices (e.g. we don’t want admin to attempt to configure them, or want our indexer to attempt to write to them).



106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 106

def related_rollover_indices(datastore_client, only_if_exists: false)
  config_indices_by_name = rollover_indices_to_pre_create.to_h { |i| [i.name, i] }

  db_indices_by_name = datastore_client.list_indices_matching(index_expression_for_search).filter_map do |name|
    index = concrete_rollover_index_for(name, {}, config_indices_by_name[name]&.time_set)
    [name, index] if index
  end.to_h

  config_indices_by_name = config_indices_by_name.slice(*db_indices_by_name.keys) if only_if_exists

  db_indices_by_name.merge(config_indices_by_name).values
end

#rollover_index_template? ⇒ Boolean

Indicates if this is a rollover index definition.

Use of this is considered a mild code smell. When feasible, it’s generally better to implement a new polymorphic API on the IndexDefinition interface, rather then branching on the value of this predicate.

Returns:

  • (Boolean) 


55
56
57
 
# File 'lib/elastic_graph/datastore_core/index_definition/rollover_index_template.rb', line 55

def rollover_index_template?
  true
end