Module: InventoryRefresh::InventoryCollection::Helpers::InitializeHelper

Included in:
InventoryRefresh::InventoryCollection
Defined in:
lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb

Instance Method Summary collapse

Instance Method Details

#init_arels(arel) ⇒ Object

Parameters:

  • arel (ActiveRecord::Associations::CollectionProxy|Arel::SelectManager) —

    Instead of :parent and :association we can provide Arel directly to say what records should be compared to check if InventoryObject will be doing create/update/delete.

    Example:

    add_collection(:cross_link_vms) do |builder|
  builder.add_properties(
    :arel        => Vm.where(:tenant => manager.tenant),
    :association => nil,
    :model_class => Vm,
    :name        => :cross_link_vms,
    :manager_ref => [:uid_ems],
    :strategy    => :local_db_find_references,
  )
end



172
173
174
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 172

def init_arels(arel)
  @arel = arel
end

#init_basic_properties(association, model_class, name, parent) ⇒ Object

Parameters:

  • association (Symbol) —

    A Rails association callable on a :parent attribute is used for comparing with the objects in the DB, to decide if the InventoryObjects will be created/deleted/updated or used for obtaining the data from a DB, if a DB strategy is used. It returns objects of the :model_class class or its sub STI.

  • model_class (Class) —

    A class of an ApplicationRecord model, that we want to persist into the DB or load from the DB.

  • name (Symbol) —

    A unique name of the InventoryCollection under a Persister. If not provided, the :association attribute is used. If :association is nil as well, the :name will be inferred from the :model_class.

  • parent (ApplicationRecord) —

    An ApplicationRecord object that has a callable :association method returning the objects of a :model_class.



16
17
18
19
20
21
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 16

def init_basic_properties(association, model_class, name, parent)
  @association            = association
  @model_class            = model_class
  @name                   = name || association || model_class.to_s.demodulize.tableize
  @parent                 = parent || nil
end

#init_changed_records_stats ⇒ Object 



283
284
285
286
287
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 283

def init_changed_records_stats
  @created_records = []
  @updated_records = []
  @deleted_records = []
end

#init_custom_procs(custom_save_block, custom_reconnect_block) ⇒ Object

Parameters:

  • custom_save_block (Proc) —

    A custom lambda/proc for persisting in the DB, for cases where it’s not enough to just save every InventoryObject inside by the defined rules and default saving algorithm.

    Example1 - saving SomeModel in my own ineffective way :-) :

    custom_save = lambda do |_ems, inventory_collection|
  inventory_collection.each |inventory_object| do
    hash = inventory_object.attributes # Loads possible dependencies into saveable hash
    obj = SomeModel.find_by(:attr => hash[:attr]) # Note: doing find_by for many models produces N+1
                                                  # queries, avoid this, this is just a simple example :-)
    obj.update_attributes(hash) if obj
    obj ||= SomeModel.create(hash)
    inventory_object.id = obj.id # If this InventoryObject is referenced elsewhere, we need to store its
                                   primary key back to the InventoryObject
 end

    Example2 - saving parent OrchestrationStack in a more effective way, than the default saving algorithm can achieve. Ancestry gem requires an ActiveRecord object for association and is not defined as a proper ActiveRecord association. That leads in N+1 queries in the default saving algorithm, so we can do better with custom saving for now. The InventoryCollection is defined as a custom dependencies processor, without its own :model_class and InventoryObjects inside:

    InventoryRefresh::InventoryCollection.new({
  :association       => :orchestration_stack_ancestry,
  :custom_save_block => orchestration_stack_ancestry_save_block,
  :dependency_attributes => {
    :orchestration_stacks           => [collections[:orchestration_stacks]],
    :orchestration_stacks_resources => [collections[:orchestration_stacks_resources]]
  }
})

    And the lambda is defined as:

    orchestration_stack_ancestry_save_block = lambda do |_ems, inventory_collection|
  stacks_inventory_collection = inventory_collection.dependency_attributes[:orchestration_stacks].try(:first)

  return if stacks_inventory_collection.blank?

  stacks_parents = stacks_inventory_collection.data.each_with_object({}) do |x, obj|
    parent_id = x.data[:parent].load.try(:id)
    obj[x.id] = parent_id if parent_id
  end

  model_class = stacks_inventory_collection.model_class

  stacks_parents_indexed = model_class
                             .select([:id, :ancestry])
                             .where(:id => stacks_parents.values).find_each.index_by(&:id)

  model_class
    .select([:id, :ancestry])
    .where(:id => stacks_parents.keys).find_each do |stack|
    parent = stacks_parents_indexed[stacks_parents[stack.id]]
    stack.update_attribute(:parent, parent)
  end
end
  • custom_reconnect_block (Proc) —

    A custom lambda for reconnect logic of previously disconnected records

    Example - Reconnect disconnected Vms

    InventoryRefresh::InventoryCollection.new({
  :association            => :orchestration_stack_ancestry,
  :custom_reconnect_block => vms_custom_reconnect_block,
})

    And the lambda is defined as:

    vms_custom_reconnect_block = lambda do |inventory_collection, inventory_objects_index, attributes_index|
  inventory_objects_index.each_slice(1000) do |batch|
    Vm.where(:ems_ref => batch.map(&:second).map(&:manager_uuid)).each do |record|
      index = inventory_collection.object_index_with_keys(inventory_collection.manager_ref_to_cols, record)

      # We need to delete the record from the inventory_objects_index and attributes_index, otherwise it
      # would be sent for create.
      inventory_object = inventory_objects_index.delete(index)
      hash             = attributes_index.delete(index)

      record.assign_attributes(hash.except(:id, :type))
      if !inventory_collection.check_changed? || record.changed?
        record.save!
        inventory_collection.store_updated_records(record)
      end

      inventory_object.id = record.id
    end
  end



261
262
263
264
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 261

def init_custom_procs(custom_save_block, custom_reconnect_block)
  @custom_save_block      = custom_save_block
  @custom_reconnect_block = custom_reconnect_block
end

#init_data(default_values) ⇒ Object

Parameters:

  • default_values (Hash) —

    A hash of an attributes that will be added to every inventory object created by inventory_collection.build(hash)

    Example: Given

    inventory_collection = InventoryCollection.new({
  :model_class    => ::Vm,
  :arel           => @ems.vms,
  :default_values => {:ems_id => 10}
})

    And building the inventory_object like:

    inventory_object = inventory_collection.build(:ems_ref => "vm_1", :name => "vm1")

    The inventory_object.data will look like:

    {:ems_ref => "vm_1", :name => "vm1", :ems_id => 10}



279
280
281
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 279

def init_data(default_values)
  @default_values = default_values || {}
end

#init_flags(complete, create_only, check_changed, update_only, use_ar_object, assert_graph_integrity) ⇒ Object

Parameters:

  • complete (Boolean) —

    By default true, :complete is marking we are sending a complete dataset and therefore we can create/update/delete the InventoryObject objects. If :complete is false we will only do create/update without delete.

  • create_only (Boolean) —

    TODO

  • check_changed (Boolean) —

    By default true. If true, before updating the InventoryObject, we call Rails ‘changed?’ method. This can optimize speed of updates heavily, but it can fail to recognize the change for e.g. Ancestry and Relationship based columns. If false, we always update the InventoryObject.

  • update_only (Boolean) —

    By default false. If true we only update the InventoryObject objects, if false we do create/update/delete.

  • use_ar_object (Boolean) —

    True or False. Whether we need to initialize AR object as part of the saving it’s needed if the model have special setters, serialize of columns, etc. This setting is relevant only for the batch saver strategy.



95
96
97
98
99
100
101
102
103
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 95

def init_flags(complete, create_only, check_changed, update_only, use_ar_object, assert_graph_integrity)
  @complete               = complete.nil? ? true : complete
  @create_only            = create_only.nil? ? false : create_only
  @check_changed          = check_changed.nil? ? true : check_changed
  @saved                  = false
  @update_only            = update_only.nil? ? false : update_only
  @use_ar_object          = use_ar_object || false
  @assert_graph_integrity = assert_graph_integrity.nil? ? true : assert_graph_integrity
end

#init_ic_relations(dependency_attributes) ⇒ Object

Parameters:

  • dependency_attributes (Hash) —

    Manually defined dependencies of this InventoryCollection. We can use this by manually place the InventoryCollection into the graph, to make sure the saving is invoked after the dependencies were saved. The dependencies itself are InventoryCollection objects. For a common use-cases we do not need to define dependencies manually, since those are inferred automatically by scanning of the data.

    Example:

    :dependency_attributes => {
  :orchestration_stacks           => [collections[:orchestration_stacks]],
  :orchestration_stacks_resources => [collections[:orchestration_stacks_resources]]
}

    This example is used in Example2 of the <param custom_save_block> and it means that our :custom_save_block will be invoked after the InventoryCollection :orchestration_stacks and :orchestration_stacks_resources are saved.



78
79
80
81
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 78

def init_ic_relations(dependency_attributes)
  @dependency_attributes = dependency_attributes || {}
  @dependees             = Set.new
end

#init_model_attributes(attributes_blacklist, attributes_whitelist, inventory_object_attributes, batch_extra_attributes) ⇒ Object

Parameters:

  • attributes_blacklist (Array) —

    Attributes we do not want to include into saving. We cannot blacklist an attribute that is needed for saving of the object. Note: attributes_blacklist is also used for internal resolving of the cycles in the graph.

    In the Example2 of the <param custom_save_block>, we have a custom saving code, that saves a :parent attribute of the OrchestrationStack. That means we don’t want that attribute saved as a part of InventoryCollection for OrchestrationStack, so we would set :attributes_blacklist => [:parent]. Then the :parent will be ignored while saving.

  • attributes_whitelist (Array) —

    Same usage as the :attributes_blacklist, but defining full set of attributes that should be saved. Attributes that are part of :manager_ref and needed validations are automatically added.

  • inventory_object_attributes (Array) —

    Array of attribute names that will be exposed as readers/writers on the InventoryObject objects inside.

    Example: Given

    inventory_collection = InventoryCollection.new({
   :model_class                 => ::Vm,
   :arel                        => @ems.vms,
   :inventory_object_attributes => [:name, :label]
 })

    And building the inventory_object like:

    inventory_object = inventory_collection.build(:ems_ref => "vm1", :name => "vm1")

    We can use inventory_object_attributes as setters and getters:

    inventory_object.name = "Name"
inventory_object.label = inventory_object.name

    Which would be equivalent to less nicer way:

    inventory_object[:name] = "Name"
inventory_object[:label] = inventory_object[:name]

    So by using inventory_object_attributes, we will be guarding the allowed attributes and will have an explicit list of allowed attributes, that can be used also for documentation purposes.

  • batch_extra_attributes (Array) —

    Array of symbols marking which extra attributes we want to store into the db. These extra attributes might be a product of :use_ar_object assignment and we need to specify them manually, if we want to use a batch saving strategy and we have models that populate attributes as a side effect.



139
140
141
142
143
144
145
146
147
148
149
150
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 139

def init_model_attributes(attributes_blacklist, attributes_whitelist,
                          inventory_object_attributes, batch_extra_attributes)
  @attributes_blacklist             = Set.new
  @attributes_whitelist             = Set.new
  @batch_extra_attributes           = batch_extra_attributes || []
  @inventory_object_attributes      = inventory_object_attributes
  @internal_attributes              = %i(__feedback_edge_set_parent)
  @transitive_dependency_attributes = Set.new

  blacklist_attributes!(attributes_blacklist) if attributes_blacklist.present?
  whitelist_attributes!(attributes_whitelist) if attributes_whitelist.present?
end

#init_references(manager_ref, manager_ref_allowed_nil, secondary_refs) ⇒ Object

Parameters:

  • manager_ref (Array) —

    Array of Symbols, that are keys of the InventoryObject’s data, inserted into this InventoryCollection. Using these keys, we need to be able to uniquely identify each of the InventoryObject objects inside.

  • manager_ref_allowed_nil (Array) —

    Array of symbols having manager_ref columns, that are a foreign key an can be nil. Given the table are shared by many providers, it can happen, that the table is used only partially. Then it can happen we want to allow certain foreign keys to be nil, while being sure the referential integrity is not broken. Of course the DB Foreign Key can’t be created in this case, so we should try to avoid this usecase by a proper modeling. Note that InventoryObject’s data has to be build with <foreign_key> => nil, it means that key cannot be missing!

  • secondary_refs (Hash) —

    TODO



58
59
60
61
62
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 58

def init_references(manager_ref, manager_ref_allowed_nil, secondary_refs)
  @manager_ref             = manager_ref || %i(ems_ref)
  @manager_ref_allowed_nil = manager_ref_allowed_nil || []
  @secondary_refs          = secondary_refs || {}
end

#init_storages ⇒ Object 



152
153
154
155
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 152

def init_storages
  @data_storage       = ::InventoryRefresh::InventoryCollection::DataStorage.new(self, @secondary_refs)
  @references_storage = ::InventoryRefresh::InventoryCollection::ReferencesStorage.new(index_proxy)
end

#init_strategies(strategy, retention_strategy) ⇒ Object

Parameters:

  • strategy (Symbol) —

    A strategy of the InventoryCollection that will be used for saving/loading of the InventoryObject objects. Allowed strategies are:

    - nil => InventoryObject objects of the InventoryCollection will be saved to the DB, only these objects
         will be referable from the other InventoryCollection objects.
- :local_db_cache_all => Loads InventoryObject objects from the database, it loads all the objects that
                         are a result of a [<:parent>.<:association>, :arel] taking
                         first defined in this order. This strategy will not save any objects in the DB.
- :local_db_find_references => Loads InventoryObject objects from the database, it loads only objects that
                               were referenced by the other InventoryCollections using a filtered result
                               of a [<:parent>.<:association>, :arel] taking first
                               defined in this order. This strategy will not save any objects in the DB.
- :local_db_find_missing_references => InventoryObject objects of the InventoryCollection will be saved to
                                       the DB. Then if we reference an object that is not present, it will
                                       load them from the db using :local_db_find_references strategy.
  • retention_strategy (Symbol) —

    A retention strategy for this collection. Allowed values are:

    • :destroy => Will destroy the inactive records.

    • :archive => Will archive the inactive records by setting :archived_at timestamp.



41
42
43
44
45
46
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 41

def init_strategies(strategy, retention_strategy)
  @saver_strategy     = :concurrent_safe_batch
  @strategy           = process_strategy(strategy)
  # TODO(lsmola) why don't we just set this strategy based on :archived_at column? Lets do that
  @retention_strategy = process_retention_strategy(retention_strategy)
end

#process_retention_strategy(retention_strategy) ⇒ Symbol

Processes passed retention strategy

Parameters:

  • retention_strategy (Symbol) —

    Passed retention strategy

Returns:

  • (Symbol) —

    Returns back the passed strategy if supported, or raises exception



326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 326

def process_retention_strategy(retention_strategy)
  unless retention_strategy
    if supports_column?(:archived_at)
      return :archive
    else
      return :destroy
    end
  end

  retention_strategy = retention_strategy.to_sym
  case retention_strategy
  when :destroy, :archive
    retention_strategy
  else
    raise "Unknown InventoryCollection retention strategy: :#{retention_strategy}, allowed strategies are "\
      ":destroy and :archive"
  end
end

#process_strategy(strategy_name) ⇒ Symbol

Processes passed strategy, modifies :data_collection_finalized and :saved attributes for db only strategies

Parameters:

  • strategy_name (Symbol) —

    Passed saver strategy

Returns:

  • (Symbol) —

    Returns back the passed strategy if supported, or raises exception



293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 293

def process_strategy(strategy_name)
  self.data_collection_finalized = false

  return unless strategy_name

  strategy_name = strategy_name.to_sym
  case strategy_name
  when :local_db_cache_all
    self.data_collection_finalized = true
    self.saved = true
  when :local_db_find_references
    self.saved = true
  when :local_db_find_missing_references
    nil
  else
    raise "Unknown InventoryCollection strategy: :#{strategy_name}, allowed strategies are :local_db_cache_all, "\
      ":local_db_find_references and :local_db_find_missing_references."
  end
  strategy_name
end

#strategy=(strategy) ⇒ Symbol

Saves passed strategy, modifies :data_collection_finalized and :saved attributes for db only strategies

Parameters:

  • strategy (Symbol) —

    Passed saver strategy

Returns:

  • (Symbol) —

    Returns back the passed strategy if supported, or raises exception



318
319
320
 
# File 'lib/inventory_refresh/inventory_collection/helpers/initialize_helper.rb', line 318

def strategy=(strategy)
  @strategy = process_strategy(strategy)
end