Class: Ci::JobDefinition

Inherits:
ApplicationRecord show all
Includes:
BulkInsertSafe, Partitionable
Defined in:
app/models/ci/job_definition.rb

Overview

The purpose of this class is to store immutable duplicate Processable related data that can be disposed after all the pipelines that use it are archived. Data that should be persisted forever, should be stored with Ci::Build model.

Constant Summary collapse

CONFIG_ATTRIBUTES_FROM_METADATA =

IMPORTANT: append new attributes at the end of this list. Do not change the order! Order is important for the checksum calculation. We have two constants at the moment because we’ll only stop writing to the p_ci_builds_metadata table via the stop_writing_builds_metadata feature flag. The tag_list and run_steps will be implemented in the future.

[
  :options,
  :yaml_variables,
  :id_tokens,
  :secrets,
  :interruptible
].freeze
CONFIG_ATTRIBUTES = 
(CONFIG_ATTRIBUTES_FROM_METADATA + [:tag_list, :run_steps]).freeze
NORMALIZED_DATA_COLUMNS = 
i[interruptible].freeze
NEW_CHECKSUM_PARTITION_THRESHOLD =

Partition ID from which we start using the new checksum approach on GitLab.com. This is set to align with new partition creation to minimize redundant job definitions. For context, see: gitlab.com/gitlab-org/gitlab/-/issues/577902

108

Constants included from BulkInsertSafe

BulkInsertSafe::ALLOWED_CALLBACKS, BulkInsertSafe::DEFAULT_BATCH_SIZE, BulkInsertSafe::MethodNotAllowedError, BulkInsertSafe::PrimaryKeySetError

Constants inherited from ApplicationRecord

ApplicationRecord::MAX_PLUCK

Constants included from HasCheckConstraints

HasCheckConstraints::NOT_NULL_CHECK_PATTERN

Constants included from ResetOnColumnErrors

ResetOnColumnErrors::MAX_RESET_PERIOD

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Partitionable

registered_models

Methods inherited from ApplicationRecord

model_name, table_name_prefix

Methods inherited from ApplicationRecord

===, cached_column_list, #create_or_load_association, current_transaction, declarative_enum, default_select_columns, delete_all_returning, #deleted_from_database?, id_in, id_not_in, iid_in, nullable_column?, primary_key_in, #readable_by?, safe_ensure_unique, safe_find_or_create_by, safe_find_or_create_by!, #to_ability_name, underscore, where_exists, where_not_exists, with_fast_read_statement_timeout, without_order

Methods included from Organizations::Sharding

#sharding_organization

Methods included from ResetOnColumnErrors

#reset_on_union_error, #reset_on_unknown_attribute_error

Methods included from Gitlab::SensitiveSerializableHash

#serializable_hash

Class Method Details

.apply_normalized_defaults!(config) ⇒ Object 



86
87
88
89
90
91
 
# File 'app/models/ci/job_definition.rb', line 86

def self.apply_normalized_defaults!(config)
  NORMALIZED_DATA_COLUMNS.each do |col|
    config[col] = config.fetch(col) { column_defaults[col.to_s] }
  end
  config
end

.extract_and_parse_tags(config) ⇒ Object

rubocop:enable Gitlab/AvoidGitlabInstanceChecks



103
104
105
106
107
108
 
# File 'app/models/ci/job_definition.rb', line 103

def self.extract_and_parse_tags(config)
  tag_list = config[:tag_list]
  return {} unless tag_list

  { tag_list: Gitlab::Ci::Tags::Parser.new(tag_list).parse }
end

.fabricate(config:, project_id:, partition_id:) ⇒ Object 



49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'app/models/ci/job_definition.rb', line 49

def self.fabricate(config:, project_id:, partition_id:)
  sanitized_config = sanitize_config(config)
  config_with_defaults = apply_normalized_defaults!(sanitized_config.deep_dup)

  if use_new_checksum_approach?(partition_id)
    # New approach: set defaults before checksum generation
    checksum = generate_checksum(config_with_defaults)
    persisted_config = sanitized_config.except(*NORMALIZED_DATA_COLUMNS)
  else
    # Old approach: generate checksum before setting defaults, persist original sanitized_config
    checksum = generate_checksum(sanitized_config)
    persisted_config = sanitized_config
  end

  new(
    project_id: project_id,
    partition_id: partition_id,
    config: persisted_config,
    checksum: checksum,
    created_at: Time.current,
    **config_with_defaults.slice(*NORMALIZED_DATA_COLUMNS)
  )
end

.generate_checksum(config) ⇒ Object 



80
81
82
83
84
 
# File 'app/models/ci/job_definition.rb', line 80

def self.generate_checksum(config)
  config
    .then { |data| Gitlab::Json.dump(data) }
    .then { |data| Digest::SHA256.hexdigest(data) }
end

.sanitize_config(config) ⇒ Object 



73
74
75
76
77
78
 
# File 'app/models/ci/job_definition.rb', line 73

def self.sanitize_config(config)
  config
    .symbolize_keys
    .slice(*CONFIG_ATTRIBUTES)
    .then { |data| data.merge!(extract_and_parse_tags(data)) }
end

.use_new_checksum_approach?(partition_id) ⇒ Boolean

rubocop:disable Gitlab/AvoidGitlabInstanceChecks – partition gating is only needed on GitLab.com

Returns:

  • (Boolean) 


94
95
96
97
98
99
100
 
# File 'app/models/ci/job_definition.rb', line 94

def self.use_new_checksum_approach?(partition_id)
  return false unless Feature.enabled?(:ci_job_definitions_new_checksum, :instance)
  # For self-managed instances, use the new approach immediately
  return true unless Gitlab.com?

  partition_id >= NEW_CHECKSUM_PARTITION_THRESHOLD
end

Instance Method Details

#job_attributes ⇒ Object

Hash containing all job attributes: config + normalized_data. Used in spec helpers, to merge with job_attributes instead of config.



121
122
123
 
# File 'app/models/ci/job_definition.rb', line 121

def job_attributes
  attributes.deep_symbolize_keys.slice(*NORMALIZED_DATA_COLUMNS).merge(config)
end

#readonly? ⇒ Boolean

Returns:

  • (Boolean) 


125
126
127
 
# File 'app/models/ci/job_definition.rb', line 125

def readonly?
  persisted?
end

#tag_list ⇒ Object

We need to re-parse the tags because there are a few records in the 106-107 partitions that were not properly parsed during the pipeline creation.



113
114
115
116
117
 
# File 'app/models/ci/job_definition.rb', line 113

def tag_list
  tags = config.fetch(:tag_list) { [] }

  Gitlab::Ci::Tags::Parser.new(tags).parse
end

#validate_config_json_schema ⇒ Object 



129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
 
# File 'app/models/ci/job_definition.rb', line 129

def validate_config_json_schema
  return if config.blank?

  validator = JsonSchemaValidator.new({
    filename: 'ci_job_definition_config',
    attributes: [:config],
    detail_errors: true
  })

  validator.validate(self)
  return if errors[:config].empty?

  Gitlab::AppJsonLogger.warn(
    class: self.class.name,
    message: 'Invalid config schema detected',
    job_definition_checksum: checksum,
    project_id: project_id,
    schema_errors: errors[:config]
  )

  errors.delete(:config) if Rails.env.production?
end