Module: Gitlab::Database::MigrationHelpers::V2

Includes:
Gitlab::Database::MigrationHelpers
Included in:
Gitlab::Database::Migration::V1_0
Defined in:
lib/gitlab/database/migration_helpers/v2.rb

Constant Summary

Constants included from Gitlab::Database::MigrationHelpers

DEFAULT_TIMESTAMP_COLUMNS, MAX_IDENTIFIER_NAME_LENGTH

Constants included from DynamicModelHelpers

DynamicModelHelpers::BATCH_SIZE

Constants included from Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers

Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers::BATCH_CLASS_NAME, Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers::BATCH_MIN_DELAY, Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers::BATCH_MIN_VALUE, Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers::BATCH_SIZE, Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers::SUB_BATCH_SIZE

Constants included from Gitlab::Database::Migrations::BackgroundMigrationHelpers

Gitlab::Database::Migrations::BackgroundMigrationHelpers::BATCH_SIZE, Gitlab::Database::Migrations::BackgroundMigrationHelpers::JOB_BUFFER_SIZE

Instance Method Summary collapse

Methods included from Gitlab::Database::MigrationHelpers

#add_check_constraint, #add_column_with_default, #add_concurrent_foreign_key, #add_concurrent_index, #add_not_null_constraint, #add_sequence, #add_text_limit, #add_timestamps_with_timezone, #backfill_conversion_of_integer_to_bigint, #backfill_iids, #change_column_type_concurrently, #check_constraint_exists?, #check_constraint_name, #check_not_null_constraint_exists?, #check_text_limit_exists?, #check_trigger_permissions!, #cleanup_concurrent_column_type_change, #column_for, #concurrent_foreign_key_name, #convert_to_bigint_column, #copy_check_constraints, #copy_foreign_keys, #copy_indexes, #create_extension, #create_or_update_plan_limit, #define_batchable_model, #disable_statement_timeout, #drop_extension, #drop_sequence, #each_batch, #each_batch_range, #ensure_batched_background_migration_is_finished, #false_value, #foreign_key_exists?, #foreign_keys_for, #index_exists_by_name?, #index_invalid?, #indexes_for, #initialize_conversion_of_integer_to_bigint, #install_rename_triggers, #postgres_exists_by_name?, #remove_check_constraint, #remove_concurrent_index, #remove_concurrent_index_by_name, #remove_foreign_key_if_exists, #remove_foreign_key_without_error, #remove_not_null_constraint, #remove_rename_triggers, #remove_text_limit, #remove_timestamps, #rename_constraint, #rename_trigger_name, #replace_sql, #restore_conversion_of_integer_to_bigint, #revert_backfill_conversion_of_integer_to_bigint, #revert_initialize_conversion_of_integer_to_bigint, #sidekiq_queue_length, #sidekiq_queue_migrate, #true_value, #undo_change_column_type_concurrently, #undo_cleanup_concurrent_column_type_change, #update_column_in_batches, #validate_check_constraint, #validate_foreign_key, #validate_not_null_constraint, #validate_text_limit

Methods included from AsyncIndexes::MigrationHelpers

#async_index_creation_available?, #prepare_async_index, #unprepare_async_index, #unprepare_async_index_by_name

Methods included from RenameTableHelpers

#finalize_table_rename, #rename_table_safely, #undo_finalize_table_rename, #undo_rename_table_safely

Methods included from DynamicModelHelpers

#define_batchable_model, #each_batch, #each_batch_range

Methods included from Gitlab::Database::Migrations::BatchedBackgroundMigrationHelpers

#delete_batched_background_migration, #finalize_batched_background_migration, #gitlab_schema_from_context, #queue_batched_background_migration

Methods included from Gitlab::Database::Migrations::BackgroundMigrationHelpers

#delete_job_tracking, #delete_queued_jobs, #finalize_background_migration, #migrate_in, #queue_background_migration_jobs_by_range_at_intervals, #requeue_background_migration_jobs_by_range_at_intervals

Methods included from Gitlab::Database::Migrations::ReestablishedConnectionStack

#with_restored_connection_stack

Instance Method Details

#cleanup_concurrent_column_rename(table, old_column, new_column) ⇒ Object

Cleans up a concurrent column name.

This method takes care of removing previously installed triggers as well as removing the old column.

table - The name of the database table. old_column - The name of the old column. new_column - The name of the new column.


166
167
168
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 166

def cleanup_concurrent_column_rename(table, old_column, new_column)
  teardown_rename_mechanism(table, old_column, new_column, column_to_remove: old_column)
end

#create_table(table_name, *args, **kwargs, &block) ⇒ Object

Creates a new table, optionally allowing the caller to add text limit constraints to the table. This method only extends Rails' `create_table` method

Example:

create_table :db_guides do |t|
  t.bigint :stars, default: 0, null: false
  t.text :title, limit: 128
  t.text :notes, limit: 1024

  t.check_constraint 'stars > 1000', name: 'so_many_stars'
end

See Rails' `create_table` for more info on the available arguments.

When adding foreign keys to other tables, consider wrapping the call into a with_lock_retries block to avoid traffic stalls.


43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 43

def create_table(table_name, *args, **kwargs, &block)
  helper_context = self

  super do |t|
    t.define_singleton_method(:text) do |column_name, **kwargs|
      limit = kwargs.delete(:limit)

      super(column_name, **kwargs)

      if limit
        # rubocop:disable GitlabSecurity/PublicSend
        name = helper_context.send(:text_limit_name, table_name, column_name)
        # rubocop:enable GitlabSecurity/PublicSend

        column_name = helper_context.quote_column_name(column_name)
        definition = "char_length(#{column_name}) <= #{limit}"

        t.check_constraint(definition, name: name)
      end
    end

    t.instance_eval(&block) unless block.nil?
  end
end

#create_table_with_constraints(*_) ⇒ Object

Superseded by `create_table` override below


10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 10

def create_table_with_constraints(*_)
  raise <<~EOM
    #create_table_with_constraints is not supported anymore - use #create_table instead, for example:

      create_table :db_guides do |t|
        t.bigint :stars, default: 0, null: false
        t.text :title, limit: 128
        t.text :notes, limit: 1024

        t.check_constraint 'stars > 1000', name: 'so_many_stars'
      end

    See https://docs.gitlab.com/ee/development/database/strings_and_the_text_data_type.html
  EOM
end

#rename_column_concurrently(table, old_column, new_column, type: nil, batch_column_name: :id) ⇒ Object

Renames a column without requiring downtime.

Concurrent renames work by using database triggers to ensure both the old and new column are in sync. However, this method will not remove the triggers or the old column automatically; this needs to be done manually in a post-deployment migration. This can be done using the method `cleanup_concurrent_column_rename`.

table - The name of the database table containing the column. old_column - The old column name. new_column - The new column name. type - The type of the new column. If no type is given the old column's

type is used.

batch_column_name - option is for tables without primary key, in this

case another unique integer column can be used. Example: :user_id

136
137
138
139
140
141
142
143
144
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 136

def rename_column_concurrently(table, old_column, new_column, type: nil, batch_column_name: :id)
  Gitlab::Database::QueryAnalyzers::RestrictAllowedSchemas.require_ddl_mode!

  setup_renamed_column(__callee__, table, old_column, new_column, type, batch_column_name)

  with_lock_retries do
    install_bidirectional_triggers(table, old_column, new_column)
  end
end

#undo_cleanup_concurrent_column_rename(table, old_column, new_column, type: nil, batch_column_name: :id) ⇒ Object

Reverses the operations performed by cleanup_concurrent_column_rename.

This method adds back the old_column removed by cleanup_concurrent_column_rename. It also adds back the triggers that are removed by cleanup_concurrent_column_rename.

table - The name of the database table containing the column. old_column - The old column name. new_column - The new column name. type - The type of the old column. If no type is given the new column's

type is used.

batch_column_name - option is for tables without primary key, in this

case another unique integer column can be used. Example: :user_id

185
186
187
188
189
190
191
192
193
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 185

def undo_cleanup_concurrent_column_rename(table, old_column, new_column, type: nil, batch_column_name: :id)
  Gitlab::Database::QueryAnalyzers::RestrictAllowedSchemas.require_ddl_mode!

  setup_renamed_column(__callee__, table, new_column, old_column, type, batch_column_name)

  with_lock_retries do
    install_bidirectional_triggers(table, old_column, new_column)
  end
end

#undo_rename_column_concurrently(table, old_column, new_column) ⇒ Object

Reverses operations performed by rename_column_concurrently.

This method takes care of removing previously installed triggers as well as removing the new column.

table - The name of the database table. old_column - The name of the old column. new_column - The name of the new column.


154
155
156
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 154

def undo_rename_column_concurrently(table, old_column, new_column)
  teardown_rename_mechanism(table, old_column, new_column, column_to_remove: new_column)
end

#with_lock_retries(*args, **kwargs, &block) ⇒ Object

Executes the block with a retry mechanism that alters the lock_timeout and sleep_time between attempts. The timings can be controlled via the timing_configuration parameter. If the lock was not acquired within the retry period, a last attempt is made without using lock_timeout.

In order to retry the block, the method wraps the block into a transaction.

When called inside an open transaction it will execute the block directly if lock retries are enabled with `enable_lock_retries!` at migration level, otherwise it will raise an error.

Examples

# Invoking without parameters
with_lock_retries do
  drop_table :my_table
end

# Invoking with custom +timing_configuration+
t = [
  [1.second, 1.second],
  [2.seconds, 2.seconds]
]

with_lock_retries(timing_configuration: t) do
  drop_table :my_table # this will be retried twice
end

# Disabling the retries using an environment variable
> export DISABLE_LOCK_RETRIES=true

with_lock_retries do
  drop_table :my_table # one invocation, it will not retry at all
end

Parameters

  • timing_configuration - [[ActiveSupport::Duration, ActiveSupport::Duration], …] lock timeout for the block, sleep time before the next iteration, defaults to `Gitlab::Database::WithLockRetries::DEFAULT_TIMING_CONFIGURATION`

  • logger - [Gitlab::JsonLogger]

  • env - [Hash] custom environment hash, see the example with `DISABLE_LOCK_RETRIES`


104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
# File 'lib/gitlab/database/migration_helpers/v2.rb', line 104

def with_lock_retries(*args, **kwargs, &block)
  if transaction_open?
    if enable_lock_retries?
      Gitlab::AppLogger.warn 'Lock retries already enabled, executing the block directly'
      yield
    else
      raise <<~EOF
      #{__callee__} can not be run inside an already open transaction

      Use migration-level lock retries instead, see https://docs.gitlab.com/ee/development/migration_style_guide.html#retry-mechanism-when-acquiring-database-locks
      EOF
    end
  else
    super(*args, **kwargs.merge(allow_savepoints: false), &block)
  end
end