Class: Aws::Glue::Types::IntegrationPartition

Inherits:
Struct
  • Object
show all
Includes:
Structure
Defined in:
lib/aws-sdk-glue/types.rb

Overview

A structure that describes how data is partitioned on the target.

See Also:

Constant Summary collapse

SENSITIVE = 
[]

Instance Attribute Summary collapse

Instance Attribute Details

#conversion_spec ⇒ String

Specifies the timestamp format of the source data. Valid values are:

  • ‘epoch_sec` - Unix epoch timestamp in seconds

  • ‘epoch_milli` - Unix epoch timestamp in milliseconds

  • ‘iso` - ISO 8601 formatted timestamp

<note markdown=“1”> Only specify ‘ConversionSpec` when using timestamp-based partition functions (year, month, day, or hour). Glue Zero-ETL uses this parameter to correctly transform source data into timestamp format before partitioning.

Do not use high-cardinality columns with the `identity` partition

function. High-cardinality columns include:

* Primary keys

  • Timestamp fields (such as ‘LastModifiedTimestamp`, `CreatedDate`)

  • System-generated timestamps

Using high-cardinality columns with identity partitioning creates

many small partitions, which can significantly degrade ingestion performance.

</note>

Returns:

  • (String) 


17017
17018
17019
17020
17021
17022
17023
 
# File 'lib/aws-sdk-glue/types.rb', line 17017

class IntegrationPartition < Struct.new(
  :field_name,
  :function_spec,
  :conversion_spec)
  SENSITIVE = []
  include Aws::Structure
end

#field_name ⇒ String

The field name used to partition data on the target. Avoid using columns that have unique values for each row (for example, ‘LastModifiedTimestamp`, `SystemModTimeStamp`) as the partition column. These columns are not suitable for partitioning because they create a large number of small partitions, which can lead to performance issues.

Returns:

  • (String) 


17017
17018
17019
17020
17021
17022
17023
 
# File 'lib/aws-sdk-glue/types.rb', line 17017

class IntegrationPartition < Struct.new(
  :field_name,
  :function_spec,
  :conversion_spec)
  SENSITIVE = []
  include Aws::Structure
end

#function_spec ⇒ String

Specifies the function used to partition data on the target. The accepted values for this parameter are:

  • ‘identity` - Uses source values directly without transformation

  • ‘year` - Extracts the year from timestamp values (e.g., 2023)

  • ‘month` - Extracts the month from timestamp values (e.g., 2023-01)

  • ‘day` - Extracts the day from timestamp values (e.g., 2023-01-15)

  • ‘hour` - Extracts the hour from timestamp values (e.g., 2023-01-15-14)

Returns:

  • (String) 


17017
17018
17019
17020
17021
17022
17023
 
# File 'lib/aws-sdk-glue/types.rb', line 17017

class IntegrationPartition < Struct.new(
  :field_name,
  :function_spec,
  :conversion_spec)
  SENSITIVE = []
  include Aws::Structure
end