Class: Aws::SageMaker::Types::S3DataSource

Inherits:
Struct
  • Object
show all
Includes:
Aws::Structure
Defined in:
lib/aws-sdk-sagemaker/types.rb

Overview

Note:

When making an API call, you may pass S3DataSource data as a hash:

{
  s3_data_type: "ManifestFile", # required, accepts ManifestFile, S3Prefix, AugmentedManifestFile
  s3_uri: "S3Uri", # required
  s3_data_distribution_type: "FullyReplicated", # accepts FullyReplicated, ShardedByS3Key
  attribute_names: ["AttributeName"],
}

Describes the S3 data source.

See Also:

Instance Attribute Summary collapse

Instance Attribute Details

#attribute_names ⇒ Array<String>

A list of one or more attribute names to use that are found in a specified augmented manifest file.

Returns:

  • (Array<String>) 


9442
9443
9444
9445
9446
9447
9448
 
# File 'lib/aws-sdk-sagemaker/types.rb', line 9442

class S3DataSource < Struct.new(
  :s3_data_type,
  :s3_uri,
  :s3_data_distribution_type,
  :attribute_names)
  include Aws::Structure
end

#s3_data_distribution_type ⇒ String

If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify ‘FullyReplicated`.

If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ‘ShardedByS3Key`. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.

Don’t choose more ML compute instances for training than available S3 objects. If you do, some nodes won’t get any data and you will pay for nodes that aren’t getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.

In distributed training, where you use multiple ML compute EC2 instances, you might choose ‘ShardedByS3Key`. If the algorithm requires copying training data to the ML storage volume (when `TrainingInputMode` is set to `File`), this copies 1/n of the number of objects.

Returns:

  • (String) 


9442
9443
9444
9445
9446
9447
9448
 
# File 'lib/aws-sdk-sagemaker/types.rb', line 9442

class S3DataSource < Struct.new(
  :s3_data_type,
  :s3_uri,
  :s3_data_distribution_type,
  :attribute_names)
  include Aws::Structure
end

#s3_data_type ⇒ String

If you choose ‘S3Prefix`, `S3Uri` identifies a key name prefix. Amazon SageMaker uses all objects that match the specified key name prefix for model training.

If you choose ‘ManifestFile`, `S3Uri` identifies an object that is a manifest file containing a list of object keys that you want Amazon SageMaker to use for model training.

If you choose ‘AugmentedManifestFile`, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. `AugmentedManifestFile` can only be used if the Channel’s input mode is ‘Pipe`.

Returns:

  • (String) 


9442
9443
9444
9445
9446
9447
9448
 
# File 'lib/aws-sdk-sagemaker/types.rb', line 9442

class S3DataSource < Struct.new(
  :s3_data_type,
  :s3_uri,
  :s3_data_distribution_type,
  :attribute_names)
  include Aws::Structure
end

#s3_uri ⇒ String

Depending on the value specified for the ‘S3DataType`, identifies either a key name prefix or a manifest. For example:

  • A key name prefix might look like this: ‘s3://bucketname/exampleprefix`.

  • A manifest might look like this: ‘s3://bucketname/example.manifest`

    The manifest is an S3 object which is a JSON file with the following format:

    ‘[`

    ‘ “s3://customer_bucket/some/prefix/”,`

    ‘ “relative/path/to/custdata-1”,`

    ‘ “relative/path/custdata-2”,`

    ‘ …`

    ‘ ]`

    The preceding JSON matches the following ‘s3Uris`:

    ‘s3://customer_bucket/some/prefix/relative/path/to/custdata-1`

    ‘s3://customer_bucket/some/prefix/relative/path/custdata-2`

    ‘…`

    The complete set of ‘s3uris` in this manifest is the input data for the channel for this datasource. The object that each `s3uris` points to must be readable by the IAM role that Amazon SageMaker uses to perform tasks on your behalf.

Returns:

  • (String) 


9442
9443
9444
9445
9446
9447
9448
 
# File 'lib/aws-sdk-sagemaker/types.rb', line 9442

class S3DataSource < Struct.new(
  :s3_data_type,
  :s3_uri,
  :s3_data_distribution_type,
  :attribute_names)
  include Aws::Structure
end