Class: Aws::Glue::Types::CreateCrawlerRequest

Inherits:

Struct

Object
Struct
Aws::Glue::Types::CreateCrawlerRequest

show all

Includes:: Structure

Defined in:: lib/aws-sdk-glue/types.rb

Overview

Note:

When making an API call, you may pass CreateCrawlerRequest data as a hash:

{
  name: "NameString", # required
  role: "Role", # required
  database_name: "DatabaseName",
  description: "DescriptionString",
  targets: { # required
    s3_targets: [
      {
        path: "Path",
        exclusions: ["Path"],
        connection_name: "ConnectionName",
      },
    ],
    jdbc_targets: [
      {
        connection_name: "ConnectionName",
        path: "Path",
        exclusions: ["Path"],
      },
    ],
    mongo_db_targets: [
      {
        connection_name: "ConnectionName",
        path: "Path",
        scan_all: false,
      },
    ],
    dynamo_db_targets: [
      {
        path: "Path",
        scan_all: false,
        scan_rate: 1.0,
      },
    ],
    catalog_targets: [
      {
        database_name: "NameString", # required
        tables: ["NameString"], # required
      },
    ],
  },
  schedule: "CronExpression",
  classifiers: ["NameString"],
  table_prefix: "TablePrefix",
  schema_change_policy: {
    update_behavior: "LOG", # accepts LOG, UPDATE_IN_DATABASE
    delete_behavior: "LOG", # accepts LOG, DELETE_FROM_DATABASE, DEPRECATE_IN_DATABASE
  },
  recrawl_policy: {
    recrawl_behavior: "CRAWL_EVERYTHING", # accepts CRAWL_EVERYTHING, CRAWL_NEW_FOLDERS_ONLY
  },
  lineage_configuration: {
    crawler_lineage_settings: "ENABLE", # accepts ENABLE, DISABLE
  },
  configuration: "CrawlerConfiguration",
  crawler_security_configuration: "CrawlerSecurityConfiguration",
  tags: {
    "TagKey" => "TagValue",
  },
}

Constant Summary collapse

SENSITIVE =

[]

Instance Attribute Summary collapse

#classifiers ⇒ Array<String>

A list of custom classifiers that the user has registered.
#configuration ⇒ String

Crawler configuration information.
#crawler_security_configuration ⇒ String

The name of the ‘SecurityConfiguration` structure to be used by this crawler.
#database_name ⇒ String

The AWS Glue database where results are written, such as: ‘arn:aws:daylight:us-east-1::database/sometable/*`.
#description ⇒ String

A description of the new crawler.
#lineage_configuration ⇒ Types::LineageConfiguration

Specifies data lineage configuration settings for the crawler.
#name ⇒ String

Name of the new crawler.
#recrawl_policy ⇒ Types::RecrawlPolicy

A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.
#role ⇒ String

The IAM role or Amazon Resource Name (ARN) of an IAM role used by the new crawler to access customer resources.
#schedule ⇒ String

A ‘cron` expression used to specify the schedule (see [Time-Based Schedules for Jobs and Crawlers]. For example, to run something every day at 12:15 UTC, you would specify: `cron(15 12 * * ? *)`.
#schema_change_policy ⇒ Types::SchemaChangePolicy

The policy for the crawler’s update and deletion behavior.
#table_prefix ⇒ String

The table prefix used for catalog tables that are created.
#tags ⇒ Hash<String,String>

The tags to use with this crawler request.
#targets ⇒ Types::CrawlerTargets

A list of collection of targets to crawl.

Instance Attribute Details

#classifiers ⇒ `Array<String>`

A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.

Returns:

(Array<String>)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#configuration ⇒ `String`

Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler’s behavior. For more information, see [Configuring a Crawler].

[1]: docs.aws.amazon.com/glue/latest/dg/crawler-configuration.html

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#crawler_security_configuration ⇒ `String`

The name of the ‘SecurityConfiguration` structure to be used by this crawler.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#database_name ⇒ `String`

The AWS Glue database where results are written, such as: ‘arn:aws:daylight:us-east-1::database/sometable/*`.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#description ⇒ `String`

A description of the new crawler.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#lineage_configuration ⇒ `Types::LineageConfiguration`

Specifies data lineage configuration settings for the crawler.

Returns:

(Types::LineageConfiguration)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#name ⇒ `String`

Name of the new crawler.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#recrawl_policy ⇒ `Types::RecrawlPolicy`

A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.

Returns:

(Types::RecrawlPolicy)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#role ⇒ `String`

The IAM role or Amazon Resource Name (ARN) of an IAM role used by the new crawler to access customer resources.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#schedule ⇒ `String`

A ‘cron` expression used to specify the schedule (see [Time-Based Schedules for Jobs and Crawlers]. For example, to run something every day at 12:15 UTC, you would specify: `cron(15 12 * * ? *)`.

[1]: docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#schema_change_policy ⇒ `Types::SchemaChangePolicy`

The policy for the crawler’s update and deletion behavior.

Returns:

(Types::SchemaChangePolicy)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#table_prefix ⇒ `String`

The table prefix used for catalog tables that are created.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#tags ⇒ `Hash<String,String>`

The tags to use with this crawler request. You may use tags to limit access to the crawler. For more information about tags in AWS Glue, see [AWS Tags in AWS Glue] in the developer guide.

[1]: docs.aws.amazon.com/glue/latest/dg/monitor-tags.html

Returns:

(Hash<String,String>)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

#targets ⇒ `Types::CrawlerTargets`

A list of collection of targets to crawl.

Returns:

(Types::CrawlerTargets)

# File 'lib/aws-sdk-glue/types.rb', line 2804

class CreateCrawlerRequest < Struct.new(
  :name,
  :role,
  :database_name,
  :description,
  :targets,
  :schedule,
  :classifiers,
  :table_prefix,
  :schema_change_policy,
  :recrawl_policy,
  :lineage_configuration,
  :configuration,
  :crawler_security_configuration,
  :tags)
  SENSITIVE = []
  include Aws::Structure
end

Class: Aws::Glue::Types::CreateCrawlerRequest

Overview

Constant Summary collapse

Instance Attribute Summary collapse

Instance Attribute Details

#classifiers ⇒ Array<String>

#configuration ⇒ String

#crawler_security_configuration ⇒ String

#database_name ⇒ String

#description ⇒ String

#lineage_configuration ⇒ Types::LineageConfiguration

#name ⇒ String

#recrawl_policy ⇒ Types::RecrawlPolicy

#role ⇒ String

#schedule ⇒ String

#schema_change_policy ⇒ Types::SchemaChangePolicy

#table_prefix ⇒ String

#tags ⇒ Hash<String,String>

#targets ⇒ Types::CrawlerTargets

#classifiers ⇒ `Array<String>`

#configuration ⇒ `String`

#crawler_security_configuration ⇒ `String`

#database_name ⇒ `String`

#description ⇒ `String`

#lineage_configuration ⇒ `Types::LineageConfiguration`

#name ⇒ `String`

#recrawl_policy ⇒ `Types::RecrawlPolicy`

#role ⇒ `String`

#schedule ⇒ `String`

#schema_change_policy ⇒ `Types::SchemaChangePolicy`

#table_prefix ⇒ `String`

#tags ⇒ `Hash<String,String>`

#targets ⇒ `Types::CrawlerTargets`