Class: Google::Cloud::Dataplex::V1::Task::SparkTaskConfig

Inherits:
Object
  • Object
show all
Extended by:
Protobuf::MessageExts::ClassMethods
Includes:
Protobuf::MessageExts
Defined in:
proto_docs/google/cloud/dataplex/v1/tasks.rb

Overview

User-specified config for running a Spark task.

Instance Attribute Summary collapse

Instance Attribute Details

#archive_uris ⇒ ::Array<::String>

Returns Optional. Cloud Storage URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Returns:

  • (::Array<::String>) —

    Optional. Cloud Storage URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#file_uris ⇒ ::Array<::String>

Returns Optional. Cloud Storage URIs of files to be placed in the working directory of each executor.

Returns:

  • (::Array<::String>) —

    Optional. Cloud Storage URIs of files to be placed in the working directory of each executor.



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#infrastructure_spec ⇒ ::Google::Cloud::Dataplex::V1::Task::InfrastructureSpec

Returns Optional. Infrastructure specification for the execution.

Returns:



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#main_class ⇒ ::String

Returns The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. The execution args are passed in as a sequence of named process arguments (--key=value).

Returns:

  • (::String) —

    The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. The execution args are passed in as a sequence of named process arguments (--key=value).



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#main_jar_file_uri ⇒ ::String

Returns The Cloud Storage URI of the jar file that contains the main class. The execution args are passed in as a sequence of named process arguments (--key=value).

Returns:

  • (::String) —

    The Cloud Storage URI of the jar file that contains the main class. The execution args are passed in as a sequence of named process arguments (--key=value).



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#python_script_file ⇒ ::String

Returns The Gcloud Storage URI of the main Python file to use as the driver. Must be a .py file. The execution args are passed in as a sequence of named process arguments (--key=value).

Returns:

  • (::String) —

    The Gcloud Storage URI of the main Python file to use as the driver. Must be a .py file. The execution args are passed in as a sequence of named process arguments (--key=value).



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#sql_script ⇒ ::String

Returns The query text. The execution args are used to declare a set of script variables (set key="value";).

Returns:

  • (::String) —

    The query text. The execution args are used to declare a set of script variables (set key="value";).



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end

#sql_script_file ⇒ ::String

Returns A reference to a query file. This can be the Cloud Storage URI of the query file or it can the path to a SqlScript Content. The execution args are used to declare a set of script variables (set key="value";).

Returns:

  • (::String) —

    A reference to a query file. This can be the Cloud Storage URI of the query file or it can the path to a SqlScript Content. The execution args are used to declare a set of script variables (set key="value";).



284
285
286
287
 
# File 'proto_docs/google/cloud/dataplex/v1/tasks.rb', line 284

class SparkTaskConfig
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods
end