Class: Google::Cloud::Bigquery::LoadJob::Updater

Inherits:
Google::Cloud::Bigquery::LoadJob show all
Defined in:
lib/google/cloud/bigquery/load_job.rb

Overview

Yielded to a block to accumulate changes for a patch request.

Attributes collapse

Attributes collapse

Schema collapse

Methods inherited from Google::Cloud::Bigquery::LoadJob

#allow_jagged_rows?, #autodetect?, #backup?, #clustering?, #clustering_fields, #csv?, #delimiter, #destination, #encryption, #ignore_unknown_values?, #input_file_bytes, #input_files, #iso8859_1?, #json?, #max_bad_records, #null_marker, #output_bytes, #output_rows, #quote, #quoted_newlines?, #schema_update_options, #skip_leading_rows, #sources, #time_partitioning?, #time_partitioning_expiration, #time_partitioning_field, #time_partitioning_require_filter?, #time_partitioning_type, #utf8?

Methods inherited from Job

#cancel, #configuration, #created_at, #done?, #ended_at, #error, #errors, #failed?, #job_id, #labels, #location, #pending?, #project_id, #reload!, #rerun!, #running?, #started_at, #state, #statistics, #status, #user_email, #wait_until_done!

Instance Attribute Details

#updates ⇒ Object (readonly)

A list of attributes that were updated.



471
472
473
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 471

def updates
  @updates
end

Instance Method Details

#autodetect=(val) ⇒ Object

Allows BigQuery to autodetect the schema.

Parameters:

  • val (Boolean) —

    Indicates if BigQuery should automatically infer the options and schema for CSV and JSON sources. The default value is false.



1073
1074
1075
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1073

def autodetect= val
  @gapi.configuration.load.update! autodetect: val
end

#boolean(name, description: nil, mode: :nullable) ⇒ Object

Adds a boolean field to the schema.

See Schema#boolean.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.boolean "active", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



695
696
697
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 695

def boolean name, description: nil, mode: :nullable
  schema.boolean name, description: description, mode: mode
end

#bytes(name, description: nil, mode: :nullable) ⇒ Object

Adds a bytes field to the schema.

See Schema#bytes.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.bytes "avatar", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



723
724
725
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 723

def bytes name, description: nil, mode: :nullable
  schema.bytes name, description: description, mode: mode
end

#check_for_mutated_schema! ⇒ Object

Make sure any access changes are saved



879
880
881
882
883
884
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 879

def check_for_mutated_schema!
  return if @schema.nil?
  return unless @schema.changed?
  @gapi.configuration.load.schema = @schema.to_gapi
  patch_gapi! :schema
end

#clustering_fields=(fields) ⇒ Object

Sets one or more fields on which the destination table should be clustered. Must be specified with time-based partitioning, data in the table will be first partitioned and subsequently clustered.

Only top-level, non-repeated, simple-type fields are supported. When you cluster a table using multiple columns, the order of columns you specify is important. The order of the specified columns determines the sort order of the data.

See Google::Cloud::Bigquery::LoadJob#clustering_fields.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"

gs_url = "gs://my-bucket/file-name.csv"
load_job = dataset.load_job "my_new_table", gs_url do |job|
  job.time_partitioning_type  = "DAY"
  job.time_partitioning_field = "dob"
  job.schema do |schema|
    schema.timestamp "dob", mode: :required
    schema.string "first_name", mode: :required
    schema.string "last_name", mode: :required
  end
  job.clustering_fields = ["last_name", "first_name"]
end

load_job.wait_until_done!
load_job.done? #=> true

Parameters:

  • fields (Array<String>) —

    The clustering fields. Only top-level, non-repeated, simple-type fields are supported.

See Also:



1437
1438
1439
1440
1441
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1437

def clustering_fields= fields
  @gapi.configuration.load.clustering ||= \
    Google::Apis::BigqueryV2::Clustering.new
  @gapi.configuration.load.clustering.fields = fields
end

#create=(new_create) ⇒ Object

Sets the create disposition.

This specifies whether the job is allowed to create new tables. The default value is needed.

The following values are supported:

  • needed - Create the table if it does not exist.
  • never - The table must already exist. A 'notFound' error is raised if the table does not exist.

Parameters:

  • new_create (String) —

    The new create disposition.



956
957
958
959
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 956

def create= new_create
  @gapi.configuration.load.update! \
    create_disposition: Convert.create_disposition(new_create)
end

#date(name, description: nil, mode: :nullable) ⇒ Object

Adds a date field to the schema.

See Schema#date.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.date "birthday", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



835
836
837
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 835

def date name, description: nil, mode: :nullable
  schema.date name, description: description, mode: mode
end

#datetime(name, description: nil, mode: :nullable) ⇒ Object

Adds a datetime field to the schema.

See Schema#datetime.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.datetime "target_end", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



807
808
809
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 807

def datetime name, description: nil, mode: :nullable
  schema.datetime name, description: description, mode: mode
end

#delimiter=(val) ⇒ Object

Sets the separator for fields in a CSV file.

Parameters:

  • val (String) —

    Specifices the separator for fields in a CSV file. BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. Default is ,.



1100
1101
1102
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1100

def delimiter= val
  @gapi.configuration.load.update! field_delimiter: val
end

#encoding=(val) ⇒ Object

Sets the character encoding of the data.

Parameters:

  • val (String) —

    The character encoding of the data. The supported values are UTF-8 or ISO-8859-1. The default value is UTF-8.



1086
1087
1088
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1086

def encoding= val
  @gapi.configuration.load.update! encoding: val
end

#encryption=(val) ⇒ Object

Sets the encryption configuration of the destination table.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"

key_name = "projects/a/locations/b/keyRings/c/cryptoKeys/d"
encrypt_config = bigquery.encryption kms_key: key_name
job = dataset.load_job "my_table", "gs://abc/file" do |job|
  job.encryption = encrypt_config
end

Parameters:

  • val (Google::Cloud::BigQuery::EncryptionConfiguration) —

    Custom encryption configuration (e.g., Cloud KMS keys).



1239
1240
1241
1242
1243
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1239

def encryption= val
  @gapi.configuration.load.update!(
    destination_encryption_configuration: val.to_gapi
  )
end

#float(name, description: nil, mode: :nullable) ⇒ Object

Adds a floating-point number field to the schema.

See Schema#float.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.float "price", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



637
638
639
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 637

def float name, description: nil, mode: :nullable
  schema.float name, description: description, mode: mode
end

#format=(new_format) ⇒ Object

Sets the source file format. The default value is csv.

The following values are supported:

Parameters:

  • new_format (String) —

    The new source format.



935
936
937
938
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 935

def format= new_format
  @gapi.configuration.load.update! \
    source_format: Convert.source_format(new_format)
end

#ignore_unknown=(val) ⇒ Object

Allows unknown columns to be ignored.

Parameters:

  • val (Boolean) —

    Indicates if BigQuery should allow extra values that are not represented in the table schema. If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. The default value is false.

    The format property determines what BigQuery treats as an extra value:

    • CSV: Trailing columns
    • JSON: Named values that don't match any column names


1122
1123
1124
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1122

def ignore_unknown= val
  @gapi.configuration.load.update! ignore_unknown_values: val
end

#integer(name, description: nil, mode: :nullable) ⇒ Object

Adds an integer field to the schema.

See Schema#integer.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.integer "age", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



609
610
611
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 609

def integer name, description: nil, mode: :nullable
  schema.integer name, description: description, mode: mode
end

#jagged_rows=(val) ⇒ Object

Sets flag for allowing jagged rows.

Accept rows that are missing trailing optional columns. The missing values are treated as nulls. If false, records with missing trailing columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. The default value is false. Only applicable to CSV, ignored for other formats.

Parameters:

  • val (Boolean) —

    Accept rows that are missing trailing optional columns.



1047
1048
1049
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1047

def jagged_rows= val
  @gapi.configuration.load.update! allow_jagged_rows: val
end

#labels=(val) ⇒ Object

Sets the labels to use for the load job.

Parameters:

  • val (Hash) —

    A hash of user-provided labels associated with the job. You can use these to organize and group your jobs. Label keys and values can be no longer than 63 characters, can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. Label values are optional. Label keys must start with a letter and each label in the list must have a different key.



1258
1259
1260
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1258

def labels= val
  @gapi.configuration.update! labels: val
end

#location=(value) ⇒ Object

Sets the geographic location where the job should run. Required except for US and EU.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |j|
  j.schema do |s|
    s.string "first_name", mode: :required
    s.record "cities_lived", mode: :repeated do |r|
      r.string "place", mode: :required
      r.integer "number_of_years", mode: :required
    end
  end
  j.location = "EU"
end

Parameters:

  • value (String) —

    A geographic location, such as "US", "EU" or "asia-northeast1". Required except for US and EU.



910
911
912
913
914
915
916
917
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 910

def location= value
  @gapi.job_reference.location = value
  return unless value.nil?

  # Treat assigning value of nil the same as unsetting the value.
  unset = @gapi.job_reference.instance_variables.include? :@location
  @gapi.job_reference.remove_instance_variable :@location if unset
end

#max_bad_records=(val) ⇒ Object

Sets the maximum number of bad records that can be ignored.

Parameters:

  • val (Integer) —

    The maximum number of bad records that BigQuery can ignore when running the job. If the number of bad records exceeds this value, an invalid error is returned in the job result. The default value is 0, which requires that all records are valid.



1137
1138
1139
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1137

def max_bad_records= val
  @gapi.configuration.load.update! max_bad_records: val
end

#null_marker=(val) ⇒ Object

Sets the string that represents a null value in a CSV file.

Parameters:

  • val (String) —

    Specifies a string that represents a null value in a CSV file. For example, if you specify \N, BigQuery interprets \N as a null value when loading a CSV file. The default value is the empty string. If you set this property to a custom value, BigQuery throws an error if an empty string is present for all data types except for STRING and BYTE. For STRING and BYTE columns, BigQuery interprets the empty string as an empty value.



1155
1156
1157
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1155

def null_marker= val
  @gapi.configuration.load.update! null_marker: val
end

#numeric(name, description: nil, mode: :nullable) ⇒ Object

Adds a numeric number field to the schema. Numeric is a fixed-precision numeric type with 38 decimal digits, 9 that follow the decimal point.

See Schema#numeric

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.numeric "total_cost", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



667
668
669
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 667

def numeric name, description: nil, mode: :nullable
  schema.numeric name, description: description, mode: mode
end

#projection_fields=(new_fields) ⇒ Object

Sets the projection fields.

If the format option is set to datastore_backup, indicates which entity properties to load from a Cloud Datastore backup. Property names are case sensitive and must be top-level properties. If not set, BigQuery loads all properties. If any named property isn't found in the Cloud Datastore backup, an invalid error is returned.

Parameters:

  • new_fields (Array<String>) —

    The new projection fields.



997
998
999
1000
1001
1002
1003
1004
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 997

def projection_fields= new_fields
  if new_fields.nil?
    @gapi.configuration.load.update! projection_fields: nil
  else
    @gapi.configuration.load.update! \
      projection_fields: Array(new_fields)
  end
end

#quote=(val) ⇒ Object

Sets the character to use to quote string values in CSVs.

Parameters:

  • val (String) —

    The value that is used to quote data sections in a CSV file. BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. The default value is a double-quote ". If your data does not contain quoted sections, set the property value to an empty string. If your data contains quoted newline characters, you must also set the allowQuotedNewlines property to true.



1173
1174
1175
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1173

def quote= val
  @gapi.configuration.load.update! quote: val
end

#quoted_newlines=(val) ⇒ Object

Allows quoted data sections to contain newline characters in CSV.

Parameters:

  • val (Boolean) —

    Indicates if BigQuery should allow quoted data sections that contain newline characters in a CSV file. The default value is false.



1060
1061
1062
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1060

def quoted_newlines= val
  @gapi.configuration.load.update! allow_quoted_newlines: val
end

#record(name, description: nil, mode: nil) {|nested_schema| ... } ⇒ Object

Adds a record field to the schema. A block must be passed describing the nested fields of the record. For more information about nested and repeated records, see Preparing Data for BigQuery .

See Schema#record.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.record "cities_lived", mode: :repeated do |cities_lived|
    cities_lived.string "place", mode: :required
    cities_lived.integer "number_of_years", mode: :required
  end
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: nil) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.

Yields:

  • (nested_schema) —

    a block for setting the nested schema

Yield Parameters:

  • nested_schema (Schema) —

    the object accepting the nested schema



873
874
875
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 873

def record name, description: nil, mode: nil, &block
  schema.record name, description: description, mode: mode, &block
end

#schema(replace: false) {|schema| ... } ⇒ Google::Cloud::Bigquery::Schema

Returns the table's schema. This method can also be used to set, replace, or add to the schema by passing a block. See Schema for available methods.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |j|
  j.schema do |s|
    s.string "first_name", mode: :required
    s.record "cities_lived", mode: :repeated do |r|
      r.string "place", mode: :required
      r.integer "number_of_years", mode: :required
    end
  end
end

Parameters:

  • replace (Boolean) (defaults to: false) —

    Whether to replace the existing schema with the new schema. If true, the fields will replace the existing schema. If false, the fields will be added to the existing schema. When a table already contains data, schema changes must be additive. Thus, the default value is false.

Yields:

  • (schema) —

    a block for setting the schema

Yield Parameters:

  • schema (Schema) —

    the object accepting the schema

Returns:



513
514
515
516
517
518
519
520
521
522
523
524
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 513

def schema replace: false
  # Same as Table#schema, but not frozen
  # TODO: make sure to call ensure_full_data! on Dataset#update
  @schema ||= Schema.from_gapi @gapi.configuration.load.schema
  if block_given?
    @schema = Schema.from_gapi if replace
    yield @schema
    check_for_mutated_schema!
  end
  # Do not freeze on updater, allow modifications
  @schema
end

#schema=(new_schema) ⇒ Object

Sets the schema of the destination table.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
schema = bigquery.schema do |s|
  s.string "first_name", mode: :required
  s.record "cities_lived", mode: :repeated do |nested_schema|
    nested_schema.string "place", mode: :required
    nested_schema.integer "number_of_years", mode: :required
  end
end
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |j|
  j.schema = schema
end

Parameters:

  • new_schema (Google::Cloud::Bigquery::Schema) —

    The schema for the destination table. Optional. The schema can be omitted if the destination table already exists, or if you're loading data from a source that includes a schema, such as Avro or a Google Cloud Datastore backup.



553
554
555
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 553

def schema= new_schema
  @schema = new_schema
end

#schema_update_options=(new_options) ⇒ Object

Sets the schema update options, which allow the schema of the destination table to be updated as a side effect of the load job if a schema is autodetected or supplied in the job configuration. Schema update options are supported in two cases: when write disposition is WRITE_APPEND; when write disposition is WRITE_TRUNCATE and the destination table is a partition of a table, specified by partition decorators. For normal tables, WRITE_TRUNCATE will always overwrite the schema. One or more of the following values are specified:

  • ALLOW_FIELD_ADDITION: allow adding a nullable field to the schema.
  • ALLOW_FIELD_RELAXATION: allow relaxing a required field in the original schema to nullable.

Parameters:

  • new_options (Array<String>) —

    The new schema update options.



1197
1198
1199
1200
1201
1202
1203
1204
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1197

def schema_update_options= new_options
  if new_options.nil?
    @gapi.configuration.load.update! schema_update_options: nil
  else
    @gapi.configuration.load.update! \
      schema_update_options: Array(new_options)
  end
end

#skip_leading=(val) ⇒ Object

Sets the number of leading rows to skip in the file.

Parameters:

  • val (Integer) —

    The number of rows at the top of a CSV file that BigQuery will skip when loading the data. The default value is 0. This property is useful if you have header rows in the file that should be skipped.



1216
1217
1218
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1216

def skip_leading= val
  @gapi.configuration.load.update! skip_leading_rows: val
end

#source_uris=(new_uris) ⇒ Object

Sets the source URIs to load.

The fully-qualified URIs that point to your data in Google Cloud.

  • For Google Cloud Storage URIs: Each URI can contain one '*' wildcard character and it must come after the 'bucket' name. Size limits related to load jobs apply to external data sources. For
  • Google Cloud Bigtable URIs: Exactly one URI can be specified and it has be a fully specified and valid HTTPS URL for a Google Cloud Bigtable table.
  • For Google Cloud Datastore backups: Exactly one URI can be specified. Also, the '*' wildcard character is not allowed.

Parameters:

  • new_uris (Array<String>) —

    The new source URIs to load.



1024
1025
1026
1027
1028
1029
1030
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1024

def source_uris= new_uris
  if new_uris.nil?
    @gapi.configuration.load.update! source_uris: nil
  else
    @gapi.configuration.load.update! source_uris: Array(new_uris)
  end
end

#string(name, description: nil, mode: :nullable) ⇒ Object

Adds a string field to the schema.

See Schema#string.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.string "first_name", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



581
582
583
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 581

def string name, description: nil, mode: :nullable
  schema.string name, description: description, mode: mode
end

#time(name, description: nil, mode: :nullable) ⇒ Object

Adds a time field to the schema.

See Schema#time.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.time "duration", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



779
780
781
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 779

def time name, description: nil, mode: :nullable
  schema.time name, description: description, mode: mode
end

#time_partitioning_expiration=(expiration) ⇒ Object

Sets the partition expiration for the destination table. See Partitioned Tables.

The destination table must also be partitioned. See #time_partitioning_type=.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"

gs_url = "gs://my-bucket/file-name.csv"
load_job = dataset.load_job "my_new_table", gs_url do |job|
  job.time_partitioning_type = "DAY"
  job.time_partitioning_expiration = 86_400
end

load_job.wait_until_done!
load_job.done? #=> true

Parameters:

  • expiration (Integer) —

    An expiration time, in seconds, for data in partitions.



1367
1368
1369
1370
1371
1372
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1367

def time_partitioning_expiration= expiration
  @gapi.configuration.load.time_partitioning ||= \
    Google::Apis::BigqueryV2::TimePartitioning.new
  @gapi.configuration.load.time_partitioning.update! \
    expiration_ms: expiration * 1000
end

#time_partitioning_field=(field) ⇒ Object

Sets the field on which to partition the destination table. If not set, the destination table is partitioned by pseudo column _PARTITIONTIME; if set, the table is partitioned by this field. See Partitioned Tables.

The destination table must also be partitioned. See #time_partitioning_type=.

You can only set the partitioning field while creating a table. BigQuery does not allow you to change partitioning on an existing table.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"

gs_url = "gs://my-bucket/file-name.csv"
load_job = dataset.load_job "my_new_table", gs_url do |job|
  job.time_partitioning_type  = "DAY"
  job.time_partitioning_field = "dob"
  job.schema do |schema|
    schema.timestamp "dob", mode: :required
  end
end

load_job.wait_until_done!
load_job.done? #=> true

Parameters:

  • field (String) —

    The partition field. The field must be a top-level TIMESTAMP or DATE field. Its mode must be NULLABLE or REQUIRED.



1333
1334
1335
1336
1337
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1333

def time_partitioning_field= field
  @gapi.configuration.load.time_partitioning ||= \
    Google::Apis::BigqueryV2::TimePartitioning.new
  @gapi.configuration.load.time_partitioning.update! field: field
end

#time_partitioning_require_filter=(val) ⇒ Object

If set to true, queries over the destination table will require a partition filter that can be used for partition elimination to be specified. See Partitioned Tables.

Parameters:

  • val (Boolean) —

    Indicates if queries over the destination table will require a partition filter. The default value is false.



1385
1386
1387
1388
1389
1390
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1385

def time_partitioning_require_filter= val
  @gapi.configuration.load.time_partitioning ||= \
    Google::Apis::BigqueryV2::TimePartitioning.new
  @gapi.configuration.load.time_partitioning.update! \
    require_partition_filter: val
end

#time_partitioning_type=(type) ⇒ Object

Sets the partitioning for the destination table. See Partitioned Tables.

You can only set the partitioning field while creating a table. BigQuery does not allow you to change partitioning on an existing table.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"

gs_url = "gs://my-bucket/file-name.csv"
load_job = dataset.load_job "my_new_table", gs_url do |job|
  job.time_partitioning_type = "DAY"
end

load_job.wait_until_done!
load_job.done? #=> true

Parameters:

  • type (String) —

    The partition type. Currently the only supported value is "DAY".



1289
1290
1291
1292
1293
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 1289

def time_partitioning_type= type
  @gapi.configuration.load.time_partitioning ||= \
    Google::Apis::BigqueryV2::TimePartitioning.new
  @gapi.configuration.load.time_partitioning.update! type: type
end

#timestamp(name, description: nil, mode: :nullable) ⇒ Object

Adds a timestamp field to the schema.

See Schema#timestamp.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
job = dataset.load_job "my_table", "gs://abc/file" do |schema|
  schema.timestamp "creation_date", mode: :required
end

Parameters:

  • name (String) —

    The field name. The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 128 characters.

  • description (String) (defaults to: nil) —

    A description of the field.

  • mode (Symbol) (defaults to: :nullable) —

    The field's mode. The possible values are :nullable, :required, and :repeated. The default value is :nullable.



751
752
753
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 751

def timestamp name, description: nil, mode: :nullable
  schema.timestamp name, description: description, mode: mode
end

#write=(new_write) ⇒ Object

Sets the write disposition.

This specifies how to handle data already present in the table. The default value is append.

The following values are supported:

  • truncate - BigQuery overwrites the table data.
  • append - BigQuery appends the data to the table.
  • empty - An error will be returned if the table already contains data.

Parameters:

  • new_write (String) —

    The new write disposition.



978
979
980
981
 
# File 'lib/google/cloud/bigquery/load_job.rb', line 978

def write= new_write
  @gapi.configuration.load.update! \
    write_disposition: Convert.write_disposition(new_write)
end