Class: Google::Cloud::Bigquery::External::JsonSource

Inherits:
DataSource
  • Object
show all
Defined in:
lib/google/cloud/bigquery/external.rb

Overview

JsonSource

JsonSource is a subclass of DataSource and represents a JSON external data source that can be queried from directly, such as Google Cloud Storage or Google Drive, even though the data is not stored in BigQuery. Instead of loading or streaming the data, this object references the external data source.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

json_url = "gs://bucket/path/to/data.json"
json_table = bigquery.external json_url do |json|
  json.schema do |schema|
    schema.string "name", mode: :required
    schema.string "email", mode: :required
    schema.integer "age", mode: :required
    schema.boolean "active", mode: :required
  end
end

data = bigquery.query "SELECT * FROM my_ext_table",
                      external: { my_ext_table: json_table }

# Iterate over the first page of results
data.each do |row|
  puts row[:name]
end
# Retrieve the next page of results
data = data.next if data.next?

Instance Method Summary collapse

Methods inherited from DataSource

#autodetect, #autodetect=, #avro?, #backup?, #bigtable?, #compression, #compression=, #csv?, #format, #ignore_unknown, #ignore_unknown=, #json?, #max_bad_records, #max_bad_records=, #sheets?, #urls

Instance Method Details

#fields ⇒ Array<Schema::Field>

The fields of the schema.

Returns:



1126
1127
1128
 
# File 'lib/google/cloud/bigquery/external.rb', line 1126

def fields
  schema.fields
end

#headers ⇒ Array<Symbol>

The names of the columns in the schema.

Returns:

  • (Array<Symbol>) —

    An array of column names.



1135
1136
1137
 
# File 'lib/google/cloud/bigquery/external.rb', line 1135

def headers
  schema.headers
end

#param_types ⇒ Hash

The types of the fields in the data in the schema, using the same format as the optional query parameter types.

Returns:

  • (Hash) —

    A hash with field names as keys, and types as values.



1145
1146
1147
 
# File 'lib/google/cloud/bigquery/external.rb', line 1145

def param_types
  schema.param_types
end

#schema(replace: false) {|schema| ... } ⇒ Google::Cloud::Bigquery::Schema

The schema for the data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

json_url = "gs://bucket/path/to/data.json"
json_table = bigquery.external json_url do |json|
  json.schema do |schema|
    schema.string "name", mode: :required
    schema.string "email", mode: :required
    schema.integer "age", mode: :required
    schema.boolean "active", mode: :required
  end
end

Parameters:

  • replace (Boolean) (defaults to: false) —

    Whether to replace the existing schema with the new schema. If true, the fields will replace the existing schema. If false, the fields will be added to the existing schema. The default value is false.

Yields:

  • (schema) —

    a block for setting the schema

Yield Parameters:

  • schema (Schema) —

    the object accepting the schema

Returns:



1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
 
# File 'lib/google/cloud/bigquery/external.rb', line 1084

def schema replace: false
  @schema ||= Schema.from_gapi @gapi.schema
  if replace
    frozen_check!
    @schema = Schema.from_gapi
  end
  @schema.freeze if frozen?
  yield @schema if block_given?
  @schema
end

#schema=(new_schema) ⇒ Object

Set the schema for the data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

json_shema = bigquery.schema do |schema|
  schema.string "name", mode: :required
  schema.string "email", mode: :required
  schema.integer "age", mode: :required
  schema.boolean "active", mode: :required
end

json_url = "gs://bucket/path/to/data.json"
json_table = bigquery.external json_url
json_table.schema = json_shema

Parameters:

  • new_schema (Schema) —

    The schema object.



1116
1117
1118
1119
 
# File 'lib/google/cloud/bigquery/external.rb', line 1116

def schema= new_schema
  frozen_check!
  @schema = new_schema
end