Class: Google::Cloud::Bigquery::Data

Inherits:
Array
  • Object
show all
Defined in:
lib/google/cloud/bigquery/data.rb

Overview

Data

Represents a page of results (rows) as an array of hashes. Because Data delegates to Array, methods such as Array#count represent the number of rows in the page. In addition, methods of this class include result set metadata such as total and provide access to the schema of the query or table. See Project#query, Google::Cloud::Bigquery::Dataset#query and Table#data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Instance Method Summary collapse

Instance Method Details

#all(request_limit: nil) {|row| ... } ⇒ Enumerator

Retrieves all rows by repeatedly loading #next until #next? returns false. Calls the given block once for each row, which is passed as the parameter.

An enumerator is returned if no block is given.

This method may make several API calls until all rows are retrieved. Be sure to use as narrow a search criteria as possible. Please use with caution.

Examples:

Iterating each rows by passing a block:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

table.data.all do |row|
  puts row[:word]
end

Using the enumerator by not passing a block:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

words = table.data.all.map do |row|
  row[:word]
end

Limit the number of API calls made:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

table.data.all(request_limit: 10) do |row|
  puts row[:word]
end

Parameters:

  • request_limit (Integer) (defaults to: nil) —

    The upper limit of API requests to make to load all data. Default is no limit.

Yields:

  • (row) —

    The block for accessing each row of data.

Yield Parameters:

  • row (Hash) —

    The row object.

Returns:

  • (Enumerator) —

    An enumerator providing access to all of the data.



440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
 
# File 'lib/google/cloud/bigquery/data.rb', line 440

def all request_limit: nil
  request_limit = request_limit.to_i if request_limit
  unless block_given?
    return enum_for(:all, request_limit: request_limit)
  end
  results = self
  loop do
    results.each { |r| yield r }
    if request_limit
      request_limit -= 1
      break if request_limit < 0
    end
    break unless results.next?
    results = results.next
  end
end

#ddl? ⇒ Boolean

Whether the query that created this data was a DDL statement.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
data = bigquery.query "CREATE TABLE my_table (x INT64)"

data.statement_type #=> "CREATE_TABLE"
data.ddl? #=> true

Returns:

  • (Boolean)

See Also:



251
252
253
254
 
# File 'lib/google/cloud/bigquery/data.rb', line 251

def ddl?
  %w[CREATE_MODEL CREATE_TABLE CREATE_TABLE_AS_SELECT CREATE_VIEW \
     DROP_MODEL DROP_TABLE DROP_VIEW].include? statement_type
end

#ddl_operation_performed ⇒ String?

The DDL operation performed, possibly dependent on the pre-existence of the DDL target. (See #ddl_target_table.) Possible values (new values might be added in the future):

  • "CREATE": The query created the DDL target.
  • "SKIP": No-op. Example cases: the query is CREATE TABLE IF NOT EXISTS while the table already exists, or the query is DROP TABLE IF EXISTS while the table does not exist.
  • "REPLACE": The query replaced the DDL target. Example case: the query is CREATE OR REPLACE TABLE, and the table already exists.
  • "DROP": The query deleted the DDL target.

Returns:

  • (String, nil) —

    The DDL operation performed.



294
295
296
297
 
# File 'lib/google/cloud/bigquery/data.rb', line 294

def ddl_operation_performed
  return nil unless job_gapi && job_gapi.statistics.query
  job_gapi.statistics.query.ddl_operation_performed
end

#ddl_target_table ⇒ Google::Cloud::Bigquery::Table?

The DDL target table, in reference state. (See Table#reference?.) Present only for CREATE/DROP TABLE/VIEW queries. (See #statement_type.)

Returns:



307
308
309
310
311
312
313
 
# File 'lib/google/cloud/bigquery/data.rb', line 307

def ddl_target_table
  return nil unless job_gapi && job_gapi.statistics.query
  ensure_service!
  table = job_gapi.statistics.query.ddl_target_table
  return nil unless table
  Google::Cloud::Bigquery::Table.new_reference_from_gapi table, service
end

#dml? ⇒ Boolean

Whether the query that created this data was a DML statement.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
data = bigquery.query "UPDATE my_table " \
                      "SET x = x + 1 " \
                      "WHERE x IS NOT NULL"

data.statement_type #=> "UPDATE"
data.dml? #=> true

Returns:

  • (Boolean)

See Also:



275
276
277
 
# File 'lib/google/cloud/bigquery/data.rb', line 275

def dml?
  %w[INSERT UPDATE MERGE DELETE].include? statement_type
end

#etag ⇒ String

An ETag hash for the page of results represented by the data instance.

Returns:

  • (String) —

    The ETag hash.



88
89
90
 
# File 'lib/google/cloud/bigquery/data.rb', line 88

def etag
  @gapi_json[:etag]
end

#fields ⇒ Array<Schema::Field>

The fields of the data, obtained from the schema of the table from which the data was read.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

data.fields.each do |field|
  puts field.name
end

Returns:



176
177
178
 
# File 'lib/google/cloud/bigquery/data.rb', line 176

def fields
  schema.fields
end

#headers ⇒ Array<Symbol>

The names of the columns in the data, obtained from the schema of the table from which the data was read.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

data.headers.each do |header|
  puts header
end

Returns:

  • (Array<Symbol>) —

    An array of column names.



199
200
201
 
# File 'lib/google/cloud/bigquery/data.rb', line 199

def headers
  schema.headers
end

#kind ⇒ String

The resource type of the API response.

Returns:

  • (String) —

    The resource type.



79
80
81
 
# File 'lib/google/cloud/bigquery/data.rb', line 79

def kind
  @gapi_json[:kind]
end

#next ⇒ Data

Retrieves the next page of data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Returns:

  • (Data) —

    A new instance providing the next page of data.



377
378
379
380
381
382
383
384
385
 
# File 'lib/google/cloud/bigquery/data.rb', line 377

def next
  return nil unless next?
  ensure_service!
  data_json = service.list_tabledata \
    @table_gapi.table_reference.dataset_id,
    @table_gapi.table_reference.table_id,
    token: token
  self.class.from_gapi_json data_json, @table_gapi, job_gapi, @service
end

#next? ⇒ Boolean

Whether there is a next page of data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Returns:

  • (Boolean) —

    true when there is a next page, false otherwise.



350
351
352
 
# File 'lib/google/cloud/bigquery/data.rb', line 350

def next?
  !token.nil?
end

#num_dml_affected_rows ⇒ Integer?

The number of rows affected by a DML statement. Present only for DML statements INSERT, UPDATE or DELETE. (See #statement_type.)

Returns:

  • (Integer, nil) —

    The number of rows affected by a DML statement, or nil if the query is not a DML statement.



322
323
324
325
 
# File 'lib/google/cloud/bigquery/data.rb', line 322

def num_dml_affected_rows
  return nil unless job_gapi && job_gapi.statistics.query
  job_gapi.statistics.query.num_dml_affected_rows
end

#schema ⇒ Schema

The schema of the table from which the data was read.

The returned object is frozen and changes are not allowed. Use Table#schema to update the schema.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"

data = table.data

schema = data.schema
field = schema.field "name"
field.required? #=> true

Returns:

  • (Schema) —

    A schema object.



152
153
154
155
 
# File 'lib/google/cloud/bigquery/data.rb', line 152

def schema
  return nil unless @table_gapi
  Schema.from_gapi(@table_gapi.schema).freeze
end

#statement_type ⇒ String?

The type of query statement, if valid. Possible values (new values might be added in the future):

Returns:

  • (String, nil) —

    The type of query statement.



229
230
231
232
 
# File 'lib/google/cloud/bigquery/data.rb', line 229

def statement_type
  return nil unless job_gapi && job_gapi.statistics.query
  job_gapi.statistics.query.statement_type
end

#token ⇒ String

A token used for paging results. Used by the data instance to retrieve subsequent pages. See #next.

Returns:

  • (String) —

    The pagination token.



98
99
100
 
# File 'lib/google/cloud/bigquery/data.rb', line 98

def token
  @gapi_json[:pageToken]
end

#total ⇒ Integer

The total number of rows in the complete table.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

sql = "SELECT word FROM `bigquery-public-data.samples.shakespeare`"
job = bigquery.query_job sql

job.wait_until_done!
data = job.data

data.count # 100000
data.total # 164656
data.each do |row|
  puts row[:word]
end
data = data.next if data.next?

Returns:

  • (Integer) —

    The number of rows.



125
126
127
128
129
 
# File 'lib/google/cloud/bigquery/data.rb', line 125

def total
  Integer @gapi_json[:totalRows]
rescue StandardError
  nil
end