Class: Gcloud::Bigquery::Dataset

Inherits:
Object
  • Object
show all
Defined in:
lib/gcloud/bigquery/dataset.rb,
lib/gcloud/bigquery/dataset/list.rb,
lib/gcloud/bigquery/dataset/access.rb

Overview

# Dataset

Represents a Dataset. A dataset is a grouping mechanism that holds zero or more tables. Datasets are the lowest level unit of access control; you cannot control access at the table level. A dataset is contained within a specific project.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery

dataset = bigquery.create_dataset "my_dataset",
                                  name: "My Dataset",
                                  description: "This is my Dataset"

Direct Known Subclasses

Updater

Defined Under Namespace

Classes: Access, List, Updater

Instance Attribute Summary collapse

Attributes collapse

Lifecycle collapse

Table collapse

Data collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Dataset

Returns a new instance of Dataset.



55
56
57
58
 
# File 'lib/gcloud/bigquery/dataset.rb', line 55

def initialize
  @service = nil
  @gapi = {}
end

Instance Attribute Details

#gapi ⇒ Object 



51
52
53
 
# File 'lib/gcloud/bigquery/dataset.rb', line 51

def gapi
  @gapi
end

#service ⇒ Object 



47
48
49
 
# File 'lib/gcloud/bigquery/dataset.rb', line 47

def service
  @service
end

Class Method Details

.from_gapi(gapi, conn) ⇒ Object 



661
662
663
664
665
666
 
# File 'lib/gcloud/bigquery/dataset.rb', line 661

def self.from_gapi gapi, conn
  new.tap do |f|
    f.gapi = gapi
    f.service = conn
  end
end

Instance Method Details

#access {|access| ... } ⇒ Gcloud::Bigquery::Dataset::Access

Retrieves the access rules for a Dataset. The rules can be updated when passing a block, see Access for all the methods available.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"

dataset.access #=> [{"role"=>"OWNER",
               #     "specialGroup"=>"projectOwners"},
               #    {"role"=>"WRITER",
               #     "specialGroup"=>"projectWriters"},
               #    {"role"=>"READER",
               #     "specialGroup"=>"projectReaders"},
               #    {"role"=>"OWNER",
               #     "userByEmail"=>"123456789-...com"}]

Manage the access rules by passing a block:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"

dataset.access do |access|
  access.add_owner_group "[email protected]"
  access.add_writer_user "[email protected]"
  access.remove_writer_user "[email protected]"
  access.add_reader_special :all
  access.add_reader_view other_dataset_view_object
end

Yields:

  • (access) —

    a block for setting rules

Yield Parameters:

Returns:

See Also:



257
258
259
260
261
262
263
264
265
266
267
268
 
# File 'lib/gcloud/bigquery/dataset.rb', line 257

def access
  ensure_full_data!
  access_builder = Access.from_gapi @gapi
  if block_given?
    yield access_builder
    if access_builder.changed?
      @gapi.update! access: access_builder.to_gapi
      patch_gapi! :access
    end
  end
  access_builder.freeze
end

#api_url ⇒ Object

A URL that can be used to access the dataset using the REST API.



124
125
126
127
 
# File 'lib/gcloud/bigquery/dataset.rb', line 124

def api_url
  ensure_full_data!
  @gapi.self_link
end

#create_table(table_id, name: nil, description: nil, fields: nil) {|table| ... } ⇒ Gcloud::Bigquery::Table

Creates a new table. If you are adapting existing code that was written for the [Rest API ](cloud.google.com/bigquery/docs/reference/v2/tables#resource), you can pass the table’s schema as a hash (see example.)

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table"

You can also pass name and description options.

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table"
                             name: "My Table",
                             description: "A description of my table."

The table’s schema fields can be passed as an argument.

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"

schema_fields = [
  Gcloud::Bigquery::Schema::Field.new(
    "first_name", :string, mode: :required),
  Gcloud::Bigquery::Schema::Field.new(
    "cities_lived", :record, mode: :repeated
    fields: [
      Gcloud::Bigquery::Schema::Field.new(
        "place", :string, mode: :required),
      Gcloud::Bigquery::Schema::Field.new(
        "number_of_years", :integer, mode: :required),
      ])
]
table = dataset.create_table "my_table", fields: schema_fields

Or the table’s schema can be configured with the block.

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"

table = dataset.create_table "my_table" do |t|
  t.schema.string "first_name", mode: :required
  t.schema.record "cities_lived", mode: :required do |s|
    s.string "place", mode: :required
    s.integer "number_of_years", mode: :required
  end
end

You can define the schema using a nested block.

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
table = dataset.create_table "my_table" do |t|
  t.name = "My Table",
  t.description = "A description of my table."
  t.schema do |s|
    s.string "first_name", mode: :required
    s.record "cities_lived", mode: :repeated do |r|
      r.string "place", mode: :required
      r.integer "number_of_years", mode: :required
    end
  end
end

Parameters:

  • table_id (String) —

    The ID of the table. The ID must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_). The maximum length is 1,024 characters.

  • name (String) (defaults to: nil) —

    A descriptive name for the table.

  • description (String) (defaults to: nil) —

    A user-friendly description of the table.

  • fields (Array<Schema::Field>) (defaults to: nil) —

    An array of Schema::Field objects specifying the schema’s data types for the table. The schema may also be configured when passing a block.

Yields:

  • (table) —

    a block for setting the table

Yield Parameters:

  • table (Table) —

    the table object to be updated

Returns:



390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
 
# File 'lib/gcloud/bigquery/dataset.rb', line 390

def create_table table_id, name: nil, description: nil, fields: nil
  ensure_service!
  new_tb = Google::Apis::BigqueryV2::Table.new(
    table_reference: Google::Apis::BigqueryV2::TableReference.new(
      project_id: project_id, dataset_id: dataset_id, table_id: table_id))
  updater = Table::Updater.new(new_tb).tap do |tb|
    tb.name = name unless name.nil?
    tb.description = description unless description.nil?
    tb.schema.fields = fields unless fields.nil?
  end

  yield updater if block_given?

  gapi = service.insert_table dataset_id, updater.to_gapi
  Table.from_gapi gapi, service
end

#create_view(table_id, query, name: nil, description: nil) ⇒ Gcloud::Bigquery::View

Creates a new view table from the given query.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
view = dataset.create_view "my_view",
          "SELECT name, age FROM [proj:dataset.users]"

A name and description can be provided:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
view = dataset.create_view "my_view",
          "SELECT name, age FROM [proj:dataset.users]",
          name: "My View", description: "This is my view"

Parameters:

  • table_id (String) —

    The ID of the view table. The ID must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_). The maximum length is 1,024 characters.

  • query (String) —

    The query that BigQuery executes when the view is referenced.

  • name (String) (defaults to: nil) —

    A descriptive name for the table.

  • description (String) (defaults to: nil) —

    A user-friendly description of the table.

Returns:



441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
 
# File 'lib/gcloud/bigquery/dataset.rb', line 441

def create_view table_id, query, name: nil, description: nil
  new_view_opts = {
    table_reference: Google::Apis::BigqueryV2::TableReference.new(
      project_id: project_id, dataset_id: dataset_id, table_id: table_id
    ),
    friendly_name: name,
    description: description,
    view: Google::Apis::BigqueryV2::ViewDefinition.new(
      query: query
    )
  }.delete_if { |_, v| v.nil? }
  new_view = Google::Apis::BigqueryV2::Table.new new_view_opts

  gapi = service.insert_table dataset_id, new_view
  Table.from_gapi gapi, service
end

#created_at ⇒ Object

The time when this dataset was created.



179
180
181
182
183
184
185
186
 
# File 'lib/gcloud/bigquery/dataset.rb', line 179

def created_at
  ensure_full_data!
  begin
    Time.at(Integer(@gapi.creation_time) / 1000.0)
  rescue
    nil
  end
end

#dataset_id ⇒ Object

A unique ID for this dataset, without the project name. The ID must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_). The maximum length is 1,024 characters.



67
68
69
 
# File 'lib/gcloud/bigquery/dataset.rb', line 67

def dataset_id
  @gapi.dataset_reference.dataset_id
end

#dataset_ref ⇒ Object

The gapi fragment containing the Project ID and Dataset ID as a camel-cased hash.



84
85
86
87
88
 
# File 'lib/gcloud/bigquery/dataset.rb', line 84

def dataset_ref
  dataset_ref = @gapi.dataset_reference
  dataset_ref = dataset_ref.to_h if dataset_ref.respond_to? :to_h
  dataset_ref
end

#default_expiration ⇒ Object

The default lifetime of all tables in the dataset, in milliseconds.



154
155
156
157
158
159
160
161
 
# File 'lib/gcloud/bigquery/dataset.rb', line 154

def default_expiration
  ensure_full_data!
  begin
    Integer @gapi.default_table_expiration_ms
  rescue
    nil
  end
end

#default_expiration=(new_default_expiration) ⇒ Object

Updates the default lifetime of all tables in the dataset, in milliseconds.



169
170
171
172
 
# File 'lib/gcloud/bigquery/dataset.rb', line 169

def default_expiration= new_default_expiration
  @gapi.update! default_table_expiration_ms: new_default_expiration
  patch_gapi! :default_table_expiration_ms
end

#delete(force: nil) ⇒ Boolean

Permanently deletes the dataset. The dataset must be empty before it can be deleted unless the ‘force` option is set to `true`.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery

dataset = bigquery.dataset "my_dataset"
dataset.delete

Parameters:

  • force (Boolean) (defaults to: nil) —

    If ‘true`, delete all the tables in the dataset. If `false` and the dataset contains tables, the request will fail. Default is `false`.

Returns:

  • (Boolean) —

    Returns ‘true` if the dataset was deleted.



291
292
293
294
295
 
# File 'lib/gcloud/bigquery/dataset.rb', line 291

def delete force: nil
  ensure_service!
  service.delete_dataset dataset_id, force
  true
end

#description ⇒ Object

A user-friendly description of the dataset.



134
135
136
137
 
# File 'lib/gcloud/bigquery/dataset.rb', line 134

def description
  ensure_full_data!
  @gapi.description
end

#description=(new_description) ⇒ Object

Updates the user-friendly description of the dataset.



144
145
146
147
 
# File 'lib/gcloud/bigquery/dataset.rb', line 144

def description= new_description
  @gapi.update! description: new_description
  patch_gapi! :description
end

#etag ⇒ Object

A string hash of the dataset.



114
115
116
117
 
# File 'lib/gcloud/bigquery/dataset.rb', line 114

def etag
  ensure_full_data!
  @gapi.etag
end

#location ⇒ Object

The geographic location where the dataset should reside. Possible values include EU and US. The default value is US.



208
209
210
211
 
# File 'lib/gcloud/bigquery/dataset.rb', line 208

def location
  ensure_full_data!
  @gapi.location
end

#modified_at ⇒ Object

The date when this dataset or any of its tables was last modified.



193
194
195
196
197
198
199
200
 
# File 'lib/gcloud/bigquery/dataset.rb', line 193

def modified_at
  ensure_full_data!
  begin
    Time.at(Integer(@gapi.last_modified_time) / 1000.0)
  rescue
    nil
  end
end

#name ⇒ Object

A descriptive name for the dataset.



95
96
97
 
# File 'lib/gcloud/bigquery/dataset.rb', line 95

def name
  @gapi.friendly_name
end

#name=(new_name) ⇒ Object

Updates the descriptive name for the dataset.



104
105
106
107
 
# File 'lib/gcloud/bigquery/dataset.rb', line 104

def name= new_name
  @gapi.update! friendly_name: new_name
  patch_gapi! :friendly_name
end

#project_id ⇒ Object

The ID of the project containing this dataset.



76
77
78
 
# File 'lib/gcloud/bigquery/dataset.rb', line 76

def project_id
  @gapi.dataset_reference.project_id
end

#query(query, max: nil, timeout: 10000, dryrun: nil, cache: true) ⇒ Gcloud::Bigquery::QueryData

Queries data using the [synchronous method](cloud.google.com/bigquery/querying-data).

Sets the current dataset as the default dataset in the query. Useful for using unqualified table names.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery

data = bigquery.query "SELECT name FROM my_table"
data.each do |row|
  puts row["name"]
end

Parameters:

  • query (String) —

    A query string, following the BigQuery [query syntax](cloud.google.com/bigquery/query-reference), of the query to execute. Example: “SELECT count(f1) FROM [myProjectId:myDatasetId.myTableId]”.

  • max (Integer) (defaults to: nil) —

    The maximum number of rows of data to return per page of results. Setting this flag to a small value such as 1000 and then paging through results might improve reliability when the query result set is large. In addition to this limit, responses are also limited to 10 MB. By default, there is no maximum row count, and only the byte limit applies.

  • timeout (Integer) (defaults to: 10000) —

    How long to wait for the query to complete, in milliseconds, before the request times out and returns. Note that this is only a timeout for the request, not the query. If the query takes longer to run than the timeout value, the call returns without any results and with QueryData#complete? set to false. The default value is 10000 milliseconds (10 seconds).

  • dryrun (Boolean) (defaults to: nil) —

    If set to ‘true`, BigQuery doesn’t run the job. Instead, if the query is valid, BigQuery returns statistics about the job such as how many bytes would be processed. If the query is invalid, an error returns. The default value is ‘false`.

  • cache (Boolean) (defaults to: true) —

    Whether to look for the result in the query cache. The query cache is a best-effort cache that will be flushed whenever tables in the query are modified. The default value is true. For more information, see [query caching](developers.google.com/bigquery/querying-data).

Returns:



650
651
652
653
654
655
656
657
 
# File 'lib/gcloud/bigquery/dataset.rb', line 650

def query query, max: nil, timeout: 10000, dryrun: nil, cache: true
  options = { max: max, timeout: timeout, dryrun: dryrun, cache: cache }
  options[:dataset] ||= dataset_id
  options[:project] ||= project_id
  ensure_service!
  gapi = service.query query, options
  QueryData.from_gapi gapi, service
end

#query_job(query, priority: "INTERACTIVE", cache: true, table: nil, create: nil, write: nil, large_results: nil, flatten: nil) ⇒ Gcloud::Bigquery::QueryJob

Queries data using the [asynchronous method](cloud.google.com/bigquery/querying-data).

Sets the current dataset as the default dataset in the query. Useful for using unqualified table names.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery

job = bigquery.query_job "SELECT name FROM my_table"

job.wait_until_done!
if !job.failed?
  job.query_results.each do |row|
    puts row["name"]
  end
end

Parameters:

  • query (String) —

    A query string, following the BigQuery [query syntax](cloud.google.com/bigquery/query-reference), of the query to execute. Example: “SELECT count(f1) FROM [myProjectId:myDatasetId.myTableId]”.

  • priority (String) (defaults to: "INTERACTIVE") —

    Specifies a priority for the query. Possible values include ‘INTERACTIVE` and `BATCH`. The default value is `INTERACTIVE`.

  • cache (Boolean) (defaults to: true) —

    Whether to look for the result in the query cache. The query cache is a best-effort cache that will be flushed whenever tables in the query are modified. The default value is true. For more information, see [query caching](developers.google.com/bigquery/querying-data).

  • table (Table) (defaults to: nil) —

    The destination table where the query results should be stored. If not present, a new table will be created to store the results.

  • create (String) (defaults to: nil) —

    Specifies whether the job is allowed to create new tables.

    The following values are supported:

    • ‘needed` - Create the table if it does not exist.

    • ‘never` - The table must already exist. A ’notFound’ error is raised if the table does not exist.

  • write (String) (defaults to: nil) —

    Specifies the action that occurs if the destination table already exists.

    The following values are supported:

    • ‘truncate` - BigQuery overwrites the table data.

    • ‘append` - BigQuery appends the data to the table.

    • ‘empty` - A ’duplicate’ error is returned in the job result if the table exists and contains data.

  • large_results (Boolean) (defaults to: nil) —

    If ‘true`, allows the query to produce arbitrarily large result tables at a slight cost in performance. Requires `table` parameter to be set.

  • flatten (Boolean) (defaults to: nil) —

    Flattens all nested and repeated fields in the query results. The default value is ‘true`. `large_results` parameter must be `true` if this is set to `false`.

Returns:



591
592
593
594
595
596
597
598
599
600
 
# File 'lib/gcloud/bigquery/dataset.rb', line 591

def query_job query, priority: "INTERACTIVE", cache: true, table: nil,
              create: nil, write: nil, large_results: nil, flatten: nil
  options = { priority: priority, cache: cache, table: table,
              create: create, write: write, large_results: large_results,
              flatten: flatten }
  options[:dataset] ||= self
  ensure_service!
  gapi = service.query_job query, options
  Job.from_gapi gapi, service
end

#table(table_id) ⇒ Gcloud::Bigquery::Table, ...

Retrieves an existing table by ID.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
table = dataset.table "my_table"
puts table.name

Parameters:

  • table_id (String) —

    The ID of a table.

Returns:



477
478
479
480
481
482
483
 
# File 'lib/gcloud/bigquery/dataset.rb', line 477

def table table_id
  ensure_service!
  gapi = service.get_table dataset_id, table_id
  Table.from_gapi gapi, service
rescue Gcloud::NotFoundError
  nil
end

#tables(token: nil, max: nil) ⇒ Array<Gcloud::Bigquery::Table>, Array<Gcloud::Bigquery::View>

Retrieves the list of tables belonging to the dataset.

Examples:

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
tables = dataset.tables
tables.each do |table|
  puts table.name
end

Retrieve all tables: (See Table::List#all)

require "gcloud"

gcloud = Gcloud.new
bigquery = gcloud.bigquery
dataset = bigquery.dataset "my_dataset"
tables = dataset.tables
tables.all do |table|
  puts table.name
end

Parameters:

  • token (String) (defaults to: nil) —

    A previously-returned page token representing part of the larger set of results to view.

  • max (Integer) (defaults to: nil) —

    Maximum number of tables to return.

Returns:



519
520
521
522
523
524
 
# File 'lib/gcloud/bigquery/dataset.rb', line 519

def tables token: nil, max: nil
  ensure_service!
  options = { token: token, max: max }
  gapi = service.list_tables dataset_id, options
  Table::List.from_gapi gapi, service, dataset_id, max
end