BigShift

BigShift uploads your data to BigQuery.

Setup

$ bundle install

Run Specs

$ bundle exec rake

Regenerating VCR Cassettes

BigQuery Workspace

A BigQuery dataset is necessary to regenerate cassettes. The dataset may be named anything, but a table named vcr_insert_table is required and must have a field named field of type string.

If the cassettes have been regenerated against an existing dataset, the table named vcr_create_table will need to be deleted prior to generating vcr cassettes again.

Environment Variables

In order to regenerate vcr cassettes, some envrionment variables are required and should be added to a .env file in the root of the project:

BIG_SHIFT_PROJECT_ID           = [project-id]
BIG_SHIFT_DATASET_ID           = [dataset-id]
BIG_SHIFT_REFRESH_TOKEN        = [refresh-token]
BIG_SHIFT_GOOGLE_CLIENT_ID     = [google-client-id]
BIG_SHIFT_GOOGLE_CLIENT_SECRET = [google-client-secret]

Find AwesomenessTv Variables

Project ID, Dataset ID

Look on the BigQuery dashboard or talk to a team member.

Client ID and Client Secret
  1. Go to the Google API Console.
  2. Log in as [email protected] (see a team member for the password).
  3. Select the Business Intelligence project (if not already there).
  4. Click on Credentials on the left nav (under APIs and auth).
  5. Client ID and Client Secret are listed under Client ID for web application.
Refresh Token (requires Client ID & Client Secret)
  1. Sign in to Google as [email protected].
  2. Go to the OAuth playground.
  3. Select the BigQuery APIs (bigquery and bigquery.insertdata scopes).
  4. Use the gear/options dropdown to set the Client ID and Client Secret.
  5. Click on Authorize APIs.
  6. Look for the Refresh token value that is generated.

Usage

All environment variables listed in the section on regenerating vcr cassettes must be set by any consumer of this gem.

Interface

All public endpoints are exposed in BigShift::Core.

Every response from the public API is wrapped in a Response object that will always have the same interface regardless of request. The Response#data attribute will be an object specific to the data requested.

create_table

This method requires only a schema to be specified:

schema = BigShift::Schema.new('MyTable')
  .add_field('field1', :string)
  .add_field('field2', :integer)
  .add_field('field3', :boolean)

BigShift.create_table schema

insert_rows

This method requires a table name and an array of objects to insert.

All objects are expected to respond to to_json.

rows = [{
  :field1 => 'field1-value-1',
  :field2 => 'field2-value-1',
}, {
  :field1 => 'field1-value-2',
  :field2 => 'field2-value-2',
}, {
  :field1 => 'field1-value-3',
  :field2 => 'field2-value-3',
}]

BigShift.insert_rows 'MyTable', rows