In general, the google-cloud-dataproc library uses Service Account credentials to connect to Google Cloud services. When running within Google Cloud Platform environments the credentials will be discovered automatically. When running on other environments, the Service Account credentials can be specified by providing the path to the JSON keyfile for the account (or the JSON itself) in environment variables. Additionally, Cloud SDK credentials can also be discovered automatically, but this is only recommended during development.
- Initialize the client.
require "google/cloud/dataproc" client = Google::Cloud::Dataproc.autoscaling_policy_service
The google-cloud-dataproc library aims to make authentication as simple as possible, and provides several mechanisms to configure your system without requiring Service Account Credentials directly in code.
Credentials are discovered in the following order:
- Specify credentials in method arguments
- Specify credentials in configuration
- Discover credentials path in environment variables
- Discover credentials JSON in environment variables
- Discover credentials file in the Cloud SDK's path
- Discover GCP credentials
Google Cloud Platform environments
When running on Google Cloud Platform (GCP), including Google Compute Engine (GCE), Google Kubernetes Engine (GKE), Google App Engine (GAE), Google Cloud Functions (GCF) and Cloud Run, Credentials are discovered automatically. Code should be written as if already authenticated.
The Credentials JSON can be placed in environment variables instead of declaring them directly in code. Each service has its own environment variable, allowing for different service accounts to be used for different services. (See the READMEs for the individual service gems for details.) The path to the Credentials JSON file can be stored in the environment variable, or the Credentials JSON itself can be stored for environments such as Docker containers where writing files is difficult or not encouraged.
The environment variables that google-cloud-dataproc
checks for credentials are configured on the service Credentials class (such as
DATAPROC_CREDENTIALS- Path to JSON file, or JSON contents
DATAPROC_KEYFILE- Path to JSON file, or JSON contents
GOOGLE_CLOUD_CREDENTIALS- Path to JSON file, or JSON contents
GOOGLE_CLOUD_KEYFILE- Path to JSON file, or JSON contents
GOOGLE_APPLICATION_CREDENTIALS- Path to JSON file
require "google/cloud/dataproc" ENV["DATAPROC_CREDENTIALS"] = "path/to/keyfile.json" client = Google::Cloud::Dataproc.autoscaling_policy_service
The path to the Credentials JSON file can be configured instead of storing it in an environment variable. Either on an individual client initialization:
require "google/cloud/dataproc" client = Google::Cloud::Dataproc.autoscaling_policy_service do |config| config.credentials = "path/to/keyfile.json" end
Or globally for all clients:
require "google/cloud/dataproc" Google::Cloud::Dataproc.configure do |config| config.credentials = "path/to/keyfile.json" end client = Google::Cloud::Dataproc.autoscaling_policy_service
This option allows for an easy way to authenticate during development. If credentials are not provided in code or in environment variables, then Cloud SDK credentials are discovered.
To configure your system for this, simply:
- Download and install the Cloud SDK
- Authenticate using OAuth 2.0
$ gcloud auth application-default login
- Write code as if already authenticated.
NOTE: This is not recommended for running in production. The Cloud SDK should only be used during development.
Creating a Service Account
Google Cloud requires Service Account Credentials to connect to the APIs. You will use the JSON key file to connect to most services with google-cloud-dataproc.
If you are not running this client within Google Cloud Platform environments, you need a Google Developers service account.
- Visit the Google Cloud Console.
- Create a new project or click on an existing project.
- Activate the menu in the upper left and select APIs & Services. From here, you will enable the APIs that your application requires.
Note: You may need to enable billing in order to use these services.
- Select Credentials from the side navigation.
Find the "Create credentials" drop down near the top of the page, and select "Service account" to be guided through downloading a new JSON key file.
If you want to re-use an existing service account, you can easily generate a new key file. Just select the account you wish to re-use, click the pencil tool on the right side to edit the service account, select the Keys tab, and then select Add Key.
The key file you download will be used by this library to authenticate API requests and should be stored in a secure location.