The following example shows several important steps to call Google Cloud APIs without using an SDK in Python. Similar code works in just about any language (c#, java, php, nodejs).
Change the source code with the filename of your service account Json file, your Google Zone and your Project ID.
This example will list the instances in one zone for the specified project. From this example you will know the framework to call almost any Google Cloud API.
This code will show you how to:
- How to load service account credentials from a Json file.
- How to extract the Private Key used to sign requests.
- How to create a JWT (Json Web Token) for Google Oauth 2.0.
- How to set the Google Scopes (permissions).
- How to sign a JWT to create a Signed-JWT (JWS).
- How to exchange the Signed-JWT for a Google OAuth 2.0 Access Token.
- How to set the expiration time. This program defaults to 3600 seconds (1 Hour).
- How to call a Google API and set the Authorization Header.
- How to process the returned Json results and display the name of each instance.
Example program in Python 3.x:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 |
############################################################ # Version 1.00 # Date Created: 2018-12-23 # Last Update: 2018-12-23 # https://www2.jhanley.com # Copyright (c) 2018, John J. Hanley # Author: John Hanley ############################################################ ''' This program lists lists the Google Compute Engine Instances in one zone ''' import time import json import jwt import requests import httplib2 # Project ID for this request. project = 'development-123456' # The name of the zone for this request. zone = 'us-west1-a' # Service Account Credentials, Json format json_filename = 'service-account.json' # Permissions to request for Access Token scopes = "https://www.googleapis.com/auth/cloud-platform" # Set how long this token will be valid in seconds expires_in = 3600 # Expires in 1 hour def load_json_credentials(filename): ''' Load the Google Service Account Credentials from Json file ''' with open(filename, 'r') as f: data = f.read() return json.loads(data) def load_private_key(json_cred): ''' Return the private key from the json credentials ''' return json_cred['private_key'] def create_signed_jwt(pkey, pkey_id, email, scope): ''' Create a Signed JWT from a service account Json credentials file This Signed JWT will later be exchanged for an Access Token ''' # Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT auth_url = "https://www.googleapis.com/oauth2/v4/token" issued = int(time.time()) expires = issued + expires_in # expires_in is in seconds # Note: this token expires and cannot be refreshed. The token must be recreated # JWT Headers additional_headers = { 'kid': pkey_id, "alg": "RS256", # Google uses SHA256withRSA "typ": "JWT" } # JWT Payload payload = { "iss": email, # Issuer claim "sub": email, # Issuer claim "aud": auth_url, # Audience claim "iat": issued, # Issued At claim "exp": expires, # Expire time "scope": scope # Permissions } # Encode the headers and payload and sign creating a Signed JWT (JWS) sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers) return sig def exchangeJwtForAccessToken(signed_jwt): ''' This function takes a Signed JWT and exchanges it for a Google OAuth Access Token ''' auth_url = "https://www.googleapis.com/oauth2/v4/token" params = { "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer", "assertion": signed_jwt } r = requests.post(auth_url, data=params) if r.ok: return(r.json()['access_token'], '') return None, r.text def gce_list_instances(accessToken): ''' This functions lists the Google Compute Engine Instances in one zone ''' # Endpoint that we will call url = "https://www.googleapis.com/compute/v1/projects/" + project + "/zones/" + zone + "/instances" # One of the headers is "Authorization: Bearer $TOKEN" headers = { "Host": "www.googleapis.com", "Authorization": "Bearer " + accessToken, "Content-Type": "application/json" } h = httplib2.Http() resp, content = h.request(uri=url, method="GET", headers=headers) status = int(resp.status) if status < 200 or status >= 300: print('Error: HTTP Request failed') return j = json.loads(content.decode('utf-8').replace('\n', '')) print('Compute instances in zone', zone) print('------------------------------------------------------------') for item in j['items']: print(item['name']) if __name__ == '__main__': cred = load_json_credentials(json_filename) private_key = load_private_key(cred) s_jwt = create_signed_jwt( private_key, cred['private_key_id'], cred['client_email'], scopes) token, err = exchangeJwtForAccessToken(s_jwt) if token is None: print('Error:', err) exit(1) gce_list_instances(token) |
Update: tokens created after October 23, 2021
If you print the access token, you might notice the token is padded with dots (period characters). Google plans to roll out security and reliability improvements. One of these changes is increasing the size of the OAuth tokens to 2048 bytes. During the testing phase tokens are padded with dots (period characters). Do not truncate or remove those padding characters.
For more details read this Google notice:
Hello Google Cloud Customer,
We’re writing to let you know that on August 23, 2021, we will roll out security and reliability improvements that will increase the sizes of OAuth 2.0 access tokens that your services may be using to call Google Cloud Platform APIs.
The access token size increase will only impact services that don’t conform to the access token size limits of 2048 bytes established on Google’s Developer guide and RFC 6749. If your service falls into this category, please continue reading and take action as outlined below.
What do I need to know?
The enhancement described above will increase the access token size issued to your services that authorize to Google Cloud APIs. Please consider that:
The overall access token size will continue to remain within the 2048 bytes limit documented in Google’s Developer guide and public documentation. Google’s OAuth 2.0 access tokens will also continue to conform to the standards defined in The OAuth 2.0 Authorization framework RFC 6749. Verification Phase
To be able to identify services that may be impacted, we are launching a verification phase. In this phase we will serve projects with access tokens that have been artificially inflated with extra padding and monitor the size of the token that we receive back from these projects.
The purpose of this phase is to identify the projects that will be affected once the enhancement described above takes place. Even if your services end up trimming this extra padding in the token to limit its size, the token will remain valid and calls to Google Cloud APIs should succeed while allowing us to pinpoint these services that do trim tokens. We will notify the project owner(s) accordingly to make the recommended changes to these services before August 23, 2021, rollout and avoid potential outage of these services.
On August 23, 2021, we will roll out security and reliability improvements that will increase the sizes of OAuth 2.0 access tokens for all projects.
Unlike the specially padded tokens that were sent during the verification phase, these tokens will not have a padding and the token itself will be larger compared to current tokens. We will also remove all projects from the exception-list and begin issuing the larger access tokens to all Google Cloud Platform (GCP) projects. After this time, if your services continue to trim the access tokens size, then their calls to Google Cloud APIs will fail. What do I need to do?
We strongly recommend that you use the documented token size limits of 2048 bytes and to remove any logic in your services/code that restricts its ability to process access tokens of certain sizes.
Please consider that:
If you haven’t built your Google Cloud APIs services’ frameworks based on access token size limits, no action is required from your part. If your services expect access tokens to meet certain size limits then these services may be impacted and their calls to Google Cloud APIs may fail. If we detect that the token sizes are being trimmed by services in this project, we will notify you by July 30, 2021, and automatically enroll the impacted projects into an exception-list. Projects in this exception-list will revert to receiving the currently available smaller access tokens until August 23, 2021. This verification phase will give you enough time to make the recommended changes to your service/code. Please use this form to enroll specific projects by June 7, 2021. For us to do the verification, we require you to follow these steps below in the specified timeline:
From now and until June 7, 2021: You can opt-in to enroll specific projects to receive the specially padded longer access tokens that could be up to 2048 bytes in size. We will monitor the access token size we send and receive back from these projects. On June 21, 2021: We will automatically enroll all GCP projects to receive the specially padded larger tokens. You can opt-out your projects from this phase by filling this form by June 7, 2021. If we don’t hear from you by June 7, 2021, we will send these projects the padded access tokens and monitor the token size we receive back from them.
Also, see my answer on Stack Overflow.
I design software for enterprise-class systems and data centers. My background is 30+ years in storage (SCSI, FC, iSCSI, disk arrays, imaging) virtualization. 20+ years in identity, security, and forensics.
For the past 14+ years, I have been working in the cloud (AWS, Azure, Google, Alibaba, IBM, Oracle) designing hybrid and multi-cloud software solutions. I am an MVP/GDE with several.
June 22, 2020 at 12:17 AM
Thank you for creating this article. I struggled for a couple of days with the authentication process before I found this. Your code is very well written and easy to read.
September 12, 2020 at 11:19 AM
Perfect, thank you!
February 28, 2021 at 4:45 PM
Code written like a beautiful poem, loved going through it !
September 14, 2021 at 2:08 PM
Hi John, this worked great for my program. I was wondering if you can help with then using the token to persist an API connection (discovery.Resource) rather than having to define the token in each call to the service? Thanks
September 14, 2021 at 4:27 PM
Hi Aaron, I am not sure what you mean by “using a token to persist an API connection”. Each connection needs authorization. For the most part, Google Cloud REST APIs do not use cookies or sessions to persist data.