• Date created: March 1, 2019
    Last updated: March 3, 2019

Note: This article is evolving as I document my deep dive.

Contents:

Introduction

“Service accounts are the keys to the cloud kingdom.”

This is the first of my “The Master Series” on Google Cloud. In this article, we will dive deep into Compute Engine Service Accounts. We will investigate service accounts, instance metadata, access scopes, identity and access management (IAM), impersonation, firewall rules, Stackdriver, auditing, logging events, alerting and best practices.

I have written a number of articles on service accounts on this site. However, this article will be different as we will investigate items that are not documented, unknown or interesting. We will experiment, do the unexpected, create scenarios and test.

Service accounts are one of the most misunderstood features in Google Cloud. One of the reasons is that Google designed service accounts with power, flexibility, and features. Service accounts are both an identity and a resource. Service accounts can act and be impersonated. Understanding service accounts are important to properly authorize and secure cloud resources.

March 1, 2019 – Day #1 – Basics and FAQ

What is a Compute Engine service account?

A service account is a special account that can be used by services and applications running on your Compute Engine instance to interact with other Google Cloud Platform APIs. Applications can use service account credentials to authorize themselves to a set of APIs and perform actions within the permissions granted to the service account and virtual machine instance. In addition, you can create firewall rules that allow or deny traffic to and from instances based on the service account that you associate with each instance. Source.

What is a Compute Engine default service account?

New projects are created with the Compute Engine default service account, identifiable using this email:

[PROJECT_NUMBER]-compute@developer.gserviceaccount.com

The default service account is created by Google and added to your account automatically but you have full control over the account.

What is a Compute Engine Service Agent aka Compute Engine System service account?

See my related article: Google Cloud – Compute Engine System Service Account.

What permissions does the Compute Engine default service account have?

Project Editor – roles/editor

Project Editor is one of the primitive roles that Google create early on in Google Cloud. In this article, I will recommend removing the Project Editor role from the Compute Engine default service account and assign specific IAM predefined or custom roles. Google also recommends this. This advice goes for any primitive role (Owner, Editor, Viewer).

FIX: Find the reference for Google recommending removing Project Editor from a service account.

What resources rely on the Compute Engine default service account?

Managed instance groups and autoscaling use the credentials of this account to create, delete, and manage instances.

Can you create a VM instance without a service account?

Yes. The instance will still be able to access most metadata, but will not be able to interact with other Google Cloud Platform APIs.

Which items do not work on VM instances without a service account?

Verifying the identity of instances will not work. More information about VM instance identity.

Can you authorize a VM instance without a Compute Engine service account?

Yes, you can authorize the instance using several methods. The first method is gcloud auth application-default login to provide user account credentials to use for Application Default Credentials. The VM instance will need Internet access to reach Google Accounts. This gcloud command will write credentials to:

~/.config/gcloud/application_default_credentials.json

The second method is to use gcloud auth login to provide user account credentials. The VM instance will need Internet access to reach Google Accounts. This gcloud gcloud command will write credentials to:

~/.config/gcloud/legacy_credentials/john.hanley@www2.jhanley.com/adc.json

User credentials persist across reboots. If your goal is security and you removed the default service account, using gcloud auth login or gcloud auth application-default login will defeat your goal of an instance with no credentials. Revoke the credentials with gcloud auth revoke or gcloud application-default revoke.

The last method, which is also the best method, is to use service account credentials in a Json file. Copy your service account file to your instance and authorize it using gcloud auth activate-service-account [ACCOUNT] --key-file=KEY_FILE.

See my article: Google Cloud – Setting up Gcloud with Service Account Credentials which goes into detail on how to correctly setup authorization with service account credentials.

What happens if you delete the default service account while a VM instance is running?

Existing running instances will error with “Invalid Credentials” for gcloud.

Instance metadata will not have the entries in /computeMetadata/v1/instance/service-accounts/

FIX – Double check: Software will fail to obtain Application Default Credentials.

What happens if you delete the default service account for new VM instances?

Creating new default VM instances will fail with an error that the service account was not found. You will be able to create new VM instances if you specify “No service account” when configuring the new VM instance.

How do I recreate the Compute Engine default service account?

You will need to contact the Google Cloud Compute Engine team to recover your service account.

Contact the Compute Engine team.

What does Google Cloud use internally for a Service Account identifier?

Google Cloud uses the unique ID assigned to a service account at creation.

This is important to know because you can create a service account, assign roles, delete the service account and then create a new service account with the same name. The role bindings are not immediately deleted. This means that you could recreate a service account and the old bindings will still be in effect for a while for the old service account (with the same name).

Let’s look at the default Compute Engine service account for my account:

Which returns:

The uniqueId is used internally.

Recommendation: Delete the roles assigned to a service account before deleting the service account.

More information here.

What are Compute Engine best practices?

 Part 1: Source:

In general, Google recommends that each instance that needs to call a Google API should run as a service account with the minimum permissions necessary for that instance to do its job. In practice, this means you should configure service accounts for your instances with the following process:

Create a new service account rather than using the Compute Engine default service account.

Grant IAM roles to that service account for only the resources that it needs.

Configure the instance to run as that service account.

Grant the instance the https://www.googleapis.com/auth/cloud-platform scope to allow full access to all Google Cloud APIs, so that the IAM permissions of the instance are completely determined by the IAM roles of the service account.

Part 2: Source

Restrict who can act as service accounts. Users who are Service Account Users for a service account can indirectly access all the resources the service account has access to. Therefore, be cautious when granting the serviceAccountUser role to a user.

Grant the service account only the minimum set of permissions required to achieve their goal. Learn about Granting roles to a service account for specific resources.

Create service accounts for each service with only the permissions required for that service.

Use the display name of a service account to keep track of the service accounts. When you create a service account, populate its display name with the purpose of the service account.

Define a naming convention for your service accounts.

Implement processes to automate the rotation of user-managed service account keys.

Take advantage of the IAM service account API to implement key rotation.

Audit service accounts and keys using either the serviceAccount.keys.list() method or the Logs Viewer page in the console.

Do not delete service accounts that are in use by running instances on Google App Engine or Google Compute Engine.

March 2, 2019 – Day #2 – Auditing, Alerting & Stackdriver

Stackdriver can provide a wealth of information about service accounts if you know how to use Stackdriver logs. Today we will cover how to use Stackdriver logs to audit events. Then we will use Pub/Sub and Cloud Functions to process Stackdriver logs looking for specific events and creating an action, such as sending an email when a specific event occurs.

Auditing

In order to perform an audit, you need to obtain information:

  • What resources do you have?
  • What resource have you had in the past?
  • What has been done using those resources?
  • What has been done to those resources?

For this deep dive, we are only interested in service account resources.

I created a new project so that the number of resources is limited. Then I enabled the Compute Engine API.

What resources do you have?

The first step is to list all of the service accounts that are currently in a project.

Which provided me with:

Only one service account in my project.

What resource have you had in the past?

Google does not provide a method to easily determine this. We will use Stackdriver to review the events for this project. Stackdriver stores events related to service accounts in the “Activity” log. The resource type within this log is “service_account”.

This provides a lot of information. Let’s save this output to a file and then parse the output.

This provides us with a list of actions on service accounts. In our case, not much has happened.

This shows that we created a service account and then created a service account key. Normal stuff. However, if you saw activity where service accounts were being created and deleted, this might indicate that someone is trying to hide their activity or grant themselves permissions for use when not at work. The only way to know is to keep track of activity on resources. When something unexpected happens, investigate. You can also see the principal email address for each activity.

Knowing who does what to whom is an important part of auditing.

From the Stackdriver logs, you can reconstruct what resources you had in the past, who created and deleted those resources.

What has been done using those resources?

Unfortunately, Google Cloud does not log all activity using service accounts. Since service accounts are the mechanism to obtain an Access Token, which authorizes API calls, the number of log entries would match the number of API calls and then some. This would result in massive log files that would be expensive to store.

However, certain admin types of activities are logged. The principal will be the service account email address that was used to create, delete, etc. other resource types. This can provide you with a higher level overview of activity by this service account.

What has been done to those resources?

By parsing the Stackdriver logs, we can see what activity has been done to a service account. Actions such as create, delete, create keys, etc. can be tracked in detail by time and who performed the action.

Audit service account usage.

Next, we will use a Compute Engine default service account to create a Compute Engine VM. A common security problem that I see is that a user is created with IAM permissions that do not allow creating VM instances, but the user is allowed to connect to VMs using SSH where the Compute Engine default service account is set to Project Editor. This service account then allows the user to bypass the IAM user account permissions and use the service account to create VM instances.

Create a new VM instance:

Connect to the new VM instance

While inside the SSH terminal session, create a new VM instance. This VM instance is created using the Compute Engine service account.

Exit the SSH session.

Now let’s look at the Stackdriver logs for Compute Engine activities. The resource type within this log is “gce_instance”. Notice I set the –freshness command line option to 1 hour since we just created the VM.

Now search for activity:

Which provided me with one action:

Looking at the logfile for this action, I can see the principalEmail that created the instance:

Which is the Compute Engine default service account. The format for Compute Engine default service accounts:

[PROJECT_NUMBER]-compute@developer.gserviceaccount.com

I create a more complicate jq command that outputs information in CSV:

Which results in this output. Notice that some lines have empty fields. This is due to events being logged at the start and the completion of an action.

This example displays the date, user email, action, and IP address.

March 3, 2019 – Day #3 – Stackdriver Logs, PubSub & Cloud Functions

Alerting

Manually looking thru or searching logfiles is not much fun. The boredom can make you overlook the obvious due to too much information to review.

Today we will enable Stackdriver export, create a Pub/Sub topic and create a Cloud Function. These combined services will automate monitoring events that involve service accounts. I will just create a simple example that you can expand upon for more serious monitoring of Stackdriver logging events.

Related Posts