How to Integrate Kubernetes and Google Cloud Service Accounts… and Why it Works

GKE Behind the Scenes: Understanding the Interaction Between Kubernetes and GCP Service Accounts Through The Metadata Server.

Image created with ChatGPT by the author

When we integrate Kubernetes with Google Cloud Platform (GCP), there is a critical interplay between Kubernetes service accounts (KSA) and GCP service accounts (GSA) through a mechanism called Workload Identity. This integration allows Kubernetes workloads to securely and efficiently access GCP services, without having to manage long-lived credentials. In this article, we’ll explore this integration process step-by-step, using tools like tcpdump and Wireshark to dive into the technical details.

First, let’s provide some context.
KSAs are managed at the namespace level and exist as ServiceAccount objects in the Kubernetes API server. Conversely, GSAs are managed at the project level using the IAM API and are used to programmatically invoke Google Cloud APIs. Workload Identity is a well-known method for GKE workloads to access Google Cloud services. It allows a KSA to authenticate to Google Cloud APIs without having to manage keys or credentials, and works by establishing a relationship between a KSA and a GSA. The KSA identifies the workload running in GKE as a user of the GSA, which in turn is configured with the necessary access to Google Cloud resources.

Setting the Stage: Creating Service Accounts and Configuring Workload Identity

Let’s start by creating the resources needed to analyze the interaction we want to focus on. We open the bash shell (when OS is Ubuntu), and after authenticating to GCP by running gcloud auth login, we start by defining an environment variable for the GCP project name: GCP_PROJECT_ID="<gcp_project_id>", a second one for the cluster name: GKE_CLUSTER_NAME="my-cluster-name", and a third one for the GSA: GSA="test-gsa". We create the GSA running:

gcloud iam service-accounts create ${GSA} --description="some description text" --display-name=${GSA}

Next, let’s create a GKE cluster with Workload Identity enabled:

gcloud container clusters create ${GKE_CLUSTER_NAME} --workload-pool=${GCP_PROJECT_ID}.svc.id.goog

By the way, if we want to use an existing cluster and check if it has Workload Identity enabled, we can run:

gcloud container clusters describe ${GKE_CLUSTER_NAME} --location=<cluster location>

and verify that the workloadIdentityConfig section is present in the output:

workloadIdentityConfig:
  workloadPool: "PROJECT_ID.svc.id.goog"

After that, we define an environment variable for the namespace: K8S_NAMESPACE=<name of the namespace>, create this namespace if it does not exist by running:

kubectl create namespace ${K8S_NAMESPACE}

and make it the default one for the current k8s context:

kubectl config set-context --current --namespace ${K8S_NAMESPACE}

Then, we define the environment variable KSA="test-ksa", create our KSA with:

kubectl create sa ${KSA}

and annotate it as follows:

kubectl annotate sa ${KSA} iam.gke.io/gcp-service-account=${KSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com

After all this, the KSA manifest should look like this:

apiVersion: v1
kind: ServiceAccount
metadata:
 annotations:
 iam.gke.io/gcp-service-account: test-gsa@<gcp project id>.iam.gserviceaccount.com
 name: test-ksa
 namespace: <name of the namespace>

The last thing we need to do is add an IAM policy binding between the roles/iam.workloadIdentityUser role and our GSA by running:

gcloud iam service-accounts add-iam-policy-binding ${GSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${KSA}]"

By following these steps, we have successfully configured a KSA to interact with a Google Cloud Service Account (GSA) using Workload Identity. On the Kubernetes side, the annotation iam.gke.io/gcp-service-account: test-gsa@<gcp project id>.iam.gserviceaccount.com did a lot of the magic.

To verify that our preparations were successful, we create the following pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  serviceAccountName: test-ksa
  containers:
  - name: app-container
    image: google/cloud-sdk:slim
    command: ["sh", "-c", "gcloud auth list && gcloud projects list"]

Then we check the logs:

kubectl logs pods/test-pod

We should find something similar in them:

Credentialed Accounts
ACTIVE ACCOUNT
* test-gsa@<your GCP project ID>.iam.gserviceaccount.com
To set the active account, run:
  $ gcloud config set account `ACCOUNT`
PROJECT_ID NAME PROJECT_NUMBER
<your GCP Project ID> <Name of your GCP Project> <Your Project Number>

This means that the KSA has successfully interacted with GCP by impersonating its GSA counterpart. You can then delete the pod used for the test.

Now let’s go back to the previous command and look at the contents of the optional --member parameter: serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${KSA}]. What does this mean? Well, this is an example of Namespace Identity, a concept introduced with GKE Workload Identity. Let's take a quick digression into this topic.

Excursus: Namespace Identity and Identity Sameness

When Workload Identity is enabled on a GKE cluster, the GCP project to which the cluster belongs is assigned a unique identity namespace in the form of PROJECT_ID.svc.id.goog. Based on this, the IAM identifies a Kubernetes service account using the following information:

PROJECT_ID.svc.id.goog[KUBERNETES_NAMESPACE/KUBERNETES_SERVICE_ACCOUNT]. In this structure:

• PROJECT_ID.svc.id.goog represents the identity namespace created by GKE.
• KUBERNETES_NAMESPACE is the Kubernetes service account namespace.
• KUBERNETES_SERVICE_ACCOUNT is the Kubernetes service account name.

In practice, if two or more different GKE clusters have a namespace with the same name, and those namespaces contain Kubernetes service accounts with the same name, those accounts will be treated the same by the GCP project identity namespace. In other words, different Kubernetes service accounts can impersonate the same GCP service account.

Let’s take an example. Imagine you have two smart buildings in different cities, each managed by a different GKE Kubernetes cluster. Both buildings are part of the same organization, and have a keyless access system that allows employees to access them using their mobile devices.

In Cluster 1, which is associated to Building A, there is a Kubernetes namespace named access and a Kubernetes service account named keyless-entry. This account has the necessary permissions to access the authentication credentials of the keyless access system for Building A.

Similarly, in Cluster 2, which is associated to Building B, there is a Kubernetes namespace called access and a Kubernetes service account named keyless-entry. This account also has permissions to access the keyless-entry system’s authentication credentials, but for Building B.

Both clusters use GKE Workload Identity, which creates a project-level identity namespace: PROJECT_ID.svc.id.goog. The Kubernetes service account in each cluster is then named as follows: PROJECT_ID.svc.id.goog[access/keyless-entry].

Because both clusters have an identical namespace (access) and an identical Kubernetes service account (keyless-entry), these accounts are treated the same by the GCP project identity namespace. This means that both Kubernetes service accounts (access/keyless-entry) in the two clusters can impersonate the same GCP service account, with the same permissions.

This system allows for simplified management where a single permission configuration can be applied to both buildings even though they are located in different cities. People in the organization can access the buildings based on the project to which they are assigned. For example, if an employee is authorized to access projects in both buildings, the identity system ensures that access permissions are consistent, providing seamless and secure access.
In this way, each cluster keeps its permissions in sync, ensuring that the organization’s employees can access buildings based on their project needs.

Diagram illustrating namespace identity and identity sameness, created by Christian Del Monte using PlantUML.

The Token Relay Race

As we’ve seen, through these operations, the Kubernetes Service Account (KSA) can now use GCP resources by impersonating the GCP Service Account (GSA). But what makes this interaction possible? The answer lies in the process of generating, managing, and exchanging tokens for KSAs and GSAs — a true relay race in which different servers pass different tokens from hand to hand.

Key Actors in the Token Relay

Before we dive into the step-by-step process, let’s introduce the key players involved:

Kubernetes API Server:

• Role: Central management point for all API requests within the Kubernetes cluster.
• Function: Authenticates requests and generates JWT (JSON Web Token) tokens for Kubernetes Service Accounts (KSAs).
• Insight: Acts as the initial gatekeeper, ensuring authenticated entities can proceed with token requests and maintaining security integrity through signed tokens.

GCP Metadata Server (GKE MDS):

• Role: Provides metadata and token handling for GCP resources and instances.
• Function: Requests an OIDC-signed JWT from the Kubernetes API server, uses it to obtain an access token from the IAM, and then exchanges it for a GCP Service Account (GSA) token.
• Insight: Ensures secure provisioning of credentials without storing them directly on the instance, which is critical for GCE and GKE environments.

Google Cloud IAM:

• Role: Manages identities and access control for GCP resources.
• Function: Validates the OIDC-signed JWT and issues an access token for the KSA.
• Insight: Enforces security policies and validates appropriate bindings on the ID namespace and OIDC provider.

GCP Resource Server:

• Role: Hosts the Google Cloud resources that the application needs to access.
• Function: Verifies the GCP SA token with the Metadata Server before granting access.
• Insight: Acts as the final gatekeeper, enforcing security policies and access controls within the GCP environment.

The Step-by-Step Process

Now, let’s go through this relay race one step at a time.

Part 1: Requesting a GSA Token

• Application requests a KSA token: Imagine an application that needs to request a KSA token from the Kubernetes API Server. Essentially, it makes an authenticated API call to the server. The Kubernetes API Server authenticates its request and, once verified, generates a JWT (JSON Web Token) for the KSA. This token contains various information, such as the identity of the service account and the duration of the token. In addition, the token is digitally signed by the Kubernetes API Server using a private key, which ensures that the token is authentic and has not been tampered with. Anyone with the appropriate public key can verify the token.
• Application sends the KSA token to GKE Metadata Server: The application sends the obtained KSA token to the GKE MDS to initiate the process of obtaining a GSA token. This step involves the application providing its Kubernetes identity to the GKE MDS for further authentication and token exchange.
• GKE Metadata Server requests a JWT: The GKE Metadata Server (GKE MDS) requests an OIDC-signed JWT from the Kubernetes API Server. This is done over a mutual TLS (mTLS) connection using node credentials.
• GKE MDS uses the JWT to request an access token: GKE MDS uses the OIDC-signed JWT to request an access token for the Identity Namespace (IDNS) or KSA from Google Cloud IAM. IAM validates the appropriate bindings on the IDNS and the OIDC provider.
• IAM validates the JWT: IAM validates the OIDC-signed JWT against the cluster’s OIDC provider and returns the access token.
• GKE MDS Sends Access Token to IAM: GKE MDS sends the access token for the IDNS/KSA back to IAM in exchange for a GCP Service Account (GSA) token. IAM validates the appropriate bindings.
• GKE MDS returns the GSA token: GKE MDS returns the GSA token to the application.

Part 2: verifiing a GSA Token

• Application sends a request to the GCP Resource Server: Now, the application needs to access a GCP resource. It sends a request to the GCP Resource Server, including the GSA token. The GCP Resource Server receives the request and the GSA token. To ensure the token is authentic and valid, it communicates with the GCP Metadata Server.
• Metadata Server validates the GSA token: The Resource Server sends the GSA token to the GCP Metadata Server for verification. The Metadata Server is responsible for authenticating the tokens and confirming that they are valid and have not expired. It uses the appropriate public keys to verify the signature of the GSA token, and also verifies that the token has not expired and was issued for the specific resources requested. If the token is valid, the Metadata Server sends a positive response to the Resource Server, confirming that the token is authentic and valid. Once the confirmation is received from the Metadata Server, the Resource Server grants the application access to the requested resources.

Sequence diagram of the process created by Christian Del Monte using PlantUML.

Capturing and Using a GCP SA Token with tcpdump

Now that we’ve explained what’s going on behind the scenes, let’s dive into the practice. We’ll demonstrate how to request and send a GSA token.

Knowing that the Metadata Server typically has the IP 169.254.169.254, we will follow these steps:

1. Capture the traffic in the Kubernetes cluster to and from the IP 169.254.169.254 using tcpdump.
2. Start a pod and let a KSA impersonate a GSA.
3. Inspect the captured traffic and find the GSA token issued to the KSA.
4. Copy and verify the captured token.

Step 1: Capture the Traffic in the Kubernetes Cluster with tcpdump

To capture the traffic that contains the GSA token, we will use tcpdump in a DaemonSet named tcpdump:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: tcpdump-daemonset
spec:
  selector:
    matchLabels:
      name: tcpdump
  template:
    metadata:
      labels:
        name: tcpdump
    spec:
      hostNetwork: true
      containers:
        - name: tcpdump
          image: google/cloud-sdk:slim
          command:
            [
              "/bin/sh",
              "-c",
              "apt-get update && apt-get install -y tcpdump && tcpdump -i any -vv host 169.254.169.254 -w /host/tmp/metadata.pcap",
            ]
          volumeMounts:
            - name: host-tmp
              mountPath: /host/tmp
          securityContext:
            capabilities:
              add: ["NET_ADMIN", "NET_RAW"]
      volumes:
        - name: host-tmp
          hostPath:
            path: /tmp

Step 2: Start a Pod and Let a KSA Impersonate a GSA

Once the DaemonSet is deployed, run a pod that uses the previously created service account (in our example, it is test-ksa) and access the Google services for which you have the necessary permissions.

Step 3: Inspect the Captured Traffic and Find the GSA Token Issued for the KSA

Next, download the captured .pcap file from a pod controlled by the DaemonSet:

kubectl cp <tcpdump-pod>:/host/tmp/combined_traffic.pcap ./combined_traffic.pcap

Open the .pcap file in Wireshark and look for requests sent to the Metadata server, with IP 169.254.169.254:

ip.addr == 169.254.169.254

Follow the HTTP stream by right-clicking on an HTTP request packet to 169.254.169.254 and selecting "Follow" → "HTTP Stream". In the HTTP stream window, locate the HTTP response from the metadata server. The token is typically found in the JSON response in the access_token field.

Screenshot of Wireshark illustrating step 3 taken by Christian Del Monte.

Step 4: Verifying the Captured Token

Now you can verify the contents of this token using the Token Info Endpoint. Google’s OAuth 2.0 token info endpoint can provide details about the token, including the associated email:

curl -H "Authorization: Bearer <GCP_OAUTH2_TOKEN>" https://www.googleapis.com/oauth2/v1/tokeninfo

The response will be in this form:

{
  "issued_to": "12345678910234456656",
  "audience": "12345678910234456656",
  "scope": "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform",
  "expires_in": 1043,
  "email": "test-gsa@<your GCP project id>.iam.gserviceaccount.com",
  "verified_email": true,
  "access_type": "online"
}

Alternative: Getting a GSA Token directly with a REST Call

As an alternative to using tcpdump and Wireshark, there is another way to illustrate this process: you can impersonate a Kubernetes Service Account (KSA) to call the metadata server using its Kubernetes token. This method works because the Kubernetes token can be used to authenticate to the metadata server, which then provides the GCP Service Account (GSA) token. Here’s how it works:

Step 1: Get the KSA Token

Retrieve the service account token from a running pod:

kubectl exec -it <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

Step2: Make a Request to the Metadata Server

Use curl to make a request to the metadata server, including the KSA token in the header:

TOKEN=$(kubectl exec -it <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -H "Authorization: Bearer $TOKEN" -H "Metadata-Flavor: Google" "http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token"

This request uses the KSA token to impersonate the Kubernetes service account and access the Metadata Server, which retrieves the GSA token associated with the KSA.

Summary

In this article, we have walked through the process of configuring Kubernetes Service Accounts (KSA) to interact with Google Cloud Service Accounts (GSA) using Workload Identity. We demonstrated how to capture and analyze token exchanges using tcpdump and Wireshark, and discussed an alternative method to obtain GSA tokens directly via REST calls. This integration enhances security and simplifies access management in Kubernetes environments, eliminating the need for long-lived credentials. By understanding and implementing these steps, you can ensure a secure and seamless connection between your Kubernetes workloads and Google Cloud resources.

Disclaimer: A newer feature called Workload Identity Federation for GKE allows KSAs to be referenced directly using a principal identifier in IAM policies without using impersonation. This is not the subject of this article.

Google Cloud Documentation on Workload Identity:

• Workload Identity Overview

Kubernetes Documentation:

• Service Accounts

GCP IAM Documentation:

• Service Accounts
• IAM Policy Binding

Tools for Network Analysis:

• tcpdump
• Wireshark