gke-workload-identity

# GKE Workload Identity Federation Troubleshooting Skill Use this skill to systematically diagnose why a Pod using **Workload Identity Federation for GKE** cannot authenticate to Google Cloud APIs. Typical symptoms: - `HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on resource` - `google.auth.exceptions ... could not find default credentials` / `ComputeEngineCredentials cannot find the metadata server` - API calls that unexpectedly use the node's default Compute Engine service account instead of the workload's identity. ## Read-only boundary This skill is **diagnostic and non-interactive**. It only **reads** cluster, IAM, and logging state and **proposes** fixes as commands or GitOps manifest changes for a human to apply. It must **never** create or modify IAM bindings, KSA annotations, node pools, or clusters automatically. When evidence is missing or the fix requires a privileged change, summarize findings and hand off to a human (see Step 7). ## Output discipline (apply to every conclusion) When you report a diagnosis, always: 1. **Name the single most-likely root cause** (not an open-ended list of possibilities). 2. **Explicitly rule out** the other plausible causes, citing the evidence that excludes them. In particular, when the cause is node-pool configuration, state plainly that it is **not** a KSA annotation or IAM binding problem; when the cause is a missing role on a target resource, state that Workload Identity itself is **not** misconfigured. 3. **Give the exact remediation as a proposed change for a human** — a concrete `gcloud` command (or GitOps manifest edit) with the real identifiers filled in — and never apply it automatically. -------------------------------------------------------------------------------- ## Diagnostic Workflow ### Step 0: Context discovery & time window Collect (from the user or the failing resource): `PROJECT_ID`, `PROJECT_NUMBER`, `CLUSTER`, cluster `LOCATION`, `NAMESPACE`, the **KSA** the Pod runs as, the **node and node pool** the Pod is scheduled on (used in Step 2), the **target resource / API** being called, and the **exact error string**. Define a time window around the first observed failure for log queries. ```bash # Resolve the project number (used in the direct-binding principal identifier). gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)" # Confirm which KSA the workload runs as. kubectl get pod "{pod_name}" -n "{namespace}" \ -o jsonpath='{.spec.serviceAccountName}' # Identify the node the Pod runs on, then the node pool that node belongs to # (Step 2 checks the node pool's Workload Identity mode). NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}') kubectl get node "$NODE" \ -o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}' ``` -------------------------------------------------------------------------------- ### Step 1: Capture the exact error signature Read the workload's own logs and the `gke-metadata-server` logs to classify the failure. ```bash kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix kubectl describe pod "{pod_name}" -n "{namespace}" ``` Equivalent via Cloud Logging (preferred for historical events). Open it as a **Logs Explorer deep link** — URL-encode the query and append the project and time window: `https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}` (encode `/` as `%2F`, or use `;duration=PT1H` for a rolling hour): ``` resource.type="k8s_container" resource.labels.namespace_name="{namespace}" resource.labels.pod_name="{pod_name}" severity>=WARNING ``` **Classify the signature:** - `iam.serviceAccounts.getAccessToken` denied / `HTTP/403` → the workload is using the **GSA impersonation** path. This is expected for the legacy setup, but if you intend to use **direct binding**, it means the KSA still carries a leftover `iam.gke.io/gcp-service-account` annotation (or the client SDK is configured to impersonate) and is unintentionally impersonating a GSA; go to Step 3. - `403 PERMISSION_DENIED` on the **target API/resource** (no `getAccessToken` in the error) → the resolved identity lacks the required IAM role on that resource; go to Step 4. - `could not find default credentials` / `cannot find the metadata server` → metadata-server connectivity or a startup race; go to Step 5. - Calls succeed but as the **node default service account** → Workload Identity is not in effect for this node pool; go to Step 2. -------------------------------------------------------------------------------- ### Step 2: Verify Workload Identity is enabled (cluster + node pool) Both the cluster and the **node pool the Pod runs on** must have Workload Identity enabled. A node pool with `GCE_METADATA` (instead of `GKE_METADATA`) causes Pods to fall back to the node's default Compute Engine service account. ```bash # Cluster must have a workload identity pool (PROJECT_ID.svc.id.goog). gcloud container clusters describe "{cluster}" --location "{location}" \ --format="value(workloadIdentityConfig.workloadPool)" # Node pool must have workloadMetadataConfig.mode = GKE_METADATA. gcloud container node-pools describe "{node_pool}" --cluster "{cluster}" \ --location "{location}" \ --format="value(config.workloadMetadataConfig.mode)" ``` - Empty workload pool → Workload Identity is **not enabled on the cluster**. - Node-pool mode is `GCE_METADATA` (or empty) → the node pool is **not** using the GKE metadata server; this is the usual cause of "runs as the node default service account". Remediation: enable `--workload-metadata=GKE_METADATA` on the node pool (propose to a human; recreates nodes). This is a **node-pool configuration** problem — not a KSA annotation or IAM binding problem — so do not change KSA annotations or IAM bindings to fix it. When the symptom is "runs as the node default service account", say so explicitly: the root cause is the node pool's `workloadMetadataConfig.mode`, and the KSA annotation and IAM bindings are **ruled out** as the cause. Propose the exact fix, e.g.: ```bash gcloud container node-pools update "{node_pool}" --cluster "{cluster}" \ --location "{location}" --workload-metadata=GKE_METADATA ``` -------------------------------------------------------------------------------- ### Step 3: Verify the KSA → identity binding There are two supported models. **Prefer direct binding** (current default); treat GSA impersonation as the **legacy** path. > **How the two models fail differently:** with **direct binding** the KSA > principal accesses resources directly, so failures show up as a plain `403 > PERMISSION_DENIED` on the target API (fix in Step 4). A `403 > iam.serviceAccounts.getAccessToken` instead means an **impersonation** attempt > — intended under the legacy path, or *unintended* if a leftover > `iam.gke.io/gcp-service-account` annotation remains on a KSA that was meant to > use direct binding. **(a) Direct KSA binding (no GSA impersonation).** The KSA principal is granted roles directly. Construct the principal identifier and search for its bindings: ``` principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME} ``` **(b) Legacy: KSA + GSA impersonation.** The KSA must be annotated to point at a GSA, and the KSA must hold `roles/iam.workloadIdentityUser` on that GSA. ```bash # The KSA annotation must reference the intended GSA. kubectl get serviceaccount "{ksa_name}" -n "{namespace}" \ -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}' # The GSA's IAM policy must bind the KSA member to workloadIdentityUser. gcloud iam service-accounts get-iam-policy \ "{gsa_name}@{project_id}.iam.gserviceaccount.com" \ --format=json # Expect a binding: role roles/iam.workloadIdentityUser, # member serviceAccount:{PROJECT_ID}.svc.id.goog[{NAMESPACE}/{KSA_NAME}] ``` > If the annotation is present but the binding is missing, the binding was > likely removed — check the `setIamPolicy` audit logs around the failure time > to find the responsible principal. -------------------------------------------------------------------------------- ### Step 4: Verify IAM permissions on the target resource Even with a correct binding, the identity (the KSA principal for direct binding, or the GSA for legacy) must hold the role required by the API call (for example `roles/storage.objectViewer`). The standard IAM Policy Troubleshooter has limited support for Workload Identity principals; use Cloud Asset Inventory to search all IAM policies for the principal instead. ```bash # Direct binding: search for the KSA principal's bindings across the project. gcloud asset search-all-iam-policies \ --scope="projects/{PROJECT_ID}" \ --query='policy:"{PROJECT_ID}.svc.id.goog"' # Legacy: search for the GSA's bindings on the target resource's project. gcloud asset search-all-iam-policies \ --scope="projects/{TARGET_PROJECT_ID}" \ --query='policy:"{gsa_name}@{project_id}.iam.gserviceaccount.com"' ``` If no binding grants the required role on the target resource, **that missing role is the root cause** (and Workload Identity itself is **not** misconfigured). Present the fix as a **proposed change for a human to apply** — the exact `add-iam-policy-binding` with the real principal and role — never applying it automatically. For example, for direct binding on a project-level resource: ```bash gcloud projects add-iam-policy-binding "{TARGET_PROJECT_ID}" \ --member="principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}" \ --role="{REQUIRED_ROLE}" # e.g. roles/storage.objectViewer ``` -------------------------------------------------------------------------------- ### Step 5: GKE metadata server & connectivity Applies when the signature is `could not find the metadata server`, `cannot find the metadata server`, or a connection/timeout error. The `gke-metadata-server` DaemonSet (in `kube-system`) brokers the token exchange on each node; requests to `169.254.169.254` are redirected to it, so a Pod fails closed if it cannot reach a healthy metadata-server Pod on its node. **(a) Check gke-metadata-server Pod health on the workload's node.** ```bash # The DaemonSet Pods must be healthy on the workload's node. kubectl get pods -n kube-system -l k8s-app=gke-metadata-server -o wide # A gke-metadata-server Pod can be OOM-evicted when the cluster has many # (>3,000) Kubernetes service accounts. Look for CrashLoopBackOff, then confirm # the eviction was OOMKilled. kubectl get pods -n kube-system | grep CrashLoopBackOff kubectl describe pod {gke_metadata_server_pod} --namespace=kube-system | grep OOMKilled ``` **(b) Inspect the `gke-metadata-server` logs** (historical, via Cloud Logging; open as a Logs Explorer deep link, see Step 1): ``` resource.type="k8s_container" resource.labels.namespace_name="kube-system" labels."k8s-pod/k8s-app"="gke-metadata-server" severity>=WARNING ``` **(c) Connectivity test** from the affected Pod — it must reach the metadata server and (for some client libraries) resolve its DNS name: ```bash # Token endpoint via the hardcoded metadata IP (a healthy path returns a token). kubectl exec {pod_name} -n {namespace} -- \ curl -sS -H 'Metadata-Flavor: Google' \ 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' ``` Possible causes: - **Startup race**: the metadata server needs a few seconds after a Pod starts. Applications that authenticate immediately may fail; the fix is application-side retries or an `initContainer` that waits for the metadata server, not a cluster change. - **DNS**: some client libraries resolve `metadata.google.internal`; broken in-cluster DNS surfaces as `cannot find the metadata server`. As a workaround, set `GCE_METADATA_HOST=169.254.169.254` to skip DNS resolution. - **Network policy / egress**: a cluster network policy must allow egress to `169.254.169.254/32` on port `80` (GKE Dataplane V2), or