agent-platform-troubleshooting

# Agent Platform Troubleshooting > **Important:** You MUST ONLY use the reference files located in this skill's > `references/` directory (`references/field-manual.md`, > `references/known-issues.md`, `references/agent-gateway.md`, > `references/policies.md`, `references/agent-registry.md`, > `references/agent-identity.md`). Do NOT search for or read external playbooks > outside this directory. Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry (Services, Agents, MCP Servers, Endpoints), Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies (SGP), explicit/transparent proxies, and tracing. This skill produces a **diagnostic report** — findings and fix recommendations. It does not apply fixes; the user owns all mutations. ## Usage Attribution **MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read and follow [Usage Attribution](references/attribution.md). Prefix direct terminal `gcloud` calls (never generated scripts, Terraform, or user code) with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-troubleshooting)"` and append `agent-skills/1.0 (skill:agent-platform-troubleshooting)` to direct HTTP/REST/MCP `User-Agent` headers. ## MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS) CRITICAL: Before generating ANY response or calling any tools, evaluate the user's prompt against Rules 1–14. When a Turn-0 rule matches, execute its directive immediately and STOP. ### Rule 1: Out-of-Scope GCP IAM / GCS Queries If the prompt asks about Compute Engine (GCE), Cloud Storage (GCS) buckets, or generic GCP IAM permissions unrelated to the Agent Platform stack: - **IMMEDIATELY DECLINE ON TURN 0. DO NOT CALL ANY TOOLS OR GIVE IAM GUIDES.** - Respond: *"I decline to troubleshoot generic GCP IAM or GCS access issues, as they are out of scope for the Agent Platform Troubleshooting skill."* ### Rule 2: Strict Prohibition on Custom Discovery Scripts If asked to write, generate, compile, or execute a custom Python or bash script to discover resources (e.g., active Agent Runtime instances): - **DO NOT CALL ANY TOOLS OR WRITE/RUN ANY SCRIPTS.** - **RESPOND ON TURN 0**: *"I cannot write or execute custom Python or bash scripts for resource discovery. Custom discovery scripts are prohibited as they consume excessive turns and cause timeouts. Instead, use standard gcloud CLI commands (see [Google Cloud SDK Installation](https://docs.cloud.google.com/sdk/docs/install.md.txt)) or curl REST API calls with application default credentials: `gcloud agent-registry agents list --location=us-central1` or `curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/reasoningEngines"`"* ### Rule 3: Consolidated Registry for Google APIs / Design Queries If asked about registering multiple Agent Runtime, Cloud Resource Manager, or Google API interfaces, or structuring services in Agent Registry: - **DO NOT CALL ANY TOOLS. RESPOND ON TURN 0 WITH**: 1. Consolidate ALL Google APIs under a single `googleapis` service entry in Agent Registry via `gcloud agent-registry services create googleapis --endpoint-spec-type=no-spec`. 2. Explicitly state: *"Do NOT register each Google API as a separate registry service entry, as separate entries cause resource clutter, complicate IAM policy management, and risk hitting registry quota limits."* 3. List all 8 required base FQDN interfaces (plus global `https://aiplatform.googleapis.com` when calling global Gemini models): - `https://agentregistry.googleapis.com` - `https://aiplatform.mtls.googleapis.com` - `https://cloudresourcemanager.mtls.googleapis.com` - `https://iamcredentials.mtls.googleapis.com` - `https://telemetry.mtls.googleapis.com` - `https://{region}-aiplatform.mtls.googleapis.com` - `https://{region}-aiplatform.googleapis.com` - `https://aiplatform.{region}.rep.googleapis.com` 4. Refer to `references/agent-registry.md` §2 for the full command. ### Rule 4: Cloud Run / Cloud Functions Egress 401/403 (OIDC vs IAP MCP) If an agent gets a `401` or `403` egress error calling an MCP server or tool on Cloud Run or Cloud Functions: - **DO NOT RUN LOG TOOLS. RESPOND ON TURN 0 WITH**: 1. Explain that direct Agent Identity (`principalSet://...` / `principal://...`) to standard Cloud Run OIDC authentication is **not natively supported**. 2. Recommend **Service Account impersonation** in agent code to obtain an OIDC ID token, granting **`roles/iam.serviceAccountTokenCreator`** to the Agent Identity on the target SA (and `roles/run.invoker` to that SA on Cloud Run). 3. Note that for Preview IAP-enabled Cloud Run MCP servers (`--iap --functional-type=mcp-server`, `run.app` short URL), authorize via IAP/UAP instead of `roles/run.invoker` (see `references/known-issues.md` BKI 21 and `references/agent-identity.md`). ### Rule 5: Telemetry & Monitoring Endpoint Blocks on Startup If Agent Runtime startup crashes (`set_up()`) with connection resets or handshake errors reaching `telemetry.mtls.googleapis.com`: - **Evidence**: Explicitly check whether all 4 monitoring/tracing endpoints are registered as Endpoints in Agent Registry (`gcloud agent-registry endpoints list`): `telemetry.mtls.googleapis.com`, `monitoring.googleapis.com`, `trace.mtls.googleapis.com`, and `cloudtrace.googleapis.com`. - **Recommended Fix**: 1. Register missing telemetry/tracing interfaces via `gcloud agent-registry services create --endpoint-spec-type=no-spec` (which projects read-only Endpoints in Agent Registry for `telemetry.mtls.googleapis.com`, `monitoring.googleapis.com`, `trace.mtls.googleapis.com`, and `cloudtrace.googleapis.com`). 2. Create or update an **`AuthorizationPolicy`** (`AuthzPolicy` / IAP / UAP) bound to the Gateway that allows the agent's principal set (identity) to access the telemetry endpoints (see `references/known-issues.md` BKI 23). ### Rule 6: IAP Denial Troubleshooting (`403` / `Egress request is not authorized`) When logs or symptoms show `403` / `Egress request is not authorized` via IAP: 1. **Check IAP Egressor bindings on the registry entry FIRST** (`roles/iap.egressor` via `gcloud iap web get-iam-policy --resource-type=agent-registry` bound to the server-generated child UUID from `registryResource`, not `SERVICE_NAME`). 2. **Create a registry entry** (`gcloud agent-registry services create`) if none exists for the exact target hostname. 3. **Grant `roles/iap.egressor`** on the registry entry (IAM v1 — `set-iam-policy` requires top-level `{"bindings": [...]}` JSON without a `"policy"` wrapper) or verify UAP `AccessPolicy` + CRM `PolicyBinding` (`iapPolicyVersion: "V2"`, `iap.googleapis.com/resources.egressViaIAP`; see `references/policies.md` §2). 4. **Check IAP audit logs** (`protoPayload.serviceName="iap.googleapis.com"`) and verify an `AuthorizationPolicy` (`AuthzPolicy`) is bound to the Gateway. 5. **Explicitly warn**: *"Do NOT use `roles/iap.tunnelResourceAccessor` (which is for classic IAP TCP tunneling to GCE VMs, not Agent Gateway egress) and do NOT bypass IAP authentication."* ### Rule 7: PSC Subnet Exhaustion Speed Rule (`PROVISIONING` Hang) When diagnosing gateway provisioning stuck or failing on subnet IP exhaustion: - **DO NOT loop across regions.** Run ONLY in the target region (default `us-central1`): 1. `gcloud network-services agent-gateways list --location=us-central1` 2. `gcloud network-services agent-gateways describe <GW> --location=us-central1` 3. `gcloud compute network-attachments describe <NA> --region=us-central1` 4. `gcloud compute networks subnets describe <SUBNET> --region=us-central1` - Check free IP count (`Usable - Allocated = Free`) **before** recommending any Network Attachment deletion; flag that a `/28` subnet is easily exhausted and recommend expanding the CIDR range to at least `/26` (`known-issues.md` BKI 19). ### Rule 8: Multi-Region Manual Registration Prohibition (`us` / `eu`) If asked about manually registering endpoints or services in `us` or `eu`: - **DO NOT CALL ANY TOOLS. RESPOND ON TURN 0**: 1. *"Manual endpoint/service registration is NOT supported in `us` or `eu` multi-region locations (`us` and `eu` are read-only roll-up locations for auto-discovered resources)."* (Explicitly name BOTH `us` and `eu`.) 2. *"Register manually in a specific region (e.g., `us-central1`) or `global`."* ### Rule 9: VPC Service Controls (VPC-SC) Perimeter & ACT `ALL_TRAFFIC` When diagnosing VPC-SC perimeter blocks or external API hangs under VPC-SC: 1. Identify a **VPC Service Controls perimeter block** / boundary enforcement. 2. Check that the Agent Connectivity Template (ACT) sets **`vpcEgress: ALL_TRAFFIC`** (native VPC-SC creation works out of the box since Sept 8, 2026 when `vpcEgress: ALL_TRAFFIC` is set) and that **Cloud NAT** is configured on the consumer VPC's PSC-I network attachment subnet. 3. For legacy or strict custom perimeters blocking Google actuation identities, recommend creating **ingress policies** allowing: - `actuation-a@networkservices-prod.iam.gserviceaccount.com` - `cloud-aiplatform-pipeline-robot-prod@system.gserviceaccount.com` 4. Explicitly state: *"Do NOT disable VPC Service Controls or delete perimeters."* Refer to `references/known-issues.md` BKI 18 & 28. ### Rule 10: UAP Policy Binding Org Policy Blocker & Blanket `DENY` Crash - **`CUSTOM_ORG_POLICY_VIOLATION` on `gcloud iam policy-bindings create`**: Caused by **`constraints/iam.managed.disableAccessPolicyBinding`** (note: some docs misspell this plural). Apply an Org Policy override (`enforce: false`) at the target resource and wait **30 to 60 seconds** for IAM propagation (BKI 24). - **Startup crash after UAP `destination.is_registered == false` `DENY`**: Explain that the blanket `DENY` blocked internal/unauthenticated SDK bootstrap and metadata calls (such as Cloud Resource Manager project lookups) during Reasoning Engine init; remove or scope the blanket `DENY` rule (BKI 5). ### Rule 11: Agent Gateway Dual-Registry & Jurisdiction Invariants When configuring or validating `registries` on an Agent Gateway: 1. An Agent Gateway supports a **maximum of two registries**. 2. When two registries are configured, **exactly one must be `global`** and the other must be **`regional` or `multi-regional`**. 3. Binding two regional registries (e.g., `us-central1` + `europe-west1`), two multi-regionals, two globals, or a mismatched jurisdiction (e.g., `us-central1` gateway + `eu` registry) is invalid and rejected with an **HTTP 400** validation error (`references/agent-gateway.md` §4, BKI 26). ### Rule 12: Cross-Project Runtime-to-Gateway Binding Diagnosis When Reasoning Engine in Project 1 fails to bind to a Gateway in Project 2: 1. **Deploying Identity**: Needs `networkservices.agentGateways.use` and `networkservices.agentGateways.get` (or `roles/networkservices.viewer`) in Project 2. 2. **Vertex AI Service Agent**: Grant `roles/networkservices.viewer` or custom role `ae_agw_cross_project_sa` (alias `ar_agw_cross_project_sa`) to `service-PROJECT_1_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com` in Project 2. 3. **Regional Colocation**: Verify both runtime and gateway are in the **same region**. 4. **CAA Opt-Out**: Set `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES` to `False` (`references/known-issues.md` BKI 10, `references/agent-gateway.md` §11). ### Rule 13: Downstream SWP + PBR vs. Explicit Forward HTTPS Proxy Distinguish the two supported consumer-VPC proxy architectures: - **Pattern A — Transparent Next-Hop SWP + PBR** (no client `HTTPS_PROXY` needed): 1. Explain that **PSC endpoints cannot be route next-hops**, **SWP behind PSC Service Attachments only supports explicit proxy mode (not transparent next-hop mode)**, and **static routes cannot be used because Network Attachments cannot have VM network tags**. 2. Deploy SWP directly in the consumer VPC with `type: SECURE_WEB_GATEWAY` and `routingMode: NEXT_HOP_ROUTING_MODE` on an internal IP on the PSC-I subnet (e.g., `10.20.1.250`), with a `/26` `REGIONAL_MANAGED_PROXY` `ACTIVE` subnet. 3. Configure **Policy-Based Routing (PBR)** matching source CIDR `10.20.1.0/24` to `--next-hop-ilb-ip=10.20.1.250` (priority 200) + fallback PBR (1000). 4. Create **Cloud NAT with `ENDPOINT_TYPE_SWG`** (`nats create ... --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG`), and allowlist `<region>-aiplatform.mtls.googleapis.com` in SWP CEL (BKI 33). - **Pattern B — Explicit Forward HTTPS Proxy (`HTTPS_PROXY` + L4 `AuthzPolicy`)**: 1. Without L4 bypass, `HTTPS_PROXY=https://proxy.corp.internal:443` fails on `HTTP CONNECT` with **HTTP `404`** because Agent Gateway's L7 HTTP manager terminates TLS and rejects nested `CONNECT`. 2. Import a **`v1beta1`** L4 `AuthzPolicy` via **`gcloud beta network-security authz-policies import`** with `action: ALLOW` and `networkRules` matching `to.operations[].snis` (omit `target.loadBalancingScheme` and `ports`; max 5 rules; omit `from` or include `7.0.0.0/8` + PSC-I CIDR in `from.sources[].ipBlocks`). 3. Listen with TLS on `:443`, allow the consumer PSC-I subnet CIDR on VPC firewall/proxy ACLs (never `7.0.0.0/8`), set `NO_PROXY`/`no_proxy` to `.googleapis.com,googleapis.com,metadata.google.internal,169.254.169.254,localhost,127.0.0.1`, bundle `agentGatewayCard.rootCertificates` + Private CA in container env vars (`SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`), and warn L4 `ALLOW` bypasses L7 IAP/UAP, Model Armor, SGP, and `httpRules` for matched SNIs (BKI 34). ### Rule 14: DNS Peering, `TrustConfig`, Model Armor, SGP, MCP, Identity, & Lifecycle - **DNS Peering & Service Agent**: Public FQDNs resolve via public recursive DNS. Private DNS requires `dnsPeeringConfig` (`domains` array with trailing `.`; ACT uses string `targetNetwork` only, whereas inline `AgentGateway.networkConfig.dnsPeeringConfig` also requires `targetProject`). Cloud DNS caps total peering hops at **2** (Gateway consumes Hop 1 → max 1 consumer peering hop; a 2nd hop fails preflight or returns `SERVFAIL` → `503`, BKI 35). Grant `roles/dns.peer`, `roles/dns.reader`, `roles/compute.networkViewer`, and `roles/compute.networkUser` to `service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com` (and `compute.networkAttachments.use` to caller, BKI 9). Never use authoritative `"."` private root zones (`"."` requires a Cloud DNS Private Forwarding Zone). - **Private CA `TrustConfig` & Immutable ACT**: Set `egressNetworkConfig.tlsConfig.trustConfig` **with `additionalRoots: PUBLICLY_TRUSTED_ROOTS`** (omitting breaks `*.googleapis.com`). ACT fields are immutable — create a new ACT and rebind via `PATCH` with `updateMask=agentConnectivityTemplate` (BKI 6). - **Model Armor (`CONTENT_AUTHZ`)**: Ingress (`CLIENT_TO_AGENT`, max 1, ADK `streamQuery` only) uses `service-AGENT_PROJECT_NUMBER@gcp-sa-aiplatform-re.iam.gserviceaccount.com`; Egress (`AGENT_TO_ANYWHERE`, max 4) uses `service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com`.