gcp-debug

# /gcp-debug — GCP Production Debugging Workflow End-to-end diagnostic workflow for incidents on a Cloud Run + Apigee stack. Built from real incident triage (Intercom freeze caused by per-request `PublisherClient` instantiation + Apigee SpikeArrest matching the wrong path). ## When to activate - User reports a prod incident: "the app is frozen", "we have latency", "5xx in service X" - User asks to "look at logs", "check metrics", "see what happened at HH:MM" - User says a third-party (webhook source like Intercom/Slack/WhatsApp) is throttling or freezing them - Any time you need to correlate Apigee gateway behavior with backend Cloud Run behavior ## Core principles - **Always confirm time window** in absolute UTC. Convert from user's local time (CDMX = UTC-6, Caracas = UTC-4) and state the conversion back to the user. - **Triangulate, don't trust one signal.** A single dashboard can lie. Cross-check Cloud Run metrics against `run.googleapis.com/requests` log lines and against Apigee LB logs. - **Look for "round" repeated values in latency** (e.g., 3.06s repeated across 8 requests = timeout signal, not random tail). This is the strongest fingerprint of a timeout/retry bug. - **`p99 = 64.229s` in `request_latencies` means the histogram bucket is saturated** — actual values may be higher and Cloud Run is timing them out. - **Never propose a fix until the chain of causation is complete.** Surface causes amplify each other; pick the one that, if removed, breaks the chain at its root. ## Gotchas you must know about this stack | Gotcha | Workaround | |---|---| | `gcloud monitoring time-series` does NOT exist | Use REST API: `https://monitoring.googleapis.com/v3/projects/$PROJ/timeSeries?...` with `gcloud auth print-access-token` | | Apigee Analytics add-on **disabled** in `cashea-apigee-prod` | The `/stats/apiproxy` API returns "Analytics add-on is disabled". Use Cloud Logging on `resource.type="http_load_balancer"` instead | | `httpRequest.latency` is empty in app logs | Use `logName="projects/$PROJ/logs/run.googleapis.com%2Frequests"` to get real request logs with latency | | Apigee `Matches "/foo/*"` matches **single segment**, not sub-paths | Use `MatchesPath "/foo/**"` for multi-segment paths | | Cloud Run `instance_count` returns `state` label: `active` and `idle` | Sum them for total provisioned capacity. `idle` = warm but no traffic | | If CPU < 60% target but instances pinned to `min`, the bottleneck is **I/O wait**, not CPU | Don't blame the autoscaler — look at LLM/DB/external HTTP latencies | ## Standard workflow ### Step 0 — Confirm scope Ask the user (if not already given): - Time window in absolute UTC - Which service / channel / endpoint they suspect - Whether there's a specific dashboard/screenshot they're looking at ### Step 1 — List what you have access to ```bash gcloud config list # which account/project gcloud projects list --limit=20 # what projects user has gcloud run services list --project=$PROJ --format="table(metadata.name,status.url,...autoscaling...)" ``` ### Step 2 — Get error logs in window ```bash gcloud logging read 'resource.type="cloud_run_revision" AND severity>=ERROR \ AND timestamp>="$START_UTC" AND timestamp<="$END_UTC"' \ --project=$PROJ --limit=200 \ --format="value(timestamp,resource.labels.service_name,severity,jsonPayload.message,textPayload)" ``` ### Step 3 — Get traffic + latency metrics (REST, not gcloud) ```bash TOKEN=$(gcloud auth print-access-token) curl -s -H "Authorization: Bearer $TOKEN" \ "https://monitoring.googleapis.com/v3/projects/$PROJ/timeSeries?\ filter=metric.type%3D%22run.googleapis.com%2Frequest_latencies%22%20AND%20resource.label.service_name%3D%22$SVC%22&\ interval.startTime=$START_UTC&\ interval.endTime=$END_UTC&\ aggregation.alignmentPeriod=300s&\ aggregation.perSeriesAligner=ALIGN_PERCENTILE_99&\ aggregation.crossSeriesReducer=REDUCE_MAX" ``` Key metric types: - `run.googleapis.com/request_latencies` — p50/p95/p99 (use `ALIGN_PERCENTILE_*`) - `run.googleapis.com/request_count` — req/s (use `ALIGN_RATE`, group by `metric.label.response_code_class`) - `run.googleapis.com/container/instance_count` — group by `metric.label.state` for active/idle - `run.googleapis.com/container/cpu/utilizations` — to check if autoscaler should be triggering - `run.googleapis.com/container/max_request_concurrencies` — to see saturation per instance ### Step 4 — Find slow individual requests ```bash gcloud logging read 'logName="projects/$PROJ/logs/run.googleapis.com%2Frequests" \ AND resource.labels.service_name="$SVC" \ AND timestamp>="$START_UTC" AND timestamp<="$END_UTC"' \ --project=$PROJ --limit=10000 \ --format="value(timestamp,httpRequest.latency,httpRequest.status,httpRequest.requestUrl,httpRequest.userAgent)" \ | awk -F'\t' '{l=$2; gsub("s","",l); print l+0"\t"$0}' \ | sort -rn | head -30 | cut -f2- ``` This finds the actual slow requests. **Look for repeated near-identical latency values** — they signal a hard timeout or retry. Random tail ≠ repeated round number. ### Step 5 — Apigee LB logs (when proxy is suspected) ```bash # Status code distribution for /your-proxy-path gcloud logging read 'resource.type="http_load_balancer" \ AND timestamp>="$START_UTC" AND timestamp<="$END_UTC" \ AND httpRequest.requestUrl=~"/your-proxy-base"' \ --project=$APIGEE_PROJ --limit=10000 \ --format="value(httpRequest.status)" | sort | uniq -c | sort -rn # Drill into specific status (e.g. 429) gcloud logging read '... AND httpRequest.status=429' --format="value(timestamp,httpRequest.userAgent,httpRequest.requestUrl,httpRequest.latency)" ``` ### Step 6 — Pull Apigee proxy bundle (config inspection) ```bash TOKEN=$(gcloud auth print-access-token) # Find active revision curl -s -H "Authorization: Bearer $TOKEN" \ "https://apigee.googleapis.com/v1/organizations/$APIGEE_ORG/environments/$ENV/apis/$PROXY/deployments" # Download bundle curl -s -H "Authorization: Bearer $TOKEN" \ -o /tmp/proxy.zip \ "https://apigee.googleapis.com/v1/organizations/$APIGEE_ORG/apis/$PROXY/revisions/$REV?format=bundle" unzip -o /tmp/proxy.zip -d /tmp/proxy-bundle ``` Then inspect XML files in `apiproxy/policies/` (look for `SpikeArrest`, `Quota`, `RateLimit`) and `apiproxy/proxies/default.xml` (look at `Condition` blocks — be wary of `Matches` vs `MatchesPath`). ### Step 7 — Build the chain of causation Lay out a timeline like: ``` HH:MM:SS.ms Event A (with source: log line / metric / proxy config) HH:MM:SS.ms Event B (consequence of A) HH:MM:SS.ms Event C (root cause - candidate fix) ``` Then identify which fix would break the chain at the highest leverage point. Surface causes (e.g., a 429 burst) often have a deeper cause (a 3s timeout that triggered the retry burst). ## Reporting format to user Always present findings as: 1. **Hallazgo** — what you found (concrete numbers, log excerpts, file:line) 2. **Cadena de causación** — timeline + how each event leads to the next 3. **Fix(es) propuestos** — minimal, prioritized by impact, with diff 4. **Pregunta antes de actuar** — confirmation needed before applying anything ## Anti-patterns - Running broad queries without time bounds — burns API quota and dumps too much - Trusting a single percentile spike without correlating with other signals - Proposing a fix at the symptom layer (e.g., raising SpikeArrest rate) without finding the root timeout/retry that caused the burst - Skipping the Apigee bundle inspection when proxy 4xx/5xx is observed — the policy XML is often where the bug is - Recommending autoscaler tuning when CPU is < 20% (the bottleneck is I/O wait, not capacity) ## Constraints - Never modify GCP infra without explicit user approval (Apigee proxy changes especially — proxy is shared across teams) - Never propose `--force`, `--no-verify`, or "raise the limit" type fixes as first resort - Never skip the time-window confirmation step — wrong window invalidates the entire diagnosis - Always state UTC + user's local time when reporting timestamps