| name | kiwi-cicd-debugging |
| description | Use when a deploy.yml run fails on Kiwi's prod CI/CD pipeline, bootstrap-cicd.sh fails against real GCP, or you're modifying deploy.yml/bootstrap-cicd.sh and want to avoid gotchas already hit. Covers gcloud beta run jobs, Cloud Run canary/ingress 404s, IAM instance-scope restrictions, OS Login setup, and a required-status-check merge blocker. |
Kiwi CI/CD Debugging
Overview
.github/workflows/deploy.yml deploys Kiwi's real prod (GCP project
kiwi-prod-502913) on every merge to main: verify → build-and-push →
migrate → deploy-api → deploy-orchestrator → deploy-frontend →
refresh-fleet. One-time GCP identity setup is
deploy/gcp/bootstrap-cicd.sh (human-run, not invoked by CI). Full design
history and every amendment below is also recorded in
docs/superpowers/specs/2026-08-10-prod-cicd-design.md — keep that file and
this skill in sync when you find something new.
Pulling real failure logs
gh run view --job <id> --log sometimes returns nothing (seen for jobs
belonging to a reusable workflow). The REST API always works and is not
truncated:
gh run list --repo RunKiwi/kiwi --workflow deploy.yml --limit 5
gh run view <RUN_ID> --repo RunKiwi/kiwi
gh api repos/RunKiwi/kiwi/actions/jobs/<JOB_ID>/logs | tail -150
gh run watch <RUN_ID> --repo RunKiwi/kiwi --exit-status --interval 15
streams a run to completion — run it with run_in_background, a full run
can take 10+ minutes.
Known failure modes
All of these were hit for real against kiwi-prod-502913 (2026-08-10/11,
PRs #333-#337) and are already fixed in the current deploy.yml/
bootstrap-cicd.sh — listed here in case a similar change reintroduces one,
or a new environment hits the same class of issue.
| Symptom | Cause | Fix |
|---|
ERROR: (gcloud.run) Invalid choice: 'jobs' | gcloud run jobs isn't in the GA command track | Use gcloud beta run jobs ... everywhere (migrate job, bootstrap script) |
| Migrate job hangs then fails: "not in an interactive session" / "requires the installation of components: [beta]" | gcloud auto-installs the beta component on first use; its confirmation prompt hangs forever on a non-interactive runner | setup-gcloud step needs with: install_components: beta |
bootstrap-cicd.sh fails: "Role roles/iap.tunnelResourceAccessor is not supported for this resource" | Compute Engine's instance-level IAM only accepts a curated allowlist — OS Login roles (osLogin/osAdminLogin) are on it, IAP tunnel accessor and most custom roles are not | Grant at project level instead: gcloud projects add-iam-policy-binding $PROJECT --role roles/iap.tunnelResourceAccessor --condition=None. Same fix for a custom role hitting the same error. |
refresh-fleet's SSH step fails: "Required 'compute.instances.setMetadata' permission" | OS Login IAM roles were granted, but OS Login itself was never enabled on the instance — gcloud silently falls back to legacy metadata-based SSH keys, which need a permission this identity deliberately lacks | gcloud compute instances add-metadata <vm> --metadata enable-oslogin=TRUE — one-time, non-disruptive to a running instance. Granting the IAM role is necessary but not sufficient. |
Canary health check 404s forever on a Cloud Run service's direct *.run.app tag URL, even though the app's /healthz works fine through its real domain | The 404 is Google's own edge page (generic "That's an error", not the app's) — the service is ingress-restricted to load-balancer-only traffic (fronted by a serverless NEG + external HTTPS LB), so direct Cloud Run URLs never reach the container | Don't chase the curl. Deploy with a plain gcloud run deploy (no --no-traffic, no tag) — it already blocks until the new revision is Ready before routing traffic, and never touches traffic if the revision fails. No HTTP check needed. All three Cloud Run services deploy this way today; there is no canary workflow in this pipeline anymore. |
Debugging something not in this table
- Pull the real log with
gh api .../logs above — the job summary in
gh run view truncates and can hide the actual error.
- Classify it: IAM/permissions ("Permission ... denied", "is not supported
for this resource"), a genuine app/config bug, or transient
(timeouts,
DeadlineExceeded) — that determines whether to fix, retry,
or dig further.
- If it's IAM, check what's already granted in
bootstrap-cicd.sh first,
and remember the instance-vs-project scope restriction above before
adding a new gcloud compute instances add-iam-policy-binding call —
it may need to be a project-level binding instead.
- Fix it, then update both the code/script and the "Amendments"
section(s) of
docs/superpowers/specs/2026-08-10-prod-cicd-design.md —
that file and this skill should never drift apart.
- Branch the fix off latest
main (not a stale branch), expect the
required-status-check gotcha above, and watch the next run with
gh run watch.
