debug-ci-pipeline-failure

name	debug-ci-pipeline-failure
description	Debugs a red CI job to root cause instead of blind-rerunning — reproduce locally in the SAME image (`act -j <job>`, `gitlab-runner exec`, `circleci local execute`, or `docker run` the exact pinned digest), read the full log + the real exit code (124=timeout, 137=OOM/SIGKILL, 143=SIGTERM, 139=segfault), then classify into flaky / env-drift / poisoned-or-stale cache / resource-OOM / missing-secret / timeout / test-ordering / network, and confirm with a targeted experiment — diff local-vs-CTRL env (`printenv | sort`, tool `--version`, lockfile hash), run clean (no cache, `--no-cache`/clear key) vs cached, isolate ONE matrix leg, bisect with `git bisect run`, re-run with debug logging (`ACTIONS_STEP_DEBUG=true`, `CI_DEBUG_TRACE=true`, `set -x`) or open an interactive runner (`tmate`/`debug with SSH`/`--privileged` shell) — and fix the cause (pin the digest, scope the cache key, raise the limit, randomize then fix test order) not the symptom.
when_to_use	A CI/CD job that passes locally is red on the runner, fails intermittently, or broke without a relevant code change — green-on-my-machine/red-on-CI, an OOM/timeout/exit-137, a cache or matrix-only failure, or you're tempted to just hit "Re-run job". Distinct from cicd-pipeline-author (designs/authors the pipeline YAML from scratch; this debugs an existing one that's failing) and debug-flaky-tests (fixes the nondeterministic TEST itself — one of several causes here; this skill first classifies whether flakiness, env drift, cache, or limits is even the cause).

When to Use

Reach for this skill when a CI job is failing and you need to find out WHY before touching anything:

• "It's green locally but red on CI" / "passes on my machine, fails on the runner"
• "The job got killed — exit 137 / OOM / 'Process completed with exit code 137'"
• "It only fails sometimes — re-running makes it go green" (don't stop there — classify it)
• "One matrix leg (py3.12 / arm64 / windows) fails, the rest pass"
• "Nothing in my diff touches this — it broke on its own" (env/cache/upstream drift)
• "The job hangs and gets cancelled after N minutes" (timeout vs deadlock)
• "I keep hitting Re-run and hoping" — STOP, reproduce and root-cause instead

NOT this skill:

• Authoring the pipeline YAML, stages, caching strategy, runners from scratch → cicd-pipeline-author (this skill debugs the pipeline it produced)
• Fixing the nondeterministic test itself (shared state, time/random, async races, order-dependence) once you've confirmed flakiness is the cause → debug-flaky-tests (this skill decides whether it's a flaky test vs env/cache/limit, then hands off)
• General "why does this code crash" root-causing unrelated to CI → debug-root-cause (this skill is CI-runner-specific: images, caches, runners, matrices)
• Choosing/pinning the toolchain & language versions as a deliverable → pin-toolchain-versions (this skill detects version drift as a cause and tells you to pin)
• Designing the cache key/layers/TTL as a strategy → caching-strategy (this skill invalidates a poisoned cache to confirm it's the cause)
• Debugging a failing K8s pod/job workload (not a CI runner) → k8s-debug-workload
• A missing secret that's really a vault/rotation/scoping problem → secrets-management (this skill detects "secret empty on CI" as a class; that one fixes how secrets are stored/injected)
• Standing up a reproducible local dev container to match CI → compose-local-dev-stack / setup-devcontainer-env
• A production incident/postmortem (not a build) → incident-response-sre

Steps

1. Read the log top-to-bottom and grab the REAL exit code before theorizing. The first red line is rarely the cause — scroll up to the first error, and check the process exit code, which names the failure class:

   Exit code	Means	Likely cause
   1 / 2	generic failure / misuse	real test/build error — read the actual assertion
   124	command timed out (timeout wrapper)	step exceeded its time budget
   137	128+9 = SIGKILL	OOM-killed (almost always memory limit) or job cancelled
   139	128+11 = SIGSEGV	native segfault (bad binary/arch mismatch)
   143	128+15 = SIGTERM	timeout/cancel signalled gracefully
   125	docker run failed	image/entrypoint problem, not your code

   In GitHub Actions add --rerun-failed-jobs only AFTER you know why. Download the raw log (gh run view <id> --log-failed, GitLab "Complete Raw") — the web UI truncates and folds groups.

2. Reproduce locally in the SAME image, not your laptop. "Green on my machine" proves nothing if your machine isn't the runner. Run the actual job in its actual container:

   CI	Local reproduce
   GitHub Actions	act -j <job> --container-architecture linux/amd64 (use the runner image: -P ubuntu-latest=catthehacker/ubuntu:act-latest)
   GitLab CI	gitlab-runner exec docker <job> or glab ci run; pull the exact image:
   CircleCI	circleci local execute --job <job>
   any	docker run --rm -it <image>@<digest> then run the steps by hand

Common Errors

• Hitting "Re-run job" until it's green and calling it fixed. That's hiding a flaky/OOM/cache bug; it recurs and erodes trust in CI. Fix: classify (step 4) and fix the cause; re-run only to confirm.
• "It passes on my machine" as proof. Your laptop isn't the runner (arch, tool version, env, cache). Fix: reproduce in the exact image/digest (step 2).
• Reading only the last red line. The real error is usually higher; the last line is often a downstream symptom. Fix: read top-down, find the FIRST error + the exit code.
• Treating exit 137 as a code bug. It's OOM/kill, not your assertion. Fix: measure RSS, raise mem or cut parallelism (step 9).
• Floating tags (node:20, ubuntu-latest, @v4 minor). They drift between your pull and CI's → "broke with no diff." Fix: pin by digest + lockfile + exact version (pin-toolchain-versions).
• npm install / non-ci installs in CI. Ignores the lockfile → different deps than local. Fix: npm ci, poetry install --no-update, --frozen-lockfile.
• Blaming the test when it's the cache. Stale restored layer fails a clean build. Fix: clean-vs-cached run (step 5) before touching the test.
• Debugging the whole matrix at once. 12 red legs hide which is the real bug. Fix: isolate the one failing leg (step 6).
• CI_DEBUG_TRACE/printing secrets to debug. Leaks credentials into logs. Fix: trace on protected branches only; print ${#SECRET} length, never the value.
• No pipefail, so a failed mid-pipe command exits 0. Silent green on a broken step. Fix: set -euo pipefail in every shell step.
• Empty secret on fork PRs read as a code bug. Secrets aren't exposed to forks/pull_request from forks by design. Fix: recognize the class, use pull_request_target carefully or a label gate, not a value hunt.
• Adding broad retries to mask flakiness. Retrying a deterministic bug just burns minutes. Fix: retry ONLY external network I/O; fix logic/ordering/resource causes.

Verify

1. Reproduced in-image: the failure reproduces with act/gitlab-runner exec/docker run @digest (or you've proven via env-diff exactly what the runner has that you don't) — not just observed in the web UI.
2. Classified, not guessed: you can name the class (flaky / env-drift / cache / OOM / secret / timeout / ordering / network) AND state the experiment that confirmed it (clean-vs-cached, isolated test, RSS measurement, env diff).
3. Exit code accounted for: you read the real exit code and it's consistent with the diagnosis (137→OOM, 124/143→timeout, 1/2→real error).
4. Root cause fix, not a re-run: the diff pins/invalidates/scopes/limits the actual cause; there's no bare "Re-run" or blanket continue-on-error standing in for a fix.
5. Determinism proven: the SAME commit is re-run ≥3× (and for a flake, ≥10×) and is green every time — not green once after N reds.
6. No new leak: debug tracing is off (or gated to protected branches), no secret value was printed, and interactive runners were torn down.
7. Matrix restored: if you isolated a leg, the full matrix is back and all legs pass.

Done = the failure was reproduced in the runner's actual image, classified into one named cause confirmed by a targeted experiment (env diff / clean-vs-cached / isolated test / RSS), and fixed at the root (pin, cache-key, limit, order) — proven by the same commit going green ≥3× with no blind re-run, no masking, and no leaked secrets.