| name | circleci-why-flaky |
| description | Classify recent CircleCI workflow failures into buckets so you can see what's flake vs what's a real bug. Use when CI on a branch is failing repeatedly and you want to know whether retries will fix it. Infers project + branch from the current git repo; defaults to last 7 days. |
CircleCI: why is it flaky?
Bucket recent CircleCI failures by root cause and tell the user which buckets are fixable flake, external outage, or real bug.
How to invoke
Two steps: the script fetches data into a directory of plain text files; you classify by iterative pattern discovery with grep.
Step 1 — set up and fetch
Generate an output directory under .claude/cache/ (gitignored) and run the fetcher in one command:
mkdir -p .claude/cache && OUT=$(mktemp -d .claude/cache/cci-flake.XXXXXX) && node .claude/skills/circleci-why-flaky/fetch.mjs --out "$OUT" && echo "$OUT"
The printed path is your working directory for this run. Remember it — every subsequent Bash call must substitute the literal path, because each Bash call runs in a fresh shell so $OUT does not persist. Below the skill writes $OUT for readability; substitute the actual path each time.
Common flags:
--branch <name> — branch to analyze (default: current branch; pass master/main for the trunk)--workflow <name> — restrict to a single workflow name (default: all workflows on the branch)--days <N> — time window (default: 7)--max <N> — cap on failed workflows to deep-analyze (default: 200)--org, --repo, --vcs — override repo inference if not in a git repo--token <token> — explicit token (overrides ~/.circleci/cli.yml)--out <dir> — required, output directory--cache-dir <dir> — raw log cache (default: .claude/cache/circleci-why-flaky-cache, shared across runs)
Progress goes to stderr. Stdout prints the output directory path on success.
Exit codes: 0 success, 2 bad input/missing flag, 3 auth needed.
Output layout:
$OUT/
├── summary.txt # PROJECT=, BRANCH=, TOTAL_WORKFLOWS=, FAILED_WORKFLOWS=, FAILURE_RATE_PCT=, FAILED_JOBS=
└── jobs/
├── 0000.txt # most recent failure (lower index = more recent)
├── 0001.txt
└── ...
Each NNNN.txt is a KEY=VALUE header block (URL=, JOB=, WORKFLOW=, STATUS=, TIMED_OUT=, TIME=, COMMIT=), a blank line, then the last ~4KB of each failed step's log.
Step 2 — classify by iterative pattern discovery
Don't read every job upfront. Discover patterns one at a time. This scales whether there are 5 failures or 500.
Keep a working list in your head as you go:
markers = [
{ marker, category, label }
...
]
category is one of fixable, external, real. marker is an extended regex (grep -E) — escape regex metacharacters (., (, [, ?, etc.) if you mean them literally.
Loop:
-
Find the next unclassified job.
First iteration (no markers yet):
ls "$OUT/jobs/" | sort | head -1
Later iterations — RE is the alternation of all collected markers:
RE='heap out of memory|TargetClosedError|ERR_PNPM_FETCH'
grep -LE "$RE" "$OUT"/jobs/*.txt | sort | head -1
grep -L lists files without a match. Empty output means every job is classified — break.
-
Read the file (cat <path>). Identify a short, distinctive marker that captures this class of failure:
- Broad enough to catch similar failures (so you don't add one marker per job).
- Narrow enough to avoid false positives across other classes.
- Prefer a stable error string from the failing tool (e.g.
heap out of memory, ERR_PNPM_FETCH, TargetClosedError), not a transient bit like a timestamp or PID.
-
Decide the category and a short label. See the category guide below.
-
Append { marker, category, label } to markers. Repeat from step 1.
Final counts. Once the loop ends, for each marker get the count and the most-recent matching example URL:
PATTERN='heap out of memory'
grep -lE "$PATTERN" "$OUT"/jobs/*.txt | wc -l
grep -lE "$PATTERN" "$OUT"/jobs/*.txt | sort | head -1 | xargs grep -m1 '^URL=' | cut -d= -f2-
If a job matches multiple markers, the earlier (broader) one wins. Pick distinct enough markers that overlap is small — if counts sum to more than FAILED_JOBS, tighten the broader markers.
Category guide
fixable — infra flake we own. Worth filing/digging into.
- Out of memory (
heap out of memory, allocation failed)
- Browser/headless test crashes mid-run (page/context/browser closed, driver disconnected)
- Hung process —
TIMED_OUT=true in the header with no external-outage signal
- Disk full (
no space left on device)
- Flaky-by-design tests (timing races, port conflicts, leaked state)
external — third-party service down. Just retry when it recovers.
- Package registry unreachable (
ERR_PNPM_FETCH, registry.npmjs.org errors)
- OS mirror down (
archive.ubuntu.com, security.ubuntu.com unreachable)
- GitHub down (auth/clone/API failures against
github.com)
- CircleCI infra hiccup (checkout failures, runner provisioning errors)
- DNS/connectivity to a clearly external host
Generic ECONNRESET / ETIMEDOUT without an identifiable third-party host is ambiguous — lean fixable unless the surrounding context names an external service.
real — code/config defect. Retry won't help.
- TypeScript compile error (
error TS<digits>:)
- ESLint error (
<N> problems (<N> errors)
- Test assertion failure (
AssertionError, Expected:/Received:)
- Build/bundler error (module not found, syntax error, unresolved import)
- Snapshot or generated-artifact drift
These are starting points, not closed sets. If a clearly recurring novel pattern appears, give it its own bucket with a short label.
Output shape
Three sections, in this order. Skip any section with zero entries.
# <PROJECT> `<BRANCH>` — last <DAYS> days
**<FAILED_WORKFLOWS>/<TOTAL_WORKFLOWS>** workflow runs failed (<FAILURE_RATE_PCT>% failure rate). **<FAILED_JOBS>** failed jobs classified.
**Fixable flake** (<sum %>):
- N times, P%, <label>, [example](workflow url)
- ...
**External outage** (<sum %>):
- N times, P%, <label>, [example](workflow url)
- ...
**<N> real issues (not flaky)** — code or config bugs, not bucketed individually.
Rules:
- Within each section, sort by count desc.
- Render the example as a clickable markdown link:
[example](https://app.circleci.com/...). Never paste the bare URL — the brackets must be present so it renders as a link.
- Use the most recent matching job's
URL value (from the file header) as the link target.
- Real issues are summarised on a single line with a total count — do not list each one.
- If you couldn't fit a job under any marker, add a bucket "unclassified (manual inspection needed)" inside Fixable flake.
- Cap the report at 10 flake buckets total across Fixable flake + External outage combined, picking the largest by count. If you had to drop any, add one trailing line:
- <N> other long-tail flake (<P>%) — smaller patterns rolled up.
End with a one-line bottom line: should the user retry, dig in, or wait for the outage to clear.
Notes
-
Don't reimplement the fetching logic; just run the script. It paginates pipelines, parallel-fetches workflows / jobs / step logs, strips ANSI, and caches raw log responses under .claude/cache/circleci-why-flaky-cache/ (shared across runs) so re-runs in the same session are fast.
-
The cache dir and per-run output dir both live under .claude/cache/, which is gitignored.
-
For private projects without a valid token, the script exits with setup instructions; relay them to the user:
brew install circleci
curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/main/install.sh | bash
circleci setup
