| name | investigate-ci-failure |
| model | sonnet |
| description | Classify a pull request's failing CI checks — a mechanical gate slip, a real code defect, a known infrastructure flake, or something unclassified — with the evidence behind each verdict. Investigates and reports; never fixes, pushes, or re-runs. |
Investigate a CI failure
Turn a red pull request into a classified, evidenced report: for each failing check, what kind of
failure it is and what says so. The caller decides what to do about it.
pr-followup step 2 is the wired caller. That step's own procedure —
read --log-failed, find the cause, fix it — suits a failure whose reason is printed in the log,
which is most ci.yml failures and few E2E ones. The on-device lanes fail with nothing in the log:
the layered diagnostics that explain them
(BE-0361,
BE-0367)
are uploaded as artifacts under runs/diagnostics/ and never appear in gh run view --log-failed
output at all. Diagnosing from the log alone there means guessing, in both directions: a host flake
read as a regression sends a pointless fix through a full CI cycle, and a real regression waved off
as "the usual flake" ships. This skill reads the artifacts instead.
What this skill does not do
- It does not change anything. No edit to the code under test, no commit, no push, no
gh run rerun. It classifies and reports; the caller acts. The single file it may write is its
own references/known-ci-failure-patterns.md (step 5),
and even then it does not commit that change.
- It does not treat
devicefarm.yml or ai-smoke.yml. Both are workflow_dispatch only, so
neither appears in a pull request's checks and neither can be the failure being investigated.
- It does not decide a verdict. Reading recorded failures to sort them is not judging a run.
Pass/fail stays where prime directive 1 puts it, in the deterministic runner.
Input
A pull request number, from pr-followup step 2.
Steps
1. Sort the failing checks by which workflow they belong to
gh pr checks <PR>
For each failing check, get its jobs:
gh run view <run-id> --json jobs --jq '.jobs[] | select(.conclusion == "failure") | .name'
Sort each failing job into one of three routes:
- the
check job of ci.yml → step 2
- a job of
ios-e2e.yml or android-e2e.yml → step 3
- a job of
web-e2e.yml → the log route, not step 3. That workflow uploads no runs/ artifact and
has no diagnostic layer, so there is nothing for step 3 to download and no web table in the
pattern reference. Read --log-failed and classify as code-defect — a Playwright traceback is
in the log — or gate-mechanical on a match.
- anything else (
pr-title, roadmap-id, mcp-wire, swift, serve-db, codeql, …) → treat as
code-defect and report the failing step with its log excerpt. These fail for a reason the log
states.
2. The gate job: mechanical, or a real defect
gh run view <run-id> --log-failed
Read which gate step failed and match it against the gate-mechanical table in
references/known-ci-failure-patterns.md. Three steps
are mechanical and have a single fix command each; every other step points at the change itself.
- A match →
gate-mechanical, and carry that entry's fix command into the report verbatim.
- No match →
code-defect, with the failing step name and the relevant log lines.
The reference file's lint-skills note matters here: that drift has two causes with different
fixes, and the reference says how to tell them apart. Do not shortcut it.
3. The E2E lanes: match against known infrastructure faults
Download the failing job's own artifact. Its path: runs/ carries runs/diagnostics/ inside it, so
one download brings all three diagnostic layers.
Artifact names follow no single pattern — list them rather than guessing. The Android lane is
uniform (android-e2e-<job>-run), but the iOS lane is not: the jobs that upload through the
bajutsu-e2e composite action pass a bare name, so run uploads as run, actuation as
actuation-run, golden as ios-golden-run, and bundled-runner as two names of its own. Only
conformance, fault-injection, visual, and network match ios-e2e-<job>-run. Guessing the
pattern fails the download on run, the lane's primary job.
gh api "repos/{owner}/{repo}/actions/runs/<run-id>/artifacts" --jq '.artifacts[].name'
gh run download <run-id> -n <the failing job's own name from that list>
Match what you find against the e2e-known-flake tables in
references/known-ci-failure-patterns.md — eight
confirmed patterns across the two on-device lanes, each with the artifact path that confirms it.
Read that file now rather than guessing from the job log.
A match → e2e-known-flake, citing the artifact path that confirmed it. No match → step 4.
Two cases the reference file settles rather than this step: a visual or golden job is a
non-gating signal whose red means re-record or investigate upstream, not re-run; and a probe whose
deadline was cut off in probe.txt is not a probe that found nothing.
4. No match: ask the run history instead
Reaching here means the symptom matches nothing known — either genuinely new, or a known fault
wearing an unfamiliar face. Rather than guess, ask whether this scenario has been flipping all
along.
Assemble a history from the same job's recent runs and hand it to
investigate-scenario-flakiness:
gh run list --workflow <workflow> -L 40 \
--json databaseId,conclusion,headBranch,createdAt \
--jq '.[] | select(.headBranch == "main" or (.headBranch | startswith("gh-readonly-queue/"))) | .databaseId'
Restrict to runs of main and the merge queue: a run of somebody's branch carries that branch's own
defects, which is noise against the question being asked. Keep the merge-queue runs specifically —
these lanes trigger on merge_group, and a queue run's headBranch is
gh-readonly-queue/main/pr-<N>-<sha>, so a bare == "main" test drops a large share of exactly the
clean history this step needs. Note that -L bounds the runs listed, not the runs kept, so it is
set above the twenty-artifact budget to leave the filter something to select from.
Download each run's artifact for the same job into one directory — resolving each run's own artifact
name the way step 3 does, since the name varies by job — then invoke the sub-skill on it.
Pass no use_ai. What this step needs is the classification field; the AI triage pass costs
ANTHROPIC_API_KEY credit and answers a question — why it flips — that this classification does
not ask. The sub-skill's default already omits it.
- The sub-skill reports
flaky → e2e-known-flake, on the strength of the history rather than a
matched pattern. Say which, in the report: a data-backed verdict and a pattern-matched one are
different kinds of evidence.
- Anything else →
e2e-unclassified.
Read that verdict as being about the run, not a named scenario. These jobs run whole suites, and
the sub-skill's ranking groups by content fingerprint and scores the run-level verdict — its name
field is only the first scenario in the run's summary. "This job's verdict flips at a constant
fingerprint" is the claim that supports e2e-known-flake, and it is enough for the classification
this step makes; "scenario X is flaky" is not what was shown, so do not put it in the report.
This step downloads up to twenty artifacts, which is why it is the last resort and not the first
move. When the sub-skill's own history is too thin to classify (unproven), report
e2e-unclassified and say the history was insufficient — not that the scenario is stable.
5. Record a newly confirmed pattern
When step 4 produced an e2e-known-flake from the history, append it to the e2e-known-flake table:
the symptom, the artifact path that shows it, and that it was confirmed from run history rather than
a prior diagnosis. The next investigation then matches it at step 3 and skips twenty downloads.
Edit the source copy, then redeploy — and stop there. The file to append to is
.apm/skills/investigate-ci-failure/references/known-ci-failure-patterns.md, named from the
repository root, never the deployed .claude/skills/… copy that a running agent's own relative
paths resolve to. CLAUDE.md forbids hand-editing the deployed tree, and the next make skills
overwrites it — discarding a pattern that cost twenty artifact downloads to establish. After the
edit run make skills, or the two trees disagree and make lint-skills fails at the caller's next
make check, manufacturing exactly the gate-mechanical drift this skill catalogs.
Report both the edit and the regenerated deployment, then stop. Do not commit and do not push: let
the caller fold both into whatever commit it is already making, so a skill never drops an unrelated
change into somebody else's pull request diff.
6. Report
One report, one entry per failing check, each carrying four fields:
| Field | Content |
|---|
| Check | The failing check and job name |
| Classification | gate-mechanical / code-defect / e2e-known-flake / e2e-unclassified |
| Evidence | The log lines, the artifact path, or the sub-skill's scenario name and flip_rate |
| Recommended action | See below |
Recommended action by classification:
gate-mechanical — the fix command from the reference table.code-defect — proceed with the normal fix flow.e2e-known-flake — a re-run is likely to be enough; no code change is indicated.e2e-unclassified — needs a person. Say what was ruled out, so the human starts where this skill
stopped rather than from the beginning.
Where a classification rests on something not checked, say so. "The artifact carried no stall
capture" and "no artifact was downloaded" support very different next moves, and a report that
blurs them sends the caller down the wrong one.
