| name | ci-triage |
| description | Diagnose and fix red CI on zaxy — fetch the right logs, reproduce locally despite the known local quirks (test_doctor hang, benchmark tables), do coverage-ratchet math correctly, and apply the test-vs-source intent rule. Use when CI is red, a check fails on a PR, "make master green", or coverage ratchet fails. |
CI triage for zaxy
CI jobs: lint (ruff+mypy), test (3.11/3.12/3.13) (pytest + coverage ratchet on
3.13 only), integration (Neo4j docker), web (astro build + link gate),
package. The authority is CI, not local runs — this machine has known quirks.
1. Identify what actually failed (per-SHA, not "latest")
gh run list --branch <branch> --workflow CI --limit 3 --json databaseId,headSha,conclusion,status
git rev-parse HEAD
gh run view <RUN_ID> --json jobs -q '.jobs[] | "\(.conclusion)\t\(.name)"'
Never pipe gh pr checks through awk/cut column hacks to decide a merge.
Job names contain spaces (test (3.13)), so column extraction shows name
tokens where you expect statuses — a red matrix once read as green this way
and got merged (the 2026-07-06 doctor-map incident). Before any merge, read
the FULL untruncated table and require every row to literally say pass:
gh pr checks <N> with no pipeline.
Beware fail-fast collateral: matrix jobs showing cancelled were killed by a
sibling's failure — they are not independent failures. Find the job that says
failure, fix it, and the cancelled ones usually follow.
2. Fetch failure logs (the incantation that works)
gh run view --log-failed is unreliable for grep. Download the job log raw:
jid=$(gh run view <RUN_ID> --json jobs -q '.jobs[] | select(.name=="test (3.13)") | .databaseId')
gh api "repos/<owner>/<repo>/actions/jobs/$jid/logs" > /tmp/job.txt
grep -aE 'FAILED tests/|short test summary|[0-9]+ failed|Coverage ratchet' /tmp/job.txt | tail -30
(-a: logs contain binary escape bytes. If the tail only shows "operation was
canceled", it's collateral — go find the real failing job.)
3. Reproduce locally — with the local quirks
ruff check src tests zaxy_benchmarks
mypy src zaxy_benchmarks
pytest -m "not integration" --no-cov -q -p no:randomly \
--ignore=tests/test_doctor.py --benchmark-disable
tests/test_doctor.py hangs/segfaults locally (ladybug native lib); it is
fine in CI. Always exclude it locally; never "fix" it locally.--benchmark-disable or the output drowns in perf tables.- A native segfault mid-suite is an environment/store problem, not the code — see
the
store-doctor skill before chasing ghosts.
4. Classify and fix
Lint/mypy — mechanical; fix and move on. mypy is --strict: annotate, don't ignore.
Test failures — apply the intent rule before touching anything:
- Read the failing assertion and the code it exercises.
- Find documented intent: CHANGELOG entry, spec in
docs/superpowers/specs/,
commit body of the change that broke it.
- Intent shipped deliberately (e.g. command re-grouped, contamination removed)
⇒ update the test to assert the new intended behavior.
- Source behavior changed accidentally (e.g. a hardening dropped
Retry-After
handling) ⇒ fix the source; the test was right.
- Ambiguous ⇒ stop and ask, presenting both readings with evidence.
Repo-specific test traps:
- Never assert dataset gold strings (benchmark contamination); assert content from
the test's own cited fixture.
CliRunner mixes stderr into stdout — deprecated-alias deprecation notices
corrupt --json assertions; invoke the canonical grouped command.- Changed an MCP tool? Regenerate
docs/examples/mcp-tool-contract.json and the
frozen tool-count assertion or the snapshot guard fails.
Coverage ratchet (Coverage ratchet failed: observed X% is below floor 92.00%):
- Only
src/zaxy counts ([tool.coverage.run] source). Tests and
zaxy_benchmarks cannot move the number. Ignore the dead
--cov-fail-under=90 in pytest addopts; 92.00 ([tool.zaxy.coverage]) is real.
- The floor is razor-thin. Order of operations:
coverage report --show-missing | grep <changed files> — find YOUR uncovered lines.- Delete dead code first (shrinks the denominator honestly) — verify it's
truly unreferenced before deleting.
- Cover the remaining new branches with real behavioral tests.
- NEVER lower the floor; never add
pragma: no cover to dodge it.
- Local totals understate CI (test_doctor excluded locally covers doctor.py in CI);
if local ≥ 92.0 you're safe, if 91.9x it's a judgment call — push and let CI measure.
Integration — needs Docker + Neo4j; usually env, check container logs before code.
Web — cd web && npm run build; it link-gates, so a moved/renamed doc breaks it.
5. Verify + ship
ruff check src tests zaxy_benchmarks && mypy src zaxy_benchmarks
pytest -q -p no:randomly --no-cov <the failing tests> && pytest -q -p no:randomly --no-cov <their whole files>
Commit with a body that names root cause per failure (this repo's style), push,
gh pr checks <N> --watch --interval 30. Green means the run for YOUR sha — recheck headSha.
