ci-failure-catalog

name	ci-failure-catalog
description	Run the MOMENT a CI check goes red in this repo, and BEFORE any push that could reintroduce a known failure. Fires on: a failed `test-shard`, `fast`, `ci-complete`, `guard`, `validate-*`, `codeql`, `secret-scan`, or `deps-audit` check; "CI error", "CI failed", "the build is red", "fix the failing check", "why did CI break", "/ci-failure-catalog". This is the LIVING catalog of every CI failure this repo has hit + its root cause + the exact fix — a growing playbook, not a static doc. It ROUTES to the specialised skills (ci-artifact-drift, ci-script-test-bootstrap, git-discipline §3/§3a, spark-selfhosted-runner) and catalogs everything else. **Every session that hits a CI failure not already listed MUST append a row after fixing it** (the append protocol below) — that is what keeps this skill useful.
metadata	{"short-description":"Living catalog of this repo's CI failures → root cause → fix; self-updating (append a row on every new one)"}

CI failure catalog (this repo) — a self-updating playbook

CI is red often here not because the code is wrong but because this repo has a dense mesh of generated artifacts, git-crypt, gitignored-but-tracked paths, script-mode tests, and locked constant sets — each with a non-obvious failure mode. This skill is the single place that maps a CI failure signature → root cause → exact fix, so the second time anyone (any Claude/Copilot/GLM /human session) hits a failure, they fix it in one step instead of re-deriving it.

It is modelled on the superpowers pattern: the catalog grows every time a new issue appears. A CI failure you fix but don't record is a failure the next session re-debugs from scratch — the #1 recurring cost in this multi-agent repo.

The append protocol (this is the self-updating mechanism — do not skip it)

When CI fails and the signature is not already in the catalog below:

1. Fix it (diagnose → repair → verify the check passes on the new head).
2. Append a row to the catalog table in this file, in the same PR as the fix (or a follow-up if the fix was elsewhere): the check name, the distinctive error string, the root cause, the exact fix, and the PR#/date. Keep the error string to the shortest span that uniquely identifies it.
3. If the failure also merits a full procedure (multi-step, its own triggers), write/extend a dedicated skill and link it from the Routes row instead of inlining a long fix here — this catalog stays a fast index; deep playbooks live in their own skills (skill-author).
4. This file is a plaintext process skill (no IP): editing it is tooling/self-merge scope. It has a .gitattributes !filter !diff exception, so any .gitattributes edit means python tools/cbm/derive_cbmignore.py in the same PR (see catalog row guard/.cbmignore), and a new/changed skill means a keyless-clone skill-index regen (see row validate-core/skill-index).

Trigger discipline: the catalog is only as good as its signatures. Record the actual string CI printed (e.g. assert 5 == 4, .cbmignore is stale), not a paraphrase — the next session greps for it.

Routes — failures that already have a dedicated skill (go there first)

Signature / check	Go to
skill-index STALE, validate-core on skills/registry/index.json, generated-artifact drift (RESULTS/manifest/wiki/ledger), make claim-check red, any .gitattributes edit → .cbmignore regen	ci-artifact-drift
ModuleNotFoundError for okf/agent/serving/… in a python tests/test_*.py run; a test that passes under pytest but fails in CI	ci-script-test-bootstrap
A check shown "failed" that is actually cancelled (superseded head); "CI seems stuck"/green-but-won't-merge; mergeable_state decode	git-discipline §3 / §3a
GPU-lane CI: self-hosted runner offline, python: command not found, PEP 668, SGLang/vLLM provisioning, one-dep-per-run whack-a-mole	spark-selfhosted-runner
An overclaim tripping lint_claims / claim_gate at the PR end-of-line	no-overclaim-merge-gate

Catalog — signature → cause → fix (append here; newest at top)

Check	Error string (grep-able)	Root cause	Fix	PR/date
Every workflow, on every push — a run with 0 jobs, conclusion: failure, event: push, and NO log (a startup failure prints nothing). Presents as ~50 spurious red rows/day; scale-120b-sequential was 145 runs / 145 failures / 0 successes.	Nothing in the run itself. actionlint names it: context "github" is not allowed here. no context is available here. [expression] pointing at on.workflow_dispatch.inputs.<id>.default	default: '${{ github.ref_name }}'. workflow_dispatch input defaults are one of the few places where NO context is available. It does not evaluate to the ref — it makes the whole FILE fail validation, so GitHub manufactures a zero-job startup-failure run for every push to any branch. Costs no runner (jobs=0) but poisons "was main green?", the first question asked when triaging an unrelated red PR. The repo's actionlint gate is diff-scoped and advisory by design (≈96 workflows carry pre-existing debt), so it never re-lints a file after the error lands — this can sit for days.	default: '', then fall back at the consumption site, where the context IS legal: IN_BRANCH: ${{ github.event.inputs.branch || github.ref_name }}. Look for a templated copy — tools/scale_120b_sequential.py --emit-workflow held the same string, so fixing only the YAML regresses on the next emit. In an f-string template a GitHub expression needs QUADRUPLED braces (${{{{ }}}} → ${{ }}); plain ${{ }} emits ${ }, which shellcheck flags as SC1072: Expected } to end the ksh-style ${ ..; } command expansion. Verify with actionlint <file> AND --emit-workflow | diff <committed>.	#1571 / 2026-07-26
secret-scan (gitleaks) — frequently on a PR that did not introduce it: the finding commit is already on main, because push-mode scans only newly-pushed commits while the PR merge-ref scan sees the whole range	RuleID: generic-api-key with Secret: Shift+ArrowUp/ArrowDown, File: apps/sophia-tui/src/components/PromptInput.test.ts	The upstream generic-api-key entropy rule read a test TITLE — test("isHistoryNavKey: Shift+ArrowUp/ArrowDown are NOT history-nav keys …") — as a credential assignment: the trailing …Key: looks like the key, the keyboard-shortcut string (entropy 3.675) like the value. No secret exists. The existing (^|/)tests/ allowlist never reached it because apps/sophia-tui co-locates tests as Foo.test.ts beside Foo.ts.	Add '''\.test\.(ts|tsx)$''' to .gitleaks.toml [allowlist] paths. Prefer this to a .gitleaksignore fingerprint — fingerprints are commit- AND line-pinned and go stale on the next edit of a live source file. Verify scope, do not assume: full-tree gitleaks dir . --config .gitleaks.toml before/after and diff the finding sets (was 4 → 3; only the target removed, nothing new suppressed). Security envelope holds because the separate repo-hygiene job greps sk-/hf_/AKIA over these paths — it excludes tests/**, NOT *.test.ts — confirm with a canary before claiming that.	#1567 / 2026-07-26
test-shard (1) + ci-complete — and it reddens every OTHER open PR too, because it is main's own breakage (check main before debugging your diff)	SystemExit: FAIL-CLOSED: tasks dir /…/external-data/arc-agi-2/data/evaluation not found. Place ARC-AGI-2 task JSON there (this tool does not vendor the dataset). and assert 0 == 10	Tests point at data that is deliberately never vendored (arc_calibration_experiment.py fails closed by design; its --split help says "public data only"). The test file's own docstring says it runs against task files "already on disk" — true on the authoring dev box, never in CI.	pytest.mark.skipif predicated on the dir existing AND containing *.json. Check the other direction too: one test (…_is_deterministic_given_same_seed) compared two EMPTY reports and passed VACUOUSLY — green in CI while asserting nothing, which is worse than the loud failures beside it. Verify BOTH states: data absent → skipped; data present → the tests EXECUTE (drop a probe *.json in to prove it), so the guard is conditional and not a permanent disable. Never weaken an assertion to reach green.	#1572 / 2026-07-26
Any workflow — a startup_failure run with 0 jobs on push, named by the file path (.github/workflows/<name>.yml)	Unrecognized named-value: 'runner' (compile-time; the run has no jobs so no logs; PyYAML parses the file fine)	A job-level env: value referenced a step-scoped context, e.g. RUN_ROOT: ${{ runner.temp }}/.... runner/steps/job/jobs are STEP-scoped only; using them in jobs.<job>.env makes GitHub reject the whole workflow at compile time → red ✗ on the commit on every push (even for a workflow_dispatch-only lane). Ordinary YAML validation never catches it.	Move the value into a step (where $RUNNER_TEMP / the runner context is valid) and export via $GITHUB_ENV for later steps — mirror how DOWNLOAD_ROOT is handled. Guarded repo-wide by tools/lint_workflow_contexts.py in fast-ci (rejects step-scoped contexts in job-level env; ignores step env, with:, environment.url, cache keys).	#1105/#1108 / 2026-07-14
deps-audit (pip-audit BLOCKING step)	PYSEC-2026-3447 (setuptools); and if you try to pin the fix → ResolutionImpossible: Cannot install -r requirements-rl.txt ... and setuptools>=83.0.0 because these package versions have conflicting dependencies	A new advisory dropped on a transitive dep whose fixed version can't be installed — another dep (vllm) caps setuptools below the patched version. Remediation-by-bump is infeasible; the pin makes pip unresolvable.	Add the advisory ID to the security.yml pip-audit ignore-baseline (IGNORE for-loop) with a written justification in the workflow comment (the baseline requires one); do NOT pin a version pip can't resolve. Remove the ID the moment the capping dep lifts its ceiling.	#1103 / 2026-07-14
deploy-hf-space (Deploy bundle to Hugging Face Space step)	httpx.HTTPStatusError: Client error '402 Payment Required' for url '.../api/repos/create' … "hosting Gradio and Docker Spaces on free cpu-basic requires a PRO subscription"	Account tier can't host the Space type — a billing state, not a code bug; no repo edit makes a 402 green. Build+smoke-test pass; only the publish 402s, reding main every push.	Gate the publish step behind repo variable HF_DEPLOY_ENABLED (default unset → step skipped → job green); keep build+smoke-test running every push. Flip HF_DEPLOY_ENABLED=true once the account can host. It's an owner billing/degrade/disable choice — ASK, don't silently degrade a deploy.	#1106 / 2026-07-14
validate-core (build_current_pointer.py --check step)	CURRENT pointer STALE: data/knowledge_graph/CURRENT.json differs from a fresh rebuild on the PR merge ref, while the branch's own pre-push check passed	A branch realign (git merge origin/main) union-merged a concurrent session's ledger/handover change UNDER the committed pointer — the pointer derives from the ledger, so the MERGE invalidated it; "my commit didn't touch pointer inputs" reasoning misses that the merge did. Watch for Auto-merging agi-proof/failure-ledger.md in the realign output.	After ANY realign/merge that touches agi-proof/failure-ledger.md or SESSION-HANDOVER-*, run python tools/build_current_pointer.py && python tools/generate_obsidian_views.py, commit the regenerated projections, and only then push; both --check forms are the pre-push gate.	#1072 / 2026-07-13
rlvr-runpod "Ingest the FRESH pod adapter-eval" step	selected by run identity -> <path> naming a file whose pod id is NOT this run's pod, and gate numbers that don't match the fresh receipt (retention-anchored seed 1: gate read after=0.4833 from cap-384's committed 3yb4xfcywzwrbv...seed1.json; fresh pod file said 0.5167)	(task, audit.effectiveSeed) is NOT run-unique: a committed archive from a PREVIOUS sweep of the same task+seed passes every identity filter, sorts first alphabetically, and first-match-wins gated it — the #658-era stale-pickup guard only defended against task/audit=None archives. Receipts also carried no podId/timestamp to disambiguate.	Three layers: select_report now REFUSES >1 identity match (fail closed, "matched MORE THAN ONE report"); the workflow step feeds only git-UNTRACKED files (git ls-files --error-unmatch skip) so committed archives are never candidates; _split_audit stamps audit.podId/runId/generatedAtUtc from env so future receipts self-identify. Regression tests in tests/test_ingest_rlvr_eval.py. When landing receipts, do NOT commit per-pod copies into runpod-rlvr/ (use rlvr-replication/<arm>/).	#870 / 2026-07-11
spark-smoke tier-a-invariants (CUDA stack probe step)	ModuleNotFoundError: No module named 'torch' in the probe step, every earlier step green	The probe ran on actions/setup-python's hosted toolchain, which has no torch (only requirements-math.txt is installed). Masked since the lane was written because its stale runs-on labels (dgx-spark, arm64) meant no runner EVER picked it up — the #858 label fix exposed it on first pickup. Blind-fix trap: pip install -r requirements-rl.txt pulls PyPI aarch64 torch = CPU-only (the Spark's working stacks are +cu130), which would "fix" the import and then fail the CUDA assert against a wrong stack.	Probe the CUDA venv the Spark lanes actually use — $GITHUB_WORKSPACE/.venv-spark-rlvr (spark-gpu live runs provision it), else the box's shared-checkout venv — and fail closed with a provisioning pointer when neither exists. Never pip-install torch from PyPI on the Spark; use the cu13x recipe (spark-selfhosted-runner skill).	#859 / 2026-07-10
workflow_dispatch (422 at dispatch — no run is created at all)	HTTP 422: Provided value 'off' for input '<name>' not in the list of allowed values (sibling of #854's Workflow does not have 'workflow_dispatch' trigger)	YAML-1.1 boolean coercion: an unquoted off/on/yes/no in a choice options: list parses as a BOOLEAN, so GitHub's allowed values become 'false'/'true' while a quoted default like 'off' is not in its own options list.	Quote boolean-like option values (options: ['off', 'run']). Guarded repo-wide by tests/test_workflows_parse.py::test_no_choice_option_is_yaml_boolean (flow + block list styles).	#859 / 2026-07-10
validate-core (script-mode test step)	FAIL test_verifiers_match_paid_harness_rules: AssertionError: (bare — no ImportError anywhere) while the SAME test passes in all 4 test-shards and locally on the Mac	The test exercises a verifier whose oracle dep is optional (sympy: "optional for the math verifier" in pyproject) and which fails CLOSED without it — so in validate-core's bare-python step (no pip install) the verifier returns False and the assert reads the fail-closed abstain as a mismatch. NOT the ci-script-test-bootstrap ModuleNotFoundError case: imports succeed, verdicts flip. The shards install -e ".[dev]" and genuinely validate the behavior.	In the test, probe the oracle dep (try: import sympy) and SKIP with a printed reason when absent (fail-closed abstain is by design); keep the real validation in the pytest shards. Do NOT add per-test pip installs to validate-core, and do NOT weaken the asserts. Verify BOTH paths before push: bare python → SKIP + exit 0; venv with the dep → full PASS.	#837 / 2026-07-10
GPU-lane job (Actions)	No log string — a timeout leaves none. The signal is API/UI metadata: run conclusion == cancelled AND run duration ≈ the job's timeout-minutes AND an empty/0-byte receipt. (This row is the exception to "record the literal string" — a timeout-minutes kill prints nothing greppable, so route by the metadata pattern.)	A judge-API-bound or generation-bound measurement exceeded a fixed Actions timeout-minutes (W4 entailment panel: ~60× the lexical path; independence generate: ~1h/family-seed). GitHub reports a timeout-minutes kill as cancelled, easily misread as a supersede.	Raise timeout-minutes to fit the slow path; add mid-measurement checkpoints so a hang is distinguishable from slow progress; consider moving API-bound work off the GPU job. Distinguish from a REAL supersede: a supersede HAS a newer run in the concurrency group; a timeout has NO newer run and duration ≈ the ceiling.	#699 / 2026-07-06
lane "Push … to results branch" step	! [rejected] HEAD -> <feature-branch> (fetch first) … remote contains work that you do not have locally	A long-running lane (qualify ~1h+) pushed its receipt back to the mutable dispatch branch, which was pushed-to / merged / deleted underneath it → non-fast-forward, receipt lost (survived only in the artifact).	Push to a run-scoped results branch <lane>-results-<run_id> (never the dispatch ref) — the #658 pattern. Note a re-run reuses the same run_id (only run_attempt bumps), so the branch may already exist: base the local branch on origin/<branch> before committing (fetch + checkout -B <b> origin/<b> if it exists) so the receipt APPENDS fast-forward; prefer re-dispatch over re-run.	#697 / 2026-07-06
test-shard	AssertionError: assert 5 == 4 / test_candidate_pairings_are_exact_predeclared_set "Extra items … mistral-small+qwen-2.5-72b"	A test locks a pre-declared set/count (CANDIDATE_PAIRINGS); a PR that legitimately grows the set didn't update the test in the same PR.	Update the count and the exact-set assertion in the same PR that changes the set; better, assert the intent (e.g. "≥1 clean pairing exists") so the test protects the property, not the arity.	#689 / 2026-07-06
guard	.cbmignore is stale vs .gitattributes. Run: python tools/cbm/derive_cbmignore.py	.cbmignore is DERIVED from .gitattributes; any filter/exception/LFS edit to .gitattributes makes it stale.	python tools/cbm/derive_cbmignore.py and commit it in the SAME PR as the .gitattributes change.	#687 / 2026-07-06
results-branch push step	The following paths are ignored by one of your .gitignore files: training/worldmodel → empty commit → push exit 1	training/worldmodel/, .claude/skills/, skills/portable/ are gitignored but hold tracked/delivered files; a plain git add <gitignored-path> is silently refused.	Use git add -f; make delivery steps fail loud on an empty commit (git diff --cached --quiet && { echo '::error::nothing staged'; exit 1; }).	#692 / 2026-07-06
validate-core / build_skill_index.py --check	skill-index STALE locally while CI is green, OR an encrypted skill's description LEAKED as plaintext in skills/registry/index.json	Regenerating the index on an unlocked checkout bakes decrypted descriptions in / drifts from the keyless CI view. CI runs keyless (ci.yml does not unlock git-crypt).	Regenerate the index from a keyless clone (git clone file://… ; build_skill_index.py) so encrypted skills mask as (encrypted — unlock to index); copy that index back. See ci-artifact-drift.	multiple / 2026-07-06
ci-complete	failure while every required check is green	ci-complete needs: a job that was cancelled (superseded run when main moved mid-run), OR a real sub-check failed.	Read the actual failed leaf job's conclusion string: cancelled → re-trigger/rebase (not a real failure); failure → fix the leaf. git-discipline §3.	git-discipline §3
any step that loads or downloads an HF model (rlvr-smt-smoke-pro6000, local train/eval)	OAuth token signature verification failed → RepositoryNotFoundError: 401 Client Error on a PUBLIC model (Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen2.5-32B-Instruct)	The box's persistent HF cache holds a token whose signature no longer verifies, and huggingface_hub sends it implicitly — so even public repos 401. Setting HF_TOKEN= does NOT stop it: the stored $HF_HOME/token file is still read.	CI: move it aside for the job — mv "$HF_HOME/token" "$HF_HOME/token.stale.$(date +%s)" + HF_TOKEN=. Ad-hoc/local: pass token=False with HF_HUB_DISABLE_IMPLICIT_TOKEN=1 (public repo), or go fully offline (HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1) for an already-cached model. Never delete the owner's token file.	#1241 / 2026-07-18
fast (P7 artifact-drift) and test-shard (RAG) — on a docs/data-only PR	CURRENT pointer STALE: /…/data/knowledge_graph/CURRENT.json differs from a fresh rebuild and/or FAIL: chunk ids changed vs committed index + chunkIds: <a> vs <b> (tests/test_rag_local_embed.py::test_committed_index_manifest_is_reproducible)	Two generated artifacts drift from content-only edits. build_current_pointer.py hashes agi-proof/failure-ledger.md, so a ledger row alone invalidates the pointer; and agent/rag_sources.collect_curated_chunks globs DATA_DIR/*.json, so a new data/<x>.provenance.json sidecar silently joins the RAG corpus and drifts rag/index/*. "My PR touches no code" is not a defence. Resetting the pointer to main makes it worse — it must be rebuilt against the branch's own content.	Before push: python tools/build_current_pointer.py && python tools/generate_obsidian_views.py (pointer FIRST, index SECOND) and python tools/build_rag_index.py --local && python tools/build_rag_index.py --verify; commit rag/index/{chunks.jsonl,embeddings.npz,embeddings.meta.json}.	#1220 / 2026-07-17

Before you push — 30-second self-scan

• Added a file under data/*.json (incl. a *.provenance.json sidecar) or appended a failure-ledger row? → build_current_pointer.py + generate_obsidian_views.py, and build_rag_index.py --local && --verify (a docs/data-only PR still reddens fast + test-shard).

• Touched .gitattributes? → derive_cbmignore.py (guard).

• Added/edited a skill? → keyless skill-index regen; new skill also needs the .gitattributes !filter !diff exception (if plaintext) + this catalog knows the drill.

• Changed a locked set/constant a test asserts (candidate pairings, verdict vocab, floors)? → grep tests/ for the value and update the test in the same PR.

• New tests/test_*.py importing repo packages? → ci-script-test-bootstrap (sys.path).

• Delivering a file under a gitignored dir from a lane? → git add -f + empty-commit guard.

• Run python tools/lint_claims.py and make claim-check locally — they are cheap and gate the PR.

Guardrail

This skill is CI/process discipline only. It makes no capability or AGI claim; canClaimAGI stays false. It never weakens a gate to make CI pass — a red gate that reflects a real overclaim, a moved threshold, or contaminated data is fixed by fixing the code, never by editing the gate.