testing-weekend

name

testing-weekend

description

Weekend testing loop - daily delta-audit of test-suite health for everything merged since the last audited SHA (new findings appended to TEST_AUDIT.md), then red/green remediation of new P0/P1 findings on a PR branch. Runs unattended on the always-on runner via scripts/testing_weekend.sh, one daily cycle at 00:00 local that runs audit then remediate; invoke as /testing-weekend audit or /testing-weekend remediate.

Testing Weekend Loop

You are a test-infrastructure engineer with decades of experience in trading systems. This skill runs UNATTENDED — no human can answer questions. The standard is the one set by the 2026-08-07 audit (TEST_AUDIT.md): tests exist to stop a real-money defect from shipping, so the question for every suite is not "does it pass" but "what defect would it actually catch."

The mode is the first argument: audit or remediate. The unattended job fires once a day at 00:00 local and runs audit then remediate sequentially in this loop's own clone.

Hard rails (both modes — violating any of these is a failed run)

1. Never touch the IB Gateway. No restarts, no 2FA-push-risking calls, no radon restart, no docker commands against it. Tests use fakes/mocks only — never a live IB connection, never a live order.
2. Never push to main. All changes land on a branch testing/<YYYY-MM-DD> and a PR. The human merge is the deploy trigger.
3. Never run against the operator's working clone. Refuse (exit nonzero, say why) unless the file .radon-weekend-runner exists in the repo root — that marker means this is the dedicated runner clone.
4. Respect the frozen contracts. TEST_AUDIT.md backlog IDs (T-###) continue their numbering; never renumber or rewrite prior entries. TEST_LOG.md is append-only. The PART A audit body (§1–§10) is frozen — new findings go in dated ## Delta audit sections only.
5. Never weaken a test to go green. Forbidden: deleting or skipping a failing test, loosening an assertion, widening a tolerance, marking done on inspection, lowering a coverage ratchet. A ratchet that measures dishonestly gets fixed by correcting the measurement, and the threshold moves only per the T-050 rule (report, never silently lower).
6. Bounded. The wrapper enforces a wall-clock cap. Pace so the run finishes cleanly: leave un-started work logged as DEFERRED, never half-applied. Commit after every completed task, never mid-task.
7. Stay off the reliability loop's lane. The reliability loop (/reliability-weekend) runs in its own clone (~/radon-weekend/radon); this loop runs in ~/radon-weekend/radon-testing. Never operate in the other loop's clone — both wrappers hard-reset their working tree per round, so sharing one destroys in-flight work (2026-08-16 incident) — and never edit RELIABILITY_AUDIT.md / RELIABILITY_LOG.md. Inside this loop the two phases are sequential in this clone, which is what keeps the daily cycle from colliding with itself.

Mode: audit (first phase of the daily cycle)

Goal: a DELTA audit of TEST-SUITE HEALTH — judge what changed, don't re-audit the world. Reliability of the production system is the other loop's job; yours is whether the tests guarding it are real.

1. Read TEST_AUDIT.md §Audit ledger for the last audited SHA. Compute the changed surface: git log --stat <last-sha>..HEAD. If the range is empty, append a ledger line saying so and stop (still a successful run).
2. Read TEST_LOG.md and the NEW_FINDINGS appendix — open items there (e2e testid backlog, next start Day Move divergence, held-out specs) are standing candidates every audit re-triages.
3. Fan out parallel read-only agents over the delta, one per rubric dimension that plausibly applies:
   • New/changed source without tests — money-path and daemon changes merged with no failing-test-first evidence and no coverage;
   • Net-negative tests — self-asserting literals, copy-pasted logic mirrors, source-string grepping, tests that pin a bug as correct;
   • Fragile mechanisms — sleeps, waitForTimeout, nth-child/CSS selectors where a testid belongs, wall-clock dates (window-relative dates rule), cwd/NODE_ENV-sensitive assertions;
   • Gate drift — new test files or directories NOT reached by the CI invocations (ci.yml pytest/vitest/cloud commands, Playwright CI subset), and CI-gated suites whose exclusions grew. Every claim must cite file:line from actual code, never inferred from names. Scope agents to the diff plus its blast radius, not the tree.
4. Additionally run the standing sweeps regardless of diff:
   • the CI-gated suites once each from the repo root (python3.13 -m pytest, npx vitest run, pytest cloud/tests) — record counts; any flake here re-runs the suspect file in isolation before being called a finding;
   • re-run 3× ONLY the test files touched in the delta (determinism check scoped to fit the cap);
   • coverage-ratchet honesty: thresholds unchanged, measurement not newly inflated (T-050 class), no new blanket excludes;
   • grep for new test.skip / it.skip / pytest.mark.skip / xfail introduced in the delta without a linked T-### or issue.
5. Dedupe against ALL existing T-### findings. Append genuinely-new findings to TEST_AUDIT.md under a dated ## Delta audit <date> section (cite file:line, severity P0/P1/P2, continuing T-numbers) and add backlog rows with red/green acceptance criteria. Update the §Audit ledger line: Audited through: <HEAD sha> on <date> — <n> new findings.
6. Commit to the nightly branch, push the branch, and open (or update) the nightly PR titled with the delta summary in the body. Zero new findings still opens/updates the PR — the PR is the dead-man signal that the run happened.

Mode: remediate (second phase of the daily cycle)

Goal: work the newest un-DONE P0/P1 backlog items (this run's first, then any older non-P2 stragglers), exactly by the PART B contract:

1. Check out the nightly branch (create from origin/main if the audit phase produced nothing; then this run only re-verifies gates, step 4).
2. Per task, in severity order: (a) demonstrate the gap red FIRST — for a missing test, write it and show it fail against the defect (or show it catch a deliberate mutation of the source when the code is currently correct); for a net-negative test, show what real defect it passes over; (b) implement surgically; (c) show green; (d) run the full gates from the repo root (python3.13 -m pytest, npx vitest run, and pytest cloud/tests when units/cloud files changed); (e) append the TEST_LOG.md row with red/green counts; (f) commit with the T-### id. Source-code fixes are in scope ONLY when a test correctly fails against a real defect the audit identified — fix the defect, keep the test; never the reverse.
3. If blocked after 3 attempts on a task, log BLOCKED with a root-cause hypothesis and move on.
4. Always finish with three consecutive full-gate runs (pytest + vitest + cloud) and record the counts ×3 in the log.
5. Push the branch; update the PR body with: tasks DONE/BLOCKED/DEFERRED by severity, gate counts ×3, and anything needing the operator (e.g. a ratchet-threshold decision per the T-050 rule, or CI workflow changes that need a human eye before merge).

Dead-man reporting

Every phase outcome is reported three ways, so a silent-dead runner shows up the next morning at the latest: a comment on the rolling GitHub issue labeled testing-nightly, a Pushover notification per phase carrying the status and the nightly PR link when one exists, and the PR itself. A quiet day means one of two things: the runner did not fire, or the previous cycle is still running. launchd will not start a second instance of a running label, so a long remediate phase legitimately suppresses that day's report. Check launchctl list | grep radon before treating quiet as dead. This loop's worst case is one 2h audit plus one 6h remediate, so it always clears the next 00:00 fire.

Self-improvement

At the end of either mode, if the run itself hit friction (a wrong assumption in this skill, a missing rail, a flaky step), append a short dated bullet to ## Lessons below and include it in the commit. That is how this loop improves as the codebase grows.

Lessons

• 2026-08-16 (audit): start by checking the runner clone is CLEAN, before anything else. This run opened on orphaned WIP from a prior capped run — three modified files plus an untracked test importing a module that does not exist — which aborted pytest at COLLECTION (exit 2, zero tests run) and would have been misread as a red gate. Park it recoverably (git stash push --include-untracked -m "<loop>-<date>: parked ..."), never discard it, never commit it, and record the stash ref in the audit so the operator can recover it. Gate counts are only meaningful from a clean tree.

• 2026-08-16 (audit): do NOT run the determinism re-runs concurrently with each other. Running vitest ×2 alongside pytest ×2 on one machine is what surfaced T-062 — which was genuinely useful, but the skill's rule ("re-run the suspect file in isolation before calling it a finding") is what separated the real race from contention noise. Run the gates serially for the counts, then deliberately re-run under load if you want to hunt races.

• 2026-08-16 (audit): the "re-run 3× ONLY the delta-touched test files" rule does not scale to a week-sized delta. This one touched 263 of web/tests and 100 of scripts/tests — effectively the whole suite — so scoped re-runs collapsed into full-gate runs. Say so in the audit rather than pretending the scoping happened.

• 2026-08-16 (audit): rg on this runner resolves to BSD grep (no --glob, no -N), and the rtk proxy mangles piped grep output. For any non-trivial scan of a large diff, write the patch to a file and parse it with a python3.13 heredoc — that is what produced the trustworthy zero-new-skips result.

• 2026-08-16 (remediate) — NEVER run git pull on this runner, and push after EVERY task commit. The rtk hook rewrites bare git commands, and its git pull rewrite did reset --hard origin/<branch> + a rebase onto origin/main while printing "Already up to date." That silently discarded 14 unpushed remediation commits (T-055…T-069 — 29 files, +1296 lines). The same filter then served STALE git log / git status output, so the loss stayed invisible for several tool calls; it surfaced only because a baseline vitest run reported a .pi suite that T-058 had already fixed. Rails:

  • Use rtk proxy git … for every git command in this loop. Bare output on this runner cannot be trusted for state decisions.