| name | incident-response |
| description | The exact Radon triage playbook for a production incident - reproduce locally, write a FAILING regression test first, fix, run the full test gate, commit a focused single-purpose commit, push once, wait for CI green, verify live in the browser. Use whenever an incident JSON from data/incidents/ or a production bug report is being worked; /incident orchestrates the parallel-analysis phase on top of this playbook. |
Incident response — the Radon playbook
The sequence is fixed. Skipping a step is how the 2026-07-08 outage happened
(rapid pushes) and how the destroy storm shipped green (regression test that
could not express the failing topology). Every step names its verification
evidence; paste that evidence before claiming the step done.
0. Read the incident + runbook case first
- Parse the incident JSON (
data/incidents/incident-*.json): case_id,
severity, evidence, fingerprint.
- Read the matching case in
docs/incident-runbook.md. If the case documents a
discriminating check (e.g. the Python Turso canary for turso-destroy-storm),
RUN IT before touching code — the same symptom has more than one cause, and
the wrong branch wastes the whole response (restart-flapping an upstream
Turso outage makes it worse).
- Confirm blast radius from live evidence, not assumption:
curl -s localhost:3000/api/service-health | jq (judge the BODY),
curl -s localhost:8321/health/lite, gh run list --workflow=ci.yml --limit 3.
- If this is a NEW failure mode, you will add a runbook case in step 6.
1. Reproduce locally
- Reproduce the failure on your machine before writing the fix. For data bugs,
reproduce against the artifact the user/production actually sees, never a
synthetic fixture (
feedback lesson: confirm against the user-visible
artifact). Verify against live Turso, not data/*.json fallbacks.
- Confirm
pwd is the repo root before any test run — cwd drift produces bogus
failures.
- If the failure cannot be reproduced locally (VPS-topology-only), reproduce the
MECHANISM in a test harness (the destroy-storm fix reproduced
agent.destroy() mid-flight aborting 10/10 concurrent requests).
2. Write the FAILING regression test FIRST (red)
- Vitest for web/lib, pytest for scripts/api. The test must fail for the same
REASON production failed, not merely fail. Encode the failing topology: the
destroy storm shipped because
db-timeout-self-heal.test.ts was single-caller
while the incident was concurrent-caller.
- Derive expected values, never arithmetic in your head.
- Run it, paste the red output. No red, no fix.
3. Fix
- Root cause, not symptom. Surgical diff — touch only what the fix requires;
no adjacent "improvements".
- Respect the standing invariants for the subsystem (runbook case lists them):
e.g. never destroy the shared undici Agent on isolated evidence; all
persistence through Turso; heartbeats on skip paths; bounded IB awaits.
4. Full test gate (green)
- Run the regression test file in isolation first, then the full gates from the
repo root, mirroring CI:
bunx vitest run --config vitest.config.ts (root config, NOT bun run test)python3.13 -m pytest scripts/tests scripts/api/tests scripts/trade_blotter
- If a test flakes, re-run the suspect file in isolation before concluding your
change caused it (test-ordering pollution is common here).
- All green, pasted, before any commit.
5. Commit — focused, single-purpose
git status FIRST. Stage ONLY the files you edited, by path
(git add path/to/file). NEVER git add -A / git add . — untracked WIP
(journals, scratch scripts, notebooks) must not be swept in.- One logical change per commit; docs/runbook updates for THIS incident belong
in the same commit. No secrets/PII. Claim in the message only what was
actually verified.
6. Update the runbook + lessons
- New failure mode → new case section in
docs/incident-runbook.md (mechanism,
detection signals, discriminating checks, fix commits, regression tests).
Known mode → append the new commit SHA to its case.
- If the incident came from a correction or a wrong first diagnosis, add the
pattern to
tasks/lessons.md.
7. Push ONCE and wait for CI green
- Check nothing is mid-deploy:
gh run list --workflow=ci.yml --limit 1.
If a run is in progress, WAIT — rapid-fire pushes cancelled an in-flight
deploy and corrupted the production build (2026-07-08).
git push origin main (main, never beta), then watch to completion:
gh run watch or poll gh run list --workflow=ci.yml --limit 1 until
completed/success. The deploy job is gated on the full test matrix and
holds a non-canceling production lock.
8. Verify live
- UI-visible fixes: verify in the real browser via chrome-cdp (fallback
Playwright) and capture a screenshot as evidence. Tests alone do not close a
UI incident.
- Data/API fixes:
curl the production route (authenticated where needed) and
verify the BODY; for persistence fixes confirm the row in live Turso.
- Confirm the incident watchdog resolves the artifact on its next cycle
(
python3.13 -m scripts.incident_watchdog --once, then check the incident
file's status flipped to resolved), or resolve-and-note manually if the
probe surface cannot see the fix.
- Only now report done, with the evidence inline.
Stand-down criteria (do NOT ship a fix)
- Upstream/platform outage (Turso-side wedge, IB farm down): remediation is
operational, not code. Follow the runbook case, alert, stand down.
- Off-hours staleness on RTH writers, anonymous 401/404 on protected routes,
unknown probe states: not incidents. Close the artifact with a note.
- Deploy in a transitional state (CI
in_progress): wait; do not "fix" a
marker mismatch that a running deploy is about to close.
