| name | incident-response |
| description | Use when production is broken or degraded — mitigation-first response, timeline capture, severity ladder, and blameless postmortems. Agent-loop failures go to agent-introspection-debugging instead. |
Incident Response
Production is broken. The goal ordering is: stop the bleeding → understand → prevent recurrence. (Agent-loop failures are a different skill: agent-introspection-debugging.)
Mitigate before you diagnose
- First move restores service, not understanding: roll back the last deploy, disable the feature/config, shed load, scale out. Root cause comes after users stop hurting.
- Rollback is the default mitigation. "Roll forward with a fix" needs justification: the fix is truly one line, verified locally, and rollback is impossible or worse.
- If the trigger is unknown, bisect reality: what changed in the last N hours — deploys, config, dependencies, traffic shape, certificates, quotas.
- One person mitigates; another investigates. The same pair of hands doing both serializes the incident.
Declare early, classify honestly
- Declare on suspicion, not certainty — a false alarm costs minutes; a late escalation costs the SLA. Nobody is punished for declaring.
- Severity ladder (adjust names to the org, keep the shape):
- SEV1 — users down / data loss in progress: all hands, comms channel opens, updates every 30 min.
- SEV2 — degraded or subset impact: owner + backup engaged, updates hourly.
- SEV3 — internal-only / cosmetic: ticket, fix in normal flow.
- Say what you know, what you don't, and when the next update comes. Silence reads as abandonment.
Capture the timeline as you act
- Timestamped notes in the incident channel while acting: what you saw, what you ran, what changed. Memory reconstructs badly and postmortems built from memory are fiction.
- Save the evidence before it rotates: log excerpts, graphs (screenshot with time range), the exact failing request. Dashboards age out; the postmortem needs the artifact.
- Every mitigation action is announced before it's taken ("rolling back api to v141 now") — two people mitigating unannounced is how incidents get worse.
Blameless postmortem (within 48h for SEV1/2)
- Structure: impact (who/what/how long) → timeline (from the notes) → contributing factors → what went well → action items.
- "Human error" is never a root cause — it's the start of the next question: what made the error easy to make and hard to catch? Fix the system, not the human.
- Contributing factors, plural. Single-root-cause narratives are almost always incomplete for real incidents.
- Action items have an owner and a date or they are wishes. "Add alert on queue age" beats "improve monitoring".
- Feed durable patterns into
LEARNINGS.md — a postmortem nobody can find prevents nothing.
When NOT to apply
Pre-production breakage: fix it via the normal bug-fix flow (evidence, RED, GREEN) — no severity theater. But if staging shares infrastructure with prod, treat infrastructure incidents as real ones.
