| name | incident-response-and-postmortems |
| description | Use when defining severity levels and on-call procedures, when creating a runbook for a critical service, when an incident is actively happening and a structured response is needed, or when writing a blameless postmortem after Sev1 or Sev2 resolution. |
Incident Response and Postmortems
Overview
Define how the team detects, mitigates, communicates about, and learns from production incidents — before the first one. Output is .forge/incident-response.md — severity definitions with response times, the declare-mitigate-communicate-resolve-review flow, runbook template, blameless postmortem template, and the comms templates (internal and external). Pairs with observability (alerts feed the declare step) and error-handling-and-resilience (the failure modes alerts watch for).
When to Use
- A service is going to production and there's no on-call process
- An incident just happened and the team handled it ad-hoc — write the process before the next one
- A postmortem is due and there's no template
- Severity levels are inconsistent across teams
- A runbook is needed for a critical service
When NOT to Use
- Localhost-only or pre-prod systems with no users
- A trivial bug fix — that's
debugging-and-recovery
- A near-miss that didn't affect users — log it, learn from it, but don't fire the full process
Common Rationalizations
| Thought | Reality |
|---|
| "We're too small for an incident process" | Incidents don't wait for headcount. The first Sev1 at 4 staff is when you learn the process you didn't build. |
| "We'll write the postmortem later" | Memory fades within 48h. Two weeks later the timeline is a guess. |
| "It was their fault" | Blame ends learning. Every "someone clicked the wrong button" is a system that allowed the wrong button to be clicked. |
| "Action items can wait until we have time" | Unowned action items are wishes. They never ship. |
| "We don't need severity levels — just fix everything fast" | Without severity, every alert pages every human. Fatigue sets in. The real Sev1 gets missed. |
| "Comms can be informal during the incident" | "Informal" means three different stories reaching three different audiences. Customers, board, and team need consistent, scheduled updates. |
Red Flags
- No documented severity definitions
- A critical service with no runbook
- A postmortem that uses words like "negligence", "should have known", "failure to follow procedure" — blame language
- Action items in a postmortem without an owner and a deadline
- The same incident class recurring (auth outage, queue backup, DB lock) without an action item from the previous postmortem closed
- No external comms during a customer-visible outage
- The on-call hand-off has no documented context (no runbook, no recent incident summary)
Core Process
Step 1: Define severity levels
In .forge/incident-response.md:
| Sev | Definition | Response time | Examples |
|---|
| Sev1 | Service down or major data integrity issue | Page on-call immediately, all-hands within 15min | Site down, DB corruption, security breach |
| Sev2 | Major feature broken or significant degradation | Page on-call, response within 30min | Login broken, checkout failing, search down |
| Sev3 | Minor feature broken or low-impact degradation | Ticket, response within 4h business | Non-critical endpoint slow, edge-case bug |
| Sev4 | Cosmetic or no user impact | Ticket, response next business day | Typo, log noise, internal tool glitch |
Step 2: Write the response flow
Five steps, in order:
- Declare — anyone can declare. Use the standard channel (
#incident-active). State: severity guess, what's broken, what's the impact.
- Mitigate — restore service first. Root-cause investigation comes after. Acceptable mitigations: rollback, feature flag off, traffic shift, scale up.
- Communicate — internal updates every 30min for Sev1, every 60min for Sev2. External status page updated within 15min of declare (Sev1) or 30min (Sev2). Comms owner is named, not "whoever notices."
- Resolve — service restored, customer-visible impact ended. Mark resolved on status page.
- Review — postmortem within 5 business days for Sev1/Sev2.
Step 3: Create the runbook template
For every critical service (anything with a Sev1 or Sev2 potential), a runbook must contain:
- Service summary — what it does, what fails when it goes down
- Owners — primary on-call, escalation chain
- Dashboards — links to the observability dashboards (cross-ref
observability skill)
- Common failure modes — top 5 with first-response steps
- Mitigation playbook — feature flag locations, rollback commands, traffic shift recipes
- Escalation criteria — when to page the next tier
- Recovery verification — how you know it's actually fixed (synthetic check, key metric returning to baseline)
Cross-reference the runbook URL in every alert (cross-ref observability rule: every alert has a runbook link).
Step 4: Write the blameless postmortem template
In .forge/incident-response.md:
# Postmortem: <one-line summary>
## Impact
- Severity: Sev1/2/3
- Duration: HH:MM start to HH:MM end (TZ)
- Affected: <number of users, % of traffic, $ if measurable>
## Timeline (UTC, all times verified from logs)
- 14:02 — Deploy v2.4.1 to production
- 14:08 — Error rate climbs above 5% on /api/checkout
- 14:11 — Alert fires; on-call paged
- 14:14 — On-call acks
- 14:22 — Mitigation: rollback to v2.4.0 initiated
- 14:26 — Rollback complete, error rate returns to baseline
- 14:31 — Incident resolved; status page updated
## Root cause
<what happened, mechanically. No blame language.>
## Contributing factors
- <missing alert, weak test, flaky dependency, slow rollback tooling, etc.>
## What went well
- <fast detection, clean rollback, good comms>
## What we learned
- <pattern that future work should respect>
## Action items
| ID | Action | Owner | Deadline | Tracking |
|---|---|---|---|---|
| AI-1 | Add integration test covering the checkout path | @owner | YYYY-MM-DD | LINK-123 |
| AI-2 | Reduce rollback time from 8min to <2min | @owner | YYYY-MM-DD | LINK-124 |
Blameless rule: the word "should have" is forbidden. Replace with "the system did not have." Every action item targets a system, not a person.
Step 5: Define comms templates
Three audiences, three templates:
- Internal status update (every 30min Sev1 / 60min Sev2 in
#incident-active): current status, mitigation in progress, ETA, next update time
- External status page (15min Sev1 / 30min Sev2): what's affected, what we're doing, when the next update is — no internal jargon
- Customer comms post-resolution (within 24h for Sev1, 72h for Sev2): impact summary, root cause in plain language, what we're doing to prevent recurrence
Verification
