| name | incident-response |
| description | Incident response — severity, communication, mitigation, post-mortem. |
Severity Levels
| Severity | Definition | Response Time | Example |
|---|
| SEV1 | Complete outage — service unavailable for all users | Immediate | Site down, payments failing 100% |
| SEV2 | Major feature down — core functionality broken for most users | 15 min | Login broken, checkout not processing |
| SEV3 | Degraded performance — feature impaired or slow for a subset of users | 1 hour | Search slow, notifications delayed |
| SEV4 | Minor issue — cosmetic bug or edge-case failure with low impact | Next business day | UI glitch, rare error for <1% users |
First 5 Minutes
- Declare the incident — state the severity, impacted surface, and known symptoms
- Assign Incident Commander (IC) — one person owns communication and coordination; others investigate
- Open an incident channel — create a dedicated Slack/chat channel (e.g.,
#inc-2026-05-10-checkout)
- Send initial status — post to status page and stakeholder channel within 5 minutes:
[SEV1 INCIDENT] - <short description>
Impact: <who is affected>
Status: Investigating
IC: @<name>
Next update: <time>
Investigation Loop
Repeat until service is restored:
Symptom → Hypothesis → Test → Mitigation → Verify
- Symptom: what the monitoring/alert is showing (error rate, latency spike, etc.)
- Hypothesis: most likely cause based on recent deploys, traffic patterns, or known issues
- Test: check logs, metrics, traces to confirm or rule out the hypothesis
- Mitigation: apply the smallest change that restores service (rollback, feature flag, traffic shift)
- Verify: confirm metrics return to baseline before closing the loop
Communication Cadence
| Severity | Update Frequency | Channels |
|---|
| SEV1 | Every 15 min | Status page + stakeholder channel + incident channel |
| SEV2 | Every 30 min | Status page + stakeholder channel |
| SEV3 | Hourly | Incident channel |
| SEV4 | On resolution | Ticket comment |
Update format:
[UPDATE - HH:MM] Impact: <current state> | Action: <what's being done> | ETA: <if known>
Mitigation-First Rule
- Restore service before finding root cause — rollback, disable the flag, reroute traffic, scale up
- Document all actions taken with timestamps in the incident channel
- Root cause analysis happens during post-mortem, not during active incident
- A working workaround is a valid mitigation; clean fixes come later
Post-Mortem
- Write within 48 hours of resolution while memory is fresh
- Blameless: the goal is to understand system failures, not assign fault to individuals
- Required for all SEV1 and SEV2 incidents; optional but encouraged for SEV3
Post-Mortem Template
## Summary
One-paragraph description of the incident and its impact.
## Timeline
- HH:MM — event / action taken
- HH:MM — event / action taken
## Root Cause
What was the direct technical cause?
## Contributing Factors
- Infrastructure / config state that made this possible
- Process gaps that delayed detection or response
- Dependencies that were involved
## Action Items
| Action | Owner | Due Date |
|--------|-------|----------|
| Fix X | @name | YYYY-MM-DD |
## What Went Well
- Detection was fast
- Rollback procedure worked as expected
On-Call Handoff
When handing off an active incident to the next on-call:
- Update the incident doc with current status and open questions
- Write a short Current State summary: what is known, what is still broken, what has been ruled out
- List everything that has been tried and whether it helped
- Tag the incoming on-call in the incident channel and confirm they have access to all relevant dashboards
