| name | incident-management |
| description | Use when handling production incidents, conducting post-mortems, or designing incident response processes. Covers severity classification, incident commander role, communication protocols, root cause analysis, and post-mortem templates. |
| user-invocable | false |
| allowed-tools | ["Read","Write","Bash","Grep","Glob"] |
Incident Management
Severity Classification
| Sev | Definition | Response | Examples |
|---|
| SEV1 | Complete outage, data loss, security breach | < 5 min, page IC + lead + VP | Service down, DB corruption, active breach |
| SEV2 | Major feature degraded, partial outage | < 15 min, page IC + lead | Payment failing, login broken for subset |
| SEV3 | Minor degradation, workaround exists | < 1 hr, notify via chat | Slow reports, non-critical errors for one tenant |
| SEV4 | Cosmetic, no user impact | < 4 hr, ticket | Dashboard glitch, dev env instability |
Auto-escalate if not acknowledged within response time.
Incident Commander (IC)
Authority: Roll back any deploy, page any engineer, escalate/downgrade severity, approve emergency changes, halt non-essential deploys during SEV1.
Duties: Own incident start-to-finish, coordinate responders, control communication cadence (SEV1: every 15min, SEV2: every 30min), declare resolution, schedule post-mortem.
Response Process
- Detect: Automated alert, user report, or engineer observation.
- Triage (0-5min): Assign severity, designate IC, open
#inc-YYYYMMDD-desc channel, update status page.
- Investigate (5-30min): Check recent deploys/config changes, review logs/metrics, identify affected components, assign parallel tracks.
- Mitigate (target 30min SEV1): Apply fastest fix, confirm via metrics, communicate status.
- Resolve: Apply permanent fix, verify in staging then production, monitor 1-2hr for recurrence.
- Review (within 48hr): Blameless post-mortem, document timeline/RCA/action items.
Triage Checklist
curl -s https://app.example.com/api/health | jq .
docker compose ps && docker stats --no-stream
df -h && free -h
git log --oneline -5 --format="%h %ai %s"
docker compose logs --since 30m app 2>&1 | grep -ci "error\|exception\|fatal"
docker compose exec -T db psql -U postgres -c "SELECT pid, now()-query_start AS duration, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 5;"
Check: error rate vs baseline, p95/p99 latency, traffic anomalies, CPU/memory/disk saturation, queue depth.
Mitigation Strategies (Speed Order)
| Strategy | Speed | Risk | When |
|---|
| Feature flag disable | Seconds | Very low | Bad feature rollout |
| Rollback deployment | 2-5 min | Low | Correlated with recent deploy |
| Traffic shifting | 1-2 min | Low | Canary/blue-green |
| Scale up/out | 3-10 min | Low | Capacity issue |
| Restart service | 1-2 min | Medium | Memory leak, hung process |
| Failover to secondary | 5-15 min | Medium | Primary infra failure |
| Hotfix deploy | 15-30 min | Higher | Rollback not possible |
Communication Templates
[INVESTIGATING] Investigating <symptom>. Impact: <user-facing>. Started: <UTC>. Next update: <time>.
[IDENTIFIED] Root cause: <brief>. Working on fix. Next update: <time>.
[MONITORING] Fix applied. Monitoring stability. Next update: <time>.
[RESOLVED] Resolved. Duration: <X>h <Y>m. Root cause: <summary>. Post-mortem within 48hr.
Root Cause Analysis
5 Whys
Problem: Users got 500 errors on invoice endpoint.
Why 1: Out-of-memory exception.
Why 2: Query loaded all line items into memory.
Why 3: No pagination, eager-loaded nested relations.
Why 4: Code review missed unbounded query.
Why 5: No static analysis rule for query size.
Root cause: Missing guardrails for query result size.
When 5 Whys diverge, use Fishbone categories: People, Process, Technology, Environment, Monitoring.
Post-Mortem Template
# Post-Mortem: <Title>
**Date/Severity/Duration/IC/Author**
## Summary: 2-3 sentences on what happened and user impact.
## Impact: Users affected, revenue impact, data impact, duration.
## Timeline (UTC): Alert -> Declared -> Identified -> Mitigated -> Resolved.
## Root Cause: Technical explanation + 5 Whys.
## Detection: How detected, time to detection, improvement opportunities.
## Response: What went well, what could improve.
## Action Items
| Priority | Action | Owner | Due | Status |
|----------|--------|-------|-----|--------|
| P0 | Prevent recurrence | name | date | Open |
| P1 | Improve detection | name | date | Open |
## Lessons Learned
Incident Metrics
| Metric | Target |
|---|
| MTTD (detect) | < 5 min |
| MTTA (acknowledge) | < 5 min SEV1, < 15 min SEV2 |
| MTTR (resolve) | < 1 hr SEV1, < 4 hr SEV2 |
| MTBF | Increasing trend |
Track action item completion: target 90% within 30 days.
On-Call
- 2-person rotation (primary + secondary), weekly, max 7 days consecutive.
- Escalation: 0min primary, 5min secondary, 10min lead, 15min manager.
- Every alert links to a runbook (what it means, diagnose, mitigate, escalate).
- Target < 5 actionable alerts per shift. Tune/delete noisy alerts.
Proactive Prevention
- Chaos engineering: Inject failures in staging, gradually increase scope.
- Game days: Quarterly simulated incidents, rotate IC role.
- Pre-mortems: Before launch ask "what could go wrong at 10x scale?"
- Target 40%+ of reliability work as proactive (not incident-triggered).
Blameless Culture
Focus on systems not people. Assume good intent. Discuss "the deploy" not "your deploy." Learning over punishment. Use language like "How can we improve detection?" not "Why didn't you catch this?"
