| name | incident-response-playbook |
| effort | high |
| description | Guides the developer-side incident lifecycle from triage through mitigation to blameless RCA — with severity classification, Slack update templates, escalation paths, and a structured post-mortem format. Use when a production issue is active or... |
| scenarios | ["We just had a production outage and need a structured incident response process","Help me create an on-call runbook for database failures","Design an incident response playbook for our SRE team","프로덕션 장애가 발생했는데 대응 프로세스가 없어","온콜 런북과 인시던트 대응 플레이북을 만들어줘"] |
| compatibility | {"recommended":["sequential-thinking","think-tool"],"optional":[],"remote_mcp_note":"sequential-thinking은 탐지 → 트리아지 → 완화 → RCA 단계를 순서대로 진행할 때 유용합니다. think-tool은 장애 원인 분석과 우선순위 판단에 활용됩니다. Claude 설정 → MCP Servers에서 remote SSE 엔드포인트를 추가하세요."} |
Standing Mandates
- ALWAYS classify severity before any other action — severity drives escalation and communication cadence.
- ALWAYS separate mitigation (restore service) from investigation (find root cause) — do not block one on the other.
- NEVER assign blame to individuals in post-mortems — focus on systemic factors.
- NEVER skip the timeline reconstruction step in RCA; gaps in the timeline are often the cause.
Incident Response Playbook
Speed matters. Clarity matters more. Blame solves nothing.
Incident Lifecycle
Detect → Triage → Communicate → Mitigate → Resolve → Learn
Each phase has a clear exit condition. Do not skip phases under pressure.
Severity Classification
| Severity | Definition | Response Time | Example |
|---|
| P0 | Full service down; all users affected; revenue/data at risk | Immediate — page on-call now | Checkout API 100% error rate |
| P1 | Major feature broken; significant user subset affected | < 15 min | Payment processing failing for 30% of users |
| P2 | Degraded performance or partial feature failure | < 1 hour | Search latency 10x normal; some timeouts |
| P3 | Minor issue; workaround available; low user impact | Next business day | Dashboard chart missing for 5% of users |
Escalate up, not down. When in doubt, treat as higher severity.
Phase 1 — Detect
Sources of detection (in order of reliability):
- Automated alert (PagerDuty, OpsGenie, CloudWatch alarm)
- Synthetic monitor / uptime check failure
- User report via support ticket or social media
- Team member notices in logs or dashboard
First action on detection: open an incident channel immediately, even if you're still diagnosing.
Phase 2 — Triage (target: < 5 min for P0/P1)
Answer these questions fast:
- What is broken? (service, endpoint, feature)
- Who is affected? (all users / subset / internal only)
- When did it start? (check deploy history, config changes, upstream alerts)
- What changed recently? (deployments, feature flags, infra changes, dependency updates)
- What is the blast radius? (data corruption? revenue loss? SLA breach?)
Triage output: severity classification + incident commander assigned.
Phase 3 — Communicate
Rule: communicate before you know the answer.
Silence is worse than "we're investigating." Update every 15–30 min for P0/P1.
Internal Slack Update Template
[P0] <Service> - <Brief description>
Status: Investigating / Identified / Mitigating / Resolved
Impact: <Who is affected and how>
Start time: <HH:MM UTC>
Last update: <HH:MM UTC>
IC: @<incident-commander>
Bridge: <link to war room>
Next update: <HH:MM UTC>
Status Page Update Template
Investigating: We are aware of an issue affecting <service/feature>.
Our team is actively investigating. We will provide an update by <time>.
Identified: We have identified the cause of the issue affecting <service/feature>.
We are working on a fix and expect resolution by <time>.
Resolved: The issue affecting <service/feature> has been resolved as of <time>.
All systems are operating normally. An RCA will be published within 48 hours.
What NOT to say publicly: internal system names, specific error messages that reveal architecture, speculation about cause before confirmed.
Phase 4 — Mitigate
Goal: reduce user impact now, even if the root cause isn't fully understood.
Common mitigation actions (fastest to slowest):
| Action | When to Use | Risk |
|---|
| Rollback last deployment | Issue started after deploy | Low if rollback is clean |
| Disable feature flag | Feature-specific failure | Low |
| Increase replica count / scale out | Overload / capacity issue | Medium (cost) |
| Enable circuit breaker / shed load | Cascading failure risk | Medium (some users see errors) |
| Redirect traffic to healthy region | Regional failure | Medium (requires DNS/LB change) |
| Restore from backup | Data loss / corruption | High — requires validation |
Mitigation ≠ Fix. Document that mitigation is a temporary measure. The real fix comes after the incident.
Phase 5 — Resolve
Exit criteria for "Resolved":
Phase 6 — Learn (RCA / Post-Mortem)
Target: publish RCA within 48–72 hours for P0/P1.
RCA Template
## Incident RCA: <Title>
**Date**: YYYY-MM-DD
**Duration**: X hours Y minutes
**Severity**: P0 / P1 / P2
**Services Affected**: <list>
**Author(s)**: <names>
**Status**: Draft / Final
---
### Summary
One paragraph. What happened, what was the user impact, how was it resolved.
### Timeline
| Time (UTC) | Event |
|------------|-------|
| HH:MM | Alert fired / issue first detected |
| HH:MM | On-call paged |
| HH:MM | Triage complete; P0 declared |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Incident resolved |
### Root Cause
Specific, technical explanation. Not "human error" — explain what made the error possible.
### Contributing Factors
- Factor 1 (e.g., no alerting on X metric)
- Factor 2 (e.g., runbook for this scenario was missing)
- Factor 3 (e.g., deploy pipeline did not catch Y in staging)
### What Went Well
- ...
### Action Items
| Item | Owner | Due Date | Priority |
|------|-------|----------|----------|
| Add alert for X | @person | YYYY-MM-DD | High |
| Update runbook for Y scenario | @person | YYYY-MM-DD | Medium |
| Add canary deploy for Z service | @person | YYYY-MM-DD | High |
Blameless Culture
Document: what systems failed, what processes were missing, what made the failure possible.
Avoid: naming individuals as the cause, language like "engineer forgot to", "someone accidentally".
Reframe: "The deploy pipeline did not have a canary stage that would have caught this" instead of "Alice pushed bad code."
People make mistakes. Systems should make mistakes hard to cause and easy to detect.
Incident Commander Checklist
Escalation Paths
