Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use when building runbooks, responding to incidents, or establishing incident response procedures.
Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use when building runbooks, responding to incidents, or establishing incident response procedures.
version
1.1.0
model
sonnet
invoked_by
both
user_invocable
true
tools
["Read","Write","Bash"]
best_practices
["Keep runbooks updated after incidents","Test runbooks regularly","Include rollback steps","Document assumptions"]
error_handling
graceful
streaming
supported
verified
true
lastVerifiedAt
"2026-02-22T00:00:00.000Z"
source
builtin
trust_score
100
provenance_sha
ff891c5f12206ab2
Mode: Cognitive/Prompt-Driven — No standalone utility script; use via agent context.
Incident Runbook Templates
Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.
# Verify service is healthy
curl -s https://api.company.com/payments/health | jq
# Verify error rate is back to normal
curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))" | jq '.data.result[0].value[1]'# Verify latency is acceptable
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))" | jq
# Smoke test critical flows
./scripts/smoke-test-payments.sh
INCIDENT: Payment Service Degradation
Severity: SEV2
Status: Investigating
Impact: ~20% of payment requests failing
Start Time: [TIME]
Incident Commander: [NAME]
Current Actions:
- Investigating root cause
- Scaling up service
- Monitoring dashboards
Updates in #payments-incidents
Status Update
UPDATE: Payment Service Incident
Status: Mitigating
Impact: Reduced to ~5% failure rate
Duration: 25 minutes
Actions Taken:
- Rolled back deployment v2.3.4 → v2.3.3
- Scaled service from 5 → 10 replicas
Next Steps:
- Continuing to monitor
- Root cause analysis in progress
ETA to Resolution: ~15 minutes
Resolution Notification
RESOLVED: Payment Service Incident
Duration: 45 minutes
Impact: ~5,000 affected transactions
Root Cause: Memory leak in v2.3.4
Resolution:
- Rolled back to v2.3.3
- Transactions auto-retried successfully
Follow-up:
- Postmortem scheduled for [DATE]
- Bug fix in progress
### Template 2: Database Incident Runbook
```markdown
# Database Incident Runbook
## Quick Reference
| Issue | Command |
|-------|---------|
| Check connections | `SELECT count(*) FROM pg_stat_activity;` |
| Kill query | `SELECT pg_terminate_backend(pid);` |
| Check replication lag | `SELECT extract(epoch from (now() - pg_last_xact_replay_timestamp()));` |
| Check locks | `SELECT * FROM pg_locks WHERE NOT granted;` |
## Connection Pool Exhaustion
```sql
-- Check current connections
SELECT datname, usename, state, count(*)
FROM pg_stat_activity
GROUP BY datname, usename, state
ORDER BY count(*) DESC;
-- Identify long-running connections
SELECT pid, usename, datname, state, query_start, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;
-- Terminate idle connections
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
AND query_start < now() - interval '10 minutes';
Replication Lag
-- Check lag on replicaSELECTCASEWHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN0ELSEextract(epoch from now() - pg_last_xact_replay_timestamp())
ENDAS lag_seconds;
-- If lag > 60s, consider:-- 1. Check network between primary/replica-- 2. Check replica disk I/O-- 3. Consider failover if unrecoverable
Disk Space Critical
# Check disk usagedf -h /var/lib/postgresql/data
# Find large tables
psql -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;"# VACUUM to reclaim space
psql -c "VACUUM FULL large_table;"# If emergency, delete old data or expand disk
## Best Practices
### Do's
- **Keep runbooks updated** - Review after every incident
- **Test runbooks regularly** - Game days, chaos engineering
- **Include rollback steps** - Always have an escape hatch
- **Document assumptions** - What must be true for steps to work
- **Link to dashboards** - Quick access during stress
### Don'ts
- **Don't assume knowledge** - Write for 3 AM brain
- **Don't skip verification** - Confirm each step worked
- **Don't forget communication** - Keep stakeholders informed
- **Don't work alone** - Escalate early
- **Don't skip postmortems** - Learn from every incident
## Resources
- [Google SRE Book - Incident Management](https://sre.google/sre-book/managing-incidents/)
- [PagerDuty Incident Response](https://response.pagerduty.com/)
- [Atlassian Incident Management](https://www.atlassian.com/incident-management)
## Iron Laws
1. **ALWAYS** write and test runbooks before an incident occurs — an untested runbook written during an active P0 incident introduces errors when cognitive load is highest.
2. **NEVER** assume operator knowledge in a runbook — write every step as if for a new on-call engineer at 3 AM with no context; assumed knowledge creates fatal gaps under stress.
3. **ALWAYS** include a rollback step for every mitigation action — mitigation steps that cannot be reversed trap teams in a worse state when the fix makes the incident worse.
4. **NEVER** resolve an incident without updating the runbook if a step failed or was missing — unupdated runbooks repeat the same failures in the next incident.
5. **ALWAYS** define explicit escalation triggers with time bounds and contact owners — runbooks without escalation criteria leave operators guessing when to escalate, causing under-escalation or over-escalation.
## Anti-Patterns
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| Runbooks written during the incident | High stress + cognitive load = errors and gaps; untested steps fail | Write and game-day runbooks before incidents; update post-incident |
| Steps that assume domain knowledge | New on-call engineers fail; team knowledge is a single point of failure | Write each step as a complete command with expected output and failure signal |
| No rollback steps | Mitigation makes incident worse; team is stuck without recovery path | Every `change X` step must have a matching `revert X` step with commands |
| Runbook not updated after incident | Same failure mode repeated in next incident | Mandate runbook review as part of postmortem action items |
| Vague escalation criteria ("escalate if needed") | Under-escalation extends MTTR; over-escalation burns on-call | Define triggers: "escalate after 15 minutes without mitigation progress" |
## Memory Protocol (MANDATORY)
**Before starting:**
Read `.claude/context/memory/learnings.md`
**After completing:**
- New pattern -> `.claude/context/memory/learnings.md`
- Issue found -> `.claude/context/memory/issues.md`
- Decision made -> `.claude/context/memory/decisions.md`
> ASSUME INTERRUPTION: If it's not in memory, it didn't happen.
