Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use when building runbooks, responding to incidents, or establishing incident response procedures.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use when building runbooks, responding to incidents, or establishing incident response procedures.
version
1.1.0
model
sonnet
invoked_by
both
user_invocable
true
tools
["Read","Write","Bash"]
best_practices
["Keep runbooks updated after incidents","Test runbooks regularly","Include rollback steps","Document assumptions"]
error_handling
graceful
streaming
supported
verified
true
lastVerifiedAt
"2026-02-22T00:00:00.000Z"
source
builtin
trust_score
100
provenance_sha
ff891c5f12206ab2
Mode: Cognitive/Prompt-Driven — No standalone utility script; use via agent context.
Incident Runbook Templates
Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.
# Verify service is healthy
curl -s https://api.company.com/payments/health | jq
# Verify error rate is back to normal
curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))" | jq '.data.result[0].value[1]'# Verify latency is acceptable
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))" | jq
# Smoke test critical flows
./scripts/smoke-test-payments.sh
INCIDENT: Payment Service Degradation
Severity: SEV2
Status: Investigating
Impact: ~20% of payment requests failing
Start Time: [TIME]
Incident Commander: [NAME]
Current Actions:
- Investigating root cause
- Scaling up service
- Monitoring dashboards
Updates in #payments-incidents
Status Update
UPDATE: Payment Service Incident
Status: Mitigating
Impact: Reduced to ~5% failure rate
Duration: 25 minutes
Actions Taken:
- Rolled back deployment v2.3.4 → v2.3.3
- Scaled service from 5 → 10 replicas
Next Steps:
- Continuing to monitor
- Root cause analysis in progress
ETA to Resolution: ~15 minutes
Resolution Notification
RESOLVED: Payment Service Incident
Duration: 45 minutes
Impact: ~5,000 affected transactions
Root Cause: Memory leak in v2.3.4
Resolution:
- Rolled back to v2.3.3
- Transactions auto-retried successfully
Follow-up:
- Postmortem scheduled for [DATE]
- Bug fix in progress
### Template 2: Database Incident Runbook
```markdown
# Database Incident Runbook
## Quick Reference
| Issue | Command |
|-------|---------|
| Check connections | `SELECT count(*) FROM pg_stat_activity;` |
| Kill query | `SELECT pg_terminate_backend(pid);` |
| Check replication lag | `SELECT extract(epoch from (now() - pg_last_xact_replay_timestamp()));` |
| Check locks | `SELECT * FROM pg_locks WHERE NOT granted;` |
## Connection Pool Exhaustion
```sql
-- Check current connections
SELECT datname, usename, state, count(*)
FROM pg_stat_activity
GROUP BY datname, usename, state
ORDER BY count(*) DESC;
-- Identify long-running connections
SELECT pid, usename, datname, state, query_start, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;
-- Terminate idle connections
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
AND query_start < now() - interval '10 minutes';
Replication Lag
-- Check lag on replicaSELECTCASEWHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN0ELSEextract(epoch from now() - pg_last_xact_replay_timestamp())
ENDAS lag_seconds;
-- If lag > 60s, consider:-- 1. Check network between primary/replica-- 2. Check replica disk I/O-- 3. Consider failover if unrecoverable
Disk Space Critical
# Check disk usagedf -h /var/lib/postgresql/data
# Find large tables
psql -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;"# VACUUM to reclaim space
psql -c "VACUUM FULL large_table;"# If emergency, delete old data or expand disk
## Best Practices
### Do's
- **Keep runbooks updated** - Review after every incident
- **Test runbooks regularly** - Game days, chaos engineering
- **Include rollback steps** - Always have an escape hatch
- **Document assumptions** - What must be true for steps to work
- **Link to dashboards** - Quick access during stress
### Don'ts
- **Don't assume knowledge** - Write for 3 AM brain
- **Don't skip verification** - Confirm each step worked
- **Don't forget communication** - Keep stakeholders informed
- **Don't work alone** - Escalate early
- **Don't skip postmortems** - Learn from every incident
## Resources
- [Google SRE Book - Incident Management](https://sre.google/sre-book/managing-incidents/)
- [PagerDuty Incident Response](https://response.pagerduty.com/)
- [Atlassian Incident Management](https://www.atlassian.com/incident-management)
## Iron Laws
1. **ALWAYS** write and test runbooks before an incident occurs — an untested runbook written during an active P0 incident introduces errors when cognitive load is highest.
2. **NEVER** assume operator knowledge in a runbook — write every step as if for a new on-call engineer at 3 AM with no context; assumed knowledge creates fatal gaps under stress.
3. **ALWAYS** include a rollback step for every mitigation action — mitigation steps that cannot be reversed trap teams in a worse state when the fix makes the incident worse.
4. **NEVER** resolve an incident without updating the runbook if a step failed or was missing — unupdated runbooks repeat the same failures in the next incident.
5. **ALWAYS** define explicit escalation triggers with time bounds and contact owners — runbooks without escalation criteria leave operators guessing when to escalate, causing under-escalation or over-escalation.
## Anti-Patterns
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| Runbooks written during the incident | High stress + cognitive load = errors and gaps; untested steps fail | Write and game-day runbooks before incidents; update post-incident |
| Steps that assume domain knowledge | New on-call engineers fail; team knowledge is a single point of failure | Write each step as a complete command with expected output and failure signal |
| No rollback steps | Mitigation makes incident worse; team is stuck without recovery path | Every `change X` step must have a matching `revert X` step with commands |
| Runbook not updated after incident | Same failure mode repeated in next incident | Mandate runbook review as part of postmortem action items |
| Vague escalation criteria ("escalate if needed") | Under-escalation extends MTTR; over-escalation burns on-call | Define triggers: "escalate after 15 minutes without mitigation progress" |
## Memory Protocol (MANDATORY)
**Before starting:**
Read `.claude/context/memory/learnings.md`
**After completing:**
- New pattern -> `.claude/context/memory/learnings.md`
- Issue found -> `.claude/context/memory/issues.md`
- Decision made -> `.claude/context/memory/decisions.md`
> ASSUME INTERRUPTION: If it's not in memory, it didn't happen.
