| name | incident-response |
| description | Respond to production incidents — outages, degradations, data issues, or security events — with a structured process that minimizes impact, coordinates the team, and drives to resolution. Use this skill when something in production is broken, degraded, or behaving unexpectedly and users or stakeholders are affected. Trigger phrases: "production is down", "something is wrong in prod", "users are reporting errors", "the site is slow", "we have an incident", "something broke after the deployment", "critical bug in production", "how do we respond to this?", "who does what during an incident". This skill handles the CRISIS — project-monitor detects the problem, debugging-rootcause finds the cause, release-management deploys the fix.
|
Incident Response Skill
Manages production incidents from detection to resolution.
During a crisis, the biggest cost is confusion — who does what, in what order,
with what information. This skill provides the structure to minimize that cost.
Core Principle
In an incident, speed of communication matters as much as speed of fix.
Affected users and stakeholders need to know what's happening.
The team needs to know who is in charge and what each person should do.
Contain first. Diagnose second. Fix third. Learn last.
Incident Severity Levels
Classify first. Response scales with severity.
SEV-1 — CRITICAL (wake people up)
Complete outage or data loss affecting all or most users.
Immediate financial, legal, or reputational impact.
Examples: site down, payment system failing, data breach.
Response time: immediate. All hands available.
SEV-2 — HIGH (respond within 30 minutes)
Major feature broken, significant degradation, subset of users affected.
Examples: checkout broken, login failing for 20% of users.
Response time: 30 minutes. Core team responds.
SEV-3 — MEDIUM (respond within 2 hours)
Non-critical feature broken, minor degradation, workaround exists.
Examples: export function failing, search slow but working.
Response time: 2 hours during business hours.
SEV-4 — LOW (next business day)
Minor issue, no user-facing impact, or cosmetic problem.
Response time: planned work, no urgency.
Response Roles
Assign these roles at the start of every SEV-1/SEV-2 incident:
INCIDENT COMMANDER (IC) — one person only
Owns the incident. Makes decisions when team disagrees.
Does NOT debug — coordinates others doing the debugging.
Keeps the timeline. Drives communication.
Declares resolution.
TECHNICAL LEAD (TL) — one person
Leads the diagnosis and fix.
Reports findings to IC.
Manages the technical work.
COMMUNICATIONS LEAD (CL) — one person
Handles external and internal communication.
Updates status page, sends stakeholder messages.
Frees IC and TL to focus on the problem.
SCRIBE — one person (can be IC if small team)
Documents everything in real time:
- What was tried and when
- What was found and when
- Decisions made and why
The post-incident review depends on this record.
Response Workflow
Step 1 — Declare and Classify (first 5 minutes)
□ Is this actually an incident? (not a false alarm / monitoring bug?)
□ Assign severity level (SEV-1 through SEV-4)
□ Assign roles (IC, TL, CL, Scribe)
□ Open incident channel / war room
□ Scribe begins timeline log
TIMELINE LOG ENTRY FORMAT:
[HH:MM] <who> — <what was observed or done>
Example:
[14:23] @alex — Users reporting 500 errors on checkout. Confirming in logs.
[14:25] @sam — Confirmed: error rate 45% on /api/orders. SEV-1 declared.
[14:26] @alex — IC: alex, TL: sam, CL: jordan. War room: #incident-2025-08-14
Step 2 — Communicate (first 10 minutes)
Do not wait until you know the cause to communicate.
INTERNAL (immediately):
Notify: engineering leadership, customer success, support team.
Message: "We have a SEV-[N] incident. [What is affected]. We are investigating.
Next update in [15/30] minutes."
EXTERNAL (within 10 minutes for SEV-1/2):
Update status page: "We are investigating reports of [symptoms]."
Do NOT: speculate on cause in external communication.
Do NOT: promise a fix time until you know what the fix is.
UPDATE CADENCE:
SEV-1: update every 15 minutes, even if "still investigating"
SEV-2: update every 30 minutes
SEV-3/4: update when there is meaningful progress
Step 3 — Contain (before diagnosing)
Limit the damage before finding the cause.
CONTAINMENT OPTIONS (apply the fastest one that works):
□ Rollback last deployment (if incident followed a deploy)
□ Disable the broken feature (feature flag off)
□ Route traffic away from affected component
□ Rate limit or block traffic causing the problem
□ Scale up resources (if resource exhaustion is the cause)
□ Failover to backup system
CONTAINMENT DECISION:
"Can we reduce user impact right now, even before knowing the root cause?"
YES → contain first, then diagnose on a contained system
NO → proceed directly to diagnosis
IMPORTANT: containment is not the fix. It buys time to diagnose safely.
Step 4 — Diagnose (use debugging-rootcause skill)
While TL diagnoses:
□ IC manages communication cadence
□ CL sends updates on schedule
□ Scribe logs everything
TL applies debugging-rootcause:
□ Gather evidence (logs, metrics, traces)
□ Form and test one hypothesis at a time
□ Find root cause before proposing fix
Step 5 — Fix and Verify
□ Fix addresses the root cause (not just the symptom)
□ Fix tested in staging if possible (even briefly)
□ Deployment follows release-management process
□ Verify: monitor for 15-30 minutes post-fix before declaring resolved
□ Confirm: error rate returns to baseline, user reports stop
Step 6 — Declare Resolution
IC declares resolution when:
□ Error rates have returned to normal for 15+ minutes
□ Affected features are confirmed working
□ No new user reports of the same issue
RESOLUTION COMMUNICATION:
Internal: "Incident resolved at [time]. Root cause: [brief]. Post-mortem: [date]."
External (status page): "This incident has been resolved. We will publish a
post-incident review within [24-48] hours."
Post-Incident Review (PIR)
Run within 48 hours of resolution. Blameless. Focus on systems, not people.
PIR AGENDA (60 minutes):
1. Timeline review (15 min): walk through what happened, minute by minute
2. Impact assessment (5 min): how many users? how long? what was lost?
3. What went well (10 min): what parts of the response worked?
4. What went poorly (10 min): what slowed us down or made things worse?
5. Root cause deep dive (10 min): 5 Whys on the technical cause
6. Action items (10 min): specific, owned, time-bounded improvements
PIR OUTPUT FORMAT:
Date/time of incident:
Duration:
Severity:
Impact summary:
Root cause:
Timeline: [detailed log from scribe]
What went well:
What could be improved:
Action items:
□ <action> — Owner: <name> — Due: <date>
□ <action> — Owner: <name> — Due: <date>
Incident Communication Templates
See references/communication-templates.md for ready-to-use templates.
Reference Files
references/communication-templates.md — Templates for every incident communication type
