incident-response

name	incident-response
description	Use when the user says "we have an incident", "production is down", "users are reporting errors", "our error rate spiked", "help me write a postmortem", or needs to structure an on-call response. Provides a systematic incident response framework for software engineers and on-call engineers.

Incident Response

Every production system will fail. The question is how quickly and cleanly you respond. Good incident response is a practiced discipline: detect, contain, diagnose, resolve, and learn. Postmortems are the learning mechanism — blameless, specific, and action-oriented.

When to Activate

• An active production incident
• Coordinating a multi-engineer response to an outage
• Writing or reviewing a postmortem
• Building an on-call runbook
• Establishing incident response processes for a team

Incident Lifecycle

DETECT → TRIAGE → CONTAIN → DIAGNOSE → RESOLVE → POSTMORTEM

Each phase has a different primary goal. Do not skip phases — diagnosing before containing extends the blast radius.

Phase 1: Detect

Detection sources (in order of reliability):

1. Automated alerts (synthetic monitoring, error rate thresholds, SLO burn rate alerts)
2. Customer reports (lagging indicator — many customers have been affected by the time the first report arrives)
3. Team member observation

Alert quality: a good alert is:

• Specific: "payment API error rate > 5% for 5 minutes" not "something is wrong"
• Actionable: the alert links to a runbook for the first response
• Signal-to-noise: a team that ignores alerts because 80% are false positives is flying blind

Phase 2: Triage

Within the first 5 minutes, answer:

• What is broken? — which service, feature, or user population
• How severe? — are users unable to use the product? Partial degradation? Internal only?
• How many users? — 1 user, 1%, 100%?
• Is it getting worse? — trending up or stabilizing?

Assign severity immediately:

Severity	Definition	Response
SEV1	Complete service outage affecting all users	Drop everything, all hands
SEV2	Major feature broken, significant user impact	Immediate response, team lead informed
SEV3	Partial degradation, workaround available	Fix within hours
SEV4	Minor issue, no significant user impact	Fix within days

Assign an Incident Commander — one person coordinates the response. Everyone else executes. Confusion about who is leading wastes time.

Phase 3: Contain

Before diagnosing root cause, stop the bleeding. Containment options:

• Rollback: if a recent deployment correlates with the incident, rollback first, diagnose later. Rollback is faster than fixing forward.
• Feature flag off: disable the broken feature if a flag exists
• Traffic shift: route traffic away from the broken region or service
• Scale up: if the issue is resource exhaustion, add capacity while investigating
• Drain the queue: if a queue consumer is crashing, stop consumption to prevent data loss

Principle: reducing user impact is more important than understanding the cause. Contain, then diagnose.

Phase 4: Diagnose

Apply the debugging methodology:

1. What changed recently? (deployments, config changes, traffic patterns, cron jobs)
2. What do the metrics show? (error rate, latency, CPU, memory, DB connections, queue depth)
3. What do the logs show? (error messages, stack traces, timing)
4. What do the traces show? (which span is slow, which service is failing)

Common first hypotheses:

• Recent deployment introduced a bug → check deployment time vs incident start time
• Traffic spike exceeding capacity → check request volume
• Dependency failure → check third-party status pages
• Data issue → check if specific user/account/record triggers the failure
• Infrastructure failure → check cloud provider status page

Investigation hygiene:

• One engineer investigates at a time (parallel investigations produce conflicting changes)
• Announce every action in the incident channel before taking it
• Document what you tried and what you observed in real time

Phase 5: Resolve

Fix the immediate cause. Verify the fix:

• Error rate returns to baseline
• Latency returns to baseline
• Synthetic monitors pass
• Customer reports stop

Do not close the incident until metrics confirm resolution. A fix that looks right but has not produced metric recovery may not be the actual fix.

Communicate resolution:

• Update the status page
• Notify stakeholders
• Set a postmortem deadline (within 48-72 hours while memory is fresh)

Phase 6: Postmortem

The postmortem is how the organization learns. A bad postmortem assigns blame. A good postmortem identifies system failures and prevents recurrence.

Blameless Culture

Humans make mistakes. The system should make mistakes recoverable, not catastrophic. A postmortem that concludes "engineer X made an error" without asking why the system allowed that error to cause an outage has failed. Ask: what made this mistake easy to make? What would have caught it?

Postmortem Template

## Incident Summary
- Date: 
- Duration: 
- Severity: 
- Impact: [N users, which features, which regions]

## Timeline
[List events chronologically with timestamps]
- 14:03 — Alert fired: payment API error rate > 5%
- 14:05 — On-call acknowledged
- 14:12 — Root cause identified: database connection pool exhausted
- 14:18 — Fix deployed
- 14:22 — Error rate returned to baseline
- 14:30 — Incident closed

## Root Cause
[One paragraph. What was the actual cause? Not symptoms.]

## Contributing Factors
[What conditions allowed the root cause to have this impact?]
- No alerting on connection pool utilization
- No circuit breaker on DB connection acquisition
- No runbook for this failure mode

## What Went Well
[Things that limited the impact or sped the response]

## What Went Poorly
[Things that made the incident worse or the response slower]

## Action Items
| Action | Owner | Due Date |
|--------|-------|----------|
| Add connection pool utilization alert | @engineer | 2026-04-17 |
| Add circuit breaker to DB client | @engineer | 2026-04-24 |
| Write runbook for connection exhaustion | @engineer | 2026-04-17 |

Action Items Must Be Tracked

A postmortem with no follow-through is worse than no postmortem — it creates the false impression that learning occurred. Track action items in the same system as engineering work. Review completion in the next incident review.

On-Call Runbooks

For every recurring alert or common failure mode, write a runbook before you need it:

## Alert: Payment API Error Rate > 5%

### Symptoms
Users unable to complete checkout. Error rate > 5% on /api/payments.

### Immediate Actions
1. Check deployment history — was there a recent deploy?
2. Check database connection pool metrics
3. Check third-party payment provider status: [link]

### If recent deployment:
→ Rollback: `kubectl rollout undo deployment/payments-api`

### If connection pool exhausted:
→ Restart the payments API: `kubectl rollout restart deployment/payments-api`
→ Scale up DB if load is the cause

### Escalate to:
- DB team if the issue is database-side: @db-team
- Payment provider if their status page shows issues: [escalation contact]

A runbook reduces mean time to resolve (MTTR) by removing the diagnostic phase for known failure modes.

Gotchas

1. Diagnosing before containing: spending 30 minutes understanding root cause while users experience the outage is the wrong priority. Contain first.

2. No incident commander: two engineers making independent changes, stepping on each other, creating confusion. One person commands. Everyone else executes.

3. Not communicating during the incident: stakeholders escalating because they have no status update creates noise during the response. Designate a communicator; send updates every 15-30 minutes.

4. Rolling back a rollback: the fix is deployed. Metrics improve. Someone rolls it back without checking — the incident recurs. Confirm recovery before any further changes.

5. Postmortems without action items: "we will be more careful" is not an action item. "We will add an alert for X by date Y, owned by Z" is.

6. On-call without observability: being paged for a problem you cannot diagnose because there are no logs or metrics is an engineering failure. Instrumentation is an on-call responsibility.

Integration

• debugging-methodology — diagnosis phase applies the full debugging loop
• performance-engineering — latency incidents require profiling to identify the bottleneck
• security-engineering — security incidents require immediate containment; postmortem must include disclosure assessment
• system-design — postmortem action items often surface system design improvements

References

• Incident Runbook Template — structured template for writing new runbooks

Skill Metadata

Created: 2026-04-10 Version: 1.0.0