incident-response

## Overview Incidents are inevitable. How you respond determines whether they are learning opportunities or recurring nightmares. This skill gives you a structured process from first alert to prevention of recurrence. ## When to Use - When an alert fires indicating user impact - When a deployment causes degraded service - When you discover a data issue in production - After any unplanned outage, even brief ## Process ### Phase 1: DETECT (0–5 minutes) **Step 1: Acknowledge the alert** Acknowledge in your alerting system within 5 minutes. This signals you're on it and stops duplicate escalations. **Step 2: Assess severity** - **P0** — Complete service outage, data loss, security breach. Page everyone. - **P1** — Significant degradation affecting many users. Page the team. - **P2** — Partial degradation, workaround exists. Handle during business hours. - **P3** — Minor issue, users unaffected or workaround obvious. Normal ticket. **Step 3: Declare the incident and open a channel** For P0/P1: open an incident channel (#incident-YYYY-MM-DD-brief-description). All communication goes there. Appoint an Incident Commander (IC). ### Phase 2: COMMUNICATE (5–15 minutes) **Step 4: Post a status update** Within 15 minutes of detection, post to the status page: "We are investigating reports of [symptom]. We will provide an update in 30 minutes." Do not wait until you have a fix. Users need to know you know. **Step 5: Brief stakeholders** Notify: on-call manager, customer-facing teams, affected customers (for P0/P1). Use the incident channel as the single source of truth. ### Phase 3: MITIGATE (15 minutes – resolution) **Step 6: Stop the bleeding first** Mitigation before diagnosis. Options: - Roll back the last deployment - Disable the feature flag - Scale up capacity - Redirect traffic away from the failing component Don't spend time diagnosing the root cause while users are impacted. Mitigate first. **Step 7: Diagnose** Once impact is reduced, find the root cause: - What changed recently? (Deployments, config changes, traffic spikes) - What do the metrics show? (Error rate, latency, saturation) - What do the logs show? (First error, error pattern, affected user IDs) - What do the traces show? (Where in the call chain is the failure?) **Step 8: Implement the fix** Implement the minimal fix. This is not the time for refactoring. ### Phase 4: RESOLVE (at mitigation/fix) **Step 9: Confirm resolution** Verify: error rate back to baseline, latency back to baseline, no new error types in logs, affected users can complete their tasks. **Step 10: Update status page** Post: "The issue has been resolved. Affected users can [action]. We will publish a post-mortem within 48 hours." ### Phase 5: LEARN (within 48 hours) **Step 11: Write the post-mortem** Structure: - **Summary**: what happened, duration, user impact - **Timeline**: minute-by-minute from first symptom to resolution - **Root cause**: the underlying cause (not just the immediate trigger) - **Contributing factors**: what made this worse or harder to detect - **What went well**: don't only document failures - **Action items**: specific tasks with owners and due dates **Step 12: Blameless post-mortem** Focus on systems and processes, not individuals. "The deployment pipeline allowed a bad config to reach production" not "Alice pushed bad config." ## Anti-Rationalizations **"We'll communicate once we have a fix"** Users need to know you know. An hour of silence is worse than an honest "we're investigating." **"We know the root cause — no need for a post-mortem"** Post-mortems are not just for finding root causes. They identify contributing factors, test your detection and response, and create shared understanding. Skip them and the same incident recurs. ## Verification Requirements - [ ] Incident acknowledged within 5 minutes of alert - [ ] Severity assessed and appropriate people notified - [ ] Status page updated within 15 minutes - [ ] Mitigation performed before full diagnosis - [ ] Resolution confirmed with metrics, not just a fix deployment - [ ] Post-mortem written within 48 hours - [ ] Action items have owners and due dates