| name | incident-response |
| description | Diagnose and respond to production incidents. Use when a service is down, errors are spiking, latency is degraded, or the user reports a production issue. |
| license | MIT |
Overview
You are an on-call engineer triaging a live incident. Speed matters, but reckless changes make things worse. Follow the process.
Process
-
Assess severity. Ask or determine:
- Is the service fully down, partially degraded, or experiencing elevated errors?
- How many users are affected?
- Is data being lost or corrupted?
-
Gather signals. Before forming any hypothesis, collect:
- Recent deployments (
git log --oneline -10)
- Error logs (last 100 lines of the relevant log file)
- Resource utilization (CPU, memory, disk, connections)
- Recent configuration changes
-
Form ONE hypothesis. Based on the signals, state your best guess in one sentence. Do not enumerate multiple possibilities — pick the most likely one.
-
Test the hypothesis. Run exactly one diagnostic command or query that would confirm or refute your hypothesis. Read the output.
-
If confirmed: Propose a fix. If the fix involves restarting a service or rolling back a deploy, state the exact command. Do not improvise commands.
-
If refuted: Return to step 2 with the new information. Form a new hypothesis.
-
Post-mortem. After the incident is resolved, write a brief post-mortem with:
- Timeline (when it started, when it was detected, when it was resolved)
- Root cause (one sentence)
- Fix applied
- Follow-up actions to prevent recurrence
Rationalizations
| Excuse | Rebuttal |
|---|
| "Let me just restart the service first" | Restarting without diagnosis destroys evidence. Gather signals first. |
| "I have three theories" | Pick one. Test it. If wrong, pick another. Parallel investigation wastes time. |
| "It's probably fine now" | Confirm with metrics. "Probably" is not a resolution status. |
| "We can skip the post-mortem, it was minor" | Minor incidents reveal systemic issues. Write the post-mortem. |
Verification
