| name | incident-diagnosis |
| description | Live-system triage — read-only evidence first, establish what/since-when/how-wide, correlate onset with changes, trace one failing request, mitigate reversibly before root-causing. Use when production is broken now, or before restarting/redeploying/deleting as a first move. |
Incident Diagnosis
Code debugging optimizes for finding the cause. Incident work optimizes for restoring service safely — the cause can wait; making things worse cannot. Different discipline, different order.
0. Do not touch anything yet
The strong first move is read-only. Restarts, redeploys, cache flushes, and deletes destroy the evidence AND can compound the failure (a restart storm on an overloaded dependency; a redeploy on top of a bad migration). Every state-changing action needs the evidence to support THAT action specifically — "a restart usually fixes it" is pattern-matching, not evidence.
1. Establish the three anchors
- What exactly is failing, observably? Error message/status/behavior, verbatim — reproduce or observe it yourself if at all possible; second-hand reports compress the detail that matters.
- Since when? Find the onset in logs/metrics/monitors. The onset time is the most valuable single fact you can collect.
- How wide? One user / one endpoint / one region / everything? A single-tenant failure and a global failure have disjoint cause spaces. Check whether it's already publicly acknowledged (status pages, existing incidents) before deep-diving.
2. Correlate the onset with changes
Most incidents are caused by a change. Diff the world at onset time: deploys, config changes, dependency/provider incidents, certificate expiries, cron jobs that fired, data that crossed a threshold (disk full, quota, integer/ID rollover, date boundaries). "Nothing changed" usually means "nothing I know about changed" — widen the search (adjacent teams, provider status, OS/infra auto-updates).
3. Localize by following one failing request
Pick ONE concrete failing request/job and trace it through the layers (edge → LB → service → dependency → data), checking at each hop: did it arrive, what did it return, how long did it take? The failure lives at the first layer where behavior diverges from normal. Resist diagnosing from aggregate dashboards alone — averages hide the failing path.
4. Mitigate first when impact justifies it
If users are hurting and a reversible mitigation exists — rollback, feature-flag off, failover, rate-limit, scale up — propose/apply it BEFORE completing root-cause analysis. Note what evidence to capture pre-mitigation (logs, a core dump, a copy of bad state) so the trail survives. Rollback of the correlated change is the mitigation with the best prior.
Mitigations must be reversible and understood: never "fix" by deleting data or forcing state you can't restore.
5. Then root-cause properly
After service is restored, the investigation becomes ordinary root-cause-debugging (reproduce, bisect, falsifiable hypothesis). An incident closed at "restarted it and it went away" WILL reoccur — say so explicitly if that's where things stand, and record what instrumentation would catch it earlier next time.
Report structure (during and after)
Timeline (times + facts, no speculation mixed in), current impact, what's confirmed vs suspected, actions taken (with before/after evidence), next action. Update as facts land. Never present a hypothesis as a finding — mislabeled confidence misroutes everyone downstream of your report.
