| name | db-incident-triage |
| description | Triage and mitigate a live production database incident — classify latency vs errors vs saturation, rank the suspects (lock contention, replication lag, connection storm, runaway query, disk-full, failover), run the per-mode diagnostic, and apply the least-blast-radius reversible mitigation first. Reach for this when a DB is slow/erroring in prod now. Pairs with zero-downtime-migration when a change caused it. |
Skill: DB incident triage
Diagnose and stop a live database incident under time pressure — safely.
Stabilize before you fully diagnose.
Step 0 — One opinion up front
Least blast radius first. Kill one query before restarting the instance;
throttle one source before failing over. Escalate drasticness only as needed, and
name what each step risks.
Step 1 — Classify
Latency, errors, or saturation? Trace
../../knowledge/dbre-decision-trees.md §4
from the symptom.
Step 2 — Rank the suspects and run the diagnostic
For the likely mode, look before you act (every mode has a view/query):
- Lock contention → inspect blocking/blocked sessions; kill the blocker.
- Replication lag/break → check replica status + lag seconds; reroute reads,
investigate apply/network.
- Connection storm → active/idle vs max connections; find the source; throttle /
pool-limit.
- Runaway query → find the expensive plan; kill it; fix the plan/index.
- Disk-full / read-only → free space, WAL/binlog growth; reclaim, expand.
- Failover event → confirm which node is primary; verify no split-brain.
Step 3 — Mitigate reversibly
Apply the safest action that stops customer harm. Confirm it helped (watch the SLI),
and confirm it didn't shift the problem elsewhere.
Step 4 — Stabilize, then hunt
Once the bleeding stops, do the full diagnosis. Resist the urge to keep changing
things once the incident is contained.
Step 5 — Postmortem
Blameless, timeline-first, contributing factors (not one root cause),
detection/mitigation gaps, and action items owned + dated. Separate the systemic fix
from the band-aid.
Step 6 — Hand off
- An architecture gap the incident exposed →
dbre-architect.
- The durable fix as planned change →
database-operations-engineer /
zero-downtime-migration.
- App-layer cause (bad deploy, N+1, retry storm) →
backend-engineering.
- Security incident →
incident-response-dfir / security-engineering.
Output
A triage path (classification → ranked suspects → per-mode diagnostic), the
least-blast-radius mitigation applied with its risk named, confirmation via the SLI,
and a blameless postmortem with owned action items.
