incident-triage-harness

name	incident-triage-harness
description	Production-style incident triage workflow for logs, metrics, code, and safe mitigations. Use when debugging alerts, regressions, outages, or suspicious runtime behavior.
license	MIT
metadata	{"version":"1.0.0","category":"debugging","sources":["Incident response and SRE triage practice","Evidence-first debugging workflows","Production mitigation and verification patterns"]}

Incident Triage Harness

Investigate production-style failures with an evidence-first loop: confirm symptoms, narrow the blast radius, correlate signals, inspect code, and prove the smallest safe mitigation.

When to Use

• User asks to debug an outage, incident, alert, regression, or mysterious failure
• Logs, traces, deployments, migrations, config drift, or runtime behavior are involved
• You need a structured loop for "hypothesis -> check -> narrow -> mitigate -> verify"

For deeper prompts, evidence templates, and mitigation checklists, also read reference.md in this skill directory.

---

Step 0: Triage The Situation

Before doing anything else, identify:

1. User-visible symptom: What is broken right now?
2. Blast radius: Which route, job, service, or user segment is affected?
3. Recent change surface: Deploys, migrations, feature flags, config changes, dependency bumps
4. Available evidence: logs, traces, dashboards, DB data, repo history, local repro, tests

Do not jump into code edits until you have at least one concrete failure hypothesis.

---

Step 1: Evidence Loop

Use this sequence:

1. Confirm the symptom
2. Correlate recent changes with the failure window
3. Form the smallest plausible hypothesis
4. Check the hypothesis with the strongest available evidence
5. Narrow the cause before changing code
6. Prefer safe mitigation before broad refactors
7. Verify the mitigation at the affected surface

---

Step 2: What To Inspect

Inspect in this order when relevant:

1. User symptom
   • broken page, failing endpoint, timeout, data corruption, auth issue
2. Runtime evidence
   • logs, traces, metrics, console errors, network failures
3. Recent change history
   • deploy timeline, feature flags, migrations, dependency changes
4. Code surface
   • likely entry points, data flow, integration boundaries
5. Persistence layer
   • DB schema, indexes, missing migrations, stale data, queue state

---

Step 3: Mitigation Bias

Prefer the smallest safe mitigation that reduces impact:

• disable or isolate the failing path
• revert a narrow change if the evidence is strong
• add or restore a missing migration or config
• patch one integration boundary rather than refactoring the whole subsystem

Do not claim a root cause until the evidence supports it.

---

Step 4: Closeout Expectations

When reporting back, make clear:

• symptom confirmed
• strongest hypothesis
• evidence checked
• mitigation applied or recommended
• verification run
• remaining unknowns

If the system is safer but the root cause is still incomplete, say so explicitly.