incident-response-sre

name	incident-response-sre
description	Drives live incident response and postmortems SRE-style: severity triage (P0–P3), log/metric/trace correlation to find what changed, safe mitigation, comms updates, and blameless postmortem with action items. Triggers during an active incident/outage, on-call triage, or writing a postmortem afterward.

When to Use

• An active incident is happening: errors spiking, latency up, partial/full outage, data not flowing, customers reporting breakage.
• On-call triage: an alert fired and you must decide severity + whether to page humans.
• A service is degraded but not down (elevated error rate, slow tail latency, queue backlog growing).
• After the incident is resolved and you need to write a blameless postmortem.

Do NOT use for routine bug fixing with no live blast radius, planned maintenance, or feature work — use normal debugging instead.

Steps

Phase A — Stabilize first, diagnose second (mitigation before root cause).

1. Declare + triage severity. Pick one and state it explicitly with scope:

   • P0 — full outage / data loss / security breach / money-affecting. Mitigate now, page on-call.
   • P1 — major feature down or broad degradation, no full workaround.
   • P2 — partial degradation, workaround exists, limited users.
   • P3 — minor / cosmetic / single-user, no urgency. Write impact in user terms: what is broken, who/how many affected, since when. If you can't quantify yet, say "scope unknown, treating as P1 until proven smaller" — never downgrade on a guess.

2. Build the "what changed" timeline. Most incidents are caused by a recent change. List, with timestamps, the last 24–48h of:

   • deploys / releases / rollbacks (check CI/CD history, git log, image tags)
   • config / feature-flag / env-var changes
   • infra changes (scaling, DB migration, cert rotation, DNS, dependency upgrade)
   • traffic shifts (spike, new client, retry storm, upstream outage) Overlay the incident start time against this. The change immediately preceding the symptom onset is your prime suspect.

3. Correlate signals — logs + metrics + traces together, not in isolation.

   • Metrics tell you where/when (which service, which endpoint, error-rate vs latency vs saturation — the golden signals). Find the first metric that broke and its exact start time.
   • Logs tell you what (the actual error string, stack trace, status code). Grep the failing service around the metric break time; read the FIRST errors, not the loudest.
   • Traces tell you which hop in the request path failed (DB? upstream dep? timeout? auth?). Pin all three to the same time window. A latency spike + DB connection-pool-exhausted logs + traces stalling at the DB call = one coherent story.

4. Hypothesis-driven correlation — don't oversimplify. Form 2–3 candidate root causes from the timeline + signals. For each, state the one observation that would confirm or kill it, then check that observation. Beware: correlation ≠ cause (a deploy at the same time may be innocent); the loudest error may be a downstream symptom, not the source. Keep going until the signal chain is coherent end-to-end — stop only when remaining hypotheses are killed.

5. Mitigate with the safest reversible action FIRST — before any permanent fix. Prefer, in order: roll back the suspect deploy/config → disable the suspect feature flag → scale up / add capacity → shed load / rate-limit / shut a non-critical path → failover. Reversible mitigation is allowed even before root cause is 100% confirmed if it's safe and likely to help. Never apply an irreversible action (drop/delete data, force-push, destructive migration) as a mitigation — escalate to a human first. Confirm recovery via the same metric that broke in step 3, not by assumption.

6. Post status comms. Short, factual, no blame, on a cadence (e.g. every 30 min while P0/P1). Each update: current impact → what's being done → next update time. On resolution: "mitigated/resolved at , root cause <known/under investigation>, monitoring." State unknowns plainly — don't speculate publicly.

Phase B — After recovery: blameless postmortem.

7. Write the postmortem (only after the incident is mitigated/closed). Required sections:

   • Summary — one paragraph: what broke, impact, duration.
   • Timeline — UTC timestamps from detection → mitigation → resolution, including key diagnostic steps.
   • Impact — quantified (users, requests, $, SLO/error-budget burn).
   • Root cause + contributing factors — the trigger AND the conditions that let it become an incident (missing alert, no rollback path, silent dependency). Multiple factors usually, not one.
   • What went well / what was slow — detection time, mitigation time, gaps in tooling/visibility.
   • Action items — each is concrete, owned, dated, and prioritized (P0/P1...). Distinguish "prevent recurrence" from "detect faster" from "mitigate faster". No vague "be more careful".

8. Blameless language. Attack the system, never the person. "The deploy step had no automated canary check" not "X deployed bad code". People act reasonably given the info they had; if a human could cause an outage with a normal action, the system lacked a guardrail.

9. Feed back into a runbook. If this incident class can recur, capture the detection signal → mitigation steps → verification as a reusable runbook so next time is faster. Convert the highest-value action item into a guardrail (alert, canary, rollback automation, flag).

Common Errors

• Chasing root cause while the system burns. Mitigation (rollback/flag-off) comes before diagnosis. A reverted deploy that buys 30 min is worth more than the perfect RCA mid-outage.
• Trusting the loudest error. The error filling the logs is often a downstream symptom (cascading timeouts, retry amplification). Trace upstream to the first failure in time, not the most frequent.
• "Deploy at the same time = the cause." Temporal coincidence is a hypothesis, not proof. A config change, traffic spike, or expiring cert can hide behind an innocent deploy. Verify the causal link.
• Premature severity downgrade. "Probably just a few users" → it's the whole region. When scope is unknown, hold the higher severity until you've measured it smaller.
• Irreversible "fix" under pressure. Deleting rows / dropping a table / force-pushing / running a destructive migration to "clear the bad state" turns a recoverable outage into permanent data loss. Escalate to a human before anything irreversible.
• Looking at one signal only. Metrics-only = you see the symptom but not why. Logs-only = you miss the blast radius. Traces-only = you miss the trend. The story lives at the intersection, pinned to one time window.
• Blameful postmortem. Naming who-did-it makes people hide info next time and kills the learning. The action item is a missing guardrail, never a missing person.
• Action items with no owner/date. "Improve monitoring" never ships. Each item needs a name, a date, and a priority, or it's decoration.
• Declaring "fixed" without watching recovery. Mitigation applied ≠ recovered. Confirm on the exact metric that originally broke before standing down.

Verify

• During incident: the metric that first broke (step 3) is back to baseline AND held there for a sustained window — not a momentary dip. Error rate / latency / saturation confirmed normal on the dashboard, not assumed.
• Mitigation safety: every action taken was reversible, or a human explicitly approved any irreversible one. You can state how to undo each.
• Severity correct in hindsight: measured impact matches (or was higher than) the declared severity — you didn't under-call it.
• Postmortem complete: all required sections present; root cause has ≥1 contributing factor beyond the trigger; every action item has owner + date + priority; language is blameless (no person named as cause).
• Loop closed: at least one action item became a real guardrail (alert/canary/rollback/flag) or a runbook so the same incident is faster/prevented next time.