incident-commander

name	incident-commander
description	Active incident response protocol for production outages. Provides severity classification, communication templates, rollback decision trees, and post-incident report generation. Distinguishes between incident response (during) and postmortem (after). Enforces structured command-and-control to prevent panic-driven fixes.

When To Use

• Alert fires: P99 latency > threshold, error rate spike, disk full, memory OOM.
• User reports: "the site is down", "payments are failing", "login is broken."
• Deployment correlates with a metric anomaly (within 15 minutes).
• Agent is asked to "fix production" or "investigate the outage."

Related: /incident-postmortem for after-action review; /rollback-plan for pre-planned rollback procedures; /investigate for non-urgent debugging.

Do not use this skill for minor bugs, pre-production issues, or planned maintenance. This is for active production incidents only.

Core Stance

• When production is down, the instinct to fix it immediately kills. It leads to fixes that make things worse, loss of forensic evidence, communication chaos, and burnout.
• Incident response is a military operation: there is a commander, there are phases, there are rules of engagement, and there is a defined end state.
• Evidence preservation comes before fixing. Communication comes before debugging. Rollback comes before hotfix.

Research Backing

• Allspaw & Robbins (2010), Etsy's Debriefing Facilitation Guide — Blameless post-incident process and structured communication during outages are critical to learning and reliability.
• Dekker (2011), Drift into Failure — Complex systems fail through normal work, not single causes; preserving evidence and resisting panic-driven fixes are essential.
• Google SRE (2016), Site Reliability Engineering — Severity classification, commander role, and structured response protocols reduce MTTR and prevent secondary incidents.

Process

Phase 1: Triage (First 2 Minutes)

Do not attempt a fix. Classify severity:

• SEV1: Complete outage, revenue-critical path down, data loss in progress.
• SEV2: Major feature degraded, workaround exists but is painful.
• SEV3: Minor feature broken, no workaround needed, low user impact.
• SEV4: Cosmetic, monitoring gap, no user impact.

Determine blast radius (which users, regions, data), trigger (last deployment, config change, traffic spike, dependency failure), and notify on-call channel: SEV[X] | [System] | [Symptom] | [Trigger suspected] | Commander: [name/agent].

Phase 2: Evidence Preservation (Minutes 2–5)

Before touching anything: capture error logs (last 100 lines), snapshot DB metrics (active connections, replication lag, slow queries), snapshot app metrics (memory, CPU, request queue depth), note deployment SHA (git rev-parse HEAD). If data corruption suspected, snapshot affected data (read-only dump) before any write.

Document in INCIDENT_YYYYMMDD_HHMM.md: severity, blast radius, trigger, evidence links.

Phase 3: Decision Tree (Minutes 5–10)

Choose ONE path based on severity and trigger:

Path A: Rollback (Preferred for SEV1/SEV2)

• Condition: Incident started within 30 minutes of deploy or config change.
• Action: Rollback to previous stable version. Do not debug the new version.
• Verify: Confirm rollback SHA is last known good deployment.
• Risk: Data written since deploy may be incompatible — check migration reversibility.

Path B: Mitigation (If no recent deploy)

• Condition: Infrastructure failure, dependency outage, traffic spike.
• Action: Scale up replicas, enable circuit breaker, redirect traffic to healthy region, disable failing feature via flag.
• Verify: Confirm mitigation reduced error rate within 5 minutes.

Path C: Hotfix (Last Resort)

• Condition: Rollback impossible (irreversible migration), mitigation ineffective, root cause known with high confidence.
• Action: Minimal fix, deploy via emergency pipeline (keeps security scan, skips non-required checks).
• Requirement: Two-person rule — another engineer reviews the fix before deploy.

Phase 4: Communication Rhythm

• SEV1: Update every 5 minutes in incident channel.
• SEV2: Update every 15 minutes.
• SEV3/4: Update every 30 minutes or at resolution.
• Template: [HH:MM] Status: [investigating/identified/mitigating/monitoring/resolved]. Action: [current step]. ETA: [time or unknown].
• Stakeholder update: every 30 minutes for SEV1, plain language, no jargon.

Phase 5: Monitoring (15 Minutes)

After error rate returns to baseline: do not declare resolved immediately. Watch for secondary failures (cascading effects, cache stampede, DB connection pool exhaustion). If metrics stay clean for 15 minutes, declare: RESOLVED | [time] | Monitoring complete. If metrics degrade, re-classify and restart.

Phase 6: Handoff to Postmortem

Append timeline of every action, why each path was chosen, data impact assessment, customer communication sent. Schedule postmortem within 24 hours (SEV1) or 48 hours (SEV2). Trigger /incident-postmortem.

Operating Rules

• If incident open >1 hour, rotate commander. Fresh eyes prevent tunnel vision.
• If fix attempt fails twice, escalate to Path A (rollback) or Path B (mitigation). No third fix attempt.
• If data corruption is possible, stop all writes to affected store until data engineer assesses.
• Never declare resolved without 15-minute clean monitoring window.
• No uncommitted hotfix code remains in production — merge to main via PR after resolution.

Output Format

Primary artifact: INCIDENT_YYYYMMDD_HHMM.md containing triage assessment, evidence links, decision tree path chosen, timeline of actions, resolution declaration, and postmortem scheduling.

Example

Alert: P99 latency 4500ms (baseline: 200ms). Deploy v1.2.3 at 14:32 UTC.

Agent: "SEV1 | User API | P99 4500ms | Deploy v1.2.3 at 14:32 | Commander: agent. Triage: all regions affected. Evidence preserved: logs + DB metrics + SHA. Trigger: deploy within 30min. Decision: Path A — Rollback to v1.1.9. Rollback initiated."