| name | incident-response |
| description | Structured procedure for investigating and resolving production incidents. Use when a service or infrastructure component is down or degraded, when alerts fire for critical thresholds (disk, CPU, connectivity), or when a user reports a production issue. |
| license | MIT |
Incident Response
Structured procedure for investigating and resolving production incidents.
When to Use
Live production degradation or outage — the on-call path: stabilise,
mitigate, communicate, then root-cause. A non-urgent code bug with no
running-system pressure is systematic-debugging instead. During an incident
you'll often use the observability signals (observability-setup) and reach
for rollback-plan to revert the triggering change.
Procedure
- Triage: Assess severity and blast radius. Who/what is affected?
- Diagnose: Verify Network → Process → Logs → Gateway. Follow the dependency chain.
- Mitigate: Apply the fastest safe fix to restore service. Permanent fix can follow.
- Communicate: Keep stakeholders informed of status and ETA.
- Document: Record timeline, root cause, and fix in
<ai_hats_dir>/sessions/retros/. Include what worked and what didn't.
- Follow up: Create tasks for permanent fix, monitoring gaps, and process improvements.
Completion
- Service restored and verified via health checks
- Incident report written in
<ai_hats_dir>/sessions/retros/
- Follow-up tasks created in backlog
Anti-Patterns
- Jumping to fix without diagnosis — leads to wrong fixes and longer outages
- Skipping communication — stakeholders assume the worst
- No follow-up tasks — same incident will repeat
