| name | incident-triage |
| description | Use during a production incident, when an alert is firing, a service is down, or when writing a postmortem. Provides a structured triage checklist, timeline capture format, and postmortem template for this project.
|
| license | MIT |
Incident triage — structured response
Severity levels
| Sev | Definition | Response time |
|---|
| Sev-1 | Production down, data loss risk, SLA breach | Immediate |
| Sev-2 | Degraded production, no data loss | < 30 min |
| Sev-3 | Non-critical feature affected | Next business day |
Triage checklist (run in order)
- Establish impact: which users/regions affected? What percentage of traffic?
- Check dashboards: Datadog service map → which service shows red?
- Check recent deploys:
helm history <service> -n <namespace> — did a deploy cause this?
- Check logs: Splunk/Datadog logs filtered to the affected service + last 30 min
- Communicate: post in #incidents with: service, impact, start time, who's investigating
Common first-response commands
kubectl get pods -n production -l app=<service>
kubectl logs -l app=<service> -n production --tail=200 --since=30m
helm history <service> -n production
git log --oneline main..HEAD
helm rollback <service> 0 -n production
Timeline capture (maintain during incident)
HH:MM - [event description]
HH:MM - [what was checked / found]
HH:MM - [action taken]
HH:MM - [result of action]
Postmortem template
## Incident postmortem: <title>
**Date**: YYYY-MM-DD
**Duration**: HH:MM to HH:MM (X minutes)
**Severity**: Sev-N
**Affected**: <what/who was impacted>
### Timeline
[paste timeline from above]
### Root cause
<What actually caused the incident — not symptoms, the root cause>
### Contributing factors
<What made the root cause possible or worse>
### Resolution
<What fixed it>
### Action items
| Item | Owner | Due date |
|------|-------|----------|
| <preventive action> | @person | YYYY-MM-DD |
