runbook

name	runbook
description	Structured incident investigation.

Runbook

Structured incident investigation: symptom intake, triage, investigation, resolution, documentation. Every step collects evidence.

Core principle: Collect evidence before forming hypotheses. Do not fix what you have not diagnosed.

When to Use

• Production alert fires and needs investigation
• Service outage or degraded performance
• Error rate spike or new error pattern
• User-reported issue with unclear cause
• Post-deploy regression

Not for: planned maintenance, feature development, or issues with obvious root cause already known.

Process

1. Intake    — Capture symptoms, timeline, blast radius
2. Triage    — Severity assessment, who to notify, immediate mitigation
3. Investigate — Systematic evidence gathering, hypothesis testing
4. Resolve   — Apply fix, verify, monitor
5. Document  — Post-incident report with timeline and learnings

1. Intake

Capture before investigating:

• What — exact symptoms, error messages, affected endpoints
• When — first occurrence, pattern (constant, intermittent, escalating)
• Where — which services, environments, regions
• Who — affected users/systems, scope estimate
• What changed — recent deploys, config changes, infrastructure changes

# Quick checks to establish timeline
git log --oneline --since="2 hours ago"   # recent deploys
# Check monitoring dashboards, alert history
# Check infrastructure change logs

2. Triage

Signal	Severity	Response
Data loss, security breach	Critical	All hands, immediate mitigation, notify stakeholders
Service fully down	High	On-call team, rollback consideration
Degraded performance, partial outage	Medium	Investigate, communicate ETA
Cosmetic, non-blocking	Low	Queue for next business day

Immediate mitigation options (before root cause is known):

• Rollback last deploy
• Scale up / restart affected services
• Enable feature flag / circuit breaker
• Redirect traffic away from affected component

3. Investigate

Systematic, not ad hoc. Work from evidence, not hunches.

Evidence gathering order:

1. Logs around incident start time (correlate across services)
2. Metrics — error rates, latency, resource usage, queue depths
3. Recent changes — deploys, config, infrastructure
4. Dependencies — upstream/downstream service health
5. Reproduce — can you trigger the issue in a controlled way?

Hypothesis testing: Form hypothesis, predict what evidence would confirm or refute, check that evidence. Do not confirmation-bias toward the first theory.

Use arsyn:debugging for deep root cause analysis when the surface investigation identifies the failing component but not the underlying cause.

4. Resolve

• Apply targeted fix (not a workaround unless time-critical)
• Verify fix in production — monitor for 15-30 min
• Confirm affected metrics return to baseline
• Remove any temporary mitigations applied in triage

5. Document

Write post-incident report. See references/incident-template.md for format.

Key sections: timeline, root cause, impact, resolution, action items. The report serves future investigations — be specific about what worked and what did not during diagnosis.

Quick Reference

Phase	Key question	Output
Intake	What exactly is happening?	Symptom summary with timeline
Triage	How bad is it?	Severity level + immediate actions
Investigate	Why is it happening?	Root cause with evidence
Resolve	Is it fixed?	Verified fix + monitoring
Document	What did we learn?	Post-incident report

Integration

• arsyn:debugging — deep root cause tracing when surface investigation is not enough
• arsyn:verifying — verify fix meets requirements before closing
• arsyn:precision — careful execution when applying production fixes

Gotchas

• Fixing before diagnosing — Applying a "likely fix" without confirming root cause often masks the real issue or introduces new ones. Collect evidence first.
• Tunnel vision on first hypothesis — The most obvious explanation is not always correct. Actively seek disconfirming evidence.
• Skipping documentation — "We'll write it up later" means it never gets written. Capture timeline during investigation, not after.
• Incomplete blast radius assessment — Check downstream consumers, async jobs, and cache layers. Visible symptoms are often a subset of actual impact.
• Premature all-clear — Monitor for at least 15 minutes after fix. Some issues resurface under load or on the next cron cycle.
• Not checking what changed — Most production incidents correlate with a recent change. Always check deploy history and config changes early.