incident-runbook

name	incident-runbook
description	Reads `seo-foundations/reference/runbooks/` and walks responder through diagnose → contain → fix → blameless postmortem. Used exclusively by `incident-response` playbook. Triggered by S1/S2 alerts from seo-health, traffic-monitor, web-vitals, indexation-monitor.
metadata	{"family":"ops","owner":"seo + eng","last_reviewed":"2026-04-29T00:00:00.000Z","version":"1.0.0","related_skills":["seo-foundations","traffic-monitor","seo-health","web-vitals","indexation-monitor","phased-seo-rollout"],"kpis":["Postmortem completion 100% within 48h of S1/S2 resolution","MTTR <4h S1, <24h S2, <1 sprint S3","0 S1/quarter; ≤1 S2/quarter (Standard E SLO)"],"marketing_pillar":4,"seo_standard":"E","kpi_tier":1,"funnel_stage":"all","maturity_stage":"prescriptive","used_by_stages":[]}

Incident Runbook

Operational tool for incident response. Walks responder through severity classification → matched runbook → containment → fix → blameless postmortem.

When to Use

• incident-response playbook — exclusive consumer.
• Triggered by S1/S2 alerts from seo-health, traffic-monitor, web-vitals, indexation-monitor.

Do Not Use When

• Routine weekly monitoring (no incident triggered) — those are weekly ops skills doing their normal job.

Severity Classification

Per .agents/skills/seo/seo-foundations/reference/excellence-targets.md:

S1 — Page-out-loud, immediate response

Examples:

• Pillar noindex regression on a launched pillar
• Sitemap omission on a launched pillar
• Build failure on develop or production deployment branch
• Schema validity drop to <95% on a launched template

Response time: <4h MTTR.

S2 — Urgent, within 24h response

Examples:

• Pillar traffic drop >30% WoW
• p75 LCP regression >500ms WoW
• Indexation drop below 70% on a launched pillar 14d post-launch
• Cannibalization spike to >10%

Response time: <24h MTTR.

S3 — Scheduled within sprint

Examples:

• Page-type drop >15%
• p75 INP regression >50ms
• CTR drop >25% on a tracked-cluster page
• Schema validity drop on a non-launched template

Response time: <1 sprint MTTR.

Workflow

1. Detect: alert from upstream ops skill (seo-health, traffic-monitor, web-vitals, indexation-monitor, or build CI).
2. Classify: match the alert to S1/S2/S3 per the table above.
3. Match runbook: pick from seo-foundations/reference/runbooks/:
   • indexation-regression.md
   • traffic-drop.md
   • schema-breakage.md
   • sitemap-drift.md
   • cwv-regression.md
   • noindex-accident.md
   • pseo-thinning.md
4. Contain: execute the runbook's containment section first (often a revert or hot-fix).
5. Fix: execute the runbook's fix section + add a regression test.
6. Verify recovery: per the runbook's recovery-verification section.
7. Postmortem: write blameless postmortem within 48h (S1/S2) using template below.

Blameless Postmortem Template

Per Etsy / Beyer et al. + 5-whys discipline. File at nextjs-app/docs/postmortems/<YYYY-MM-DD>-<slug>.md.

# Postmortem: <one-line summary>

**Date**: <YYYY-MM-DD>
**Severity**: S1 | S2 | S3
**Status**: Resolved | Mitigated | Open
**Author**: <responder>
**Reviewers**: <names>

## 1. Timeline (UTC, with sources)

| Time | Event | Source |
|---|---|---|
| 14:00 | Detected via `ops/seo-health` weekly run | `_index.json record link` |
| 14:05 | Severity classified S2 | `incident-response gate_id link` |
| 14:30 | Containment deployed (`commit-sha` revert) | `PR link` |
| 15:00 | Recovery verification started | `verification report link` |
| 18:00 | Resolved (re-verified clean) | `resolution link` |

## 2. Impact

- Who: <pillar / page-type / cluster>
- How long: <duration>
- Magnitude: <metric — e.g., "1,200 affected pages, ~8% of total Career Hub traffic for ~4h">
- Visibility loss: <if quantifiable from GSC>
- User-facing impact: <if any>

## 3. Root Cause (5-whys)

1. Why did the regression happen? <answer>
2. Why <answer 1>? <answer>
3. Why <answer 2>? <answer>
4. Why <answer 3>? <answer>
5. Why <answer 4>? <answer — usually a systems/process gap>

## 4. Contributing Factors

- <process gap>
- <ambiguous runbook>
- <missing test>
- <overloaded reviewer>
- <etc>

## 5. What Went Well

- Detection (how fast we caught it)
- Response (how the team coordinated)
- Containment (how quickly we stopped the bleeding)

## 6. What Went Poorly

- Where the response was slow, wrong, or unclear
- Things we wish we'd had

## 7. Action Items

| Owner | Action | Due | Ticket |
|---|---|---|---|
| @SEO-lead | Add regression test for `filterLaunchedEntries` | 2026-05-15 | <link> |
| @perf-lead | Tighten Lighthouse CI gate to fail on >500ms LCP regression | 2026-05-22 | <link> |
| @brand | Update DBA prevalence baseline before enforcing | 2026-05-30 | <link> |

Minimum 1 systemic prevention item.

## 8. Glossary

- <any unfamiliar terms used in the postmortem — useful when revisited later>

---

**No name-and-shame. Focus on systems, not people.**

Output Contract

After every S1/S2 incident:

1. Postmortem file at nextjs-app/docs/postmortems/<YYYY-MM-DD>-<slug>.md within 48h.
2. Memory layer record in _index.json with agent: ops/incident-runbook, triggered_by: incident, gate_id (if any).
3. Risk register update at nextjs-app/docs/RISK_REGISTER.md if the incident reveals a new risk.
4. Slack notification post (no gate) summarizing resolution.

Cross-References

• All runbooks live at .agents/skills/seo/seo-foundations/reference/runbooks/README.md
• DORA MTTR targets in seo-foundations/reference/excellence-targets.md
• Risk register at nextjs-app/docs/RISK_REGISTER.md