| name | postmortem-writing |
| description | Template and process for writing incident reports (IRs) with causal chain analysis, action items, and mandatory skill/checklist updates |
| version | 1.0.0 |
| author | Diana |
Postmortem Writing
Use this skill when authoring or reviewing an incident report (IR) for the Lorekeeper project. Every incident must produce durable process improvements — not just a fix but encoding the lesson into review patterns and checklists.
When to Use
- A production incident occurs (P0/P1)
- A CI pipeline breaks on
main
- A bug with systemic root cause (not a one-off typo)
- The fix changes architecture patterns, not just code
Mandatory Deliverables
Every IR MUST deliver all of:
| # | Deliverable | Location | Why |
|---|
| 1 | docs/incidents/IR-NNN-description.md | Repo docs/incidents/ | Permanent record |
| 2 | BLOCKER pattern(s) added to lorekeeper-code-reviewer | .hermes/skills/lorekeeper-code-reviewer/SKILL.md | Prevents same bug class passing review |
| 3 | Checklist items added to github-code-review-checklist | Skills directory | Catches in general review too |
| 4 | Engineering docs (if the pattern needs explanation) | docs/engineering/ | Reference for future readers |
| 5 | Action items tracked to completion | IR action items table | Closure |
IR Template
YAML Frontmatter
---
**Status:** [Resolved | In Progress | Monitoring]
**Severity:** [P0/P1/P2]
**Date:** YYYY-MM-DD
**Author:** [Name]
**Fix PR:** #[number]
---
Sections
1. Summary
One paragraph: what happened, why it was bad, what fixed it. Include the key insight in the first sentence.
2. Timeline
Table of events in chronological order: time + event. Include first symptom, investigation start, fix identified, fix merged, recovery confirmed.
3. Root Causes — Causal Chain
This is the most important section. Model the incident as a chain of independent links, where each link is survivable alone but together they create the failure.
Format each link as:
### 🥇 Link N: [Title]
[Technical explanation — what the link is]
**Why this caused today's crash:** [How this specific link contributed]
**Evidence:** [Log lines, metrics, code diffs, process output]
After the chain, show how the fixes break specific links:
| Fix | Which link it breaks | Why it suffices alone |
| ----------------------- | -------------------- | --------------------- |
| `busy_timeout=5000` | Link 2 | ... |
4. Impact
- Duration: [start to end — how long users were affected]
- Scope: [who/what was affected]
- Data integrity: [any data loss?]
- User-facing: [what users experienced]
5. Detection
How the incident was first noticed (user report, alert, dashboard). Include evidence commands (e.g. ps aux, curl checks).
6. Fixes Applied
Group by PR. For each PR, list changes as a table: what changed, file(s), what it does.
Separate emergency fixes (stop the bleeding) from structural fixes (eliminate the class).
7. Action Items
Table: # | Action | Owner | Status
Every action item must be concrete and assigned. No "investigate further" — each item has a clear done condition.
Must include:
- Skill patches (lorekeeper-code-reviewer, github-code-review-checklist)
- Engineering docs if new pattern
- Any deferred items with explicit reason
8. Lessons Learned
Numbered list of principles extracted from this incident. These go into the causal chain analysis section and become future review heuristics.
9. Related
Links to: PRs, commits, related skills, engineering docs, previous IRs.
Causal Chain Technique
The causal chain is the key insight from IR-002. Rules:
- Each link must be independently survivable — if link 1 existed alone, no incident.
- Number links by causal proximity — 🥇 is the first domino, 🥈 is second, etc.
- Provide evidence for each link — logs, code, process output.
- End with a fix/chain-break table — prove each fix independently breaks the chain.
- Include the dormant link — the bug that didn't cause this incident but would cause the next one (e.g. a missing
commit() that passed all tests).
Chain diagram
Optionally include an ASCII causal chain diagram:
[Link 1 crash] ─→ [Link 2 instant error]
│ │
│ └──→ [User-visible symptom]
│
└──→ [Link 3 infinite retries] ─→ [Link 4 zombies amplify]
│
└──→ [Every init hits contention]
Post-Incident Checklist
After the IR is written, verify:
