| name | quality-postmortem |
| description | Analyze escaped defects and test suite health through blameless postmortems. Covers bug pattern analysis, test suite health reviews, 5 Whys root cause analysis, process improvement cycles, and postmortem/retro meeting templates with action item tracking. Use when: "QA retro," "escaped bugs," "postmortem," "quality incident," "defect analysis," "improvement cycle." Not for: live release go/no-go decisions — use release-readiness. Not for ongoing metric dashboards — use qa-metrics. Not for reviewing existing test code quality — use ai-qa-review. Related: qa-metrics, test-reliability, test-strategy. |
| license | MIT |
| metadata | {"author":"kindlmann","version":"2.0","category":"process"} |
Analyze escaped defects, test suite health, and quality process gaps through structured, blameless postmortems. Every postmortem produces 1-3 concrete, tracked action items -- not vague commitments to "be more careful." The goal is systemic improvement, not individual blame.
Quick Route
| You have... | Go to | Template |
|---|
| One escaped bug to dissect | Bug Pattern Analysis | Escaped Bug Analysis (references/templates.md) |
| 10+ escaped bugs, looking for themes | Aggregating Patterns Over Time | — |
| A proactive quarterly check (no incident) | Test Suite Health Review | — |
| A P0/P1 production incident | Postmortem Template for Quality Incidents | references/templates.md |
| A recurring sprint/monthly review | Retro Meeting Template | references/templates.md |
Discovery Questions
Check .agents/qa-project-context.md in the project root first — it carries quality goals, risk areas, and test suite details that anchor any postmortem. Use it and skip anything already answered there. Then clarify:
-
Do you have a regular retro cadence? Per-sprint, monthly, or only after incidents? Regular cadence catches slow-burn problems. Incident-only cadence misses patterns until they explode.
-
What triggered this postmortem? A production incident? A pattern of escaped bugs? A feeling that the test suite is not catching enough? Test suite degradation? The trigger determines the focus.
-
What data is available? Bug tracker with severity and discovery phase? CI history with pass rates? Flaky test reports? Coverage trends? Without data, postmortems devolve into opinion sessions.
-
What happened with previous postmortem action items? Were they completed? Tracked? Forgotten? If past action items are abandoned, the team has learned that postmortems do not matter. Fix the follow-through before running another postmortem.
-
Who should participate? Engineers who worked on the affected area. QA who tested (or did not test) it. Product owner if the impact was user-facing. Engineering manager if systemic changes are needed. Keep the group to 4-8 people.
-
What are the current test suite health concerns? Rising flakiness? Slow execution? Coverage gaps in critical areas? Stale quarantine? Health reviews are proactive postmortems -- they prevent incidents instead of reacting to them.
Core Principles
1. Blameless Means Systemic
Blameless does not mean "no one is accountable." It means the analysis focuses on systems, processes, and tools rather than individual performance. "Why did the system allow this defect to escape?" is a blameless question. "Why did the developer not write a test?" is a blame question that stops the analysis too early. The developer did not write a test because: the test framework was hard to use, the PR checklist did not require it, there was no pairing to transfer knowledge, or time pressure made it feel optional. Those are systemic issues with systemic fixes.
2. Focus on Patterns, Not Incidents
A single escaped bug is an anecdote. Three escaped bugs in the same feature area over two months is a pattern. Postmortems should aggregate incidents to find recurring themes: same root cause, same team, same test gap, same phase of the pipeline. Patterns are actionable. Individual incidents are just fire-fighting.
3. Every Postmortem = 1-3 Concrete Action Items
An action item is concrete when it has: a specific deliverable ("add integration tests for the coupon API"), an owner ("assigned to Alex"), a deadline ("by end of sprint 14"), and a verification method ("PR merged, tests passing in CI"). "Improve testing" is not an action item. "Write 5 integration tests for the payment service edge cases by March 30" is.
4. Track to Completion
Action items that are not tracked are not completed. Use the team's existing work tracker (Jira, Linear, GitHub Issues). Tag them (postmortem-action or equivalent). Review completion status at the start of the next postmortem. If items are consistently abandoned, either the items are too large (break them down) or they are not prioritized (make them sprint commitments).
5. Measure Improvement With Two Metrics, Not One
After implementing action items, measure whether the problem recurred. If the postmortem identified a gap in payment testing and the action was to add integration tests, track: did another payment bug escape? Without measurement, postmortems are rituals, not tools.
Track two metrics together:
- Defect escape rate (did similar bugs reappear?)
- Action-item-closure rate (what fraction of action items shipped within their committed window?)
A high closure rate with rising escape rate means the team is doing the work but doing the wrong work. A low closure rate means the postmortems are theater. Modern incident response platforms (incident.io, Rootly, FireHydrant) track action-item follow-through natively — owner, due date, completion status — so derive both numbers from what's already there before building a dashboard.
6. AI Drafts the Timeline; a Human Owns the Judgment
If your team uses AI SRE tooling (Rootly AI SRE, incident.io's AI SRE / auto-drafted post-mortems), let it draft the incident timeline and propose candidate root causes from logs and traces. Then a named blameless RCA owner — distinct from the incident commander who managed the response — runs the 5 Whys, picks the real root cause, and writes the action items. AI is good at correlation across noisy data; it is bad at deciding what mattered. Treat AI output as a starting deck, not the conclusion. For cheap timeline drafting, Sonnet 4.6 is sufficient; reserve heavier models for ambiguous causation.
Bug Pattern Analysis
Categorizing Escaped Defects
When a bug reaches production, classify it along three dimensions to identify prevention opportunities. The single-bug worksheet (Escaped Bug Analysis) lives in references/templates.md.
Dimension 1: Root Cause Category
| Category | Description | Example |
|---|
| Logic error | Business logic incorrect or incomplete | Discount not applied for edge case currency |
| Integration failure | Two components do not communicate correctly | API returns different format than frontend expects |
| Data issue | Unexpected data shape, null values, encoding | User with emoji in name breaks CSV export |
| Race condition | Timing-dependent behavior | Two concurrent checkouts oversell last item |
| Configuration | Environment-specific settings wrong | Feature flag enabled in staging, disabled in prod |
| Regression | Previously working behavior broken | Refactor removed null check, old bug returns |
| Missing requirement | Behavior not specified, gap in product spec | No error handling for expired OAuth tokens |
| Performance | Functional but too slow under load | Search timeout with 100K+ records |
Dimension 2: Which Test Level Should Have Caught It
| Level | What it catches | If it escaped this level |
|---|
| Unit | Logic errors, edge cases, boundary conditions | Tests exist but missing edge case? Or no tests at all? |
| Integration | API contracts, data flow, service interactions | Integration tests exist? Do they cover error responses? |
| E2E | User journey failures, UI state management | Is this critical path covered? Was the specific scenario tested? |
| Manual/Exploratory | Visual issues, usability problems, unusual workflows | Was exploratory testing performed? Was the area in scope? |
| Monitoring | Performance degradation, error rate spikes | Are alerts configured? Are thresholds correct? |
Dimension 3: Prevention Opportunity
| Opportunity | Action | Example |
|---|
| Add test | Write a test at the appropriate level | Add unit test for currency rounding edge case |
| Improve existing test | Existing test was too narrow | Extend checkout E2E to include coupon + international currency |
| Add quality gate | CI check would have caught it | Add schema validation for API responses in CI |
| Improve requirements | Spec was ambiguous or incomplete | Add acceptance criteria for error states to story template |
| Add monitoring | Detect sooner even if not prevented | Add alert for error rate > 1% on payment endpoint |
| Training/Process | Knowledge gap or process gap | Run a session on defensive coding for nullable fields |
Aggregating Patterns Over Time
After analyzing 10+ escaped bugs, look for patterns:
Escaped Bug Summary: [Q1 2026]
═══════════════════════════════
Total escaped bugs: 14
By root cause:
Logic error: 5 (36%) ← unit tests needed
Integration failure: 4 (29%) ← API contract tests needed
Data issue: 3 (21%) ← input validation gaps
Configuration: 2 (14%) ← env parity issues
By area:
Checkout: 6 (43%) ← highest risk, needs investment
User management: 4 (29%)
Reporting: 2 (14%)
Settings: 2 (14%)
By should-catch level:
Unit: 5 (36%) ← developers not testing edge cases
Integration: 4 (29%) ← missing integration test layer
E2E: 3 (21%)
Monitoring: 2 (14%)
Top action themes:
1. Add integration tests for checkout API (covers 4 of 14 bugs)
2. Mandate unit tests for all calculation/validation logic (covers 5 of 14)
3. Add currency and encoding edge cases to test data fixtures (covers 3 of 14)
This aggregation reveals where investment has the highest return: fixing one systemic issue (integration tests for checkout) would have prevented 29% of all escaped bugs. Patterns that recur across multiple quarters belong in the test-strategy doc, not just the next sprint's action items — promote them so the strategy reflects where defects actually escape.
Test Suite Health Review
A proactive postmortem for the test suite itself. Conduct quarterly or when symptoms appear.
Flaky Test Trends
Flaky Test Trend Review
═══════════════════════
Current flaky rate: _____ % (target: <2%)
Trend (last 3 months):
Month 1: _____ %
Month 2: _____ %
Month 3: _____ %
Direction: [ ] Improving [ ] Stable [ ] Worsening
Top 5 flakiest tests (by failure count):
1. _____________________ — _____ failures — root cause: _____
2. _____________________ — _____ failures — root cause: _____
3. _____________________ — _____ failures — root cause: _____
4. _____________________ — _____ failures — root cause: _____
5. _____________________ — _____ failures — root cause: _____
Quarantine:
Tests in quarantine: _____ count
Oldest quarantine: _____ days (target: <14)
Quarantine resolved this month: _____ count
Execution Time Trend
Track current full suite duration, 3-month trend, and the 5 slowest tests. If duration is increasing, check for: tests that can move to nightly, sequential stages that can parallelize, slow test data setup (use API instead of UI), large test files that need splitting for better shard distribution.
Coverage Gap Review
Track overall coverage (lines/branches), critical paths with insufficient coverage (payments, auth, data export should be 90%+), recently changed code without test updates (cross-reference git log --since="30 days ago" with the coverage report), and features shipped without E2E coverage.
Disabled/Skipped Test Inventory
Audit all skipped/disabled tests by age and reason. Tests skipped < 1 week are likely in progress. Tests skipped 1-4 weeks need a ticket and timeline. Tests skipped 1-3 months are overdue -- fix or delete. Tests skipped > 3 months should be deleted -- they will never be fixed. For each: fix and unskip, delete (obsolete), or move to quarantine with a ticket link.
Process Improvement Cycles
The Improvement Sprint
Dedicate a fixed portion of each sprint (10-15% of capacity) to quality improvement, drawn from postmortem action items and health review findings.
Structure:
1. IDENTIFY — Top 3 pain points from latest retro/postmortem
2. ROOT CAUSE — 5 Whys analysis for the #1 pain point
3. PROPOSE — Solution with effort estimate (S/M/L)
4. IMPLEMENT — One improvement per sprint (start small)
5. MEASURE — Did the metric improve? By how much?
6. ITERATE — If not improved, dig deeper. If improved, tackle #2.
5 Whys Root Cause Analysis
The 5 Whys technique peels back surface symptoms to reveal systemic causes. The key discipline: keep asking "why" until you reach a process, system, or structural cause -- not an individual's action.
Example: Payment bug escaped to production
Problem: Users were charged twice for a single purchase.
Why 1: The payment API was called twice on form submit.
Why 2: The submit button was not disabled after the first click.
Why 3: The frontend developer did not implement button disabling.
Why 4: The acceptance criteria did not mention double-submit prevention.
Why 5: The story refinement process does not include edge case review
for payment-related stories.
Root cause: Process gap — payment stories are not reviewed for transaction
safety edge cases before development begins.
Action: Add a "Payment Safety Checklist" to the story template for any
story touching payment flows. Checklist includes: idempotency,
double-submit prevention, partial failure handling, timeout behavior.
Owner: [Product Manager] — Due: [Next sprint]
5 Whys guidelines:
- Stop when you reach something the team can change (process, tool, structure). Asking "why is the budget limited?" goes too far.
- The chain may branch -- one symptom may have multiple contributing causes. Follow the most impactful branch.
- Verify each "why" with evidence, not assumption. "The developer did not write tests" -- is that true? Check the PR. Maybe tests existed but were insufficient.
- If you reach "human error" as a root cause, you have not gone far enough. Humans make errors. The system should make errors difficult or detectable.
Proposing Solutions with Effort Estimates
For each root cause, propose 1-3 solutions at different effort levels. Example for a recurring flaky-test problem:
Root Cause: E2E tests fail intermittently on async-loaded content
Solution A (Small — 1 day):
Replace fixed waitForTimeout calls with explicit wait-for-condition
assertions in the 5 flakiest specs.
+ Quick to implement, kills the most common flake source
− Manual, one spec at a time; new flakes can creep back in
Solution B (Medium — 1 sprint):
Solution A across the suite + add a flaky-test detector to CI that
reruns failures once and tags any test that passes on retry.
+ Automated detection, surfaces flakes before they erode trust
− Requires CI config change; reruns add pipeline time
Solution C (Large — 2 sprints):
Solution B + auto-quarantine tagged tests and route them to an
owner-assigned backlog with a 14-day fix-or-delete SLA.
+ Self-healing trust in the green build; flakes can't block releases silently
− Needs quarantine infrastructure and ownership process buy-in
Recommendation: Start with A immediately, implement B this sprint,
plan C for next quarter as strategic work.
Postmortem & Retro Templates
Two heavy, copy-paste formats live in references/templates.md:
- Postmortem Template for Quality Incidents — for P0/P1 production bugs, data loss, security issues, or outages from a code change. Summary, severity/impact, UTC timeline table, root cause, 5 Whys, what tests existed / were missing, detection, immediate/short-term/long-term action tables, lessons learned.
- Retro Meeting Template — for recurring sprint/monthly quality retros: a 7-section, 30-60 minute agenda (previous action item review → data review → went well → needs improvement → root cause discussion → new action items → close) plus facilitator notes.
Both open by reviewing the previous retro's action items — that closed loop is the accountability mechanism; without it, items vanish silently.
Anti-Patterns
Blame-Driven Postmortems
Focusing on who made the mistake rather than what system allowed the mistake to reach production. Blame creates fear. Fear creates hiding. Hiding creates bigger incidents. When the question is "who wrote this bug?" people learn to avoid visibility. When the question is "what process gap allowed this?" people learn to improve the process.
Postmortems Without Action Items
A cathartic discussion that produces understanding but no change. If the meeting ends without specific, assigned action items, the same problem will recur. Worse, the team learns that postmortems are therapy sessions, not improvement tools.
Action Items Without Follow-Through
Generating action items that go into a backlog and are never prioritized. This is worse than no action items because it creates the illusion of improvement. If postmortem actions are not completed within 2 sprints, escalate. Action items die for predictable reasons — audit your closure rate against this checklist before blaming "we forgot":
- No owner. Items assigned to a team rather than a person become nobody's job. Fix: assign to a named person with enough context to start.
- No due date. "Soon" is not a date. Fix: a specific sprint or calendar date.
- Scope too big. "Refactor the test framework" cannot land in a sprint. Fix: break it into items that each fit a single PR.
- No review at the start of the next retro. Without a forced check-in, items vanish silently. Fix: a standing calendar slot that opens every retro with a closed-loop review.
- No metric attached. If completing the item doesn't move a number you can name, you can't tell whether it worked.
Counter-pattern: open every retro with a 5-minute "previous action items" review. Mark each as Done / In Progress (with current ETA) / Dropped (with reason).
Postmortems Only After Incidents
Waiting for a production fire to conduct a quality review. Proactive health reviews (test suite health, coverage trends, flaky test inventory) prevent incidents. Conduct proactive reviews monthly. Reactive incident postmortems supplement the proactive cadence — they do not replace it.
Root Cause Analysis That Stops Too Early
"The developer did not write a test" is not a root cause. It is a symptom. Why did they not write a test? Was the framework hard to use? Was there no time? Was there no requirement? Was there no pairing or review? Stopping at the individual level prevents systemic improvement.
Vague Action Items
"Improve test coverage" and "be more careful with deployments" are not action items. They cannot be tracked, measured, or verified. Compare: "Add integration tests for payment webhook handling, covering success, failure, and timeout scenarios. Owner: Alex. Due: Sprint 14. Verification: PR merged with 3 new integration tests passing in CI."
Data-Free Retros
Running quality retrospectives based on feelings and opinions rather than data. "It feels like we have more bugs lately" might be true or might be recency bias. Check the data: is the escaped bug count actually increasing? Where are the bugs concentrated? Without data, the team solves the loudest problem, not the most important one.
Verification
The artifact is the written postmortem plus its tracked, closed-loop action items. Prove it landed — smallest check first.
gh issue list --label postmortem-action --json number,title,assignees,milestone \
| jq '[.[] | select(.assignees == [] or .milestone == null)]'
git log --since="<introduced-date>" --until="<detected-date>" --oneline -- <affected/path>
git log --grep="<INCIDENT-ID>" --oneline
<your test runner> <new regression test path>
Then confirm by reading: the 5 Whys ends on a process/tool/structure cause (not "developer didn't write a test"), and both the action-item-closure rate and the escaped-defect rate are recorded — not just one.
Done When
- Escaped defect timeline reconstructed (introduced, released, detected, resolved) with supporting evidence from commit history and bug tracker.
- 5 Whys root cause analysis completed and stopped at a systemic cause (process / tool / structure), not at "developer didn't write a test."
- Test gap identified and mapped to a specific coverage hole (missing test type, missing scenario, or missing area).
- Action items assigned with named owners and due dates, added to the team's work tracker with a postmortem tag.
- Findings shared with the team in a written summary — not siloed in QA or lost in a private document.
- Action-item-closure-rate tracked alongside escaped-defect rate (both metrics, not one); for incident postmortems, both are derivable from the incident platform's native follow-through tracking.
- If AI SRE tooling is in use, the AI-drafted timeline and candidate root causes are recorded as input, and a named blameless RCA owner (not the incident commander) signed off on the human-authored 5 Whys and action items.
Related Skills
- qa-metrics — Provides the data (defect escape rate, flakiness rate, coverage trends) that postmortems analyze and act upon. Use it for the ongoing dashboard; use this skill for the analysis session.
- test-reliability — Flaky test classification and quarantine management, which feeds into test suite health reviews.
- test-strategy — When postmortems reveal systemic gaps that recur across quarters, the test strategy is the document that gets updated.
- shift-left-testing — Many postmortem action items are shift-left practices: earlier testing, better requirements, dev/QA pairing.
- release-readiness — Quality gates and release criteria should be updated based on postmortem findings; use it for live go/no-go, not retrospective analysis.
Reference Files (in references/)
- templates.md — Copy-paste Postmortem Template for Quality Incidents, Retro Meeting Template + facilitator notes, and the single-bug Escaped Bug Analysis worksheet.
