| name | blind-survivorship-bias |
| description | Use when analyzing metrics, user feedback, success stories, benchmarks, churn, error logs, or any dataset — before drawing conclusions. Applies Abraham Wald's WWII bomber insight: the data you have was filtered by a survival process, and the filter is the finding. |
Survivorship Bias
Overview
Abraham Wald (Statistical Research Group memoranda, 1943; exposited in Mangel & Samaniego, JASA 1984): estimating aircraft vulnerability from returning planes is wrong unless you account for the planes that never came back — the returning planes survived their hits; the unhit areas on survivors mark where hits were fatal. (The vivid scene of commanders wanting to armor the bullet holes is a later retelling, via Wallis's 1980 memoir and Ellenberg — not in Wald's papers — but it dramatizes his math faithfully.) The dataset "planes we can inspect" was silently filtered by the very outcome under study.
Every dataset arrived through a filter. Analyzing the data without modeling the filter analyzes the survivors, not the population.
When to Use
- User feedback / NPS / reviews (only non-churned, non-silent users respond)
- Error logs and crash reports (crashes that prevent reporting don't report)
- "What do successful X do?" pattern studies (failures did those things too — unobserved)
- Benchmarks, case studies, testimonials, A/B results, retention metrics
- Any conclusion of the form "our users want…" / "top performers all…"
The Procedure
- Name the filter: what process decided which data points reached you? (Self-selection, churn, crash-before-report, publication, sales pipeline.)
- Describe the ghosts: who/what would NOT appear? Estimate the invisible population's size — sometimes it dwarfs the visible one.
- Ask Wald's question: does the conclusion flip if the missing data is systematically different? (Armor the clean spots.) If churned users hated the feature that survey-respondents love, the survey is inverted evidence.
- Hunt one ghost directly: interview churned users, log the failure path, find companies that did the same playbook and died. One recovered non-survivor is worth 100 more survivors.
Quick Reference
| Visible data | Invisible counterpart to check |
|---|
| Feature requests from active users | Why departed users left |
| Errors in the log | Failures that never reached logging |
| Practices of successful founders/projects | Same practices among the failed |
| 5-star and 1-star reviews | The silent middle who just left |
| Long-tenured employees love the culture | Everyone who quit in year one |
Common Mistakes
- Adding more survivor data to fix a survivorship problem (bigger N, same filter).
- Treating absence from the data as absence in reality ("no complaints about X" — or no channel that captures them?).
- Studying only failures instead (reverse survivorship) — you need both tails to find what actually discriminates.
