| name | dont-fool-yourself |
| description | Use when building, validating, or reporting any new empirical claim — a statistical method, trading signal, backtest, model, or "we found an effect" result. Enforces a falsification battery (negative controls, proper counterfactuals, multiple-testing correction, out-of-sample validation, honest intervals, tail modeling) plus pre-registration and append-only record-keeping, so a finding has to earn belief before it counts. Domain-agnostic (finance, ecology, epidemiology, social science). Triggers include: new method, new signal, new backtest, "is this real", validate result, check for overfitting, p-hacking, multiple testing, register a prediction. |
Don't Fool Yourself
"The first principle is that you must not fool yourself — and you are the easiest person to fool." — Feynman
A new finding is guilty until proven innocent. Before any empirical claim is allowed to count — a signal, an effect, a backtest, a "this works" — it must survive the gates below. The default answer is "not yet shown," not "looks significant."
When to run this
Whenever you (or a delegated agent) are about to:
- add a new statistical method / factor / signal / model, or
- report "we found X" / "this beats baseline" / "this is significant", or
- ship a backtest or any out-of-sample claim.
First move: state the null and the kill criterion
Before looking at results, write down:
- The null — what the world looks like if there is nothing here.
- The kill criterion — the specific result that would make you abandon the claim. If you can't name one, you're not testing, you're decorating.
- Pre-register — log the hypothesis, the test, and the decision rule before running it. Append-only (see the ledger rule below).
Falsification battery
Run the gates relevant to the claim. Each gate is a way the finding can die. A finding that survives all relevant gates has earned tentative belief — nothing more.
| Gate | What it kills | The rule |
|---|
| Negative control / placebo | spurious structure, leakage, bugs | Re-run on shuffled labels / fake event dates / a variable that can't matter. The effect must vanish. If the placebo "works," your pipeline is broken — fix it first. |
| Proper counterfactual (event-study DiD) | confounding, trends as effects | Compare treated vs. control around the event; check parallel pre-trends. No parallel trends → no causal read. |
| Multiple-testing correction (FDR) | forking paths, cherry-picking | Count every test you ran across the whole family — including abandoned ones — and control FDR (Benjamini–Hochberg / –Yekutieli). One "hit" out of 40 is noise. |
| Out-of-sample (walk-forward) | overfitting, in-sample illusion | Validate on data the method never touched, rolling forward. In-sample performance is not evidence. |
| Backtest deflation (Deflated Sharpe / CPCV) | backtest overfitting under many trials | Adjust for the number of configurations tried, sample length, and non-normality. Use purged, combinatorial CV to avoid leakage. |
| Honest intervals (conformal) | false precision | Report distribution-free intervals with finite-sample coverage, not intervals that assume a model you haven't validated. |
| Tail modeling (EVT) | "it never happens" risk | Model tails with extreme-value theory (GPD / block maxima); don't extrapolate a Gaussian into the tail. |
STOP conditions — report, don't decide
Halt and surface the problem instead of quietly proceeding when:
- a placebo / negative control also shows the effect (the pipeline is lying to you),
- a calibration curve is non-monotonic or inverted,
- the only "significant" results are the ones that survived many silent choices,
- making the result look good requires dropping data, shifting the window, or redefining the metric after seeing outcomes.
In these cases, stop and report what you saw. Do not patch it into looking right.
The append-only ledger
- Log predictions / decisions before outcomes are known.
- Never edit history. Corrections are new rows, not rewrites. A record you can quietly change is not a record.
Anti-patterns — what fooling yourself looks like
- "Let me just try a few more specifications" (and reporting the best one).
- Reading significance off the in-sample fit.
- Correcting one test as if it were the only test you ran.
- Tuning on the test set, then calling it out-of-sample.
- Deleting the failed prediction because "it was a mistake."
Depth for each gate — when it applies, how to run it, its failure signature, and the key reference — is in references/honesty-gates.md.
