| name | experiment-analysis |
| description | Analyze a completed A/B test or experiment defensibly — check it against the pre-registered plan, run the primary-metric test, report effect size + CI (not just p), check guardrail metrics, apply a multiple-comparison correction across metrics/segments, and screen for the peeking/p-hacking pitfalls before declaring a winner. Used by `applied-statistician` (primary). |
Skill: experiment-analysis
Invoked by: applied-statistician (primary). Pairs with ../power-and-sample-size/SKILL.md (run first, at design time) and ../choose-statistical-test/SKILL.md (names the underlying test).
When to invoke: "is this A/B winner real?"; "the experiment finished — what does it say?"; "can we ship variant B?"
Output: a verdict (ship / don't ship / inconclusive) backed by the primary-metric effect size + CI, the guardrail check, the multiplicity correction, and an explicit pitfall screen.
Procedure
- Recover the analysis plan. Was there a pre-registered primary metric, test, and stopping rule (
../../templates/analysis-plan.md)? If analysis decisions were made after seeing data, flag the p-hacking exposure (pitfall #1) and treat findings as exploratory.
- Check the stopping rule. Was the test stopped at the planned sample, or stopped early when it "looked significant"? Early stopping on a fixed-horizon test = peeking (pitfall #3) → the nominal p-value is not valid; downgrade confidence.
- Run the primary-metric test (via
choose-statistical-test). Report the effect size + confidence interval as the headline — the p-value is secondary (pitfall #6). "Significant but the CI includes trivially small effects" is not a ship signal.
- Check guardrail metrics. A primary-metric win that degrades a guardrail (latency, churn, revenue/user) is not a win. State each guardrail's movement + CI.
- Correct for multiplicity. If you tested several metrics or segments, apply Holm/Bonferroni (confirmatory) or Benjamini-Hochberg (exploratory) — see
../../knowledge/statistical-pitfalls.md. A "winning segment" found after slicing 10 ways is usually noise.
- Sanity-check for Simpson's paradox (pitfall #4): does the aggregate result hold within key subgroups, or does the group mix drive it?
- Verdict in plain language: ship / don't ship / inconclusive (underpowered) — with the effect, CI, and the main caveat.
Output shape
Primary metric: <metric> — variant B <+X% / +X units>, 95% CI [<lo>, <hi>], p = <p>
Decision-relevant? <effect vs the MDE that justified the test>
Guardrails: <metric: movement + CI; pass/fail> ...
Multiplicity: <# metrics/segments tested; correction applied>
Pitfall screen: pre-registered? stopped at planned n? peeking? Simpson's?
Verdict: SHIP / DON'T SHIP / INCONCLUSIVE — <one-line reason + main caveat>
Guardrails
- The headline is the effect size + CI, never a bare "p < 0.05."
- Early-stopped fixed-horizon tests and post-hoc segment-mining both produce false winners — name them when present.
- An inconclusive (underpowered) result is an honest verdict; don't manufacture a winner.
