| name | alterlab-test-selection-guard |
| description | Enforces statistical-test selection as a discipline, holding the Iron Law NO TEST CHOSEN AFTER SEEING THE P-VALUE: routes the choice through a fixed decision tree (outcome type -> groups -> paired? -> normality) terminating in named tests (t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis, Wilcoxon, Friedman, chi-square, Pearson/Spearman, regression), gates interpretation behind a mandatory Shapiro-Wilk/Levene/linearity assumption check, blocks test-shopping with an Excuse-vs-Reality table and Red-Flags-STOP list, and applies a 3+-tests escalation gate forcing Bonferroni/FDR correction or an exploratory label. Use when choosing or switching a statistical test, asking which test to run, dropping a test after a non-significant result, or running many tests hunting for significance. For executing the chosen test prefer alterlab-statistical-analysis or alterlab-statsmodels; for the broader frozen-plan discipline see alterlab-preregistration-discipline. Part of the AlterLab Academic Skills suite. |
| license | MIT |
| allowed-tools | Read Bash(python:*) |
| compatibility | No API key required. A discipline-enforcing guidance skill; the optional decision-tree helper runs locally via `uv run python` (standard library only). |
| metadata | {"skill-author":"AlterLab","version":"1.0.0","last_updated":"2026-06-06","depends_on":"alterlab-statistical-analysis (executes the chosen test + assumption checks), alterlab-statsmodels, alterlab-preregistration-discipline"} |
Test-Selection Guard — Iron Law: No Test Chosen After Seeing the P-Value
Skill type: DISCIPLINE-ENFORCING. This is a thin discipline layer, not a stats
engine. It does not run tests, fit models, or compute power — it forces the order of
operations so the test is chosen by the data's shape, never by the result it produces.
For the actual computation, hand off to the skills below.
REQUIRED BACKGROUND — this skill orchestrates, it does not reimplement:
alterlab-statistical-analysis — runs the chosen test, the assumption checks
(its assumption_checks.py: Shapiro-Wilk, Levene, Q-Q), power, and APA reporting.alterlab-statsmodels — fits the model (OLS/GLM/mixed/ARIMA) once the test is fixed.alterlab-scientific-thinking — judges design validity, biases, confounders upstream.alterlab-preregistration-discipline — the frozen analysis plan this gate presumes.
When the user needs execution, route there. This skill's whole job is what happens
before the first scipy.stats call.
The Iron Law
NO TEST CHOSEN AFTER SEEING THE P-VALUE.
The test is a function of the research question and the data's structure — outcome
type, number of groups, pairing, and assumption checks — fixed before any p-value is
visible. Choosing or switching a test in response to its significance is p-hacking. It
inflates the Type I error rate by an unknown amount and makes the reported p-value a lie.
Violating the letter of the pre-specified test is violating the spirit of the inference.
A non-significant result is not a reason to try another test. It is a finding.
When to Use This Skill
Trigger this guard whenever a test is being selected, defended, or swapped:
- "Which statistical test should I use for this design / these variables?"
- "My t-test wasn't significant — should I try Mann-Whitney instead?" (← the canonical trap)
- "I ran ANOVA, t-tests, and a regression on the same hypothesis; which do I report?"
- "Can I drop these outliers / add this covariate and re-run to get under .05?"
- "Is it OK to switch to a non-parametric test now?"
The skill runs the decision tree, names the test, and — critically — .
