Enforces pre-registration discipline with the Iron Law NO DATA ANALYSIS WITHOUT A PRE-REGISTERED ANALYSIS PLAN FIRST, a spirit-vs-letter line, an Excuse-vs-Reality rationalization table, and a Red-Flags-STOP list (HARKing, optional stopping, post-hoc covariates, outlier-dropping, test-shopping). Runs a PLAN/COLLECT/CONFIRM/EXPLORE workflow that freezes hypotheses, tests, exclusions, and stopping rules before data, then forces unplanned findings to be labeled exploratory (their p-values lose confirmatory status, per COS confirmatory/exploratory model). Orchestrates, not replaces, alterlab-open-science (OSF/AsPredicted registration), alterlab-statistical-analysis (test selection, assumptions), and alterlab-scientific-thinking (bias grading). Use when analyzing data without a frozen plan, switching the primary outcome or adding covariates after seeing results, weighing early stopping, dropping outliers post-hoc, pre-registering a study, or rationalizing deviation. Part of the AlterLab Academic Skills suite.
Enforces pre-registration discipline with the Iron Law NO DATA ANALYSIS WITHOUT A PRE-REGISTERED ANALYSIS PLAN FIRST, a spirit-vs-letter line, an Excuse-vs-Reality rationalization table, and a Red-Flags-STOP list (HARKing, optional stopping, post-hoc covariates, outlier-dropping, test-shopping). Runs a PLAN/COLLECT/CONFIRM/EXPLORE workflow that freezes hypotheses, tests, exclusions, and stopping rules before data, then forces unplanned findings to be labeled exploratory (their p-values lose confirmatory status, per COS confirmatory/exploratory model). Orchestrates, not replaces, alterlab-open-science (OSF/AsPredicted registration), alterlab-statistical-analysis (test selection, assumptions), and alterlab-scientific-thinking (bias grading). Use when analyzing data without a frozen plan, switching the primary outcome or adding covariates after seeing results, weighing early stopping, dropping outliers post-hoc, pre-registering a study, or rationalizing deviation. Part of the AlterLab Academic Skills suite.
license
MIT
allowed-tools
Read Write Edit Bash(python:*)
compatibility
No API key required. Judgment/discipline skill; the optional helper runs locally via `uv run python` (stdlib only). Defers all registration mechanics and statistics to the sibling skills it orchestrates.
Skill type: DISCIPLINE-ENFORCING. This is not a how-to and not a reference. It
exists to stop one specific failure mode — exploiting researcher degrees of freedom
after seeing the data, then reporting the result as if it were planned. It governs
when you are allowed to claim a confirmatory result; it does not teach you how to
pick a test or fill an OSF form. Those are sibling skills (see the routing table).
THE IRON LAW
NO DATA ANALYSIS WITHOUT A PRE-REGISTERED ANALYSIS PLAN FIRST.
The frozen hypothesis + analysis plan is the research analog of a failing test
written before the implementation: you commit to what would count as a result
before you can see whether you got one. Run an unplanned analysis and the rule is
absolute — it is exploratory. Label it, timestamp it, and you cannot report its
p-value as confirmatory. (Per the Center for Open Science confirmatory/exploratory
model: in exploratory work, p-values lose their diagnostic value and findings require
independent replication. Verified at https://www.cos.io/initiatives/prereg.)
Violating the letter of the pre-registration is violating the spirit of the science.
That line pre-empts the predictable defense — "pre-registration is bureaucratic, I'm
following the scientific spirit." There is no spirit-following exception. Iterate
freely in the EXPLORE phase; the confirmatory claim needs the frozen plan.
When to Use This Skill
Use it the moment a confirmatory claim is on the table without a frozen plan behind
it, or the moment the plan is being bent after data are visible:
About to analyze a dataset and there is no written, pre-data analysis plan.
Switching the primary outcome, adding/removing a covariate, or changing the test
after seeing results or p-values.
Peeking at accruing data and weighing whether to stop now ("it's already significant").
Dropping outliers, transforming variables, or changing exclusions that were not
pre-specified.
Drafting an OSF / AsPredicted / PROSPERO pre-registration (route the mechanics to
alterlab-open-science; this skill enforces that the plan is actually frozen and
honored).
Rationalizing a deviation from an existing registration.
A deadline-pressure ask like "just p-hack this borderline result into something
publishable." — refuse the framing; offer the PLAN/CONFIRM/EXPLORE path instead.
Does NOT Trigger
This skill is the gatekeeper, not the toolbox. It orchestrates siblings and routes
everything mechanical away. Route adjacent requests as follows:
Proposal authoring, not pre-registration enforcement.
Research ethics approval / KVKK / data-management compliance
alterlab-tr-research-ethics / alterlab-kvkk-dmp
Ethics & data governance, distinct from analysis-plan freezing.
REQUIRED BACKGROUND (this skill orchestrates, it does not reimplement):
alterlab-open-science for the registration artifact, alterlab-statistical-analysis
for test choice + assumptions, alterlab-scientific-thinking for bias framing. When a
step needs any of those, hand off — do not restate their content here.
The Workflow: PLAN → COLLECT → CONFIRM → EXPLORE
This is the research analog of red → green → refactor. Each phase has an exit gate; you
may not enter the next phase until the current gate passes.
1. PLAN — freeze it before any data are visible
Write down, and lock, all of:
The hypotheses (directional where applicable).
The primary outcome and any secondary outcomes — ranked.
The exact test(s) for each hypothesis (defer the choice to
alterlab-statistical-analysis; record the decision here).
Inclusion/exclusion rules and the outlier rule, pre-specified.
The stopping rule / planned sample size (so optional stopping is off the table).
Covariates, transformations, and how missing data are handled.
Register the artifact via alterlab-open-science (OSF Registries / AsPredicted /
PROSPERO). Gate: the plan exists, is timestamped, and nothing in it depends on
having seen the outcome data. If you cannot answer "what result would falsify this?"
you are not done planning.
2. COLLECT — data, untouched
Collect according to the stopping rule. Do not peek at the outcome to decide whether
to keep going. If an interim look is genuinely needed, it had to be a pre-specified
sequential design (record that in PLAN). Gate: data collection matched the frozen
rule; no outcome-dependent stopping occurred.
3. CONFIRM — run exactly the planned tests
Run the pre-specified tests, in the pre-specified order, on the pre-specified sample.
Assumption checks (via alterlab-statistical-analysis) run and are reported before
interpreting the result — you do not get to swap the test because an assumption failed
unless the swap was pre-specified; an unplanned swap demotes the result to EXPLORE.
Gate: every confirmatory number traces to a line in the frozen plan.
4. EXPLORE — everything else, explicitly flagged
Anything not in the frozen plan lives here: new subgroups, post-hoc covariates,
alternative tests, interesting patterns. This work is valuable — it generates the
next study's hypotheses — but it is reported under an Exploratory heading, its
p-values are descriptive not confirmatory, and it carries a "requires replication"
caveat. Gate: every exploratory finding is labeled as such; none is laundered into
the confirmatory narrative.
The predictable rationalizations, and what is actually happening. (Seeded from the
documented researcher-degrees-of-freedom literature — see
references/researcher_degrees_of_freedom.md.)
Excuse
Reality
"The data suggested a better test/model."
You are fitting noise. That is HARKing — hypothesizing after results are known. Report it as exploratory.
"We only peeked once."
Optional stopping inflates the Type I error rate. Report the (pre-specified) sequential design, or stop peeking.
"Pre-registration is rigid; science is iterative."
Iterate in the EXPLORE section. The confirmatory claim still needs the frozen plan.
"We dropped 3 outliers to meet normality."
Outlier rules must be pre-specified, or reported as a sensitivity analysis — not a silent edit.
"This covariate obviously belongs in the model."
"Obvious" post-hoc = a researcher degree of freedom. Pre-specify it, or flag it as exploratory.
"We switched to Mann–Whitney because the t-test wasn't significant."
Choosing a test by its p-value is test-shopping. Pre-register the decision rule or label the result exploratory.
"The deviation follows the spirit of the registration."
Violating the letter is violating the spirit. There is no spirit exception.
"It's only exploratory, so registration is overkill."
Then label every finding exploratory and claim no confirmatory p-values. You don't get confirmatory credit without the plan.
Red Flags — STOP
If you catch yourself (or the user) thinking any of these, STOP:
"Let me just try a different test and see."
"I'll drop these outliers and rerun."
"The effect is there if I add this one covariate."
"We can stop collecting now — it's already significant."
"I'll just report the analyses that worked."
"This subgroup is fascinating (we didn't predict it, but…)."
"Let me change the primary outcome to the one that came out."
"Pre-registration is overkill for a study like this."
All of these mean the same thing: you are exploiting researcher degrees of freedom.
Either return to the frozen plan, or label the work exploratory and forfeit the
confirmatory claim. There is no third option.
The Multiple-Comparisons Escalation Gate
A hard, countable rule (the analog of "3 failed fixes = wrong architecture"):
Ran 3+ tests on the same hypothesis searching for significance? STOP. This is
multiple comparisons / test-shopping. Either correct for all of them (Bonferroni /
Benjamini–Hochberg FDR — via alterlab-statistical-analysis) or declare the whole
analysis exploratory. Do not run test #4 to find p < .05.
Deviation Disclosure Gate
If a deviation from the frozen plan is unavoidable (a genuine error in the plan, an
impossible assumption), it is not silently absorbed. Run, in order:
STATE the original frozen specification (quote the plan).
STATE the deviation and the concrete reason it was forced.
CLASSIFY the affected result as exploratory unless the deviation provably
cannot have been outcome-driven.
DISCLOSE the deviation in the manuscript's transparency/deviations section
(route reporting completeness to alterlab-results-transparency).
You can run a quick structured self-audit of a plan-vs-actual story with the optional
helper:
uv run python skills/methodology/alterlab-preregistration-discipline/scripts/prereg_check.py \
--plan plan.json --actual actual.json
It is a stdlib-only checklist scorer (no network, no third-party APIs) — it does not
run statistics. See its --help.
Self-Check Before Claiming a Confirmatory Result
Does a frozen, timestamped plan exist, written before the outcome data were visible?
Does every confirmatory number trace to a line in that plan?
Were assumption checks run and reported before interpretation?
Is every unplanned analysis under an Exploratory heading with a replication caveat?
Are all deviations disclosed, not absorbed?
Any "no" → the result is exploratory, not confirmatory. Say so plainly.
references/researcher_degrees_of_freedom.md —
the documented failure modes (HARKing, optional stopping, p-hacking, the garden of
forking paths) this skill is built to catch, with sources.
