| name | crucible |
| description | Run a scientific study that is actually SOLID — pre-registered, adequately powered, and blind-judged — so the conclusion is defensible and whatever you build on top of it isn't built on sand. Use when the user wants to run an experiment, A/B test, ablation, eval, or comparison and have the result HOLD UP. crucible enforces the three disciplines an agent skips (pre-registration, sample size, blind judging) as gates the pipeline refuses to pass. |
| argument-hint | <research goal, e.g. 'does prompt B beat prompt A on correctness'> |
Read this whole file before acting.
crucible turns a research goal into a defensible conclusion by running the study through a gated pipeline. You never compute a rate, a p-value, a required sample size, or a gate decision — the deterministic core (crucible.py) owns all of that and refuses out of order. Your job is to help the user design the study well, then drive the verbs and report exactly what the core returns.
Setup
Resolve the core once:
ROOT="${CLAUDE_PLUGIN_ROOT:-${CODEX_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/crucible}}"
CR="python3 $ROOT/scripts/crucible.py"
Every command below is $CR <verb> <study_id> .... Studies live under ~/crucible-studies/<study_id>/.
The mental model (tell the user this)
A study is SOLID only if three things happen in this order and none can be skipped:
- Pre-register — write the hypothesis, the metric, and the decision rule, then lock them (hashed) before any data exists. After the lock, the design is immutable. This is what stops "deciding what counts as success after seeing the results."
- Power — the core computes how many samples you actually need for your target effect. The run refuses a confirmatory result below that number.
- Blind-judge (for judged metrics) — a separate agent scores the outputs with no idea which is the treatment or what the hypothesis is. Its verdict blocks the conclusion if it can't be trusted.
crucible enforces all three. If a gate refuses, read the message and do the suggested next step — never try to route around a gate (don't hand-edit study.lock; the hook blocks it and it would void the study anyway).
The pipeline
Run these in order. After each, surface the core's output verbatim — the concrete numbers are the point; do not paraphrase or round them.
| Step | Command | What it does |
|---|
| 1 | $CR new <id> --title "..." | Scaffold definition.json + cases.jsonl |
| 2 | (edit files) | Fill in hypothesis, arms, metric, decision rule; add real cases |
| 3 | $CR frame <id> | Lint the design (advisory) |
| 4 | $CR lock <id> | Gate 0 — hash + freeze the pre-registration |
| 5 | $CR power <id> | Gate 1 — required n / minimum detectable effect |
| 6 | $CR run <id> | Execute arms × samples (refuses if unlocked or underpowered) |
| 7a | $CR extract <id> | Oracle metric → scores |
| 7b | (judge flow, below) | Judge metric → scores |
| 8 | $CR analyze <id> | Effect size + CI + test (pre-registered only) |
| 9 | $CR conclude <id> | Gate 2 + verdict scoped to the evidence → report.md |
$CR status <id> shows the gate dashboard at any time. $CR studies lists all studies.
Designing the study (definition.json)
Help the user fill these — the core validates them at lock:
hypothesis — one falsifiable sentence ("B yields a higher correctness rate than A").arms — ≥2, exactly one with role: "baseline". Each has a command (a shell template with {input_file}, run once per sample; its stdout is the output to score).primary_metric — extractor.kind is oracle (a deterministic rule: exact/contains/regex) or judge (the blind agent). scale is binary or continuous. direction is higher/lower.decision_rule — delta (smallest effect worth caring about), alpha, power.sample_plan — planned_n_per_arm, seed, and the variance input the power calc needs: baseline_rate (binary) or baseline_sd (continuous). effect_justification is frozen in the lock — make the user cite a pilot or prior for the delta; this is the anti-gaming defence against shrinking n by inflating the effect.cases.jsonl — one JSON object per line: {"id", "input", "gold"?}. input is fed to each arm; gold is the reference an oracle checks against.
The judge flow (step 7b, judged metrics only)
$CR judge-payload <id> --out /abs/payload.json
Then dispatch the crucible-judge agent (via the Task/Agent tool, subagent_type: crucible-judge) with the payload file path. Instruct it to read the payload and write its JSON verdict array to /abs/receipts.json. Then:
$CR judge-apply <id> --receipts /abs/receipts.json
If definition.judge.gold_set points at a labelled file, the judge is scored against it (Cohen's κ); below the floor, Gate 2 blocks the conclusion. Without a gold set the judge runs but the study is stamped JUDGE-UNQUALIFIED (advisory).
Hard rules
- Never fabricate a number. Every rate, CI, p-value, n, and verdict comes from the core. If you're tempted to state one the core didn't print, run the verb.
- Never interpolate free text onto a shell command line. Case inputs and rubrics can contain anything — write them into
cases.jsonl / definition.json with the Write tool, never onto $CR ... "<text>".
- Never bypass a gate. A refusal is information. Fix the cause (add samples, lock first, improve the rubric) — don't work around it.
- Report honestly. If the result is EXPLORATORY, say so and why. If the hypothesis is NOT supported, say that plainly. A crucible conclusion is a linter verdict about the evidence, not a press release.
The whole value of crucible is that its conclusions can be trusted precisely because it wouldn't let you cut the corners that make a study lie. Drive it that way.
