crucible

Read this whole file before acting. crucible turns a research goal into a defensible conclusion by running the study through a gated pipeline. **You never compute a rate, a p-value, a required sample size, or a gate decision** — the deterministic core (`crucible.py`) owns all of that and refuses out of order. Your job is to help the user *design* the study well, then *drive* the verbs and *report exactly what the core returns*. ## Setup Resolve the core once: ```bash ROOT="${CLAUDE_PLUGIN_ROOT:-${CODEX_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/crucible}}" CR="python3 $ROOT/scripts/crucible.py" ``` Every command below is `$CR <verb> <study_id> ...`. Studies live under `~/crucible-studies/<study_id>/`. ## The mental model (tell the user this) A study is SOLID only if three things happen in this order and none can be skipped: 1. **Pre-register** — write the hypothesis, the metric, and the decision rule, then **lock** them (hashed) *before any data exists*. After the lock, the design is immutable. This is what stops "deciding what counts as success after seeing the results." 2. **Power** — the core computes how many samples you actually need for your target effect. The run **refuses** a confirmatory result below that number. 3. **Blind-judge** (for judged metrics) — a separate agent scores the outputs with no idea which is the treatment or what the hypothesis is. Its verdict **blocks** the conclusion if it can't be trusted. crucible enforces all three. If a gate refuses, **read the message and do the suggested next step — never try to route around a gate** (don't hand-edit `study.lock`; the hook blocks it and it would void the study anyway). ## The pipeline Run these in order. After each, surface the core's output verbatim — the concrete numbers are the point; do not paraphrase or round them. | Step | Command | What it does | |---|---|---| | 1 | `$CR new <id> --title "..."` | Scaffold `definition.json` + `cases.jsonl` | | 2 | *(edit files)* | Fill in hypothesis, arms, metric, decision rule; add real cases | | 3 | `$CR frame <id>` | Lint the design (advisory) | | 4 | `$CR lock <id>` | **Gate 0** — hash + freeze the pre-registration | | 5 | `$CR power <id>` | **Gate 1** — required n / minimum detectable effect | | 6 | `$CR run <id>` | Execute arms × samples (refuses if unlocked or underpowered) | | 7a | `$CR extract <id>` | Oracle metric → scores | | 7b | *(judge flow, below)* | Judge metric → scores | | 8 | `$CR analyze <id>` | Effect size + CI + test (pre-registered only) | | 9 | `$CR conclude <id>` | **Gate 2** + verdict scoped to the evidence → `report.md` | `$CR status <id>` shows the gate dashboard at any time. `$CR studies` lists all studies. ## Designing the study (`definition.json`) Help the user fill these — the core validates them at `lock`: - **`hypothesis`** — one falsifiable sentence ("B yields a higher correctness rate than A"). - **`arms`** — ≥2, exactly one with `role: "baseline"`. Each has a `command` (a shell template with `{input_file}`, run once per sample; its stdout is the output to score). - **`primary_metric`** — `extractor.kind` is `oracle` (a deterministic rule: `exact`/`contains`/`regex`) or `judge` (the blind agent). `scale` is `binary` or `continuous`. `direction` is `higher`/`lower`. - **`decision_rule`** — `delta` (smallest effect worth caring about), `alpha`, `power`. - **`sample_plan`** — `planned_n_per_arm`, `seed`, and the variance input the power calc needs: `baseline_rate` (binary) or `baseline_sd` (continuous). **`effect_justification` is frozen in the lock** — make the user cite a pilot or prior for the delta; this is the anti-gaming defence against shrinking n by inflating the effect. - **`cases.jsonl`** — one JSON object per line: `{"id", "input", "gold"?}`. `input` is fed to each arm; `gold` is the reference an oracle checks against. ## The judge flow (step 7b, judged metrics only) ```bash $CR judge-payload <id> --out /abs/payload.json # blind: only {item_id, input, output} ``` Then **dispatch the `crucible-judge` agent** (via the Task/Agent tool, `subagent_type: crucible-judge`) with the payload file path. Instruct it to read the payload and write its JSON verdict array to `/abs/receipts.json`. Then: ```bash $CR judge-apply <id> --receipts /abs/receipts.json # de-anonymizes, qualifies the judge ``` If `definition.judge.gold_set` points at a labelled file, the judge is scored against it (Cohen's κ); below the floor, **Gate 2 blocks the conclusion**. Without a gold set the judge runs but the study is stamped `JUDGE-UNQUALIFIED` (advisory). ## Hard rules - **Never fabricate a number.** Every rate, CI, p-value, n, and verdict comes from the core. If you're tempted to state one the core didn't print, run the verb. - **Never interpolate free text onto a shell command line.** Case inputs and rubrics can contain anything — write them into `cases.jsonl` / `definition.json` with the Write tool, never onto `$CR ... "<text>"`. - **Never bypass a gate.** A refusal is information. Fix the cause (add samples, lock first, improve the rubric) — don't work around it. - **Report honestly.** If the result is EXPLORATORY, say so and why. If the hypothesis is NOT supported, say that plainly. A crucible conclusion is a linter verdict about the evidence, not a press release. The whole value of crucible is that its conclusions can be trusted precisely because it wouldn't let you cut the corners that make a study lie. Drive it that way.