hypothesis-batch

# Workflow: Claim Stage — Behavior Discovery × Mechanism Discovery Orchestrate the claim stage for: **$ARGUMENTS**. ## Overview This skill chains sub-skills into a single automated, non-interactive pipeline: ``` /research-lit → /idea-creator → /novelty-check → /impact-check → /research-review → /research-refine-pipeline (survey) (brainstorm) (verify novel) (verify it (critical feedback) (refine method + plan) matters) ``` **Batch shape: 30 in, 10 out.** Phase 2 brainstorms **~30** candidate ideas across three angle-partitioned `/idea-creator` rounds. Novelty (Phase 3) is a hard gate; impact (Phase 3.5) and external review (Phase 4) then score **every survivor**, and Phase 4.25 makes the single cut to **10** — vetoing fatal design flaws outright, then ranking by impact first, reviewer score second, novelty last. From Phase 4.5 on, each of the 10 is worked **independently and in parallel**, and each gets its own directory. **Deliverables.** ``` idea-stage/ RESEARCH_LIT.md # raw retrieval dump (audit-only) LANDSCAPE.md # synthesized landscape IDEA_REPORT.md # all ~30 ideas, ranked, with eliminations — the batch index claims/ 01_<name>/ # one directory per surviving idea, rank-ordered FINAL_PROPOSAL.md EXPERIMENT_PLAN.md claim.json # ⭐ the deliverable a human reviewer reads 02_<name>/ … 10_<name>/ ``` `claim.json` is the point of the whole run: a self-contained paper skeleton written **for a human expert reviewer**, who sees that file and nothing else. Everything above it is working material. Its specification is Phase 4.8 and is binding. ## Pipeline ### Phase 1: Literature Survey Invoke `/research-lit` to map the research landscape: ``` /research-lit "$ARGUMENTS" ``` **What this does:** - Search Zotero, Obsidian, local PDFs, the web, and the cloud mechanic-db SEARCH service (via the skill `/mechanic-db-search`) for relevant papers - Build a landscape map: sub-directions, approaches, open problems - Identify structural gaps and recurring limitations - Output two files: `idea-stage/RESEARCH_LIT.md` (raw retrieval dump, audit-only) and `idea-stage/LANDSCAPE.md` (synthesized landscape — Phase 2 reads this from disk) ### Phase 1.75: Mechanism Strategy Load This phase loads strategy into context; it writes **no artifact**. It is a *reference* step — its job is to put the relevant strategy skills in context so Phase 2 (hypothesis direction) and Phase 4.5 (experiment plan) can apply them. Both skills are principles only: read them, do not execute their phases, and do not expect an output file from either. 1. **Behavior strategy** — invoke `/mechanism-behavior-discovery` and read its `SKILL.md` in full (the standard for finding, sharpening, and validating a *new* behavioral phenomenon — the five bars Real / Non-obvious / Specific / Robust / Tractable, the discovery strategies, and the validation discipline). It decides *what behavior is worth explaining*. Log `[behavior-strategy] discovery — loaded /mechanism-behavior-discovery`. 2. **Mechanism strategy** — invoke `/mechanism-explore` via the Skill tool and read its `SKILL.md` in full; the strategy shapes *how to explain* the behavior (the hypothesis direction in Phase 2 and the experiment plan in Phase 4.5). It fixes the direction only — the concrete family is left unbound. Log `[mechanism-strategy] discovery — loaded /mechanism-explore`. Hold the loaded guidance in context for the rest of the run — do **not** copy it into any output file. What the loaded strategy is used for (advisory — it never relaxes `/idea-creator`'s novelty / feasibility / pilot filtering): - **Phase 2 (hypothesis direction)** — first surface and *sharpen* candidate phenomena (per `/mechanism-behavior-discovery`), then frame candidate ideas as *falsifiable mechanistic hypotheses* (name the implicated internal object and the predicted causal relation) with at least one "boring" null (memorization / surface feature / shortcut), favoring ideas on a climbable ladder of evidence. The claim **assumes** the chosen phenomenon exists and attaches the mechanistic hypothesis — its *actual* existence is tested by the M0 gate when the plan is run, not asserted here. - **Phase 4.5 (experiment plan)** — shape the plan to climb the ladder of evidence: a cheap correlational/attribution screen to localize candidates → a causal intervention (ablation / patching / steering) to confirm the survivors → matched-control + off-target + confound checks for specificity. Each intervention milestone records the expected **sign**, **magnitude / dose-response**, and a **specificity control**. The plan **opens with a phenomenon-validation gate (M0)** that the mechanism milestones all depend on (see Phase 4.5). ### Phase 2: Produce the Candidate Ideas This phase writes `idea-stage/IDEA_REPORT.md`, the ranked candidate pool that Phases 3–3.5 cut down to 10. **Three rounds, not one call.** `/idea-creator` generates 8–12 ideas per invocation — that count is fixed inside its own prompt template and is not a parameter, so a single call cannot produce 30. Run it **three times** and merge: ``` /idea-creator "$ARGUMENTS — angle: <round assignment>; do not regenerate or closely vary: <titles from earlier rounds>" ``` Assign each round a **different slice of the search space**, so the rounds diverge by construction rather than by luck: | Round | Assignment | |---|---| | 1 | The landscape's **structural gaps** — one idea aimed at each gap `LANDSCAPE.md` names. | | 2 | The **mechanism directions from `/mechanism-explore` not yet used** in round 1 (Location / Causal Intervention / Tuning & Editing / Formation Tracing / Unit Interpretation / Decision Auditing). | | 3 | The **phenomena classes underrepresented** after rounds 1–2 — read the two merged rounds, name what kind of behavior is missing, and aim the third round there. | Pass the titles already generated as an explicit exclusion in each later round. Round 3 is written **after** reading rounds 1–2, so its assignment is derived, not guessed. **What each round does:** - Read `idea-stage/LANDSCAPE.md` from disk (written by Phase 1 / `/research-lit`), then brainstorm 8–12 concrete ideas via the external LLM reviewer (llm-chat). - Filter by feasibility, compute cost, quick novelty search. - Deep-validate the leading ideas (full novelty check + devil's advocate). - Rank by empirical signal. **Merge into `idea-stage/IDEA_REPORT.md`.** Concatenate the three rounds into one `## Ranked Ideas` section, then **deduplicate**: two ideas are the same idea when they share a phenomenon *and* an internal object, whatever their titles say. Keep the better-stated one, record the merge. Target **~30** candidates; 24–36 is fine, and a merged pool below ~20 means the rounds collapsed onto each other — run a fourth round on whatever the dedupe revealed as over-covered. **Breadth is the point.** The pool exists to be cut, so 30 near-duplicates of one framing is a failed Phase 2. Overshoot rather than undershoot: the pool must survive a novelty gate and a design veto and still leave 10. **Apply the Phase 1.75 strategy.** Pass `/mechanism-explore`'s framing into `/idea-creator` so each generated idea is stated as a falsifiable mechanistic hypothesis (internal object + predicted causal relation + at least one boring null), and bias ranking toward ideas with a climbable ladder of evidence. First apply `/mechanism-behavior-discovery` to surface and *sharpen* candidate *phenomena* from the landscape (state each crisply per the five bars and screen plausibility) — the claim **assumes** the chosen phenomenon exists and attaches a mechanistic hypothesis; the phenomenon's actual existence is tested by the M0 gate in the experiment stage, not claimed here. This shapes idea *framing* only — novelty, feasibility, and pilot filtering are unchanged. ### Phase 3: Deep Novelty Verification Run a thorough novelty check on **every surviving candidate** — one `/novelty-check` per idea, dispatched in parallel: ``` /novelty-check "[idea N description]" # for each surviving idea ``` **What this does:** - Multi-source literature search (arXiv, Scholar, Semantic Scholar) - Cross-verify with the external LLM reviewer - Check for concurrent work (last 3-6 months) - Identify closest existing work and differentiation points **Update `idea-stage/IDEA_REPORT.md`** with deep novelty results. Eliminate any idea that turns out to be already published, and record for each survivor its **three nearest works** with the specific finding each established — Phase 4.8's `Related Work` field and checklist item 1 both need exactly this, and re-deriving it later wastes a second literature pass. ### Phase 3.5: Deep Impact Verification For each idea that survived novelty (Phase 3), run an impact check on the **problem/behavior** it studies — one `/impact-check` per idea, in parallel: ``` /impact-check "[idea N — behavior/problem + hypothesis]" # for each survivor ``` **What this does:** judges whether the studied problem/behavior is *important* — solves a real problem, is likely to be used/cited, could shift a field's direction, helps applications/industry/society/cross-disciplinary work, or reveals an important phenomenon even with a simple method. Outputs `Impact: X/10` + `PROCEED / PROCEED WITH CAUTION / DEPRIORITIZE`. **Update `idea-stage/IDEA_REPORT.md`.** Add an `Impact: X/10` line to each surviving idea. **No cut happens here** — impact is one of three signals the Phase 4.25 selection combines. ### Phase 4: External Critical Review Review **every idea that survived novelty** — not just a shortlist. One `/research-review` per idea, dispatched in parallel: ``` /research-review "[idea N with hypothesis + evidence]" # for each survivor ``` **What this does:** - The external LLM reviewer acts as a senior reviewer (NeurIPS/ICML level) - Scores the idea, identifies weaknesses, suggests minimum viable improvements - Provides concrete feedback on experimental design Reviewing the full pool rather than a pre-selected 10 is deliberate: novelty and impact both judge *whether the question is worth asking*, and neither can tell whether the proposed design could actually answer it. That is the reviewer's lens, and it is worth having **before** the batch is fixed rather than after. **Update `idea-stage/IDEA_REPORT.md`** with each idea's reviewer score and feedback. The feedback has two distinct downstream uses, and Phase 4.25 depends on telling them apart: - **Fatal design flaws** — the hypothesis is not falsifiable as stated, the measurement cannot reach the quantity it claims to measure, a confound is structural rather than controllable, the intervention cannot be run on any accessible model. Refinement cannot rescue these; they are grounds for elimination. - **Ordinary weaknesses** — incremental framing, execution risk, thin baselines, unclear presentation. These are exactly what Phase 4.5's refinement is for and must **not** eliminate anything. Classify each review as one or the other when recording it. ### Phase 4.25: Select the Batch of 10 This is the only cut that produces the batch. It combines all three signals. **Step 1 — Apply the veto.** Eliminate every idea whose Phase 4 review found a **fatal design flaw**, regardless of its impact score. An idea whose measurement cannot answer its own question does not become shippable by being important. Record it under `## Eliminated Ideas` with reason `vetoed at external review — <the flaw>`. **Step 2 — Rank the rest.** Order by **impact first, reviewer score second, novelty third**: - **Impact** dominates — a less-novel idea on an important problem outranks a novel idea nobody needs. - **Reviewer score** breaks impact ties and pulls down ideas the reviewer found weak-but-fixable. - **Novelty** is the last tiebreak only. Everything still standing already passed Phase 3's novelty gate, so it has little spread left to discriminate on. **Step 3 — Take the top 10.** Everything below the cut moves to `## Eliminated Ideas` with reason `below rank 10`. This ranking fixes the folder numbering in Phase 4.5 — nothing downstream re-orders it. **Step 4 — Check the spread.** If the top 10 collapse onto two or three phenomena, swap the lowest-ranked duplicates for the highest-ranked candidates that cover a distinct phenomenon, and note each swap. Ten skeletons on one phenomenon is a worse batch than ten on eight, even at slightly lower average score. **If fewer than 10 survive**, run **one** top-up round: return to Phase 2, generate additional ideas aimed at the gaps the eliminations exposed, and put them through Phases 3 → 4.25. If the batch is still short after that round, ship what survived and state the shortfall plainly in `IDEA_REPORT.md` — never pad the batch with an idea that was already eliminated, never overturn a novelty verdict or a design veto to reach 10. ### Phase 4.5: Per-Claim Refinement + Experiment Planning **From here on the batch splits.** Each of the 10 ideas is refined **independently, in its own directory**, and nothing is shared between them except the landscape. Run the 10 in parallel. **Step 0 — Create the directories.** For each idea, in the Phase 4.25 rank order: 1. Derive `<name>`: a **snake_case slug, ≤ 6 words, naming the phenomenon** — the same slug that becomes `claim.json`'s `Name` field in Phase 4.8. Derive it once here and reuse it verbatim; the directory name and the `Name` field must not diverge. 2. Create `claims/<NN>_<name>/`, where `NN` is the zero-padded rank (`01`…`10`). 3. Log `[claim] <NN>_<name> — "<idea title>"`. Every artifact from this phase onward is written **inside that directory**. Nothing goes to a shared `refine-logs/`. **Step 1 — Refine each idea.** For each of the 10, invoke `/research-refine-pipeline` with that idea's title, description, evidence, and its Phase 4 reviewer feedback: ``` /research-refine-pipeline "[idea <NN>: title + description + evidence + reviewer feedback]" ``` and write its outputs to `claims/<NN>_<name>/FINAL_PROPOSAL.md` and `claims/<NN>_<name>/EXPERIMENT_PLAN.md`. **What `/research-refine-pipeline` does** (per idea): - Freeze a **Problem Anchor** (the idea's problem statement) to prevent scope drift. - Iteratively refine the **method / testing approach** via external LLM review (up to 5 rounds, until score ≥ 9). - Generate a claim-driven experiment roadmap with ablations, budgets, and run order. - **(Mechanism milestones — ladder of evidence, from the Phase 1.75 strategy):** shape the roadmap to climb the ladder of evidence — a cheap correlational/attribution screen to localize candidates → a causal intervention (ablation / patching / steering) to confirm the survivors → matched-control + off-target + confound milestones for specificity — and have each intervention milestone record the expected sign, magnitude / dose-response, and its specificity control. Because the plan commits to a *direction*, not to a concrete mechanism **submethod**, any milestone field whose correct value depends on that submethod — typically `n_pairs`, intervention `sites`, the exact effect `metric`, and the per-run GPU-hours estimate — is necessarily provisional. Tag each such milestone with a `method_sensitive: [<field>, ...]` line (English machine field, written verbatim regardless of report language) marking the fields that must be re-bound once a submethod is picked. - **(Phenomenon-validation gate M0):** the claim only **assumes** the phenomenon exists; whether it is *real* is settled by running the plan, not by writing it. So the roadmap's **first milestone** is a hard gate that **must carry the machine marker `kind: phenomenon-validation`** (the title may be phrased / localized freely — the marker is what identifies the gate mechanically). With an explicit pass criterion: - reproduces across **paraphrase, seed, and decoding settings** (not one exact string); - **confounds controlled** (length, token frequency, position, format, label identity, few-shot leakage); - **statistical reality** — sample size large enough to clear noise (honor any project minimums; ≥ ~50 by default), with variability reported, not a single number; - the **trivial-explanation check** ruled out (tokenizer / sampling / eval-bug). All mechanism milestones (M1…Mn) **declare `depends_on: [M0]`**, and the M0 block spells out the **four-state verdict** that governs them: `established` → run the mechanism milestones; `conditional` → restrict mechanism analysis to the conditions where the phenomenon holds and tag the claim `conditional`; `not-established` → stop and report a negative result; `inconclusive` (the M0 test itself is broken or underpowered) → fix and re-run M0. Never run mechanism work on an untested phenomenon. - Output: `claims/<NN>_<name>/FINAL_PROPOSAL.md`, `claims/<NN>_<name>/EXPERIMENT_PLAN.md`. **Failure isolation.** If refinement fails for one idea, that failure stays local: record it in that directory and in `IDEA_REPORT.md`, and continue with the other nine. One bad claim never aborts the batch. **`EXPERIMENT_PLAN.md` shape.** Each milestone block MAY declare these fields: - `depends_on: [<milestone-name>, ...]` — this milestone's runs must wait until every listed upstream milestone is complete before launching. Use for teacher → student chains, baseline → ablation chains, or any milestone that needs an upstream artifact (checkpoint, dataset, calibration result). The names refer to other milestone blocks in the same `EXPERIMENT_PLAN.md`. - `grid: { <param-name>: [<value>, <value>, ...], ... }` — Cartesian-product expansion. Each combination produces one run with the params substituted via `${param-name}` placeholders into the milestone's `cmd:` template. Use for multi-seed sweeps (e.g., `grid: { seed: [42, 200, 201], n_hidden: [64, 128, 256], n_train_subset: [50000, 150000, 500000] }` → 27 runs). When `grid:` is present, the milestone's `cmd:` is treated as a template (rather than a literal command) and `id:` may also be a template (e.g., `s${seed}_N${n_hidden}_n${n_train_subset}`). - `kind: phenomenon-validation` — **stable machine marker** identifying the M0 gate, so it can be found by field rather than by title. The marker is an English machine field (per the Output-language protocol) and must be written verbatim regardless of the report language, leaving the title heading free to be phrased / localized. Every plan has exactly one `kind: phenomenon-validation` milestone (M0). - `method_sensitive: [<field>, ...]` — **per intervention milestone.** Lists the fields whose correct value depends on the mechanism submethod (typically `n_pairs`, `sites`, `metric`, `gpu_hours`) and therefore must be re-bound before the milestone can run. In addition to the per-milestone fields above, `EXPERIMENT_PLAN.md` carries the top-metadata machine marker `mechanism_strategy:` (the strategic-direction block above). Example milestone block: ```markdown ### M2: Multi-seed main method **Depends on**: M1 (baselines) **Grid**: seed: [42, 200, 201] n_hidden: [64, 128, 256, 512] n_train_subset: [50000, 150000, 500000] **Cmd template**: `python run_method.py --seed ${seed} --n_hidden ${n_hidden} --n_train_subset ${n_train_subset} --backbone softmax --K 500 --L 96` **Expected output (template)**: `results/main_N${n_hidden}_seed${seed}_n${n_train_subset}.json` **Priority**: MUST-RUN **Estimated GPU-hours per run**: 1.5h ```