auto-experiment

name	auto-experiment
description	Workflow 1.5: Bridge between idea discovery and auto review. Reads EXPERIMENT_PLAN.md, routes mechanism family inline (Phase 1.5), implements experiment code, deploys to GPU, and collects initial results. Use when user says "implement experiments", "experiment", "deploy the plan", or has an experiment plan ready to execute.
argument-hint	["research-direction"]
allowed-tools	Bash(*), Read, Write, Edit, Grep, Glob, WebSearch, WebFetch, Agent, AskUserQuestion, Skill, mcp__llm-chat__chat

Workflow 1.5: Experiment

Implement and deploy experiments from plan: $ARGUMENTS

Overview

This skill bridges Workflow 1 (idea discovery + method refinement) and Workflow 2 (auto review loop). It takes the experiment plan and turns it into running experiments with initial results.

Workflow 1 output:                    This skill:                                    Workflow 2 input:
refine-logs/EXPERIMENT_PLAN.md   →   implement → LLM review → deploy → collect → initial results ready
refine-logs/EXPERIMENT_TRACKER.md     code        (cross-model)    /run-experiment     for /auto-iteration-loop
refine-logs/FINAL_PROPOSAL.md

Constants

• RESEARCH_DOMAIN = auto — Project domain tag (free-form, e.g. mechanistic-interpretability, vision-encoders, rl-policy-eval). Consumed by Phase 1.5 only as a routing constraint to /mechanism-skills — see Phase 1.5 Step 2's domain: arg. When null or auto, Phase 1.5 infers from FINAL_PROPOSAL.md; on ambiguous inference, silently default to general and log [research-domain] inference ambiguous — defaulted to general (this fallback bypasses AUTO_PROCEED by design — see /auto's flag-table row for the canonical statement). To force a specific domain, pass it explicitly on the CLI. (Note: Phase 1.1 routes through /experiment-tips using its own symptom-level trigger table and does not consume this constant.)
• MECHANISM_ROUTING = auto — Phase 1.5 mechanism-family routing mode. auto (default): invoke /mechanism-skills, write refine-logs/MECHANISM_ROUTING.md, present 2–3 candidates and let the caller pick (auto-select #1 when AUTO_PROCEED=true; otherwise block on the caller's AskUserQuestion). skip: assume routing already exists (or is not applicable) and proceed. not-applicable: explicitly mark behavioral-only proposal and skip without invoking. When called from /auto, the orchestrator's mini-prompt fills CHOSEN_FAMILY so this skill is re-entered with MECHANISM_ROUTING=skip.
• CHOSEN_FAMILY = none (dynamic — not in config; forwarded by /auto's orchestrator — from MECHANISM=given (the user's chosen_mechanism captured by the claim stage), the AUTO_PROCEED=false family mini-prompt, or an explicit family: pin in task.md (cross-round Rule 2), after any settled-pin conflict is resolved) — When set, commits this family/submethod combo from MECHANISM_ROUTING.md before implementation (Phase 1.5 Mode B).
• CODE_REVIEW = true — external LLM reviewer checks experiment code before deployment. Catches logic bugs before wasting GPU hours. Set false to skip.
• AUTO_DEPLOY = true — Automatically deploy experiments after implementation + review. Set false to manually inspect code before deploying. Treated as a standing approval for the deploy step: when AUTO_DEPLOY=true, the Phase 4 deploy proceeds even if AUTO_PROCEED=false.
• AUTO_PROCEED = true — Whether the Phase 4 Experiment Gate may skip the UI prompt. When true (default) and AUTO_DEPLOY=true, the gate proceeds silently. When false and AUTO_DEPLOY=false, the gate calls AskUserQuestion (approve / narrow-scope / abort) and blocks until the user answers. AUTO_DEPLOY=true overrides AUTO_PROCEED=false for this gate (standing approval). Forwarded from /auto.
• SANITY_FIRST = true — Run the sanity-stage experiment first (smallest, fastest) before launching the rest. Catches setup bugs early.
• MAX_PARALLEL_RUNS = 4 — Maximum number of experiments to deploy in parallel (limited by available GPUs). For Phase 4's queue dispatch path (Phase 4.B), this becomes max_parallel: in the /experiment-queue manifest. For the direct dispatch path (Phase 4.A), it's the in-skill concurrency cap on /run-experiment calls.
• BATCH_DISPATCH = auto — Phase 4 dispatch routing rule. auto (default): per the Phase 4.0 table — milestones with ≥ 10 runs, depends_on, grid expansions, or ≥ 3-seed × ≥ 3-config multi-seed sweeps go to /experiment-queue; smaller ad-hoc milestones go to /run-experiment. queue: force every milestone to /experiment-queue (use when you know the workload benefits from OOM retry + stale cleanup even at small sizes). direct: force every milestone to /run-experiment (use only when debugging the queue scheduler itself; emits a warning if any milestone would have triggered the queue rule under auto). Forwarded from /auto.
• BASE_REPO = null — GitHub repo URL to use as base codebase. When set, clone the repo first and implement experiments on top of it. When null, write code from scratch or reuse existing project files.
• COMPACT = false — When true, (1) read idea-stage/IDEA_CANDIDATES.md instead of full idea-stage/IDEA_REPORT.md if available, (2) append experiment results to EXPERIMENT_LOG.md after collection.
• RESUME = false — When true, each phase checks if its primary artifact already exists non-empty and skips itself if so (see "Resume protocol" below). Useful for picking up after a crash. Default false = every phase always runs from scratch and overwrites prior artifacts. Resume never deletes pre-existing files.
• GPU_ID = auto — GPU device(s) to use for sanity and full-suite runs. auto (default) inherits from environment / launcher. A single id (0) or comma-list (4,5,6,7) causes Phase 3 (sanity) and Phase 4 (deploy) to pass CUDA_VISIBLE_DEVICES=<GPU_ID> as the first positional argument to /run-experiment — the run-experiment skill then exports this env var before launching the experiment subprocess (do not treat it as a shell prefix; /run-experiment is a Skill invocation, not a shell command). Also record the effective CUDA_VISIBLE_DEVICES into each run's run.sh so reproductions land on the same devices. Override: — gpu-id: 4,5,6,7. When GPU_ID lists multiple devices and MAX_PARALLEL_RUNS > 1, partition devices across concurrent runs (e.g., GPU_ID=4,5,6,7 + 2 parallel → run A on 4,5, run B on 6,7); do not co-schedule two runs on the same device unless memory measurements confirm fit. Forwarded from /auto and from agents/experiment.md; /auto-verify follows the same convention for verify variants.

Standalone overrides: /auto-experiment "EXPERIMENT_PLAN.md" — compact: true, base-repo: https://github.com/org/project, research-domain: vision-encoders, mechanism-routing: auto.

Resource-Fidelity Harness (the reproduction combination)

Active only when the resource_fidelity: strict marker is present in the top metadata of refine-logs/FINAL_PROPOSAL.md / refine-logs/EXPERIMENT_PLAN.md — which /auto-claim stamps iff BEHAVIOR_SOURCE=given AND MECHANISM=given (the reproduction combination). There is no flag and no override: activation is purely marker-driven (only the reproduction combination stamps it; every other combination never does). When the marker is absent (any cost-aware combination), the existing cost-aware behavior is unchanged.

Scope: the main experiment only. The harness binds the main-experiment runs this skill (/auto-experiment) produces. It does not bind /auto-verify, whose deliberate model/dataset/method swaps are how it measures robustness — verify ignores the marker by design. When active, every phase of this skill obeys all five rules:

1. Exact models. Instantiate the precise model id(s) / size(s) the plan specifies. Never substitute a smaller / cheaper / distilled / quantized variant to save compute or memory (quantization or reduced precision is allowed only if the plan / task.md explicitly calls for it).
2. Exact data. Use the full specified dataset(s) and used_n / split. Never subset, down-sample, cap, or truncate the data to save time. A run that consumed less than the specified used_n is recorded failed, not done, and surfaced — never papered over with a subset note.
3. No must-run skipped. Run every must-run experiment at full specified scale (seeds, configs, grid points). Cost (GPU-hours) is not grounds to skip or thin a must-run milestone.
4. OOM → auto-scale-up across GPUs, never downscale. On OOM, resolve it only by science-neutral memory techniques — batch size ↓ (with gradient accumulation to preserve the effective batch), gradient checkpointing, sequence chunking, and automatically adding more GPUs to the run: query free GPUs (nvidia-smi, memory.used < 500 MiB), add them and enable sharding, relaunch. If the script is single-GPU (model.cuda() / .to(device)), auto-convert its loading to device_map="auto" / FSDP / CPU-or-disk offload — and before trusting it on the full model, verify the conversion is wired correctly with a numeric-equivalence check on a fit-on-one-GPU proxy slice (single-GPU vs sharded outputs/loss must match within tolerance; sharding is a mathematically equivalent transform, so this step only catches a mis-wired edit, not the sharding itself). Keep adding free GPUs up to OOM_MAX_GPUS (default 4) while leaving MAX_PARALLEL_RUNS headroom for sibling runs. Only when the cap is reached (or no free GPU remains) and CPU/disk offload is also exhausted do you HALT and report — write Pipeline status = halted-at-experiment: strict-OOM-after-scaleup with the scale-up trace (started on N GPUs, auto-expanded to M, sharding mode, equivalence-check result, final memory shortfall) and next-round options (add GPUs / bigger machine / explicitly authorize quantization in task.md / explicitly down-scope the plan / switch to discovery). Never shrink the model or subset the data to make it fit — that is the one forbidden remedy, even in full-auto.
5. Sanity is exempt but not a substitute. The smoke test may run at a tiny scale (its only job is catching setup bugs). The real runs that produce results must use the exact resources above; never report a sanity-scale run as a reproduction result.

Inputs

This skill expects one or more of:

1. refine-logs/EXPERIMENT_PLAN.md (best) — claim-driven experiment roadmap from /experiment-plan
2. refine-logs/EXPERIMENT_TRACKER.md — run-by-run execution table
3. refine-logs/FINAL_PROPOSAL.md — method description for implementation context
4. idea-stage/IDEA_CANDIDATES.md — compact idea summary (preferred when COMPACT: true) (fall back to ./IDEA_CANDIDATES.md if not found)
5. idea-stage/IDEA_REPORT.md — full brainstorm output (fall back to ./IDEA_REPORT.md if not found)

If none exist, ask the user what experiments to implement.

Workflow

Resume protocol (only when RESUME = true)

Skip entirely if RESUME = false (default). When true, each phase below begins with a skip-if-present check:

Phase	Primary artifact (skip key)	Notes
1	refine-logs/EXPERIMENT_PLAN.md already loaded into memory (no separate file)	Always cheap; just re-parse.
1.1	refine-logs/EXPERIMENT_TIPS.md with committed: true	Skip routing if already committed; matched tips must still be re-read into the current context each Phase 2 invocation, since context doesn't survive across calls.
1.25	refine-logs/EXPERIMENT_RESULTS.md with phenomenon_status: set to a terminal value (not-established, or inconclusive after the retry budget)	The M0 gate already ran and ended the stage — do not re-run M0 or proceed to mechanism. Re-emit the same terminal phenomenon_status in the return so /auto re-applies the early exit. If phenomenon_status is established / conditional (or M0 partially ran), fall through and let Phase 1.5+ resume normally. Only applies when the plan has a milestone with kind: phenomenon-validation (BEHAVIOR_SOURCE ∈ {given-validation, discovery}); otherwise this phase doesn't exist.
1.5	refine-logs/MECHANISM_ROUTING.md with committed: true (or routing: not-applicable) and a ## Plan reconciliation section whose reconciliation_status is set (ok/escalate/n/a)	Skip routing entirely only if both hold. If committed: true but reconciliation is missing (a prior run crashed between commit and Step 7), do not skip — resume at Step 7 to complete reconciliation before Phase 2. If reconciliation_status: escalate, HALT (do not proceed to build).
2	runs/<run_id>_<short_purpose>/ directory non-empty for every run referenced by the plan	Per-run granularity: skip code generation for runs whose directory already has code. Path convention matches the "Output Directory Naming" section below.
2.5	reviewer-approved marker in run directory (e.g., )

Log every skip as [resume] phase <N> skipped — <reason>. Resume never deletes pre-existing files. To force a phase to re-run, delete its primary artifact (or, for Phase 4, the run directory).

Phase 1: Parse the Experiment Plan

Read EXPERIMENT_PLAN.md and extract:

1. Run order and milestones — which experiments run first (sanity → baseline → main → ablation → polish)
2. For each experiment block:
   • Dataset / split / task
   • Compared systems and variants
   • Metrics to compute
   • Setup details (backbone, hyperparameters, seeds)
   • Success criterion
   • Priority (MUST-RUN vs NICE-TO-HAVE)
3. Compute budget — total estimated GPU-hours
4. Method details from FINAL_PROPOSAL.md — what exactly to implement
5. Mechanism routing hints (under mechanism_strategy:, written by the claim stage from cross-round memory) — the chosen direction; any families_already_settled: [<families>] (families already confirmed/refuted for this behavior+direction, to be excluded at routing — Phase 1.5); and any explicit family: pin. The list is absent (or omitted) when there is nothing to exclude — round 1, a new phenomenon, or a non-mechanism plan.

Present a brief summary:

📋 Experiment plan loaded:
- Milestones: [N] (sanity → baseline → main → ablation)
- Must-run experiments: [N]
- Nice-to-have: [N]
- Estimated GPU-hours: [X]

Proceeding to implementation.

Data Rules — load and check now (data design). Must load skills/data-rule/SKILL.md and validate the plan's data against its four rules (provenance / splits / labels / sample-size floor). Loaded once here, it governs all data use through the rest of the workflow — unconditional, not the Phase 1.1 symptom routing. Surface any violation before proceeding.

Phase 1.1: Experiment-Tips Routing

Before any code is written, route the parsed plan through /experiment-tips — the routing entry point under skills/experiment-tips/. Tips encode hard-won conventions that prevent silent reproducibility / overclaim failure modes. Catching them at plan-implementation time is free; catching them after a full deploy wastes GPU hours.

Routing flow:

1. Read the artifacts that drive the routing: refine-logs/EXPERIMENT_PLAN.md, refine-logs/FINAL_PROPOSAL.md. Extract: task description, datasets, models, declared n_pairs / n_examples, intervention sites (block / layer / token / residue), steering parameter names (α, dose, magnitude, scale, coefficient), and any adapter-based fine-tune milestone (LoRA / QLoRA / PEFT).

2. Invoke /experiment-tips as the routing entry point. It matches the plan against the symptom-level trigger table in skills/experiment-tips/SKILL.md and returns a list of tip folders to load.

3. For each matched tip, load skills/experiment-tips/<tip>/SKILL.md in full into working memory. Hard requirement: the routing previews in experiment-tips/SKILL.md are deliberately thin — every implementation detail lives in the tip's SKILL.md, not in the routing file. Acting on the preview alone is forbidden.

4. Write refine-logs/EXPERIMENT_TIPS.md with the routing decision:

   # Experiment Tips Routing
   
   <!-- Metadata block (parsed by /auto orchestrator resume check). -->
   committed: true
   matched_tips:
     - <tip-folder-1>
     - <tip-folder-2>
   
   ## Matches
   
   1. **<tip-folder>** — <one-line trigger that fired>
      - convention to adopt: <one-line summary from the tip's own SKILL.md — not from the routing preview>
   2. **<tip-folder>** — <one-line trigger that fired>
      - convention to adopt: <one-line summary from the tip's own SKILL.md — not from the routing preview>
   
   ## No-match log
   <if a borderline trigger didn't fire, note it here for audit>
   

   If no tip matches, write a stub EXPERIMENT_TIPS.md containing committed: true and matched_tips: [] plus a one-line note explaining why none matched (e.g., "behavioral-only proposal, no representation interventions"). Always write the file — it's the audit anchor for Phase 2 to confirm tips were considered.

Hard requirement: before Phase 1.25, the experiment agent's transcript must show an actual /experiment-tips invocation. Bypassing the skill or fabricating refine-logs/EXPERIMENT_TIPS.md by hand is forbidden. If the transcript shows no invocation, or the file is absent or hand-written, re-run Phase 1.1.

Hard requirement: every matched_tips entry in EXPERIMENT_TIPS.md and its convention to adopt must be adopted in all downstream experiments.

Phase 1.25: Phenomenon-Validation Gate (BEHAVIOR_SOURCE ∈ {given-validation, discovery})

Runs only when EXPERIMENT_PLAN.md contains a milestone carrying the machine marker kind: phenomenon-validation (conventionally titled M0) — i.e. the claim stage ran with BEHAVIOR_SOURCE ∈ {given-validation, discovery}. Detect by the kind: phenomenon-validation field, never by the milestone's title (the title may be phrased or localized freely; only the field is the stable contract). If no milestone has that marker (a BEHAVIOR_SOURCE=given run, or any non-phenomenon plan), skip this phase entirely and set phenomenon_status = n/a. The principle: claim only assumed the phenomenon exists and proposed a mechanism for it — this gate is where the assumption is actually tested, before any mechanism compute is spent.

Run M0 first and alone at the plan's specified used_n, then decide a four-state phenomenon_status. Concretely:

• Implement only M0's code (the Phase 2 implementation step, scoped to M0's milestone). If CODE_REVIEW=true, M0's code gets one cross-model review pass (Phase 2.5) like any other run. M0's scale is set by the plan's used_n — the agent may not subset, cap, or down-sample M0's data to save compute. An M0 run that consumed less than the planned used_n is recorded failed, not done (Phase 5's used_n rule).
• Deploy M0 via /run-experiment (one run). This creates M0's runs/<run_id>_*/ directory and flips its EXPERIMENT_TRACKER.md row to done/failed through the normal mechanism — so the later Phase 4 deploy automatically skips re-running M0 (its per-run skip key is already satisfied), and resume is consistent. Do not hand-roll separate bookkeeping.
• Forward GPU_ID to M0's run exactly as Phase 4 would.

Then decide the verdict:

1. Lightweight integrity check on M0 before trusting its verdict: invoke /experiment-audit scoped to M0 (the same audit verify's Phase 2 uses). This separates "the phenomenon is genuinely absent" from "the M0 test itself was broken/underpowered".

2. Assign phenomenon_status:

   • established — M0 audit is clean AND the behavior reproduces per its pass criterion (paraphrase/seed/decoding robust, confounds controlled, adequate n, trivial-explanation ruled out). → Proceed to Phase 1.5 and the mechanism milestones (M1…Mn) normally.

   • conditional — M0 audit clean, but the behavior holds only under a subset of conditions. → Runtime-scope; do not edit the plan (EXPERIMENT_PLAN.md is owned by claim Phase 4.5). Run the mechanism milestones with inputs restricted to the condition subset where the phenomenon holds, record phenomenon_status: conditional + the boundary in EXPERIMENT_RESULTS.md, and tag the claim conditional — holds under <X> so /auto carries it into the ledger. Formally narrowing the claim text and re-planning is the iteration loop's type-③ job, not this stage. AUTO_PROCEED: true auto-continues with the narrowed scope; false → AskUserQuestion (continue with narrowed scope / terminate) and block.

   • inconclusive — M0's integrity audit FAILs; the phenomenon is untested, not disproven. → Do not terminate. Diagnose, fix at the lowest sufficient level, and re-run M0. Log every fix under M0's block in EXPERIMENT_RESULTS.md (what changed, old → new, audit rationale).

     • Script bug (crash, wrong path, tensor-shape error, eval bug) → auto-debug M0 (≤ 3 attempts), then re-run.
     • Run-level methodology (under-sampled within plan limits, seed not fixed, decoding drift) → adjust the M0 run invocation only; leave the plan alone.
     • Plan-level methodology (wrong metric, used_n too low, mis-specified coefficient / threshold / hyperparameter, inadequate dataset size or split) → most commonly under-tuned knobs (finetune lr / LoRA rank / epochs / batch, steering α / target layer, thresholds, decoding temperature). — not a general re-audit — passing M0's realized evidence (loss / effect size / refusal rate / seed variance) and routing to the matching sub-skill: for fine-tune knobs, for α / β / dose of any additive intervention (steering / CAA / DAS / SAE / ROME), for the target layer / site / window. Apply the returned search range / scan grid by editing in place, confined to the implicated field(s); do not touch the claim, phenomenon description, or milestone graph.

🚨 Phase 1.25 hard constraints (non-negotiable):

1. ~20 GPU-hours are pre-allocated. Time/GPU cost is not a valid reason to skip a sweep or retry — only the terminal triggers below (5-iteration budget / no actionable sub-skill / trigger (c)) may stop iteration.
2. Terminal not-established / inconclusive skips all downstream mechanism milestones (M1…Mn). You must exhaust matching tuning sub-skills and retries within the 5-attempt budget before settling on either verdict.

Mechanism milestones declare depends_on: [M0], so even outside this gate the queue will not launch them before M0 completes; the gate adds the verdict-based branch (run status alone is not enough — a clean M0 that shows no effect must stop the pipeline, which depends_on cannot express).

Phase 1.5: Mechanism-Family Routing

A committed mechanism family is a hard precondition for any Phase 2 code that touches an internal object (layer, head, neuron, SAE feature, weight, input feature).

CHOSEN_FAMILY takes precedence over MECHANISM_ROUTING. Whenever CHOSEN_FAMILY is set to a real family (anything other than unset / none / not-applicable), go straight to the routing flow's Mode-B commit (Step 6) regardless of the MECHANISM_ROUTING value — Step 6 commits CHOSEN_FAMILY (and builds the routing scaffold around it when MECHANISM_ROUTING.md is absent or does not list it). This is what makes all three direct-commit sources work: the first-class MECHANISM=given path and the family: pin (no Mode A ran → file may be absent → Step 6 builds it), and the AUTO_PROCEED=false post-mini-prompt build call (Mode A already wrote the candidates → Step 6 flips committed). A bare MECHANISM_ROUTING=skip therefore never silently proceeds without a committed family when CHOSEN_FAMILY is set. CHOSEN_FAMILY = not-applicable is the behavioral-only sentinel (a behavioral-only proposal, incl. the MECHANISM=given behavioral-only reproduction): treat it exactly like MECHANISM_ROUTING = not-applicable — write the routing: not-applicable stub, run no mechanism milestone — regardless of the MECHANISM_ROUTING value.

When CHOSEN_FAMILY is unset, behavior is gated by MECHANISM_ROUTING:

• MECHANISM_ROUTING = skip — assume refine-logs/MECHANISM_ROUTING.md already exists (or is intentionally absent for a behavioral proposal). Read it if present, otherwise log [routing] skipped — no manifest, proceeding without routing and continue to Phase 2.
• MECHANISM_ROUTING = not-applicable — write a stub refine-logs/MECHANISM_ROUTING.md containing routing: not-applicable and a one-line justification pulled from FINAL_PROPOSAL.md, then continue.
• MECHANISM_ROUTING = auto (default) — run the routing flow below.

Routing flow (auto mode):

1. Use the Phase 1 parse already in memory to identify routing inputs: chosen claim(s), the internal objects the method targets (layer / head / neuron / SAE feature / weight / input feature), the read sites, any mechanism families already named in FINAL_PROPOSAL.md, and the cross-round routing hints from EXPERIMENT_PLAN.md (item 5) — the families_already_settled: [<families>] avoid-set (Rule 1) and any family: pin. No new file reads needed at this step.

2. Invoke /mechanism-skills via the Skill tool — this is a hard requirement, not a description of behavior. The manifest must be derived from the catalog this skill loads at routing time, not from prior training knowledge. You are not permitted to write MECHANISM_ROUTING.md until you have actually read, in this turn, via the Skill tool:

   • skills/mechanism-skills/SKILL.md (the routing entry point listing all eleven families)
   • The SKILL.md of every family you intend to list as a candidate
   • The SKILL.md of every submethod underneath those families

   If any of these files is unread, re-invoke /mechanism-skills before generating candidates — do not write the manifest from memory.

   Pass RESEARCH_DOMAIN as the routing constraint:

   • If set to a specific tag (e.g., mechanistic-interpretability), pass it as — domain: <RESEARCH_DOMAIN> so candidates are restricted to the matching families.
   • If auto, do not pass — domain:; ask /mechanism-skills to infer from FINAL_PROPOSAL.md. On ambiguous inference, fall back to domain: general silently (same rule as the Constants section).

3. Produce 2–3 candidate family/submethod combinations from the catalog. Each candidate must use the canonical family name as it appears in skills/mechanism-skills/SKILL.md — one of: Causal Attribution, Circuit Discovery, Probing, Magnitude Analysis, Gradient Detection, Feature Dictionary Learning, Representation and Parameter Analysis, Vocabulary Projection, SHAP, Neural Feature Learning, Multi-Modal.The chosen_family and candidate_paths metadata fields must use canonical catalog names.

   Cross-round avoid-set (Rule 1): if (item 5) lists any families, them from the candidate list — they were already / for this behavior+direction, so re-routing to them redoes settled work. A family left is settled and still be proposed (ideally with a refined submethod). Note the exclusion in . If excluding leaves no viable catalog candidate for the direction, do not silently fall back to a settled family — surface it (the direction may itself be exhausted; a signal to pick a different direction or behavior next round).

Hard requirement: before Phase 2 implements any code that touches an internal object, refine-logs/MECHANISM_ROUTING.md must exist and contain a committed candidate (or routing: not-applicable) with a completed ## Plan reconciliation whose reconciliation_status: ok (or n/a when no method_sensitive fields were declared). If absent, fall back to MECHANISM_ROUTING=auto and re-run this phase; if reconciliation_status: escalate, HALT per Step 7.

Phase 2: Implement Experiment Code

Data precondition — every data loader / split / labeling / sample-size choice obeys the four Data Rules already loaded in Phase 1 (skills/data-rule/SKILL.md). Re-read that file if it has fallen out of context.

Routing precondition — refine-logs/MECHANISM_ROUTING.md was produced by Phase 1.5; load the committed candidate and use its listed scripts/ and references/ as reference material. Do not copy verbatim. If the manifest says routing: not-applicable, skip mechanism-specific scaffolding.

If BASE_REPO is set — clone the repo first:

git clone <BASE_REPO> base_repo/
# Read the repo's README, understand its structure, find entry points
# Implement experiments by modifying/extending this codebase

Script granularity — default to one script per dispatch unit (model × dataset), not per milestone. Loads the model once, keeps activations in memory across phases, makes Phase 4.A trivially nohup ... & wait-parallel. As a reference pattern, a single ~400-line script that runs Block 1+2+3 together keeps activations resident across phases. Only split when milestones need different runtime stacks (e.g., one needs vllm, another transformers hooks) or share < 30% code.

For each dispatch unit (in plan order), write the experiment script(s):

1. Check existing code — scan the project (or cloned base_repo/) for existing experiment scripts, model code, data loaders. Reuse as much as possible.

2. Implement missing pieces:

   • Training scripts with proper argparse (all hyperparameters configurable)
   • Evaluation scripts computing the specified metrics
   • Data loading / preprocessing if needed
   • Baseline implementations if not already present
   • Fixed random seeds for reproducibility
   • Results saved to JSON/CSV for later analysis
   • Proper logging (wandb if configured in CLAUDE.md)

3. Follow the plan's run order — within a single monolithic script, the milestone sequence becomes function-call order (sanity sub-routine → baseline → main → ablations). Across scripts, dispatch order still matches the plan.

4. Self-review before deploying:

   • Are all hyperparameters from EXPERIMENT_PLAN.md reflected in argparse?
   • Is the random seed fixed and controllable?
   • Are results saved in a parseable format (JSON/CSV)?
   • Does the code match FINAL_PROPOSAL.md's method description?
   • Could two of the scripts you just wrote be merged into one --mode {a,b} script without losing clarity? (If yes, merge them — see granularity guidance above.)

Phase 2.5: Cross-Model Code Review (when CODE_REVIEW = true)

Skip this step if CODE_REVIEW is false.

Before deploying, send the experiment code to the external LLM reviewer for review:

mcp__llm-chat__chat:
  prompt: |
    Review the following experiment implementation for correctness.

    ## Experiment Plan:
    [paste key sections from EXPERIMENT_PLAN.md]

    ## Method Description:
    [paste from FINAL_PROPOSAL.md]

    ## Implementation:
    [paste the experiment scripts]

    Check for:
    1. Does the code correctly implement the method described in the proposal?
    2. Are all hyperparameters from the plan reflected in the code?
    3. Are there any logic bugs (wrong loss function, incorrect data split, missing eval)?
    4. Is the evaluation metric computed correctly?
    5. **CRITICAL: Does evaluation use the dataset's actual ground truth labels — NOT another model's output as ground truth?** This is a common and severe bug.
    6. **Does the scorer match the answer format?** Strict exact-match on free-form QA, or a 0.5 binarization on a near-zero-variance label, will collapse the label distribution. If the experiment scores LLM-generated answers, the scorer must be chosen to match the answer format (e.g., multiple-choice → exact-match on the option token; short free-form → normalized exact-match or token-F1; long free-form → LLM-judge or substring containment with a documented rubric). Confirm a pilot **label-floor check** has been run: on a held-out 50–100 example slice, the score distribution should not collapse to a near-constant (variance ≥ 0.05 on a [0,1] metric). A collapsed pilot is **not** a normal halt — it is a **Round-End Decision** (`ended-needs-decision (experiment: scorer-invalid)`): stop before the full run, do **not** auto-swap in another scorer (choosing the scorer that matches the answer format is a judgment call — an auto-swap risks trading one bad scorer for another), and surface the collapse evidence (measured variance, the scorer/answer-format mismatch) plus the recommended scorer fix so the user fixes it and re-runs. See `auto/SKILL.md` → "Round-End Decision".
    7. Any potential issues (OOM risk, numerical instability, missing seeds)?

    For each issue found, specify: CRITICAL / MAJOR / MINOR and the exact fix.

On review results:

• No CRITICAL issues → proceed to Phase 3
• CRITICAL issues found → fix them, then re-submit for review (max 2 rounds)
• llm-chat MCP unavailable → skip silently, proceed to Phase 3 (graceful degradation)

Phase 3: Sanity Check (if SANITY_FIRST = true)

GPU pinning. If GPU_ID is not auto, pass CUDA_VISIBLE_DEVICES=<GPU_ID> as the first positional arg to every /run-experiment invocation from this phase onward (sanity below, the full suite in Phase 4, and any auto-debug re-runs). /run-experiment itself parses this leading positional and exports it as an env var in the experiment subprocess. Record the effective CUDA_VISIBLE_DEVICES in each run's run.sh so reproductions land on the same devices.

Before deploying the full experiment suite, run the sanity-stage experiment:

/run-experiment CUDA_VISIBLE_DEVICES=<GPU_ID> [sanity experiment command]

(When GPU_ID = auto, drop the CUDA_VISIBLE_DEVICES=... positional and let the launcher decide.)

Wait for completion. Verify:

• Training loop runs without errors
• Metrics are computed and saved correctly
• GPU memory usage is within bounds
• Output format matches expectations

If sanity fails → auto-debug before giving up (max 3 attempts):

1. Read the error — parse traceback, stderr, and log files

2. Diagnose — classify the failure:

   • OOM → reduce batch size or enable gradient checkpointing
   • ImportError → install missing package
   • FileNotFoundError → fix path or download data
   • CUDA error → check GPU availability, reduce model size
   • NaN/divergence → reduce learning rate, check data preprocessing

   Resource-Fidelity Harness override (when active): the "reduce model size" remedy above is forbidden, and so is any data subsetting. Resolve OOM by the harness's auto-scale-up path — batch size ↓ / gradient accumulation / gradient checkpointing / sequence chunking, plus auto-adding free GPUs with sharding (auto-converting a single-GPU script to device_map="auto" / FSDP / offload, equivalence-checked on a fit-on-one-GPU proxy slice) up to OOM_MAX_GPUS. Only after the cap (or no free GPU) and offload are exhausted, HALT and report rather than swapping in a smaller model or trimming the data. (See "Resource-Fidelity Harness" rule 4.)

3. Fix and re-run — apply the fix, re-run sanity

4. Attempt 2+ still failing? → Call in Codex rescue (if Codex plugin installed): Before the next retry, invoke /codex:rescue to get a second opinion on the root cause. Codex independently reads the code and error logs — it may spot issues Claude missed (wrong tensor shapes, subtle import shadowing, config mismatches, etc.). Apply its suggested fix, then re-run.

   • If /codex:rescue is not available (plugin not installed), continue with Claude's own diagnosis

5. Still failing after 3 attempts? → stop, report the failure with all attempted fixes and error logs. Do not proceed with broken code.

Never give up on the first failure. Most experiment crashes are fixable without human intervention.

Phase 4: Deploy Full Experiments

Deploy experiments following the plan's milestone order. Carry forward the GPU pinning from Phase 3 — every command below gets CUDA_VISIBLE_DEVICES=<GPU_ID> as the leading positional arg to /run-experiment when GPU_ID is not auto:

/run-experiment CUDA_VISIBLE_DEVICES=<GPU_ID> [experiment commands]

Phase 4.0: Route by milestone size and dependencies

Before dispatching, classify each milestone into one of two dispatch paths:

Condition (any one triggers)	Dispatch path
Milestone has ≥ 10 runs OR declares depends_on: in EXPERIMENT_PLAN.md OR uses a grid expansion (seeds: [...] × params: [...]) OR is a multi-seed sweep (≥ 3 seeds × ≥ 3 configurations)	/experiment-queue (this milestone)
All other milestones (≤ 5 ad-hoc runs, no dependencies, no grid)	/run-experiment (this milestone, one dispatch per run)
6–9 runs, no dependencies	/experiment-queue recommended; /run-experiment acceptable — emit warning [phase-4] milestone <name> has <N> runs (6-9): prefer /experiment-queue for OOM safety, falling back to /run-experiment per MAX_PARALLEL_RUNS=<n>. Default behavior: still use /run-experiment (backward compatible). To force queue, set BATCH_DISPATCH=queue on the CLI.

BATCH_DISPATCH knob (optional, forwarded from /auto or default auto):

• auto (default) — follow the table above.
• queue — force every milestone to /experiment-queue regardless of size.
• direct — force every milestone to /run-experiment; emit a [phase-4] BATCH_DISPATCH=direct overriding the recommended /experiment-queue for <N>-run milestone — OOM safety net disabled warning if any milestone would have triggered the queue rule. Use only when intentionally bypassing the queue (e.g., debugging the scheduler itself).

Log the routing decision per milestone: [phase-4] milestone=<name> runs=<N> deps=<depends_on or none> → /<run-experiment|experiment-queue>.

Phase 4.A: Direct dispatch (small milestones — /run-experiment path)

Launch all parallel runs of the milestone in ONE Bash call using nohup ... & wait (template in /run-experiment Step 4 "Parallel runs"). Shell wait joins direct children only — do not use pgrep -f '<script>.py' (it self-matches the polling bash's own cmdline). Remote/SSH: filter screen -ls by session name. After wait returns, call /monitor-experiment to finalize each runs/<run-id>/cost.json.

Phase 4.B: Queue dispatch (large milestones — /experiment-queue path)

For each milestone routed to the queue path:

1. Build the manifest. Derive a YAML grid spec from EXPERIMENT_PLAN.md's milestone block. If the plan declares parameters as lists (e.g., seeds: [42, 200, 201], n_hidden: [64, 128, 256]), pass them as grid: axes; otherwise expand each run as a single jobs: entry. Carry forward:

   • gpus: from GPU_ID (split into list; auto → omit and let the queue auto-discover)
   • max_parallel: from MAX_PARALLEL_RUNS
   • gpu_free_threshold_mib: 500 (queue default; raise via plan if known shared)
   • oom_retry: {delay: 120, max_attempts: 3} (queue defaults)
   • cwd: and conda: from CLAUDE.md / environment
   • depends_on: from the plan, naming the upstream milestone(s)

2. Invoke /experiment-queue with the manifest:

   /experiment-queue <path-to-manifest.json or grid_spec.yaml>
   

3. Wait for batch completion. Poll $REMOTE_RUN_DIR/queue_state.json until every job is completed or stuck (see /experiment-queue Step 4 for the jq one-liner). Do NOT proceed to Phase 5 with stuck jobs unresolved — surface them in the deploy report.

4. Finalize per-run cost manifests. For each completed job in queue_state.json, invoke /monitor-experiment on its run ID to write runs/<job-id>/cost.json (Step 3.6 in monitor — see "Coordination with cost.json and queue_state.json" in /experiment-queue for why this split exists).

5. Continue to Phase 5 with the union of all milestones' results.

Multi-GPU partitioning. When GPU_ID lists multiple devices and MAX_PARALLEL_RUNS > 1, split the device list across concurrent runs (one disjoint subset per run). Example: GPU_ID=4,5,6,7 with MAX_PARALLEL_RUNS=2 → run A gets CUDA_VISIBLE_DEVICES=4,5, run B gets CUDA_VISIBLE_DEVICES=6,7. Do not co-schedule two runs on the same device unless memory measurements confirm it fits.

Auto-clamp when fewer devices than parallelism. If GPU_ID is an explicit list with len(GPU_ID) < MAX_PARALLEL_RUNS (e.g., GPU_ID=0 + default MAX_PARALLEL_RUNS=4), automatically lower the effective parallelism to len(GPU_ID) rather than co-scheduling — multiple runs on one GPU usually OOM. Log [gpu-clamp] MAX_PARALLEL_RUNS lowered from <orig> to <len(GPU_ID)> — only <len(GPU_ID)> device(s) requested via GPU_ID. Skip the clamp when GPU_ID = auto (the launcher decides).

🚦 Experiment Gate (before deploy). Branch on AUTO_DEPLOY × AUTO_PROCEED:

AUTO_DEPLOY	AUTO_PROCEED	Action
true	true	Deploy silently. Log [experiment-gate] proceeded — AUTO_DEPLOY=true, AUTO_PROCEED=true.
true	false	Deploy silently — AUTO_DEPLOY=true is treated as standing approval. Log [experiment-gate] proceeded — AUTO_DEPLOY=true overrides AUTO_PROCEED=false.
false	true	Deploy silently — AUTO_PROCEED=true is full-auto and overrides AUTO_DEPLOY=false. Log [experiment-gate] proceeded — AUTO_PROCEED=true overrides AUTO_DEPLOY=false (full-auto mode). (Print the checkpoint summary below into the deploy log for the human reader, but do not block.)
false	false	Call AskUserQuestion with three options — approve / narrow scope / abort — and block until the user answers. On narrow scope, ask which milestones to keep, then re-enter the gate. On abort, stop the skill with [experiment-gate] aborted by user.

The checkpoint summary (logged in every cell for the human reader; only blocks in the AUTO_DEPLOY=false AND AUTO_PROCEED=false cell via AskUserQuestion):

🔧 Code implementation complete. Ready to deploy:

Milestone 0 (sanity): [status — passed/pending]
Milestone 1 (baseline): [N experiments, ~X GPU-hours]
Milestone 2 (main method): [N experiments, ~X GPU-hours]
Milestone 3 (ablations): [N experiments, ~X GPU-hours]

Total estimated: ~X GPU-hours on [N] GPUs

Deploy now? Or review the code first?

Use reconciled cost. The GPU-hours shown here must be the post-reconciliation estimate (the gpu_hours: ... → revised ~Y value from MECHANISM_ROUTING.md's ## Plan reconciliation, Phase 1.5 Step 7), not the stale plan figure — the committed submethod may cost more or less than the claim stage could estimate before the method was known. When no field was re-bound, the plan figure stands.

Implementation note: AskUserQuestion has no timeout. Only call it in the AUTO_DEPLOY=false AND AUTO_PROCEED=false cell so unattended overnight runs cannot deadlock here.

Phase 5: Collect Initial Results

As experiments complete:

1. Parse output files (JSON/CSV/logs) for key metrics

2. Training quality check — if W&B data is available (CLAUDE.md has wandb: true and wandb_project), invoke /training-check to detect NaN, loss divergence, plateaus, or overfitting. If W&B is not configured, skip silently.

3. Update refine-logs/EXPERIMENT_TRACKER.md — EXPERIMENT_TRACKER.md is owned by /auto-claim Phase 4.5 (it writes the initial plan-level table with every row at Status: pending). This phase updates rows in place: flip each plan run's Status column (pending → running → done / failed) and fill in Notes / result columns. Do not overwrite the file wholesale — the plan rows from Phase 4.5 are the audit trail. New rows are only appended by Phase 5.6 when ablations get planned post-baseline; never by Phase 5 itself.

   Supersede on re-task (no parallel narratives). When this run is a re-task of a rejected earlier attempt (the orchestrator re-dispatched under a corrected requirement, or a resume redoing a milestone), do not leave the old result standing beside the new one. Flip the milestone's tracker row to the new run's terminal status in place, and in EXPERIMENT_RESULTS.md rewrite that milestone's section in place, marking the prior narrative superseded: <old run-id> (one line naming what changed and why). The reader — and the downstream ledger — must see exactly one current narrative per milestone. Appending a second, contradictory result section is the dirty-data failure this rule exists to prevent.

   Update cadence: flip Status to running before launch and done/failed immediately after each run returns — never batch updates to the end of Phase 5. Makes hang detection possible (a row stuck on running past 2× ETA is the signal).

4. Check success criteria from EXPERIMENT_PLAN.md — did each experiment meet its bar?

5. Record the data actually used — for each claim/block, the realized (the number of examples/pairs the runs actually consumed), reconciled against the plan's and from EXPERIMENT_PLAN.md. Carry over /; flag and correct them only if the realized data differed from plan, and note any subsetting. realized MUST equal the specified — any shortfall means the run is , not ; record it as failed and surface it rather than entering a subset note. (The "Subset note" column should read for every row under the active harness; a non-empty subset note is a harness violation.)

# Initial Experiment Results

**Date**: [today]
**Plan**: refine-logs/EXPERIMENT_PLAN.md

## Data Actually Used
Per claim/block, reconciled against the *planned* data in EXPERIMENT_PLAN.md (provenance: `existing` used as-is / `adapted` from existing / `constructed` from scratch):

| Claim/Block | Provenance | Source | Available N (total) | Used N (actual) | Subset note (if used < available) |
|-------------|-----------|--------|---------------------|-----------------|-----------------------------------|
| C1 / Block 1 | existing | [name] | [N] | [N] | [how/why subsetted, or —] |

## Results by Milestone

### M0: Sanity — PASSED
- [result]

### M1: Baselines
| Run | System | Key Metric | Status |
|-----|--------|-----------|--------|
| R001 | baseline_1 | X.XX | DONE |

### M2: Main Method
| Run | System | Key Metric | Status |
|-----|--------|-----------|--------|
| R003 | our_method | X.XX | DONE |

### M3: Ablations
...

## Summary
- [X/Y] must-run experiments completed
- Main result: [positive/negative/inconclusive]
- Ready for /auto-verify: [YES/NO]

## Next Step
→ /auto-verify

Phase 5.5: Write Compact Log (when COMPACT = true)

Skip entirely if COMPACT is false.

Append each completed experiment to EXPERIMENT_LOG.md:

## [Run ID] — [timestamp]
- **System**: [method name]
- **Config**: [key hyperparameters]
- **Result**: [primary metric = X.XX]
- **Verdict**: [positive / negative / inconclusive]
- **Reproduce**: `python train.py --config configs/run_id.yaml --seed 42`

This structured log survives session recovery — downstream skills read it instead of parsing screen output.

Phase 5.6: Auto Ablation Planning

After main experiments (M2) complete with positive results, invoke /ablation-planner to design ablation studies:

• Read the main results and method description
• Generate a claim-driven ablation plan: which components to remove, what to compare, expected outcomes
• Append ablation blocks to refine-logs/EXPERIMENT_PLAN.md and refine-logs/EXPERIMENT_TRACKER.md
• If main results are negative or inconclusive, skip ablation planning and note in the summary

If /ablation-planner is not available, skip silently — the existing EXPERIMENT_PLAN.md ablation blocks (if any) remain unchanged.

Phase 6: Handoff

Present final status:

🔬 Experiment bridge complete:
- Implemented: [N] experiment scripts
- Deployed: [N] experiments on [M] GPUs
- GPU devices: [effective CUDA_VISIBLE_DEVICES used, or "auto (inherited from launcher)" when GPU_ID=auto]
- Completed: [X/Y] must-run, [A/B] nice-to-have
- Main result: [one sentence]

Results: refine-logs/EXPERIMENT_RESULTS.md
Tracker: refine-logs/EXPERIMENT_TRACKER.md

Next step: /auto-verify to stress-test the passed claims.

Output Directory Naming

Every run from this skill writes to a dedicated subdirectory under runs/. Use the format:

runs/<run_id>_<short_purpose>/

• <run_id> — the Run ID from the Quick-Glance Run Order table in EXPERIMENT_PLAN.md (e.g., A1, B1, C1). Keeps the directory cross-referenceable with the plan, tracker, and results tables.
• <short_purpose> — at most 4 snake_case words summarizing what the run does (no spaces, no hyphens, no punctuation). Stay descriptive over clever: cache_activations, layer_sweep, intervention_variants, label_permutation_null, seed_stability, ood_transfer, loglik_shift.

Examples (good):

runs/A1_cache_activations/
runs/B1_layer_sweep/
runs/D1_intervention_variants/
runs/G1_seed_stability/

Examples (bad):

runs/A1/                       # opaque, no purpose
runs/run_A1/                   # redundant "run_" prefix
runs/cache_activations/        # no run_id — can't cross-reference plan
runs/A1-cache-activations/     # use underscores, not hyphens
runs/A1_a_thorough_cache_of_all_layer_activations_for_sst2/   # too long

When writing EXPERIMENT_TRACKER.md and EXPERIMENT_RESULTS.md, the Run column still uses the bare ID (A1), but every output path reference in prose / code uses the full runs/A1_cache_activations/ form.

Key Rules

• Large file handling: If the Write tool fails due to file size (typical when EXPERIMENT_RESULTS.md accumulates many runs, or when EXPERIMENT_TRACKER.md grows past Write's limits), immediately retry using Bash with a heredoc to write in chunks:

  cat << 'EOF' > refine-logs/EXPERIMENT_RESULTS.md
  ...content...
  EOF
  

  Do NOT ask the user for permission — just do it silently. This protocol mirrors auto, auto-claim, and auto-iteration-loop.

• Data Rules. Obey skills/data-rule/SKILL.md (loaded in Phase 1) for all data — provenance, splits, labels, sample-size floor. Most critical: evaluation ground truth comes from the dataset, NEVER from another model's output.

• Follow the plan. Do not invent experiments not in EXPERIMENT_PLAN.md. If you think something is missing, note it but don't add it.

• Sanity first. Never deploy a full suite without verifying the sanity stage passes.

• Reuse existing code. Scan the project before writing new scripts. Extend, don't duplicate.

• Save everything as JSON/CSV. The auto-review-loop needs parseable results, not just terminal output.

• Update the tracker. EXPERIMENT_TRACKER.md should reflect real status after each run completes.

• No pgrep -f wait loops. They self-match the polling bash's own cmdline → infinite loop (caused a 5h hang on 2026-05-18). Use shell wait (joins direct children only), screen -ls filtered by session name, or /experiment-queue polling queue_state.json. Any fallback polling must include self-exclusion (grep -v $$) and a hard timeout (≤ 2× milestone ETA).

• GPU budget (a mandate to spend, not just a ceiling). A declared budget does two things, and both bind. (1) Never abandon: do not reject or skip an experiment for insufficient GPU budget unless the session's cumulative GPU-hours have reached a maximum explicitly agreed in task.md; if no maximum is agreed, treat the budget as unlimited. (2) Never simplify to save cost while under budget: in a cost-aware run (no resource_fidelity: strict marker) where declares a budget that covers the fuller run, prefer the option — do swap in a smaller / cheaper / distilled / quantized model, subset or down-sample the data, or drop a planned must-run experiment . A generous budget the default scale (the larger of the planned model / data, more seeds, more grid / checkpoint points) up to what the budget covers; only genuinely exceeding the cap forces trimming, and then you rather than silently downscaling (per the Hard-constraints rule). Cost-aware therefore means — not . this does forbid the ladder-of-evidence staging (a cheap correlational / attribution screen the causal confirmation) — that is a scientific design choice and each rung still runs at its proper scale; what's forbidden is shrinking the model / trimming the data / cutting must-run runs to save cost. Still track GPU-hours for the Phase 6 summary.

Composing with Other Skills

/auto-claim "direction"              ← Workflow 1: find + refine + plan
/mechanism-skills                    ← mechanism family + submethod routing — folded into this skill's Phase 1.5 (callable standalone for routing-only use; otherwise no need to chain separately)
/auto-experiment                     ← you are here (Workflow 1.5: route + implement + deploy)
/auto-verify                         ← Workflow 1.75: stress-test passed claims
/auto-iteration-loop "direction"     ← Workflow 2: review + iterate

Or use /auto for the autonomous idea → routing → experiments → verify → iteration chain.