auto

# Auto Pipeline: Idea → Experiments → Verify → Review End-to-end autonomous run for: **$ARGUMENTS** (when provided) or the project's `task.md` (when `$ARGUMENTS` is empty). Each stage runs in a dedicated agent (isolated context + configurable model). The orchestrator only sees each agent's final summary and the files on disk — it uses those to fire the gates between stages. > **⏰ First action — register the hourly notification timer.** Before any pipeline work, if `task.md` opts into notifications (see the Notifications rule in [Key Rules](#key-rules)), register a **recurring scheduled task** with `CronCreate` (cron `<off-minute> * * * *` — one call per hour, pick a minute ≠ 0/30; prompt: `/notify hourly`). This fires the hourly briefing on a real wall-clock timer instead of relying on the orchestrator remembering to poll during long waits. The timer is the **sole** source of the hourly cadence — the orchestrator never separately polls or manually fires `/notify hourly`. Register it **once, up front** — first `CronList` to check an equivalent `/notify hourly` job is not already scheduled (e.g. on resume), and skip if so. When `task.md` does **not** opt into notifications, skip this entirely (no timer). The timer covers only the hourly cadence; the event-driven `/notify` touchpoints (progress / done / halted / approval-needed) are separate and stay orchestrator-initiated (see [Key Rules](#key-rules)). **Direction source resolution:** - `$ARGUMENTS` non-empty → use it as the research direction; if `task.md` also exists, the claim agent treats `task.md` as authoritative detailed context (current behavior). - `$ARGUMENTS` empty + `task.md` exists → use `task.md` as the sole direction source; pass `direction: ""` to the claim agent so it relies entirely on `task.md`. - `$ARGUMENTS` empty + `task.md` absent → stop and report `[direction] no input — provide $ARGUMENTS or create task.md`. Do not invent a direction. ## Defaults | Flag | Default | Effect | |---|---|---| | `AUTO_PROCEED` | `true` | When `true`, gates skip the UI prompt entirely and directly pick the recommended option. When `false`, the orchestrator calls `AskUserQuestion` and waits for the user. | | `RESUME` | `false` | When `true`, the orchestrator skips any stage whose final artifacts already exist non-empty on disk, and forwards `resume: true` to each invoked agent so sub-skills can do phase-level skipping too. Useful for picking up after a crash, or for re-running only the missing stages. Default `false` = always run every stage from scratch (and overwrite previous artifacts). | | `REVIEW_LOOP` | `true` | Run iteration after verify; set `false` to stop at verify. | | `MODEL` | _none_ | Global model **family** alias applied to every stage whose `<STAGE>_MODEL` is unset. Accepts `opus` / `sonnet` / `haiku` (case-insensitive). Per-stage `<STAGE>_MODEL` always wins over this. When both are unset, the stage uses its frontmatter `model:` line if present, else inherits the session model (`claude --model …`). | | `<STAGE>_MODEL` | _none_ | Per-stage model **family** override for any of `CLAIM`, `EXPERIMENT`, `VERIFY`, `ITERATION`. Accepts **only family aliases**: `opus`, `sonnet`, `haiku` (case-insensitive). When unset, falls back to the global `MODEL`, then the agent's frontmatter `model:` line if present, else the session model (`claude --model …`). **Pinning lives only in `agents/<name>.md` frontmatter** — add a `model:` line to pin a stage, leave it absent to inherit the session model. CLI/frontmatter aliases resolve to the family's *latest* version (`opus` → newest opus, not `claude-opus-4-x`), so running a stage at *exactly* the session model requires an absent frontmatter `model:`. | | `DIMENSIONS` | `model` | Verify swap axes — and therefore **the variant count per picked claim**, since verify runs exactly one swap per listed axis. List or comma-separated subset of `{method, dataset, model}`. Default `model` → 1 variant/picked-claim (fast, single-axis model swap). Broaden with `dimensions: method,dataset,model` → 3 variants/picked-claim (full stress test). Forwarded to verify agent. | | `TARGET_CLAIMS` | `all` | Which claims verify stress-tests: `all` (default; covers both main-experiment-supported and main-experiment-rejected claims so robustness is checked in both directions) / `passed` (only main-experiment-supported = `claim_supported = pass`) / `failed` (only main-experiment-rejected = `claim_supported = fail`) / a specific claim id. Note `passed ∪ failed = all`. Forwarded to verify agent. | | `MAX_VERIFY_CLAIMS` | `1` | Cap on how many Stage-1-admitted claims proceed into Stage 2 (swap variants). Stage 1 (main-experiment integrity audit) **always audits every target claim regardless** — the cap only gates Stage 2 entry. When the admitted pool exceeds the cap, `/auto-verify`'s Phase 3 step 0 picks the top-K by importance judgment (reading each admitted claim's statement against upstream narrative like `IDEA_REPORT.md` / `## Rationale`; row order is NOT a priority signal). Un-picked admitted claims are marked `INTEGRITY_ONLY` with `stage2_skip_reason: max_verify_claims_cap` in `VERIFY_REPORT.md`; user can swap-test them later via `/auto-verify <id> — resume: true` (Stage 1 audit is reused via RESUME). Forwarded to verify agent. At the default cap of 1 with default `DIMENSIONS=model`, verify launches 1 × 1 = 1 variant run per `/auto` pass. | | `ROBUSTNESS_THRESHOLD` | `0.5` | A claim PASSes verify iff `robustness ≥ ROBUSTNESS_THRESHOLD`. Set higher (e.g., `0.67`) for stricter publication-ready verification, lower (`0.33`) for exploratory work. Forwarded to verify agent. | | `MIN_VARIANTS_FOR_VERDICT` | `1` | Minimum number of integrity-clean variants required to issue a PASS / FAIL verdict on a claim. Default `1` means a single eligible variant still yields a verdict; `N_eligible < MIN_VARIANTS_FOR_VERDICT` triggers **ZERO_ELIGIBLE_VARIANTS** (a distinct terminal state from INCONCLUSIVE — see `auto-verify/SKILL.md` Phase 10). Raise to `2`/`3` for stricter projects where you want multiple independent axes to agree before issuing PASS / FAIL. Forwarded to verify agent. | | `BASE_REPO` | _none_ | GitHub repo URL to clone before implementing. | | `RESEARCH_DOMAIN` | `auto` | Project domain tag (e.g. `mechanistic-interpretability`, `vision-encoders`, `rl-policy-eval`). Used to gate mechanism-family routing and downstream auxiliary decisions. Default `auto`: orchestrator forwards `auto` to the experiment agent and the sub-skill infers from `FINAL_PROPOSAL.md`; if inference is ambiguous, silently falls back to `general` regardless of `AUTO_PROCEED` (no UI prompt). To force a specific domain, set this flag explicitly on the CLI. | | `COMPACT` | `false` | Generate compact summary artifacts: `idea-stage/IDEA_CANDIDATES.md` after claim, `refine-logs/EXPERIMENT_LOG.md` after experiment, skip per-claim `verify/<claim_dir>/ROBUSTNESS.md` (where `<claim_dir>` = `<claim_id>_<short_claim>` on disk; see `skills/auto-verify/SKILL.md` "Directory Layout"). Forwarded to claim / experiment / verify agents. | | `CODE_REVIEW` | `true` | External LLM reviewer checks experiment + verify-variant code before deployment. Set `false` to skip. Forwarded to experiment + verify agents. | | `SANITY_FIRST` | `true` | Run the smallest/cheapest run first to catch setup bugs before launching the full suite. Forwarded to experiment + verify agents. | | `AUTO_DEPLOY` | `true` | Auto-deploy after implementation + review; set `false` to pause for manual inspection. Forwarded to experiment + verify agents. | | `MAX_PARALLEL_RUNS` | `4` | Max concurrent `/run-experiment` calls dispatched within the experiment / verify stages. Becomes `max_parallel:` in the `/experiment-queue` manifest when Phase 4 of experiment routes to the queue path. Forwarded to experiment + verify agents. | | `BATCH_DISPATCH` | `auto` | Phase 4 dispatch routing rule (experiment stage). `auto` (default) lets `/auto-experiment` auto-pick `/run-experiment` for small milestones (≤ 5 ad-hoc runs) and `/experiment-queue` for large or dependency-laden ones (≥ 10 runs, `depends_on`, grid expansions, ≥ 3-seed × ≥ 3-config sweeps). `queue` forces every milestone to `/experiment-queue`; `direct` forces every milestone to `/run-experiment` (debug-only — emits a warning if it overrides the queue rule). Forwarded to experiment agent. | | `REF_PAPER` | `false` | Reference paper for the claim stage to summarize first. Accepts a local PDF path, an arXiv abs URL, or any paper URL. When set, claim Phase 0.5 writes `idea-stage/REF_PAPER_SUMMARY.md` and downstream idea generation builds on it. Forwarded to claim agent as `ref_paper:`. | | `BEHAVIOR_SOURCE` | `given` | **Behavior stage** — controls where the behavior to study comes from and whether it is validated. Three values: **`given`** (default): the behavior is already specified in the direction / `task.md` and **assumed to hold** — no ideation, no novelty check, **no M0 validation**; the claim stage faithfully captures it and goes straight to the mechanism. **Precondition enforced by the Given-Behavior Comprehension Gate** (see the claim stage): `task.md` must name a *concrete* behavior; if it names only a topic, the orchestrator asks the user to specify a behavior or switch to `discovery` (this gate always waits — even in full-auto). **`given-validation`**: the behavior is given the same way (faithfully captured from `task.md`, no mining, no ideation, no novelty) **but its existence is validated first** — the experiment plan opens with a hard **M0 phenomenon-validation gate** that the experiment stage runs before any mechanism compute. **`discovery`**: the behavior itself is mined — load `/mechanism-behavior-discovery` to *sharpen* a *new* candidate phenomenon and run the full ideation pipeline (research-lit → idea-creator → novelty-check → impact-check → research-review → research-refine-pipeline; final idea chosen by impact-first, novelty-second), then the experiment plan opens with the **M0 gate** too. The M0 four-state verdict (`established`/`conditional` proceed; `not-established` ends the pipeline with a negative-result report skipping verify+iteration; `inconclusive` re-runs M0) applies to both `given-validation` and `discovery`. Accepts `given` / `given-validation` / `discovery` (case-insensitive). Forwarded to the claim agent as `behavior_source:`. | | `MECHANISM` | `discovery` | **Mechanism stage** — controls who picks the mechanism method. **`discovery`** (default): the system selects the mechanism family — the experiment stage runs `/mechanism-skills` routing (route_only → auto-select the recommended candidate, or the family mini-prompt when `AUTO_PROCEED=false`) and the claim stage loads `/mechanism-explore` to shape the hypothesis direction + experiment plan. **`given`**: the user has specified the mechanism method/family in `task.md` — the claim stage captures it and the experiment stage commits it **directly** (Phase 1.5 Mode B, `CHOSEN_FAMILY=<the method>`), bypassing routing and the family mini-prompt; `task.md` **must** name a concrete mechanism method/family, else the claim stage halts (report it back). Accepts `given` / `discovery` (case-insensitive). Forwarded to the claim agent as `mechanism:`. **Resource fidelity:** the claim stage stamps `resource_fidelity: strict` into `FINAL_PROPOSAL.md` + `EXPERIMENT_PLAN.md` **iff `BEHAVIOR_SOURCE=given` AND `MECHANISM=given`** (the reproduction combination) — the experiment stage then enforces the **Resource-Fidelity Harness** on the main experiment (exact models/datasets at full scale — no smaller-model swap, no data subsetting, no skipped must-run runs; OOM handled by batch↓/grad-accum/sharding/offload, HALT rather than downscale; verify is exempt, its swaps are intentional). Every other combination leaves the marker unstamped, so model/dataset choice is unconstrained (cost-aware). | | `MAX_ITERATIONS` | `6` | Iteration agent: max total back-edge actions (variant fix / plan+script fix / claim-stage re-entry) before stopping. Counts every back-edge uniformly; new claims produced by claim-stage re-entry inherit the same budget (no fresh allocation). Forwarded to iteration agent. Legacy alias `MAX_ROUNDS` is accepted at the CLI for one release and silently normalized to `MAX_ITERATIONS`. | | `MAX_CLAIM_REENTRIES` | `2` | Iteration agent: sub-budget within `MAX_ITERATIONS` for claim-stage re-entries (action type ③ — see `auto-iteration-loop/SKILL.md`). Prevents the failure mode where the reviewer keeps requesting claim rewrites without anyone fixing the experiments behind them. When exhausted, the iteration loop refuses further ③ actions even if iterations remain. Forwarded to iteration agent. | | `TARGET_SCORE` | `6` | Iteration agent: stop early when score ≥ this AND verdict is `ready`/`almost` AND no FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS claim remains (three-dimensional STOP rule). | | `GPU_ID` | `auto` | GPU device(s) to pin every experiment / verify-variant / iteration-round run to. `auto` lets each sub-skill inherit from the environment / launcher (no pinning). A single id (`0`) or comma-list (`4,5,6,7`) causes the sub-skill to **pass `CUDA_VISIBLE_DEVICES=<GPU_ID>` as the first positional argument to `/run-experiment`** (and sanity runs); `/run-experiment` is a Skill, not a shell command, so it cannot accept a shell prefix — instead it parses this positional arg and exports `CUDA_VISIBLE_DEVICES` itself before launching the experiment subprocess. Forwarded to experiment + verify + iteration agents (iteration's Phase-C `/run-experiment` dispatches inherit the same pin). | | `OOM_MAX_GPUS` | `4` | **`resource_fidelity: strict` only (the `BEHAVIOR_SOURCE=given` + `MECHANISM=given` reproduction combo).** Upper bound on how many free GPUs the Resource-Fidelity Harness may auto-add to a single OOM'd run before halting. On OOM the harness queries free GPUs (`memory.used < 500 MiB`), auto-adds them to the run and enables sharding — auto-converting a single-GPU script (`model.cuda()` / `.to(device)`) to `device_map="auto"` / FSDP / CPU-or-disk offload, first verified numerically equivalent on a fit-on-one-GPU proxy slice — repeating up to this cap while leaving `MAX_PARALLEL_RUNS` headroom for sibling runs. Only after the cap is hit (or no free GPU remains) **and** CPU/disk offload is exhausted does it HALT — and **never** by downscaling (smaller model / subset data stay forbidden, even in full-auto). Set higher to let one big run spread across more devices; `auto` lets it use all currently-free GPUs. Forwarded to experiment + verify + iteration agents. Ignored when the strict marker is absent (every non-reproduction combination). | | `UNDERPOWER` | `tag` | **Active whenever the strict marker is absent** (ignored under `resource_fidelity: strict` — the reproduction combo — which already forbids subsetting). Guards against an under-powered cheap run's weak/negative verdict being mistaken for a *real* negative. After the experiment, a claim whose main-experiment verdict is weak (`not-supported` / `partial` / null) **and** whose realized scale is materially below the plan (`used_n` shortfall, fewer seeds, or fewer grid/checkpoint points than `EXPERIMENT_PLAN.md`) is flagged **suspected under-power**. `tag` (default): tag the claim `[suspected under-power: used_n X/Y, seeds A/B, grid P/Q]` (recorded as a **provisional** caveat carried into the ledger + verify + iteration, so the negative is treated as provisional, not a confirmed falsification) and proceed — respecting the cost-aware design; under `AUTO_PROCEED=false` the experiment agent instead asks (full re-run / targeted-milestone re-run / accept demo-scale). `stop`: treat a suspected-under-power claim as a **Round-End Decision** (`ended-needs-decision (experiment: suspected-under-power)`) even under `AUTO_PROCEED=true`, so you decide whether to re-run at full scale before verify trusts it. `off`: disable the check. Forwarded to experiment agent. | | `LEDGER_FIGURES` | `auto` | Whether the final ledger render should call `/paper-figure` to produce per-claim plots **and tables** embedded into `CLAIMS_LEDGER.md` (image figures = PNG inline + PDF link; tables = Markdown inline + `.tex` link). Tri-state: `auto` (default) generates figures only for claims with at least one plottable or tabulable data source; `true` forces an attempt for every non-deferred claim (an unsupported claim still degrades to a `skipped` entry, never a halt); `false` disables the hook entirely and the ledger renders without a `Figures` bullet. Fires **once per pipeline run**, at the final ledger hook only (`iteration:final`, or `verify` when `REVIEW_LOOP=false`); intermediate hooks never re-invoke `/paper-figure`. See [Ledger Figures hook](#ledger-figures-hook). | Override on the CLI, e.g. `/auto "direction" — auto-proceed: false, claim-model: opus, verify-model: sonnet, dimensions: method,dataset, code-review: false`. **Canonical CLI form is hyphen-separated** (`auto-proceed`, `claim-model`, `code-review`, ...). The parser also accepts underscore (`auto_proceed`) and uppercase env-style (`AUTO_PROCEED`) for compatibility — see "Argument parsing" below. Pass each flag to the agent(s) listed in its "Forwarded to" hint as the corresponding arg in `agents/<name>.md`'s Invocation contract (e.g., `COMPACT` → `compact:`, `CODE_REVIEW` → `code_review:`). ### Argument parsing `$ARGUMENTS` is shaped as `"<direction>" — <key>: <value>, <key>: <value>, ...`. Both the direction and the option list are independently optional. The orchestrator parses it as follows: 1. **Direction (positional, optional)** — everything before the first ` — ` (em dash with spaces) or `--` is the research direction; pass it to the claim agent as `direction:`. When omitted (empty `$ARGUMENTS`, or `$ARGUMENTS` starts directly with ` — ` / `--`), pass `direction: ""` and rely on `task.md` (see "Direction source resolution" above). If both are absent, stop and report rather than inventing a direction.