auto-iteration-loop

# Auto Iteration Loop: Adversarial Review with Bounded Back-Edges Autonomously iterate over `/auto-verify`'s per-claim verdicts: review → implement fixes (variant-integrity, main-experiment-script, or claim-stage rewrite) → re-review, until the external reviewer gives a positive assessment with all FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS claims resolved, or the iteration budget is exhausted. ## Context: $ARGUMENTS ## Constants - **MAX_ITERATIONS = 6** — Hard upper bound on total back-edge actions across the whole loop. **Each of the following consumes 1 iteration**, regardless of which Phase A→E cycle they fall in: - ① variant-only fix → re-run `/auto-verify <claim-id> — resume: true` - ② main-experiment-script / `EXPERIMENT_PLAN.md` fix → re-run `/auto-experiment` (optionally followed by `/auto-verify` chain) - ③ claim-stage re-entry → rewrite claim (full `/auto-claim` invocation OR lightweight in-loop rewrite) → re-run downstream stages Pure reviewer cycles that change nothing on disk (no new experiments, no claim changes) do **not** consume iterations. PASS-claim consistency checks and deferred-claim recording do not consume iterations. **New claims produced by ③ inherit the same budget — they do not get a fresh `MAX_ITERATIONS` of their own.** Override via `MAX_ITERATIONS:` arg from `agents/iteration.md`. > **Resource-Fidelity invariant (all back-edges).** If `refine-logs/FINAL_PROPOSAL.md` / `refine-logs/EXPERIMENT_PLAN.md` carry a `resource_fidelity: strict` marker (the reproduction combination — `behavior-source:given` + `mechanism:given`), every back-edge MUST preserve it: when editing `EXPERIMENT_PLAN.md` (type ②) or any claim rewrite (type ③), keep the top-of-file `resource_fidelity:` line **verbatim** — never drop, comment out, or change it. Dropping the marker silently **disables the harness**, re-opening the door to smaller-model swaps and data subsetting the harness exists to prevent. For type-③ full re-entry, detect the original run's axes from `idea-stage/IDEA_REPORT.md`'s `**Behavior-source**:` / `**Mechanism**:` headers (or directly from the `resource_fidelity: strict` marker) and pass **both** `behavior_source` and `mechanism` into the queued `auto-claim` call so a reproduction-combo re-entry stays `given`+`given` (the new run re-stamps the `resource_fidelity: strict` marker automatically — it is marker-driven, not a flag). - **MAX_CLAIM_REENTRIES = 2** — Sub-budget within `MAX_ITERATIONS` for action type ③ (claim-stage re-entry). Prevents the failure mode where the reviewer keeps asking "rewrite the claim" without anyone actually fixing the experiments behind it. When `claim_reentries_consumed >= MAX_CLAIM_REENTRIES`, the loop refuses further ③ actions even if iterations remain — at that point the only legal back-edges are ① and ②, and if those are also exhausted the loop terminates and writes the final report. - **TARGET_SCORE = 6** — Stop when reviewer score >= this AND verdict matches a `POSITIVE_VERDICT_TERMS` entry AND no claim is still in FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS (three-dimensional STOP rule — see Phase B). - **POSITIVE_VERDICT_TERMS = [ready, almost]** — Case-insensitive substrings that count the verdict as positive. Phase B canonicalizes the reviewer's free-form verdict to one of `{ready, almost, not ready}` before this check, so any reviewer-supplied synonym (e.g., `accept`, `sufficient`, `good enough`) must be mapped to `ready` / `almost` by Phase B's parser — see Phase B's STOP rule for the canonical check. - **REVIEW_DOC = `review-stage/AUTO_REVIEW.md`** — Per-iteration append-only audit log (fall back to `./AUTO_REVIEW.md` for legacy projects). This is the raw audit trail; the narrative report is `FINAL_REPORT_DOC` below. - **REVIEWER_MEMORY_DOC = `review-stage/REVIEWER_MEMORY.md`** — Reviewer's persistent suspicion log across iterations. Append-only: each iteration's Phase B.5 adds a section under `## Iteration N`, never deletes prior iterations (audit trail). Phase A of iteration 2+ prepends the full file to the reviewer prompt so the reviewer can check whether prior-iteration suspicions were genuinely addressed or sidestepped. Counters the failure mode where Claude curates a comfortable subset of context for the reviewer each iteration. - **FINAL_REPORT_DOC = `review-stage/AUTO_ITERATION_FINAL_REPORT.md`** — Narrative final report written **once** in Termination (after the loop ends for any reason). Organized per-claim by their original `/auto-verify` category (PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS / DEFERRED), with the per-claim iteration journey, experiment/script diffs, and any claim rewrites laid out inside each claim's section. Distinct from `REVIEW_DOC` (which is the chronological audit log). See "Final Report" section below for the full template. - **GPU_ID = `auto`** — GPU device(s) to pin every Phase-C `/run-experiment` dispatch to. `auto` lets the launcher / environment decide (no extra export). A single id (`0`) or comma-list (`4,5,6,7`) causes this skill to **pass `CUDA_VISIBLE_DEVICES=<GPU_ID>` as the first positional arg to every `/run-experiment` invocation** (the run-experiment skill exports it internally; do not treat as a shell prefix), and record the effective `CUDA_VISIBLE_DEVICES` in each run's `run.sh`. Forwarded from `agents/iteration.md`'s `gpu_id:` arg (which the orchestrator forwards from `/auto`'s `GPU_ID`). - **RESUME = false** — When `true`, read `review-stage/REVIEW_STATE.json` and pick up from `iterations_consumed + 1` instead of starting fresh. Append new iterations to the existing `AUTO_REVIEW.md` rather than overwriting it. Iteration budget and claim-reentry sub-budget are **inherited** across resume — they are never reset. If `REVIEW_STATE.json` already shows `status: completed` (positive verdict, iterations exhausted, or claim-reentry budget exhausted), return immediately without running another iteration. If `status: awaiting_upstream`, the orchestrator (see `auto/SKILL.md`) is expected to run the pending upstream calls first, then resume this loop. Default `false` = always start fresh and overwrite prior review state. Resume never deletes pre-existing review state. Schema fields read here: `iterations_consumed`, `claim_reentries_consumed`, `status`, `last_verdict`, `last_score` — see "State Persistence" below. ## Reviewer LLM Configuration (mandatory, read first) This skill calls an external LLM reviewer. **Never hardcode a model name and never read the reviewer model from `task.md` / project READMEs / source comments.** Project-level files may list available API keys for unrelated purposes (e.g., LLM-as-judge inside experiment code); those are *not* the reviewer config. Resolve `LLM_MODEL`, `LLM_BASE_URL`, `LLM_API_KEY` strictly in this priority order before any reviewer call: 1. **Project MCP config** — `${PROJECT_ROOT}/.mcp.json`, field `mcpServers["llm-chat"].env.{LLM_MODEL,LLM_BASE_URL,LLM_API_KEY}`. 2. **User MCP config** — `~/.claude/settings.json`, same field. 3. **Shell environment** — `$LLM_MODEL`, `$LLM_BASE_URL`, `$LLM_API_KEY`. ### Pre-flight check (run before Phase A of every iteration, mandatory) ```bash LLM_MODEL_SRC="" if [ -f .mcp.json ] && jq -e '.mcpServers["llm-chat"].env.LLM_MODEL' .mcp.json >/dev/null 2>&1 ; then export LLM_MODEL=$(jq -r '.mcpServers["llm-chat"].env.LLM_MODEL' .mcp.json) export LLM_BASE_URL=$(jq -r '.mcpServers["llm-chat"].env.LLM_BASE_URL' .mcp.json) export LLM_API_KEY=$(jq -r '.mcpServers["llm-chat"].env.LLM_API_KEY' .mcp.json) LLM_MODEL_SRC="project .mcp.json" elif [ -f ~/.claude/settings.json ] && jq -e '.mcpServers["llm-chat"].env.LLM_MODEL' ~/.claude/settings.json >/dev/null 2>&1 ; then export LLM_MODEL=$(jq -r '.mcpServers["llm-chat"].env.LLM_MODEL' ~/.claude/settings.json) export LLM_BASE_URL=$(jq -r '.mcpServers["llm-chat"].env.LLM_BASE_URL' ~/.claude/settings.json) export LLM_API_KEY=$(jq -r '.mcpServers["llm-chat"].env.LLM_API_KEY' ~/.claude/settings.json) LLM_MODEL_SRC="user ~/.claude/settings.json" elif [ -n "$LLM_MODEL" ] && [ -n "$LLM_BASE_URL" ] && [ -n "$LLM_API_KEY" ] ; then LLM_MODEL_SRC="shell env" fi echo "[reviewer-config] LLM_MODEL=$LLM_MODEL LLM_BASE_URL=$LLM_BASE_URL source=$LLM_MODEL_SRC" ``` **Hard-fail rule**: If `LLM_MODEL` is empty after this resolution (none of the three sources provides it), the skill MUST abort with: > "Reviewer model not configured. Add `mcpServers.llm-chat.env.{LLM_MODEL,LLM_BASE_URL,LLM_API_KEY}` to `.mcp.json` (project) or `~/.claude/settings.json` (user)." Do not guess a default. Do not fall back to a model name read from `task.md` or any other project file. > The earlier `## LLM Configuration` example block in older versions of this skill (which duplicated the same `llm-chat` MCP setup) has been removed — the authoritative config resolution is the priority order above. For a worked `.mcp.json` example, see `mcp-servers/llm-chat/` in the project repo. ## API Call Method **Primary: MCP Tool** ``` mcp__llm-chat__chat: prompt: | [Review prompt content] model: "${LLM_MODEL}" # resolved per "Reviewer LLM Configuration" priority order above — never hardcode system: "You are a senior ML reviewer..." ``` **Fallback: curl** ```bash curl -s "${LLM_BASE_URL}/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${LLM_API_KEY}" \ -d '{ "model": "${LLM_MODEL}", "messages": [ {"role": "system", "content": "You are a senior ML reviewer..."}, {"role": "user", "content": "[review prompt]"} ], "max_tokens": 4096 }' ``` ## State Persistence (Compact Recovery) Persist state to `review-stage/REVIEW_STATE.json` after each iteration (i.e., every Phase E): ```json { "iterations_consumed": 2, "claim_reentries_consumed": 1, "status": "in_progress", "last_score": 5.0, "last_verdict": "not ready", "consecutive_noop_count": 0, "iteration_breakdown": [ {"i": 1, "type": "variant_fix", "target_claims": ["C1"], "produced_claims": []}, {"i": 2, "type": "claim_reentry", "target_claims": ["C2"], "produced_claims": ["C2_v2"]} ], "claim_stage_reentries": ["C2"], "pending_upstream_calls": [], "runs_total": 5, "gpu_hours_total": 12.4, "thread_id": "th_01ABC123def456", "timestamp": "2026-03-15T10:00:00" } ``` - `iterations_consumed` is the cumulative count of back-edge actions (types ①/②/③) since iteration 1 — what's compared against `MAX_ITERATIONS`. - `claim_reentries_consumed` is the cumulative count of type ③ actions only — what's compared against `MAX_CLAIM_REENTRIES`. - `iteration_breakdown` records each iteration's action type, the claim(s) it targeted, and any new claim ids produced by claim-stage re-entry (so the final report can attribute new claims to the right ancestor). - `claim_stage_reentries` is the flat list of claim ids that triggered ③ — surfaced to `agents/iteration.md` so the orchestrator can record which original claims got rewritten in the Ledger's `journey_summary.iteration` line and `open_items[]`. - `pending_upstream_calls` is non-empty only when `status = awaiting_upstream`: a list of `{skill, args}` calls the orchestrator must execute before resuming this loop. - `runs_total` / `gpu_hours_total` are cumulative across all iterations; they monotonically increase and are the canonical budget source `agents/iteration.md` reports back to the orchestrator. **Field consumer map** (so future maintainers know which fields are external API vs internal state): | Field | Written by | Read by | Notes | |---|---|---|---| | `iterations_consumed` | this skill | `auto/SKILL.md` (inherits on resume), `agents/iteration.md` | external; compared against `MAX_ITERATIONS` | | `claim_reentries_consumed` | this skill | `auto/SKILL.md` (inherits on resume), `agents/iteration.md` | external; compared against `MAX_CLAIM_REENTRIES` | | `status` | this skill | `auto/SKILL.md` resume check, `agents/iteration.md` | external; values `in_progress` / `awaiting_upstream` / `completed` | | `last_verdict` | this skill | `auto/SKILL.md` resume check | external; matched against `POSITIVE_VERDICT_TERMS` | | `last_score` | this skill | `auto/SKILL.md` resume check | external; numeric 1–10 | | `iteration_breakdown` | this skill | this skill (Termination, final-report assembly); `agents/iteration.md` summary | external; ordered list, append-only | | `claim_stage_reentries` | this skill | `auto/SKILL.md` (for `CLAIMS_LEDGER.md` `journey_summary` + `open_items[]`), Termination | external | | `pending_upstream_calls` | this skill | `auto/SKILL.md` orchestrator only | external; consumed (cleared) by orchestrator before resuming this skill | | `runs_total` | this skill | `agents/iteration.md` summary, `auto/SKILL.md` budget tracking | external | | `gpu_hours_total` | this skill | `agents/iteration.md` summary, `auto/SKILL.md` budget tracking | external | | `timestamp` | this skill | this skill's resume gate (24h staleness check — see Workflow Initialization) | internal — drives the stale-state cutoff; archive instead of delete on age-out | | `thread_id` | this skill (Phase A iteration 1) | this skill (Phase A iteration 2+) | **internal** — opaque conversation handle returned by `mcp__llm-chat__chat` after iteration 1's call. Phase A iteration 2+ passes it back to the MCP server so the reviewer keeps the same conversation thread (cheaper context + better cross-iteration coherence than re-pasting summaries). Absent / `null` when the iteration-1 reviewer call used the curl fallback (curl doesn't return a thread handle), or when the MCP server rejected thread continuation — in either case Phase A iteration 2+ silently falls back to the paste-summary path. | **Write this file at the end of every Phase E** (after documenting the iteration). **On completion**, set `"status": "completed"` and continue to Termination (which generates `FINAL_REPORT_DOC`).