Use when the user wants an autonomous ML research loop that does more than blindly try changes. After every training run the agent analyses what actually happened inside the model — gradients, activations, embeddings, errors, data — and grounds the next change in that evidence. A `<literature>` on/off dial adds scientific-literature grounding: off behaves as a pure analysis-first loop; on searches papers, grades the evidence, and implements only what prior work supports. One change per run; loops forever until interrupted. Not for one-off training runs or hyperparameter sweeps.
Use when the user wants an autonomous ML research loop that does more than blindly try changes. After every training run the agent analyses what actually happened inside the model — gradients, activations, embeddings, errors, data — and grounds the next change in that evidence. A `<literature>` on/off dial adds scientific-literature grounding: off behaves as a pure analysis-first loop; on searches papers, grades the evidence, and implements only what prior work supports. One change per run; loops forever until interrupted. Not for one-off training runs or hyperparameter sweeps.
compatibility
Requires Python 3.9+
metadata
{"version":"0.1.0"}
ML Autoresearch Loop
This loop is analysis-first: every experiment is followed by a diagnostic pass that examines what
happened inside the model, and the next change is a hypothesis grounded in that evidence — not a guess.
The feedback signal is <metric> read from the run log; the analysis is the spine that decides what to
change. A <literature> dial (on/off) optionally grounds each change in prior work via the sibling
literature-search skill. One change per iteration, so each metric move is attributable.
You are the researcher. Do not pause to ask for permission once the loop is running.
When to use
Use for an open-ended, autonomous ML research campaign where you want each change motivated by analysis
of the model's actual behaviour. Set <literature> = off for a self-contained analysis-and-score loop;
set <literature> = on to additionally ground changes in the scientific literature (paper search,
evidence grading, a reusable findings backlog). Not for a single training run, a fixed sweep, or tasks
with no measurable scalar metric. Default to off unless the user wants literature grounding or the
problem is a known, well-published one where prior recipes will pay off.
Setup
Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it and skip to
the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available — record <host> =
claude-code) infer a likely value for each binding from the project and present it as the recommended
option; on other hosts (<host> = other) ask each as a quoted plain-text prompt. Then write
loop.run.yaml (format: examples/run.example.yaml) and confirm every value with the user before
creating any other files. For branches strategy, create git checkout -b autoresearch/<run_tag>
(tag from today's date; branch must not exist). For time gating, write <sandbox_root>/run_with_timeout.sh
(timeout $(( <budget> * 60 )) <entrypoint> "$@") and use it as the run command, hard-killing at
2 × <budget> min; for epochs, patch the epoch cap in an <editable_files> file.
one-phrase problem domain; seeds query phrasing only, never filters
—
infer from data/model/task
<lit_skill_dir>(on only)
install dir of the literature-search skill
sibling of this loop
~/.claude/skills/literature-search/ (adjust per host)
<lit_py>(on only)
Python ≥3.9 interpreter for the lit helper (stdlib-only)
python3
independent of <run_cmd>
FILE EDIT GUARD: before touching any file at any point — setup or loop — confirm it is in
<editable_files>, because everything else is read-only ground truth (the eval harness defines
<metric>). No exceptions.
Literature toolchain (only when <literature> = on)
Paper search goes through the sibling literature-search skill (stdlib-only, no installs):
<lit> = <lit_skill_dir>/tools/lit_search.py (note the tools/ segment). Reuse one cache by
appending --cache-dir <sandbox_root>/literature/.cache after the subcommand. Subcommands print JSON;
on failure they print {"error","fallback"} and exit non-zero — then degrade to the host's
WebSearch/WebFetch (never fabricate citations). Smoke-test <lit> --help at setup; if the skill is
absent, tell the user and offer to install it (cp -r <repo>/loops/literature-search ~/.claude/skills/)
or proceed degraded. For onboarding and API keys (all optional; a free S2_API_KEY is recommended),
run <lit> keys --init — it manages the shared gitignored keys.env at the project root and reports
presence as booleans (secrets never enter chat). Persist the live tiers to loop.run.yaml.
Initialise the sandbox
Create the layout (extra literature/ tree only when <literature> = on) and write the ledger headers:
<sandbox_root>/
├── loop.run.yaml ← resolved bindings (written now)
├── results.tsv ← experiment ledger, header only (written now)
├── literature/ ← (on only)
│ ├── corpus.tsv ← findings ledger, header only (written now)
│ ├── .cache/ ← lit_search on-disk cache
│ ├── pdfs/ ← fallback PDF reads
│ └── text/ ← extracted LaTeX section text
└── iter1/ ← created at loop start
results.tsv header (tab-separated; the literature_basis column exists only when <literature> = on):
iter <metric> status analysis_summary [literature_basis ]description
The loop (LOOP FOREVER — until interrupted)
Iteration 1 is always the unmodified baseline: skip change-planning and research (no diagnostics
yet to ground a change), but still write a baseline plan.md and run the mandatory analysis — it
produces the first empirical anchor that iteration 2 builds on. Everything in <editable_files> is fair
game (architecture, optimizer, hyperparameters, data pipeline, loss); the only constraints are that the
code runs without crashing and finishes within <budget>. Simplicity criterion: all else equal,
simpler is better — a 0.001 gain that adds 20 lines of hacky code is not worth it; a 0.001 gain (or an
equal metric) from deleting code is a keep.
Copy this checklist each iteration and tick items off:
1. Look at the state.branches: git log --oneline -5. snapshots: confirm iter<N>/
doesn't exist. Read iter N-1's analysis summary; (on) skim corpus.tsv for unimplemented keepers.
2. Plan one change (iteration 1: SKIP — run baseline unmodified). Grounded in iter N-1's
analysis. See Planning a change below; (on) it also runs the literature step.
3a. Snapshot / commit, then apply the one change.snapshots: create
iter<N>/{code_snapshot,analysis,results}/, copy every <editable_files> into code_snapshot/,
copy loop.run.yaml to iter<N>/, then apply. branches: apply, then git commit -am "<desc>".
When implementing a published/library technique, ground it in a real current example or the actual
library in the repo (read it first) — never write the API from memory.
3b. Write the analysis plan BEFORE the run + add any instrumentation it needs. See The
analysis plan below.
4. Run the experiment, redirecting everything (never tee):
<entrypoint> > <sandbox_root>/iter<N>/<run_log> 2>&1 (or run_with_timeout.sh when time-gated).
If it overruns, kill it and treat as a crash.
5. Read the metric: grep '^<metric>:' <sandbox_root>/iter<N>/<run_log>. If empty,
tail -n 50 <run_log>, read the trace, attempt one trivial fix (typo/import); if fundamentally
broken, log crash and continue.
6. Analyse the results — MANDATORY, produces real artifact files. See Analysing below.
7. Log to the ledger(s) (untracked — never commit). See Ledger.
8. Keep or revert (the change ran this iteration). Improved per <metric_direction> → keep,
update current-best. Equal/worse/crash → discard/crash; branchesgit reset --hard HEAD~1,
snapshots restore <editable_files> from iter<N>/code_snapshot/. Apply the simplicity
criterion before logging discard. On a crash/OOM, fix with the minimal change that preserves
the intent (OOM → smaller batch + grad-accum to hold effective batch) — never mutate the
experiment into something the plan didn't call for.
9. Go to step 1 — the analysis from step 6 is the primary input to the next hypothesis.
Planning a change (step 2 — iterations 2+)
The latest analysis sets the direction; it is the master input every iteration. Decide exactly one
lever, grounded in iter N-1's analysis. Other vetted ideas are queued, not bundled into one run.
State explicitly before applying:
the one change and which <editable_files> it touches;
the empirical anchor — a specific file + value/pattern from iter N-1's results/ that motivates
it. Every non-simplification change must cite an anchor; theoretical reasoning alone is insufficient.
A swing to a different architecture is anchored too (a ceiling/structural finding, e.g. "the family
plateaus at X with headroom" or "it fails exactly on cases needing Y"), not a local pathology.
what you predict will happen and why the finding supports it.
When <literature> = off: that anchor is the whole basis — pick the change directly from the
analysis. Before writing the plan, scan prior analyses (ls <sandbox_root>/iter*/analysis/*.py) so you
don't repeat a diagnostic without a comparison reason.
When <literature> = on: after fixing the anchor, ground the change in the literature —
2a. Retire drift, then consult the backlog as a cache. If the last kept change altered the
architecture family (e.g. CNN→transformer), set result=stale for every unimplemented keeper whose
scope is a non-matching architecture tag; scope=agnostic keepers (schedules, weight decay,
augmentation, init philosophies) survive. Then check corpus.tsv for an unimplemented keep
targeting the analysis's direction — reuse it only if it still passes the gate (2c) against the
CURRENT architecture (re-validate now; a finding that no longer applies is retired, not forced in).
2b. Research the direction (the default unless 2a yielded a still-valid lever). Turn the
analysis's limitations into questions (tie limitations to questions), record them in
iter<N>/questions.md, then dispatch research subagents — see roles/research-subagent.md
(spawn-or-degrade: real isolated subagents on Claude Code, otherwise run the research inline in this
context) at the dial's depth/effort (see that file's dial table for <research_scale>).
Level 1 = high-level (architecture fit, prior approaches, does the literature show success); research
L1 first — if it surfaces a compelling new direction that becomes the lever. Level 2 = specific
micro-opts (init, weight-decay dynamics, attention/cache for the sequence length, norm placement,
schedule). Anti-rut: a keeper passed over for ~3 iterations, or no longer on any live direction, is
retired (result=stale) so it stops resurfacing.
2c. Evidence gate (re-validated; consults history). Keep/reuse a finding only if all hold:
evidence.strength is moderate/strong (not weak); applicability_to_our_setup transfers to
the CURRENT data/model/budget; every files_to_edit is within <editable_files> and fits <budget>;
and it is not materially equivalent to a change already logged no-effect/hurt/crashed in
corpus.tsv (unless the finding explains why it differs now). Tag each kept finding's scope
(agnostic or the current architecture tag) so drift retirement is mechanical. Record every finding
in corpus.tsv (keep or reject); kept-but-unchosen findings stay queued (implemented=n,
result=pending).
2d. Choose the single lever from the keepers (backlog + new) — highest evidence × applicability ×
gain-per-complexity. Add to the explicit statement above: the literature basis (kept finding +
citation(s)), or none for a pure analysis-driven simplification/ablation; and the verify-todo
(the metric/log/analysis from the finding's how_to_verify that will tell whether it helped). The
literature basis is additive grounding — it never replaces the empirical anchor.
The analysis plan (step 3b — written BEFORE the run)
Write <sandbox_root>/iter<N>/analysis/plan.md as a deliverables table, not prose — each row is a
hard commitment: a script you will write and run, the exact output file it produces, the question it
answers, and what each outcome would look like:
script
output file
question it answers
expected if change helped
expected if not
The plan must include at least one diagnostic dimension not measured in any prior iteration
(the history scan enforces this); the "expected if not" column is mandatory (it forces you to define
falsification before seeing the result). When <literature> = on, also include the verify-todo
from 2d, and periodically (and whenever a swing is on the table) a ceiling/headroom probe — is the
current architecture near its limit? (train/val plateau, representation collapse, structural failure).
Instrumentation (mandatory when needed). If a planned measurement needs a metric/log not currently
produced (e.g. per-layer grad norms, per-class accuracy, val_loss alongside val_acc) and the
producing script is in <editable_files>, add that logging NOW, before the run. Writing the plan
first is what forces this — don't discover at analysis time that the number you need wasn't logged.
Analysing (step 6 — MANDATORY; produces real artifacts)
This is the spine of the loop. Do not proceed to step 7 until every row in plan.md has a
corresponding non-empty file in iter<N>/results/. Analysis that didn't write a file did not happen.
6a. Execute the plan. For each row, write the script in iter<N>/analysis/ and run it,
redirecting to iter<N>/results/. Diagnostic dimensions to draw from (choose what the hypothesis and
results call for, not a fixed rotation): gradient norms/flow, activation stats/saturation, embeddings
(PCA/CKA/collapse), error & confusion analysis, loss dynamics & headroom, weight/parameter stats,
data profiling (auditing the data itself is one of the highest-yield diagnostics), compute profiling.
6b. Checklist.ls iter<N>/results/ against plan.md; write and run any missing deliverable now.
6c. Interpret against the plan. For each row, did it match "expected if helped" or "if not"? A
mismatch is information, not a failure.
6d. (on) Check the verify-todo from 2d — did the change do what the literature predicted? This
sets the finding's result in step 7.
6e. Opportunistic follow-up. If something unexpected appears (odd loss shape, a consistently-wrong
class, a layer behaving differently), write one more script to chase it — the most novel findings
come from here.
Write a concise analysis summary (3–8 bullets): what you examined, the single most important
finding (expected or not), (on) whether the architecture is near its ceiling, and the specific
empirical anchor (file + value/pattern) that will motivate the next plan.
Empirical-justification gate. Before step 7, you must be able to complete this sentence with a real
file reference and a concrete value/pattern — not a theoretical argument:
"The next change will be X because the analysis showed Y (from results/<file>, value/pattern Z)."
If you cannot, go back to 6a and run more analysis until you can. Proceeding without an empirical anchor
is the exact failure this loop is built to prevent.
Forward-looking instrumentation. After analysing, ask "what would I wish I had logged?" — if the
producing script is in <editable_files>, add it now (best-epoch checkpoint, extra per-epoch
diagnostics) so future analyses are richer. Adding instrumentation is a valid iteration on its own.
Ledger
<sandbox_root>/results.tsv, tab-separated, never commas in free text. The literature_basis column
is present only when <literature> = on. Status ∈ {keep, discard, crash}; use 0.000000 for the
metric on crashes. For branches, the first column is the 7-char commit instead of the iter number.
iter <metric> status analysis_summary literature_basis description
1 0.6320 keep grad norms clean; no pathologies none (baseline) baseline
2 0.7050 keep overfit on small data per val/train gap Dosovitskiy 2020 ViT — aug+warmup add RandAug + 3-ep warmup
3 0.7010 discard warmup ok but LR too high; unstable Loshchilov 2017 — decoupled wd switch to AdamW wd=0.05
<sandbox_root>/literature/corpus.tsv (only when <literature> = on; tab-separated): a backlog of
vetted findings reused across iterations instead of re-researched. The orchestrator (not the subagent)
writes it and applies the final evidence gate, so verdicts are consistent across questions.
iter level paper_id title scope relevance verdict implemented result
2 649def… 1 An Image is Worth 16x16 Words vit high keep y helped
3 a1c2… 2 Decoupled Weight Decay Regularization agnostic high keep y no-effect
4 7d9e… 1 ConvNeXt blocks cnn low keep n stale
Columns: level ∈ {1,2}; scope = agnostic or an architecture tag (drives drift retirement);
relevance ∈ {high,med,low}; verdict ∈ {keep,reject}; implemented ∈ {y,n}; result ∈
{helped,no-effect,hurt,pending,stale} (stale = retired on drift/expiry). Last row: a CNN keeper
retired after the architecture moved to ViT; the agnostic weight-decay finding survives drift.
Report the best iteration (not necessarily the last) when summarising progress. Do not commit
results.tsv or literature/ to git — leave them untracked.
Constraints
Only edit files in <editable_files> — confirm before every edit, because everything else is
read-only ground truth and the eval harness defines <metric>.
One change per iteration, so each metric delta is attributable to one lever.
Analysis is mandatory every iteration and must produce real files in results/ (each plan.md
row → a non-empty output file). A change with no analysis behind it is not allowed — it degrades the
loop into blind iteration (or, when on, recipe-following that plateaus blind).
Every non-simplification change must cite an empirical anchor (file + value/pattern) from that
analysis; when on, the literature basis is additive, never a substitute. Record both in results.tsv.
Do not install packages or add dependencies the project lacks; helper scripts (and the lit helper)
stay stdlib-only.
Always redirect training output to <run_log>; never use tee. The sandbox is self-contained — no
../ escapes.
(on) Never print or commit API keys — keys reports presence only; keys.env lives at the project
root and must stay gitignored. When a lit tool returns {"error","fallback"}, degrade to
WebSearch/WebFetch; never fabricate citations — every cited paper must come from a real tool result.
Stops
This loop runs forever until the human interrupts it — do not pause to ask "should I continue?" or
"is this a good stopping point?". The human may be away and expects autonomous work indefinitely. A
working result is the start of the next iteration, not the end. If you run out of ideas: re-read the
in-scope files for missed angles, go deeper on the analysis (gradients/activations/embeddings/errors
always surface something), combine previous near-misses from results.tsv, try more radical changes —
and when on, go back to the literature: crawl citation graphs deeper, read methodology of recent
work citing your approach, mine corpus.tsv for queued keepers, and combine recipes across papers.
