| name | autoresearch |
| description | Autonomous hill-climbing on any measurable problem: qualify the four legs, freeze an eval referee, then loop mutate-evaluate-keep/revert until the stopping condition fires. Works on tests, latency, bundle size, cost. |
| disable-model-invocation | true |
Autoresearch
Autonomous hill-climbing over a mutable artifact, judged by a frozen
referee, advanced by a git ratchet: keep what measurably improves,
revert what doesn't, never stop until the stopping condition fires. The human
programs the protocol; you run the experiments; the referee decides.
Generalized from karpathy/autoresearch.
Continue mode: if invoked as continue (or the repo already has a
PROTOCOL.md and results.tsv on an autoresearch/* branch), read both,
skip to phase 4, and resume the loop under the existing protocol.
1. Qualify
The pattern stands on four legs. Name each concretely before anything else:
| Leg | Question | Example |
|---|
| Mutable artifact | Which files may the loop edit? | src/translator/** |
| Quantitative metric | One number; which direction is better? | tests passed, higher |
| Bounded eval | One command, fixed budget, prints the metric | pnpm eval, ~30s |
| Keep/discard | Can a bad change be fully reverted? | git commit / reset |
A missing metric or eval command is normal: you construct it in phase 3. But
if the artifact isn't revertible or nothing is measurable even in principle,
say so and stop. This problem does not hill-climb.
2. Align: grill the protocol out of the user
Invoke the grill-with-docs skill on the proposed run (fall back to
grilling if unavailable). The grilling must pin down, at minimum:
- Objective: the single metric and its direction. Secondary metrics are
constraints, never co-objectives.
- Stopping condition: a target value, an experiment budget, a wall-clock
budget, or a plateau (no keep in the last K experiments). "Run until
interrupted" is allowed only if the user says so explicitly.
- Frozen zone: everything the loop must never touch (eval harness, data,
dependencies, rule checks).
- Eval budget: fixed time per experiment, so results stay comparable.
- Constraints and the simplicity criterion: resource ceilings, rules, and
how much added complexity a marginal gain is worth. Deleting code for equal
results is a win.
Write the answers into PROTOCOL.md at the target repo root, using the
protocol template. The protocol is the
human's program: the human iterates on PROTOCOL.md, the loop iterates on
the artifact. Done when the user confirms the protocol.
3. Freeze the referee
Build the evaluation harness. Domain recipes, the referee contract, noise
handling, and the anti-Goodhart checklist live in
harness recipes; consult it before building.
Hard gates, all of them, before the loop may start:
- One command prints one machine-parseable metric line (
metric: <value>)
and exits non-zero on failure.
- Baseline twice: run the referee two or three times on the unchanged
artifact. The spread is the noise floor; if noise swamps the expected
effect size, fix the harness before proceeding.
- The referee and
PROTOCOL.md are now frozen: only the human may change
them. If mid-loop you feel the urge to "fix" the referee, that is Goodhart
knocking. Stop and ask.
Record the baseline as the first journal row.
4. The loop
One-time setup: branch autoresearch/<tag> off the current branch; journal
results.tsv (tab-separated, untracked, never committed) with header
commit metric status description.
LOOP until the stopping condition fires:
- Pick the next idea: protocol idea seeds, near-misses in the journal, or
your own judgment.
- Edit the artifact.
git commit.
- Run the referee:
<eval-cmd> > run.log 2>&1, then grep the metric line.
Never stream run output into context.
- Empty grep means crash: read the log tail. A dumb bug (typo, import), fix
and rerun once or twice. A fundamentally broken idea, journal
crash,
reset, move on.
- Journal the row: short commit hash, metric (0 for crash), status
(
keep/discard/crash), one-line description.
- Improved beyond the noise floor: keep, the ratchet advances. Anything
else: discard,
git reset --hard to the last keep.
- Stopping condition unmet: loop. Do not pause to ask whether to continue;
the protocol already decided. Out of ideas means think harder: re-read the
artifact, combine near-misses, try something more radical.
Kill any run exceeding 2x the eval budget and treat it as a crash.
Long runs: to keep the loop alive across sessions or overnight, tell the
user to start it as /loop /autoresearch continue (self-paced re-invocation;
continue mode picks the run back up each time).
5. Report
When the stopping condition fires, report: baseline vs best metric, counts of
keeps/discards/crashes, the cumulative diff of surviving changes
(git diff <baseline-commit>..HEAD --stat), and the journal. The branch holds
the result; merging it is the user's call.
