| name | scoreboard-loop |
| description | Defines a cheap-to-evaluate score and one mutable workspace so agents iterate from numbers; separates immutable prep from editable experiment files. Use when designing overnight experiment loops, autoresearch-style harnesses, or metric-gated agent iteration. |
Scoreboard Loop
Distilled pattern: give agents (or humans) a cheap-to-score objective and a single mutable workspace; let iteration be driven by numbers, not vibes.
When to use
- You want continuous improvement (training code, prompts, configs, pipelines) without hand-running every experiment.
- You can define a metric that is efficient to evaluate relative to the cost of proposing changes.
Steps
- Freeze the interface — One script or file is the contract: edit here, run
train / eval, read score. Everything else is optional. When you need hygiene, split immutable prep (data, constants) from mutable experiment surface so diffs stay reviewable.
- Publish the scoreboard — Stdout table, log file, or wandb-style trail; the loop must make “better / worse” unambiguous.
- Let the optimizer be dumb but persistent — Agents propose diffs; the metric decides survival. Novelty comes from breadth of search, not from charisma in the proposal.
- Keep the baseline intentionally bare — Frame the first version as a template for a pattern, not a finished product—so others fork the loop, not your hyperparameters.
- Human owns objective and constraints — What is optimized, what is forbidden, and what requires review stays outside the agent.
Anti-patterns
- Fuzzy goals (“make it better”) with no scalar or rubric.
- Letting agents change eval code and training code in one step without separation of duties.
- Chasing leaderboard metrics you admit are gamed or meaningless—pair with epistemic honesty (see Hype–Discipline Pairing).
Output shape
- Objective function (one paragraph).
- Command(s) to reproduce one experiment.
- What the agent is allowed to edit vs read-only.
- One paragraph: what this loop does not solve.
