| name | auto-research |
| description | Use when running or designing a measured self-improvement loop on a YURI subsystem — "climb this metric", "optimize the resolver", "autoresearch", "improve X until the score goes up" — or when deciding whether a subsystem is loop-improvable at all. Also use before touching an evaluator, benchmark, or scorer while a loop is in flight. |
| triggers | ["/autoresearch","autoresearch","self-improvement loop","frozen evaluator","climb the benchmark"] |
| scope | harness |
| invocation | workflow |
Auto-Research — measured self-improvement loops
Authority: _SYSTEM/yuri-origin.md → Loop Discipline states WHAT is required. This skill is the
operational HOW. It does not restate that doctrine and must not contradict it — on conflict, origin wins.
Harness-neutral. Claude Code, Codex CLI, OMP, and Cursor all run the same loop; nothing here depends on
one harness's permission layer (that is exactly the point of the enforcement ordering below).
Worked references in this repo: _SYSTEM/eval/atlas-score.mjs (navigation quality, hit@k over a 40-item
JSONL benchmark) and _SYSTEM/eval/build-score.mjs (build health, weighted components with --runs=N
median + variance for noise). Read one before writing a new scorer — the shape is already settled.
The entry gate
A subsystem is loop-improvable exactly when it has an immutable scorer. No frozen benchmark → no
loop. Build the benchmark first, or do not run. Writing benchmark ground truth is an owner judgment; the
lane being measured does not get to author what counts as correct.
Before iteration 1, all four must hold:
- A scorer exists as a separate artifact from everything the loop may edit.
- The scorer is frozen (all four layers below armed).
- The benchmark passes the construct-validity check.
- You are on a scratch branch, not
main.
Construct validity — check before you trust a number
Verify the benchmark measures what the system claims to do, not merely something adjacent that is
easy to score. This is the expensive failure mode because it looks exactly like success: a reproducible,
defensible number answering the wrong question.
The Atlas case: every question in atlas-benchmark.jsonl asks "what file does X?", while Atlas claims
faceted region navigation. That metric can only ever reward a filename index. A genuine-looking negative
result meant nothing — the system was never asked to do the thing it exists to do.
Run these three, in order, on any benchmark before it gates anything:
- Claim mapping. Write the subsystem's claimed capability in one sentence. Map each question type to
a clause of it. Unmapped clauses = untested capability; unmapped questions = you are measuring
something else.
- Leakage scan. Grep question text for tokens of the expected answer's path.
q037 leaked
"graduation ladder" → alpha-factor-library/graduation.mjs and produced a hit that proved nothing
about retrieval.
- Reachability check. Can the system reach each expected answer at all? A coverage ceiling (the
answer is not in the index) and a ranking failure (it is, but ranked low) are different diagnoses with
different fixes. Conflating them sends the loop after the wrong knob.
Keep unwinnable questions. A question the system cannot answer is a mapped blind spot. Pruning to a
flattering set converts a diagnostic into a scoreboard.
The four enforcement layers, ordered by universality
Order matters: layers 1–2 bind all four harnesses and every model. Layer 4 binds one harness and only
while it is loaded.
chmod 444 on the eval directory. OS file mode. Binds every process, harness, agent, and script —
nothing negotiates with the filesystem.- The loop self-aborts. Refuse to start, and re-check every iteration, if
git diff HEAD -- _SYSTEM/eval/ is non-empty. Cheap, deterministic, harness-independent.
- Git pre-commit hook rejecting any commit touching the eval dir without an explicit unfreeze env
var. Catches the drift that survives 1–2 (someone chmod'd back).
- Harness permission config (Claude
settings.json deny, Codex sandbox, etc.). A bonus layer,
never load-bearing. A deny rule the other three harnesses never read is not enforcement — it is a
note to one reader.
Never rely on layer 4 alone. That is the whole reason the ordering is written down.
Verifier isolation
The scorer runs in a fresh process with no access to the proposal, the diff, or the reasoning. A
checker sharing the maker's context inherits its blind spots and rubber-stamps them. See the ISOLATION
RULE comment block at the top of atlas-score.mjs for the concrete contract: it reads the benchmark file
and the resolver's stdout, nothing else. If you find yourself wanting to pass "context about what
changed" into the scorer, stop — that urge is the failure mode.
The loop
scratch branch → freeze (4 layers) → construct-validity check → baseline score
↓
propose ONE change → measure (fresh process) → keep if metric improved, revert if equal or worse
→ append iteration, knob, before, after, verdict to an append-only results log → repeat
Single-knob mutation. One change per iteration. Two knobs and the attribution of any gain is lost —
you learn the score moved, not what moved it, and the next iteration builds on a guess.
Append-only log. Never rewrite past entries. The log is the only record of which hypotheses died, and
a loop that can edit its history can launder a regression.
Noise. For a metric with run-to-run variance (tests, timing), take a median over N runs and report
variance — build-score.mjs --runs=N does this. A single run cannot separate a real regression from a
flake, and a loop that reverts on noise wanders.
Legitimate vs illegitimate evaluator interaction
Same files touched, opposite epistemics. The test: can you justify the change from theory or first
principles, without reference to which questions currently fail? If the justification requires the
failure list, it is optimizing the evaluator.
| Legitimate | Fraud |
|---|
| Add IDF weighting derived from corpus statistics — principled, derivable before seeing any score | Twiddle constants until the benchmark passes |
| Fix a construct-validity defect by adding question types that test the claimed capability, removing no hard question | Delete the questions the system failed |
| Fix a leaking question so it stops giving away its own answer | Add a question shaped around what the system already does well |
| Version the scorer explicitly, re-baseline, and mark historical scores incomparable | Silently retune weights mid-run |
Any change to the benchmark or scorer ends the current run. Re-freeze, re-baseline, note the version
break in the log. There is no such thing as adjusting the ruler and keeping the measurements.
When NOT to loop
- No frozen scorer. Build it first, or do the work by hand.
- Behavioural or noisy metrics with no stable ground truth ("does the output read better") — model
opinion is not a measurement.
- Fewer than ~25 benchmark items. Thin benchmarks get exploited: a handful of items is memorizable,
and one lucky item swings the score more than any real improvement.
- The change is a one-shot fix. A loop is for climbing a gradient, not for applying a known patch.
Anti-rationalization
| Excuse | Reality |
|---|
| "I'll just tweak the scorer to be more fair" | You are the thing being scored. Apply the theory test: if you cannot justify the tweak without naming which questions fail, it is fraud. |
| "The benchmark is obviously wrong here" | Sometimes true — and it still ends the run. Fix it, re-freeze, re-baseline, mark the version break. Do not fix it and keep climbing. <!-- @anchor: v1 |
| "This question is unwinnable so it should go" | Unwinnable = mapped blind spot, the most valuable item in the set. Removing it raises the number and lowers the information. <!-- @anchor: v1 |
| "This hit proves retrieval works" | Check for leakage first. q037 contained "graduation ladder" and the expected answer was graduation.mjs — the hit measured string overlap, not navigation. <!-- @anchor: v1 |
| "I'll freeze it after I get a good baseline" | The baseline is the measurement most worth trusting and the one you just took with an editable ruler. Freeze first, always. |
| "Component tests passed so the pipeline works" | Green-in-isolation says nothing about the seam. atlas-regions.mjs never read edges.json; both sides were fine and the pipeline was not. Test the connection explicitly. <!-- @anchor: v1 |
| "One harness enforces it, that's enough" | Harness config is layer 4. Codex, OMP, and Cursor never read Claude's settings.json. Layers 1–2 or it is not frozen. |
| "Two knobs, but they're independent" | Then run two iterations. Independence is a claim; the log records only that the score moved. |
Red flags — STOP the run
- About to edit anything under
_SYSTEM/eval/ mid-loop
- Justifying a scorer or benchmark change by naming which items fail
- The score improved and you cannot name the single knob that moved it
- Running on
main
- The scorer is being handed the diff, the proposal, or "context"
- Baseline taken before the freeze
- Rewriting or pruning past entries in the results log
Any one of these: revert to the last logged good state, re-freeze, re-baseline.
Provenance
The loop pattern (frozen evaluator, propose→measure→keep-or-revert, single-knob mutation, append-only
log) is adopted from the publicly discussed autoresearch pattern as an unprotectable idea. karpathy/ autoresearch carries no license (GitHub API returns license: null) — all rights reserved. No text
from that repo is copied or closely paraphrased here or in _SYSTEM/eval/*; the wording, vocabulary, and
enforcement model are YURI's own.
Session Notes
2026-07-26 — Skill authored. Tools: Read/Grep over _SYSTEM/yuri-origin.md (Loop Discipline),
.claude/rules/skill-creation.md, skills/writing-skills/SKILL.md, _SYSTEM/eval/atlas-score.mjs,
_SYSTEM/eval/build-score.mjs, _SYSTEM/eval/atlas-benchmark.jsonl. Verified locally before citing:
q037 leakage (grep of the benchmark line — question text contains "graduation ladder", expected path
graduation.mjs); the seam miss (edges.json referenced only by atlas-edges.mjs, the writer —
atlas-regions.mjs never reads it). Corrections: none. Errors: none. Notes: /autoresearch alias file
created at .claude/commands/autoresearch.md per the skill-creation checklist step 2. No RED baseline
run yet — the rationalization table is anchored to the 2026-07-26 Atlas findings, not to subagent
pressure-test transcripts; a baseline run per writing-skills RED-GREEN-REFACTOR is the open item.
