auto-research

name	auto-research
description	Use when running or designing a measured self-improvement loop on a YURI subsystem — "climb this metric", "optimize the resolver", "autoresearch", "improve X until the score goes up" — or when deciding whether a subsystem is loop-improvable at all. Also use before touching an evaluator, benchmark, or scorer while a loop is in flight.
triggers	["/autoresearch","autoresearch","self-improvement loop","frozen evaluator","climb the benchmark"]
scope	harness
invocation	workflow

Auto-Research — measured self-improvement loops

Authority: _SYSTEM/yuri-origin.md → Loop Discipline states WHAT is required. This skill is the operational HOW. It does not restate that doctrine and must not contradict it — on conflict, origin wins.

Harness-neutral. Claude Code, Codex CLI, OMP, and Cursor all run the same loop; nothing here depends on one harness's permission layer (that is exactly the point of the enforcement ordering below).

Worked references in this repo: _SYSTEM/eval/atlas-score.mjs (navigation quality, hit@k over a 40-item JSONL benchmark) and _SYSTEM/eval/build-score.mjs (build health, weighted components with --runs=N median + variance for noise). Read one before writing a new scorer — the shape is already settled.

The entry gate

A subsystem is loop-improvable exactly when it has an immutable scorer. No frozen benchmark → no loop. Build the benchmark first, or do not run. Writing benchmark ground truth is an owner judgment; the lane being measured does not get to author what counts as correct.

Before iteration 1, all four must hold:

1. A scorer exists as a separate artifact from everything the loop may edit.
2. The scorer is frozen (all four layers below armed).
3. The benchmark passes the construct-validity check.
4. You are on a scratch branch, not main.

Construct validity — check before you trust a number

Verify the benchmark measures what the system claims to do, not merely something adjacent that is easy to score. This is the expensive failure mode because it looks exactly like success: a reproducible, defensible number answering the wrong question.

The Atlas case: every question in atlas-benchmark.jsonl asks "what file does X?", while Atlas claims faceted region navigation. That metric can only ever reward a filename index. A genuine-looking negative result meant nothing — the system was never asked to do the thing it exists to do.

Run these three, in order, on any benchmark before it gates anything:

• Claim mapping. Write the subsystem's claimed capability in one sentence. Map each question type to a clause of it. Unmapped clauses = untested capability; unmapped questions = you are measuring something else.
• Leakage scan. Grep question text for tokens of the expected answer's path. q037 leaked "graduation ladder" → alpha-factor-library/graduation.mjs and produced a hit that proved nothing about retrieval.
• Reachability check. Can the system reach each expected answer at all? A coverage ceiling (the answer is not in the index) and a ranking failure (it is, but ranked low) are different diagnoses with different fixes. Conflating them sends the loop after the wrong knob.

Keep unwinnable questions. A question the system cannot answer is a mapped blind spot. Pruning to a flattering set converts a diagnostic into a scoreboard.

The four enforcement layers, ordered by universality

Order matters: layers 1–2 bind all four harnesses and every model. Layer 4 binds one harness and only while it is loaded.

1. chmod 444 on the eval directory. OS file mode. Binds every process, harness, agent, and script — nothing negotiates with the filesystem.
2. The loop self-aborts. Refuse to start, and re-check every iteration, if git diff HEAD -- _SYSTEM/eval/ is non-empty. Cheap, deterministic, harness-independent.
3. Git pre-commit hook rejecting any commit touching the eval dir without an explicit unfreeze env var. Catches the drift that survives 1–2 (someone chmod'd back).
4. Harness permission config (Claude settings.json deny, Codex sandbox, etc.). A bonus layer, never load-bearing. A deny rule the other three harnesses never read is not enforcement — it is a note to one reader.

Never rely on layer 4 alone. That is the whole reason the ordering is written down.

Verifier isolation

The scorer runs in a fresh process with no access to the proposal, the diff, or the reasoning. A checker sharing the maker's context inherits its blind spots and rubber-stamps them. See the ISOLATION RULE comment block at the top of atlas-score.mjs for the concrete contract: it reads the benchmark file and the resolver's stdout, nothing else. If you find yourself wanting to pass "context about what changed" into the scorer, stop — that urge is the failure mode.

The loop

scratch branch → freeze (4 layers) → construct-validity check → baseline score
  ↓
propose ONE change → measure (fresh process) → keep if metric improved, revert if equal or worse
  → append iteration, knob, before, after, verdict to an append-only results log → repeat

Single-knob mutation. One change per iteration. Two knobs and the attribution of any gain is lost — you learn the score moved, not what moved it, and the next iteration builds on a guess.

Append-only log. Never rewrite past entries. The log is the only record of which hypotheses died, and a loop that can edit its history can launder a regression.

Noise. For a metric with run-to-run variance (tests, timing), take a median over N runs and report variance — build-score.mjs --runs=N does this. A single run cannot separate a real regression from a flake, and a loop that reverts on noise wanders.

Legitimate vs illegitimate evaluator interaction

Same files touched, opposite epistemics. The test: can you justify the change from theory or first principles, without reference to which questions currently fail? If the justification requires the failure list, it is optimizing the evaluator.

Legitimate	Fraud
Add IDF weighting derived from corpus statistics — principled, derivable before seeing any score	Twiddle constants until the benchmark passes
Fix a construct-validity defect by adding question types that test the claimed capability, removing no hard question	Delete the questions the system failed
Fix a leaking question so it stops giving away its own answer	Add a question shaped around what the system already does well
Version the scorer explicitly, re-baseline, and mark historical scores incomparable	Silently retune weights mid-run

Any change to the benchmark or scorer ends the current run. Re-freeze, re-baseline, note the version break in the log. There is no such thing as adjusting the ruler and keeping the measurements.

When NOT to loop

• No frozen scorer. Build it first, or do the work by hand.
• Behavioural or noisy metrics with no stable ground truth ("does the output read better") — model opinion is not a measurement.
• Fewer than ~25 benchmark items. Thin benchmarks get exploited: a handful of items is memorizable, and one lucky item swings the score more than any real improvement.
• The change is a one-shot fix. A loop is for climbing a gradient, not for applying a known patch.

Anti-rationalization

Excuse	Reality
"I'll just tweak the scorer to be more fair"	You are the thing being scored. Apply the theory test: if you cannot justify the tweak without naming which questions fail, it is fraud.
"The benchmark is obviously wrong here"	Sometimes true — and it still ends the run. Fix it, re-freeze, re-baseline, mark the version break. Do not fix it and keep climbing. <!-- @anchor: v1
"This question is unwinnable so it should go"	Unwinnable = mapped blind spot, the most valuable item in the set. Removing it raises the number and lowers the information. <!-- @anchor: v1
"This hit proves retrieval works"	Check for leakage first. q037 contained "graduation ladder" and the expected answer was graduation.mjs — the hit measured string overlap, not navigation. <!-- @anchor: v1
"I'll freeze it after I get a good baseline"	The baseline is the measurement most worth trusting and the one you just took with an editable ruler. Freeze first, always.
"Component tests passed so the pipeline works"	Green-in-isolation says nothing about the seam. atlas-regions.mjs never read edges.json; both sides were fine and the pipeline was not. Test the connection explicitly. <!-- @anchor: v1
"One harness enforces it, that's enough"	Harness config is layer 4. Codex, OMP, and Cursor never read Claude's settings.json. Layers 1–2 or it is not frozen.
"Two knobs, but they're independent"	Then run two iterations. Independence is a claim; the log records only that the score moved.

Red flags — STOP the run

• About to edit anything under _SYSTEM/eval/ mid-loop
• Justifying a scorer or benchmark change by naming which items fail
• The score improved and you cannot name the single knob that moved it
• Running on main
• The scorer is being handed the diff, the proposal, or "context"
• Baseline taken before the freeze
• Rewriting or pruning past entries in the results log

Any one of these: revert to the last logged good state, re-freeze, re-baseline.

Provenance

The loop pattern (frozen evaluator, propose→measure→keep-or-revert, single-knob mutation, append-only log) is adopted from the publicly discussed autoresearch pattern as an unprotectable idea. karpathy/ autoresearch carries no license (GitHub API returns license: null) — all rights reserved. No text from that repo is copied or closely paraphrased here or in _SYSTEM/eval/*; the wording, vocabulary, and enforcement model are YURI's own.

Session Notes

2026-07-26 — Skill authored. Tools: Read/Grep over _SYSTEM/yuri-origin.md (Loop Discipline), .claude/rules/skill-creation.md, skills/writing-skills/SKILL.md, _SYSTEM/eval/atlas-score.mjs, _SYSTEM/eval/build-score.mjs, _SYSTEM/eval/atlas-benchmark.jsonl. Verified locally before citing: q037 leakage (grep of the benchmark line — question text contains "graduation ladder", expected path graduation.mjs); the seam miss (edges.json referenced only by atlas-edges.mjs, the writer — atlas-regions.mjs never reads it). Corrections: none. Errors: none. Notes: /autoresearch alias file created at .claude/commands/autoresearch.md per the skill-creation checklist step 2. No RED baseline run yet — the rationalization table is anchored to the 2026-07-26 Atlas findings, not to subagent pressure-test transcripts; a baseline run per writing-skills RED-GREEN-REFACTOR is the open item.