| name | autoresearch |
| description | Autonomous keep/discard optimization loop — Karpathy autoresearch pattern generalized to any measurable metric (eval pass rate, bundle KB, web vitals, LLM cost, accessibility score) |
| allowed-tools | Read, Bash, Glob, Grep |
autoresearch Skill
What This Is
The Karpathy autoresearch pattern applied to CKS. An autonomous agent runs an overnight optimization loop:
- Mutate one target file (the lever)
- Run one metric command (the judge)
- If metric improved:
git commit (keep)
- If metric regressed or flat:
git reset --hard (discard)
- Repeat until budget exhausted
The value is the discipline: one lever, one metric, branch isolation, no human gates inside the loop.
Deterministic vs Indeterministic Design
This is the core architecture. Violating this split produces non-reproducible experiments.
| Component | Nature | Rule |
|---|
| Loop step sequence | Deterministic | Always: checkpoint → mutate → run → compare → keep/reset → log → check-budget |
| Keep/discard decision | Deterministic | Purely numerical. Improved = keep. Flat or regressed = reset. No judgment. |
| Exit conditions | Deterministic | Budget exhausted OR 3 consecutive crashes OR STOP file exists |
| Metric command | Deterministic | Same command string every iteration — no variations |
| TSV schema and timing | Deterministic | Append one row after every iteration, regardless of outcome |
| Consent gate | Deterministic | Fires exactly once at loop start. Never again. |
| Baseline measurement | Deterministic | Run metric once before iteration 1. Record as iteration=0, status=baseline |
| Crash threshold | Deterministic | 3 consecutive crashes → exit. Reset counter on any non-crash. |
| Mutation content | Indeterministic | Agent decides what to change. Informed by program.md but creative. |
| TSV description | Indeterministic | Natural language summary of what was tried. Generated fresh each iteration. |
| program.md learning | Indeterministic | What the agent extracts as learnings. No template — genuine reflection. |
| Hypothesis selection | Indeterministic | Which optimization to try next. Guided by history, not constrained. |
The deterministic shell makes experiments reproducible. The indeterministic core explores the search space. Never mix them.
Core Concepts
program.md — agent's steering document at .autoresearch/<tag>/program.md. Describes what to try, what has worked, what has failed. Agent updates every 5 iterations. User can edit between sessions to steer.
Branch isolation — loop runs on autoresearch/<tag>. Main is never touched. Branch is disposable.
Keep/discard discipline — a change is worth keeping only if the metric strictly improved. The number decides.
Consent contract — user approves the loop once. That covers all in-loop resets. See .claude/rules/autoresearch.md.
TSV log — .autoresearch/<tag>/results.tsv. Schema:
iteration commit metric_value delta status description
status: kept | reset | crash | baseline
Metric Taxonomy
See references/metric-adapters.md for command patterns.
| Category | Examples | Direction |
|---|
| Eval quality | cks:evals pass rate, test pass % | higher = better |
| Bundle size | webpack bundle KB, gzip size | lower = better |
| Web vitals | LCP ms, FID ms | lower = better |
| LLM cost | tokens/request | lower = better |
| Accessibility | Lighthouse a11y score | higher = better |
Loop Engine
Step-by-step execution: workflows/loop-engine.md. Load before starting.
program.md Template
# Optimization Program: <tag>
## Goal
Improve `<metric command>` by modifying `<target file>`.
Direction: <higher is better | lower is better>
## What to Try
- Pass 1: Conservative changes (config flags, quick wins)
- Pass 2: Moderate changes (restructuring, algorithm tweaks)
- Pass 3: Aggressive changes (larger refactors if budget remains)
## What Has Worked
(agent fills in every 5 iterations)
## What Has Failed
(agent fills in every 5 iterations)
Common Rationalizations
| Rationalization | Reality |
|---|
| "I'll also change this other file while I'm here" | One target only. Multi-file mutations hide the causal signal. |
| "The metric barely improved, I'll keep it anyway" | Keep = strictly improved. Flat = reset. |
| "I should ask the user before this reset" | Consent was given at loop start. Mid-loop prompts violate the contract. |
| "I should vary the metric command for a cleaner signal" | The metric command is deterministic. Same string every iteration. |
Verification
