Autonomous experiment loop: edit code, commit, run benchmark, extract metrics, keep improvements or revert, repeat forever. Use this skill when the user asks to "run autoresearch", "start an experiment loop", "optimize a metric autonomously", "autonomous experiments", "autoresearch setup", "benchmark loop", "keep/discard experiments", "optimize test speed", "optimize bundle size", "optimize build time", "run experiments overnight", "speed up my tests", "make my build faster", "reduce compile time", "optimize this automatically", "keep trying until it's faster", "run experiments while I sleep", "overnight optimization", "edit-measure-keep loop", "cancel autoresearch", "stop autoresearch", "autoresearch status", "how many experiments", or mentions "autoresearch", "experiment loop", "autonomous optimization". Always use this skill when the user wants to iteratively and autonomously improve any measurable metric — even if they don't use the word "autoresearch". Also use when the user asks about the status of a ru
Autonomous experiment loop: edit code, commit, run benchmark, extract metrics, keep improvements or revert, repeat forever. Use this skill when the user asks to "run autoresearch", "start an experiment loop", "optimize a metric autonomously", "autonomous experiments", "autoresearch setup", "benchmark loop", "keep/discard experiments", "optimize test speed", "optimize bundle size", "optimize build time", "run experiments overnight", "speed up my tests", "make my build faster", "reduce compile time", "optimize this automatically", "keep trying until it's faster", "run experiments while I sleep", "overnight optimization", "edit-measure-keep loop", "cancel autoresearch", "stop autoresearch", "autoresearch status", "how many experiments", or mentions "autoresearch", "experiment loop", "autonomous optimization". Always use this skill when the user wants to iteratively and autonomously improve any measurable metric — even if they don't use the word "autoresearch". Also use when the user asks about the status of a running autoresearch session or wants to cancel/stop one.
version
0.2.0
argument-hint
[GOAL] [--max-iterations N]
Autoresearch: Autonomous Experiment Loop
An autonomous optimization loop where Claude edits code, runs a benchmark, measures a metric, and keeps improvements or reverts — repeating forever until stopped.
Core Concept
The loop is simple: edit → commit → run → measure → keep or discard → repeat.
Primary metric is king. Lower (or higher, depending on direction) is better. Improved → keep the commit. Equal or worse → .
git revert
State survives context resets via autoresearch.jsonl (append-only log) and autoresearch.md (living session document).
Domain-agnostic. Works for any measurable target: test speed, bundle size, LLM training loss, Lighthouse scores, build times, etc.
Be careful not to overfit to the benchmarks and do not cheat on the benchmarks. Optimize the real workload, not the measurement harness.
Setup Phase
When the user triggers autoresearch, gather the following (ask if not provided):
Goal — what to optimize (e.g., "reduce unit test runtime")
Command — the benchmark to run (e.g., pnpm test, uv run train.py)
Primary metric — name, unit, and direction (lower or higher is better)
Secondary metrics — optional additional metrics to track for tradeoff monitoring (e.g., memory, compile time)
Files in scope — which files can be modified
Constraints — time budget, off-limits files, correctness requirements
Optionally check for .claude/autoresearch-ai-plugin.local.md in the project root for persistent configuration:
---
enabled: true
max_iterations: 50
working_dir: "/path/to/project"
benchmark_timeout: 600
checks_timeout: 300
---
# Autoresearch Configuration
Additional context or notes for this project's autoresearch setup.
enabled — whether autoresearch is active (default: true)
max_iterations — stop after N experiments (default: 0 = unlimited)
working_dir — override directory for experiment files (default: current directory)
benchmark_timeout — benchmark timeout in seconds (default: 600)
checks_timeout — correctness checks timeout in seconds (default: 300)
If the file doesn't exist, use defaults. The file should be added to .gitignore (.claude/*.local.md).
Then execute these setup steps:
Create a branch: git checkout -b autoresearch/<goal>-<date>
Ensure session files are gitignored (critical — git revert will fail if autoresearch.jsonl is tracked):
Parse metrics from output (lines matching METRIC name=value)
Record baseline in autoresearch.jsonl (with "type":"config" header first, then baseline result)
Begin the experiment loop
The Experiment Loop
LOOP FOREVER. Never ask "should I continue?" — just keep going.
The user might be asleep, away from the computer, or expects you to work indefinitely. If each experiment takes ~5 minutes, you can run ~12/hour, ~100 overnight. The loop runs until the user interrupts you, period.
Each iteration:
1. Read current git state and autoresearch.md
2. Choose an experimental change (informed by past results and ASI notes)
3. Edit files in scope
4. git add <files> && git commit -m "experiment: <description>"
5. Run: bash autoresearch.sh > run.log 2>&1
6. Parse METRIC lines from output
7. If autoresearch.checks.sh exists, run it (separate timeout, default 300s)
8. Decide: keep or discard
9. Log result to autoresearch.jsonl (include ASI annotations)
10. If discard/crash: git revert $(git rev-parse HEAD) --no-edit
11. Update autoresearch.md with learnings (every few experiments)
12. Repeat
Crash or checks failed → discard (revert, note the failure in ASI)
Simpler code for equal perf → keep (removing complexity is a win)
Catastrophic secondary metric regression → consider discard even if primary improved (e.g., 1% speed gain but 10x memory usage)
If stuck → think deeper, try a different approach. Consult autoresearch.ideas.md if it exists. Re-read source files for new angles. Try combining previous near-misses. Try more radical changes. Read any papers or docs referenced in the code.
Simplicity Criterion
All else being equal, simpler is better. Weigh complexity cost against improvement magnitude:
A 0.001 improvement that adds 20 lines of hacky code? Probably not worth it.
A 0.001 improvement from deleting code? Definitely keep.
Equal performance with much simpler code? Keep.
Handling User Messages During Experiments
If the user sends a message while the loop is running:
Finish the current experiment cycle (don't abandon mid-run)
Address the user's feedback or question
Resume the loop immediately after — do not wait for permission
Secondary metrics are tracked in the JSONL log and help guide future experiments, but they rarely affect keep/discard decisions (only discard if a catastrophic secondary regression accompanies a marginal primary improvement).
Output instrumentation data — phase timings, error counts, cache rates, domain-specific signals. This data guides the next iteration and helps identify where optimization effort should focus.
Actionable Side Information (ASI)
ASI is structured annotation per experiment that survives reverts. When code changes are discarded, only the description and ASI remain — making them the only structured memory of what happened.
Record ASI for every experiment:
{"hypothesis":"Reducing loop iterations by breaking early","result":"Marginal speedup but code readability suffered","next_action_hint":"Try vectorization instead of loop unrolling","bottleneck":"Memory bandwidth on L2 cache misses"}
ASI fields are free-form — use whatever keys are useful:
hypothesis — what you expected
result — what actually happened
next_action_hint — guidance for the next experiment
bottleneck — identified performance bottleneck
error_details — crash/failure diagnostics
Any other domain-specific observations
Logging to autoresearch.jsonl
Config Header (written once at setup)
{"type":"config","name":"Optimize unit test runtime","metricName":"total_ms","metricUnit":"ms","bestDirection":"lower"}
Experiment Results (appended after each run)
Each experiment appends one JSON line:
{"run":5,"commit":"abc1234","metric":4230,"metrics":{"compile_ms":1200,"memory_mb":512},"status":"keep","description":"parallelized test suites","timestamp":1700000000,"segment":0,"confidence":2.3,"asi":{"hypothesis":"parallel tests reduce wall time","next_action_hint":"try worker pool size tuning"}}
Fields:
run — experiment number (1-indexed, sequential)
commit — short git commit hash (7 chars)
metric — primary metric value
metrics — secondary metrics dict (optional)
status — one of: keep, discard, crash, checks_failed
description — brief description of what was tried
timestamp — Unix timestamp (seconds)
segment — session segment index (0-based, incremented when optimization target changes)
confidence — MAD-based confidence score (null if < 3 experiments)
asi — Actionable Side Information dict (optional, omit if empty)
Use ${CLAUDE_SKILL_DIR}/scripts/log-experiment.sh to append entries:
