autoresearch

name	autoresearch
description	Set up and run an autonomous experiment loop for any optimization target. Use when asked to "run autoresearch", "optimize X in a loop", "set up autoresearch for X", "start experiments", or "benchmark and optimize".
allowed-tools	Bash(*), Read, Write, Edit, Grep, Glob

Autoresearch

Autonomous experiment loop: try ideas, keep what works, discard what doesn't, never stop.

Helper Scripts

All scripts are in ${CLAUDE_SKILL_DIR}/scripts/. Use these instead of raw bash commands — they handle timing, output capture, JSONL persistence, git commits, and error handling automatically.

• init-experiment.sh — configure session (name, metric, unit, direction). Call again to re-initialize with a new baseline when the optimization target changes.

  bash "${CLAUDE_SKILL_DIR}/scripts/init-experiment.sh" "<name>" "<metric_name>" "<metric_unit>" "<direction>"
  

• run-experiment.sh — runs command, times it, captures output. Outputs structured JSON.

  bash "${CLAUDE_SKILL_DIR}/scripts/run-experiment.sh" "<command>" [timeout_seconds] [checks_timeout_seconds]
  

• log-experiment.sh — records result. keep auto-commits. discard/crash/checks_failed reverts via git checkout -- ..

  bash "${CLAUDE_SKILL_DIR}/scripts/log-experiment.sh" "<status>" "<metric_value>" "<description>" ['{"secondary": 123}']
  

• dashboard.sh — generates and opens an interactive HTML dashboard in the browser.

  bash "${CLAUDE_SKILL_DIR}/scripts/dashboard.sh"
  

• summary.sh — quick terminal summary of experiment progress.

  bash "${CLAUDE_SKILL_DIR}/scripts/summary.sh"
  

Setup

1. Ask (or infer from context): Goal, Command, Metric (name, unit, direction: lower/higher), Time budget (seconds per run, default 300), Files in scope, Constraints, Run tag (short label, e.g., mar24, bundle-opt).
2. git checkout -b autoresearch/<tag> (e.g., autoresearch/mar24)
3. Read ALL source files in scope. Understand the workload deeply — what the CPU/GPU spends time on, where the bottlenecks are, what the hot paths are. The best experiments come from deep understanding, not random variations.
4. Write autoresearch.md (include Experiment Protocol and Decision Rules sections — see template) and autoresearch.sh. Commit both.
5. Run init-experiment.sh → run baseline with run-experiment.sh → record with log-experiment.sh → start looping immediately.

autoresearch.md Template

This is the heart of the session. A fresh agent with no context should be able to read this file and run the loop effectively. Invest time making it excellent.

# Autoresearch: <goal>

## Objective
<Specific description of what we're optimizing and the workload.>

## Metrics
- **Primary**: <name> (<unit>, lower/higher is better)
- **Secondary**: <name>, <name>, ...

## How to Run
`./autoresearch.sh` — outputs `METRIC name=number` lines.
Time budget: <N> seconds per run (hard kill after timeout).

## Files in Scope
<Every file the agent may modify, with a brief note on what it does.>

## Off Limits
<What must NOT be touched.>

## Constraints
<Hard rules: tests must pass, no new deps, etc.>

## Experiment Protocol

### Setup
1. Create branch `autoresearch/<tag>` (e.g., `autoresearch/mar24`)
2. Read ALL source files in scope. Understand the workload deeply before writing anything.
3. Run baseline with `run-experiment.sh` → record with `log-experiment.sh`

### Loop (FOREVER — never stop, never ask)
1. Study the code. Think about what the CPU/GPU is actually doing.
2. Form a hypothesis. Make a focused edit (prefer single-file changes).
3. Run: `bash "${CLAUDE_SKILL_DIR}/scripts/run-experiment.sh" "./autoresearch.sh" <timeout>`
4. Evaluate result. If crashed: read output, fix if trivial, else move on.
5. Log: `bash "${CLAUDE_SKILL_DIR}/scripts/log-experiment.sh" "<status>" "<metric>" "<description>"`
   keep = commit stays. discard/crash/checks_failed = revert to last commit.
6. GOTO 1

## Decision Rules
- Primary metric is king. Improved → keep. Worse/equal → discard.
- Simpler code for equal perf = always keep. Removing code is a win.
- Ugly complexity for tiny gain = probably discard.
- Don't thrash — repeated reverts on same idea → try structurally different approach.
- Think longer when stuck. Re-read source, profile, reason about bottlenecks.
  The best ideas come from deep understanding, not random variations.
- Prefer focused, single-file changes. Easier to evaluate and revert.
- Consider: will this improvement transfer to larger configurations?
  Fundamental improvements > clever tricks.

## What's Been Tried
<Update periodically with: key wins, dead ends, architectural insights,
current run count, kept count, best metric so far.
A fresh agent should be able to read this section alone and know where to pick up.>

Update autoresearch.md periodically — especially the "What's Been Tried" section — so resuming agents have full context.

autoresearch.sh Template

Bash script (set -euo pipefail) that: pre-checks fast (syntax errors in <1s), runs the benchmark, outputs METRIC name=number lines. Keep it fast — every second is multiplied by hundreds of runs. Update it during the loop as needed.

#!/bin/bash
set -euo pipefail

# Pre-checks (fast, <1s)
# e.g., syntax check, build check

# Run the benchmark
# e.g., pnpm test --run

# Output metrics (REQUIRED — one per line)
# METRIC time=12.345
# METRIC memory_kb=4096

autoresearch.checks.sh (optional)

Bash script (set -euo pipefail) for backpressure/correctness checks: tests, types, lint, etc. Only create this file when the user's constraints require correctness validation (e.g., "tests must pass", "types must check").

When this file exists:

• Runs automatically after every passing benchmark in run-experiment.sh.
• If checks fail, run-experiment.sh reports it clearly — log as checks_failed.
• Its execution time does NOT affect the primary metric.
• You cannot keep a result when checks have failed.
• Has a separate timeout (default 300s, configurable via argument).

Keep output minimal. Only the last 80 lines of checks output are fed back. Suppress verbose progress/success output and let only errors through.

#!/bin/bash
set -euo pipefail
# Example: run tests and typecheck — suppress success output, only show errors
pnpm test --run --reporter=dot 2>&1 | tail -50
pnpm typecheck 2>&1 | grep -i error || true

Loop Rules

LOOP FOREVER. Never ask "should I continue?" — the user expects autonomous work.

• Deep understanding first. Re-read source files, study profiling data, reason about what the CPU/GPU is actually doing. The best experiments come from deep understanding, not random variations. When stuck, think longer before trying more things.
• Primary metric is king. Improved → keep. Worse/equal → discard. Secondary metrics rarely affect this decision.
• Simpler is better. Removing code for equal perf = always keep. Ugly complexity for tiny gain = probably discard.
• Focused changes. Prefer single-file, focused edits. They're easier to evaluate, revert, and understand.
• Respect the time budget. Every second of overhead is multiplied by hundreds of runs. Keep the benchmark fast.
• Don't thrash. Repeatedly reverting the same idea? Try something structurally different.
• Transfer thinking. Consider: will this improvement transfer to larger configurations? Fundamental improvements > clever tricks.
• Crashes: fix if trivial, otherwise log and move on. Don't over-invest.
• Resuming: if autoresearch.md exists, read it (it contains the full protocol) + autoresearch.jsonl + git log, continue looping.

NEVER STOP. The user may be away for hours. Keep going until interrupted.

Ideas Backlog

When you discover complex but promising optimizations that you won't pursue right now, append them as bullets to autoresearch.ideas.md. Don't let good ideas get lost.

On resume (context limit, crash), check autoresearch.ideas.md — prune stale/tried entries, experiment with the rest. When all paths are exhausted, delete the file and write a final summary.

User Messages During Experiments

If the user sends a message while an experiment is running, finish the current run-experiment.sh + log-experiment.sh cycle first, then incorporate their feedback in the next iteration. Don't abandon a running experiment.

Example Domains

Domain	Metric	Command
Test speed	seconds ↓	pnpm test
Bundle size	KB ↓	pnpm build && du -sb dist
LLM training	val_bpb ↓	uv run train.py
Build speed	seconds ↓	pnpm build
Lighthouse	perf score ↑	lighthouse http://localhost:3000 --output=json