autoresearch

name	autoresearch
description	Autonomous performance-research loop for the markdown previewer fork. Picks an idea, edits ext/markdown-fast/, builds, runs the bench, logs to results.tsv, and advances or reverts the branch based on TTFP. Runs forever until interrupted. Use when the user says "continue autoresearch", "keep iterating on perf", "run the loop", or after `program.md` setup is complete.
argument-hint	["continue | <run-tag>"]
allowed-tools	Bash, Read, Write, Edit, Glob, Grep, Agent

autoresearch — perf experiment loop

Drive the autonomous TTFP-minimization loop defined in program.md at the repo root. The seed idea backlog lives in docs/plans/20260502-perf-improvement-ideas.md (5 fully-specified ideas + common prereq). After those are exhausted, generate new ideas yourself.

Read first (every invocation, in this order):

1. program.md — full rules of the loop. Source of truth.
2. README.md — repo goal, metric (TTFP), bench commands.
3. docs/plans/20260502-perf-improvement-ideas.md — seed backlog.
4. results.tsv — what's already been tried. Don't repeat.

If program.md or results.tsv is missing, the loop hasn't been set up. Run the Setup section of program.md with the user, then proceed.

Pre-flight (once per session)

• Confirm we're on an autoresearch/<tag> branch. If not and the user passed a tag arg, git checkout -b autoresearch/<tag>. If no tag arg and not on such a branch, ask for a tag.
• Confirm tmp/corpus/*.md exists (else npm run gen:corpus).
• Confirm benchmark/results/e2e-baseline-vscode.json exists (else npm run bench:e2e).
• Confirm results.tsv exists with the header commit\tmedian_ttfp_ms\tp95_ttfp_ms\tflags\tstatus\tdescription. If not, create it (untracked — gitignored).
• Confirm ext/markdown-fast/node_modules exists (else (cd ext/markdown-fast && npm install)).

The loop

LOOP FOREVER (until human interrupts):

1. Pick the next idea

• If the seed backlog has unattempted items (compare against results.tsv descriptions), take the next one in the recommended order from the plan file: common prereq → Idea 3 → Idea 1 → Idea 2 → Idea 4-A → Idea 5.
• Otherwise generate a new idea. Sources: latest [BENCH] markers in benchmark/results/e2e-*.json (slowest phase is the target), upstream tmp/vscode-markdown/extensions/markdown-language-features/, combinations of previous wins, ablations, more radical rewrites.
• Announce the idea in one sentence before editing.

2. Implement

Delegate the edit to a general-purpose subagent. Brief it with:

• the idea (hypothesis + flag name + files to touch from the plan file or your own scoping),
• the constraint that only ext/markdown-fast/ may be edited — never benchmark/, never tmp/vscode-markdown/,
• new code paths must be gated behind an env flag in ext/markdown-fast/src/optFlags.ts (default off),
• run npm run build:fork at the end and report any compile errors,
• return a one-line summary of what changed.

The subagent does the edits; you stay in the loop.

3. Commit

git add -A
git commit -m "<short idea summary>"

Capture the short SHA: git rev-parse --short HEAD.

4. Build

npm run build:fork > tmp/build.log 2>&1

If non-zero exit, tail -n 50 tmp/build.log. Easy fix → fix and re-run. Else log crash, git reset --hard HEAD~1, go back to step 1.

5. Bench

Pick the right command for the change:

• Host-side only (parse/render path, before IPC): start with npm run bench:offline > tmp/run.log 2>&1 for fast iteration. This is a pre-check, not a substitute for E2E.
• Always end with E2E:

  BENCH_FORK=1 <FLAGS> BENCH_N=5 \
    BENCH_TARGETS="prose-5MB.md,mixed-5MB.md,code-5MB.md,table-5MB.md" \
    npm run bench:e2e:fork > tmp/run.log 2>&1
  

  <FLAGS> is whatever this experiment needs (e.g. BENCH_INLINE_BODY=1 BENCH_LAZY_HL=1). Result file naming follows benchmark/results/e2e-fork-<variant>.json per README.

Hard timeout: 5 minutes. If it doesn't return, kill, treat as crash.

Do not tee. Do not cat the full log. Read only what you need.

6. Read out results

Parse the right JSON for the variant you ran. Median + p95 of ttfp_e2e_ms across iterations:

node -e 'const f=process.argv[1]; const j=require(f); const xs=j.iterations.map(i=>i.ttfp_e2e_ms).sort((a,b)=>a-b); const q=p=>xs[Math.floor(xs.length*p)]; console.log(JSON.stringify({median:q(0.5), p95:q(0.95), n:xs.length}))' benchmark/results/e2e-fork-<variant>.json

If the JSON is missing or iterations is empty, the run failed. tail -n 80 tmp/run.log and decide: easy fix → fix + retry; else log crash + revert.

7. Decide

Compare median + p95 against the previous good entry in results.tsv (or the fork-baseline row if this is the first non-baseline experiment).

• Win: median improved ≥3% AND p95 did not regress >5% → status keep, branch advances (do nothing — the commit stays).
• Wash / regress: status discard, git reset --hard HEAD~1.
• Crash: status crash, git reset --hard HEAD~1.

If the result is borderline (1–3% median delta), re-run the bench once before deciding — TTFP has noise.

8. Log

Append one tab-separated row to results.tsv:

<sha>	<median_ttfp_ms>	<p95_ttfp_ms>	<flags>	<status>	<description>

Use 0 for ttfp values on crash. flags is e.g. none, lazyhl, inline+lazyhl. Keep description short, no commas, no tabs. Do not commit results.tsv.

9. Loop

Go back to step 1. Do not stop. Do not ask the user if they want to continue. The user is autonomous-mode and may be away from the keyboard.

Stop conditions

Only stop when:

• The user explicitly interrupts.
• You hit 3 consecutive crashes you can't diagnose — write a short status paragraph, then stop.
• The repo is in a dirty state you can't safely reset (e.g. a merge conflict from an unexpected upstream change) — stop and ask.

Otherwise: keep going. Out of seed ideas? Generate. Out of generated ideas? Re-read the plan, re-read fresh [BENCH] markers, try ablations and combinations. The point of this loop is volume.

Anti-patterns

• ❌ Editing benchmark/, tmp/vscode-markdown/, or the baseline JSON files.
• ❌ Skipping the env-flag gate "just for a quick test" — every change must be revertable by toggling a flag, not by reverting code.
• ❌ Calling a 1ms median delta a win.
• ❌ Asking the user "should I keep going?".
• ❌ Committing results.tsv or tmp/.
• ❌ Running tests/builds without redirecting output (floods context).
• ❌ Doing host-side and webview-side changes in the same commit — keep experiments atomic so revert is clean.