autoresearch

name	autoresearch
description	Autonomous Goal-directed Iteration. Apply Karpathy's autoresearch principles to ANY task. Loops autonomously — modify, verify, keep/discard, repeat. Supports optional loop count via Claude Code's /loop command. Invoking /autoresearch <free-form goal> builds a real-data benchmark harness, captures a baseline, and iterates with a regression gate until the goal is hit.
version	1.0.4

Claude Autoresearch — Autonomous Goal-directed Iteration

Inspired by Karpathy's autoresearch. Applies constraint-driven autonomous iteration to ANY work — not just ML research.

Core idea: You are an autonomous agent. Modify → Verify → Keep/Discard → Repeat.

Subcommands

Subcommand	Purpose
/autoresearch <goal>	Default path — parse free-form goal, build harness, capture baseline, loop until goal met
/autoresearch	Run the autonomous loop (default)
/autoresearch:plan	Interactive wizard to build Scope, Metric, Direction & Verify from a Goal
/autoresearch:security	Autonomous security audit: STRIDE threat model + OWASP Top 10 + red-team (4 adversarial personas)

Default Path: /autoresearch

When the user invokes /autoresearch <goal> with any free-form string after the command, parse the goal into seven slots, print the parsed-slot dump back for user visibility, then run the harness protocol in references/benchmark-harness.md before entering the loop.

Goal-parsing rubric:

Slot	Extraction rule	Fallback
metric	First measurable noun (latency, reliability, coverage, flakiness, bundle size, p95, accuracy, error-rate, LOC, build time)	Ask user (1 sentence)
direction	reduce/lower/below/under/minimise/to 0% + cost-word → minimise; increase/raise/above/over/maximise/to 100% + quality-word → maximise	minimise for cost/time/size/error, maximise for coverage/score/throughput
target	Number + unit in goal (500ms, 95%, 0%, <200KB)	"best achievable" — unbounded loop
scope	Grep repo for goal's domain terms (API, test, build); propose globs	Whole repo minus node_modules, .venv, dist, target
corpus_source	If goal names inputs (signals, queries, PRs, logs) → find source; if absent → ASK, never fabricate	corpus_required=false only when metric is purely structural (LOC, build time, bundle size)
verify_cmd	Single shell command that prints metric: <float> on stdout — typically python benchmark.py or equivalent single-file rig	Constructed during harness build
regression_cmd	Auto-detect: first of pytest -q, npm test, cargo test, go test ./... whose config exists	Ask user

Worked examples:

/autoresearch reduce API p95 latency to 200ms
→ metric=p95_latency_ms, direction=minimise, target=200, scope=src/api/**,
  corpus_source=prod log tail or fixtures, verify_cmd=python benchmark.py,
  regression_cmd=pytest -q

/autoresearch reduce test flakiness to 0%
→ metric=flaky_test_rate, direction=minimise, target=0, scope=tests/**,
  corpus_source=CI run history, verify_cmd=python benchmark.py (N reruns),
  regression_cmd=pytest -q

/autoresearch increase signal-parser reliability to 99%
→ metric=reliability, direction=maximise, target=0.99, scope=src/parser/**,
  corpus_source=autoresearch/data/signals.jsonl, verify_cmd=python benchmark.py,
  regression_cmd=pytest -q

Print the parsed slot dump to the user before any action — this is the single confirmation checkpoint before the harness protocol begins.

/autoresearch:security — Autonomous Security Audit (v1.0.3)

Runs a comprehensive security audit using the autoresearch loop pattern. Generates a full STRIDE threat model, maps attack surfaces, then iteratively tests each vulnerability vector — logging findings with severity, OWASP category, and code evidence.

Load: references/security-workflow.md for full protocol.

What it does:

1. Codebase Reconnaissance — scans tech stack, dependencies, configs, API routes
2. Asset Identification — catalogs data stores, auth systems, external services, user inputs
3. Trust Boundary Mapping — browser↔server, public↔authenticated, user↔admin, CI/CD↔prod
4. STRIDE Threat Model — Spoofing, Tampering, Repudiation, Info Disclosure, DoS, Elevation of Privilege
5. Attack Surface Map — entry points, data flows, abuse paths
6. Autonomous Loop — iteratively tests each vector, validates with code evidence, logs findings
7. Final Report — severity-ranked findings with mitigations, coverage matrix, iteration log

Key behaviors:

• Follows red-team adversarial mindset (Security Adversary, Supply Chain, Insider Threat, Infra Attacker)
• Every finding requires code evidence (file:line + attack scenario) — no theoretical fluff
• Tracks OWASP Top 10 + STRIDE coverage, prints coverage summary every 5 iterations
• Composite metric: (owasp_tested/10)*50 + (stride_tested/6)*30 + min(findings, 20) — higher is better
• Creates security/{YYMMDD}-{HHMM}-{audit-slug}/ folder with structured reports: overview.md, threat-model.md, attack-surface-map.md, findings.md, owasp-coverage.md, dependency-audit.md, recommendations.md, security-audit-results.tsv

Flags:

Flag	Purpose
--diff	Delta mode — only audit files changed since last audit
--fix	After audit, auto-fix confirmed Critical/High findings using autoresearch loop
--fail-on {severity}	Exit non-zero if findings meet threshold (for CI/CD gating)

Usage:

# Unlimited — keep finding vulnerabilities until interrupted
/autoresearch:security

# Bounded — exactly 10 security sweep iterations
/loop 10 /autoresearch:security

# With focused scope
/autoresearch:security
Scope: src/api/**/*.ts, src/middleware/**/*.ts
Focus: authentication and authorization flows

# Delta mode — only audit changed files since last audit
/autoresearch:security --diff

# Auto-fix confirmed Critical/High findings after audit
/loop 15 /autoresearch:security --fix

# CI/CD gate — fail pipeline if any Critical findings
/loop 10 /autoresearch:security --fail-on critical

# Combined — delta audit + fix + gate
/loop 15 /autoresearch:security --diff --fix --fail-on critical

Inspired by:

• Strix — AI-powered security testing with proof-of-concept validation
• /plan red-team — adversarial review with hostile reviewer personas
• OWASP Top 10 (2021) — industry-standard vulnerability taxonomy
• STRIDE — Microsoft's threat modeling framework

/autoresearch:plan — Goal → Configuration Wizard

Converts a plain-language goal into a validated, ready-to-execute autoresearch configuration.

Load: references/plan-workflow.md for full protocol.

Quick summary:

1. Capture Goal — ask what the user wants to improve (or accept inline text)
2. Analyze Context — scan codebase for tooling, test runners, build scripts
3. Define Scope — suggest file globs, validate they resolve to real files
4. Define Metric — suggest mechanical metrics, validate they output a number
5. Define Direction — higher or lower is better
6. Define Verify — construct the shell command, dry-run it, confirm it works
7. Confirm & Launch — present the complete config, offer to launch immediately

Critical gates:

• Metric MUST be mechanical (outputs a parseable number, not subjective)
• Verify command MUST pass a dry run on the current codebase before accepting
• Scope MUST resolve to ≥1 file

Usage:

/autoresearch:plan
Goal: Make the API respond faster

/autoresearch:plan Increase test coverage to 95%

/autoresearch:plan Reduce bundle size below 200KB

After the wizard completes, the user gets a ready-to-paste /autoresearch invocation — or can launch it directly.

When to Activate

• User invokes /autoresearch <goal-string> (anything after the command) → parse with Default Path rubric, then build harness per references/benchmark-harness.md
• User types /autoresearch with no argument → ask for a one-sentence goal OR suggest /autoresearch:plan
• User invokes /autoresearch or /ug:autoresearch → run the loop
• User invokes /autoresearch:plan → run the planning wizard
• User invokes /autoresearch:security → run the security audit
• User says "help me set up autoresearch", "plan an autoresearch run" → run the planning wizard
• User says "security audit", "threat model", "OWASP", "STRIDE", "find vulnerabilities", "red-team" → run the security audit
• User says "work autonomously", "iterate until done", "keep improving", "run overnight" → run the loop
• Any task requiring repeated iteration cycles with measurable outcomes → run the loop

Optional: Controlled Loop Count

By default, autoresearch loops forever until manually interrupted. However, users can optionally specify a loop count to limit iterations using Claude Code's built-in /loop command.

Requires: Claude Code v1.0.32+ (the /loop command was introduced in this version)

Usage

Unlimited (default):

/autoresearch
Goal: Increase test coverage to 90%

Bounded (N iterations):

/loop 25 /autoresearch
Goal: Increase test coverage to 90%

This chains /autoresearch with /loop 25, running exactly 25 iteration cycles. After 25 iterations, Claude stops and prints a final summary.

When to Use Bounded Loops

Scenario	Recommendation
Run overnight, review in morning	Unlimited (default)
Quick 30-min improvement session	/loop 10 /autoresearch
Targeted fix with known scope	/loop 5 /autoresearch
Exploratory — see if approach works	/loop 15 /autoresearch
CI/CD pipeline integration	/loop N /autoresearch (set N based on time budget)

Behavior with Loop Count

When a loop count is specified:

• Claude runs exactly N iterations through the autoresearch loop
• After iteration N, Claude prints a final summary with baseline → current best, keeps/discards/crashes
• If the goal is achieved before N iterations, Claude prints early completion and stops
• All other rules (atomic changes, mechanical verification, auto-rollback) still apply

Setup Phase (Do Once)

1. Parse the goal using the Default Path rubric → print slot dump
2. Read all in-scope files for full context before any modification
3. Ingest corpus per references/benchmark-harness.md Phase A — refuse if synthetic or empty
4. Build single-file harness per Phase B — must print metric: <float> on stdout
5. Capture baseline per Phase C — record as iteration #0 in autoresearch/results.tsv
6. Establish regression gate per Phase D — detect existing test suite, record pass count in autoresearch/.regression-baseline
7. Read hot path per Phase E — list candidate ideas (never apply more than one per iteration)
8. Confirm and go — show setup summary, BEGIN THE LOOP

The Loop

Read references/autonomous-loop-protocol.md for full protocol details.

LOOP (FOREVER or N times):
  1. Review: Read current state + git history + results log
  2. Ideate: Pick next change based on goal, past results, what hasn't been tried
  3. Modify: Make ONE focused change to in-scope files
  4. Commit: Git commit the change (before verification)
  5. Verify: Run the mechanical metric (tests, build, benchmark, etc.)
  5a. Regress: Run regression_cmd. If passed < N_pre → STATUS=discard (skip Decide).
  6. Decide:
     - IMPROVED → Keep commit, log "keep", advance
     - SAME/WORSE → Git revert, log "discard"
     - CRASHED → Try to fix (max 3 attempts), else log "crash" and move on
  7. Log: Record result in results log
  8. Repeat: Go to step 1.
     - If unbounded: NEVER STOP. NEVER ASK "should I continue?"
     - If bounded (N): Stop after N iterations, print final summary

Critical Rules

1. Loop until done — Unbounded: loop until interrupted. Bounded: loop N times then summarize.
2. Read before write — Always understand full context before modifying
3. One change per iteration — Atomic changes. If it breaks, you know exactly why
4. Mechanical verification only — No subjective "looks good". Use metrics
5. Automatic rollback — Failed changes revert instantly. No debates
6. Simplicity wins — Equal results + less code = KEEP. Tiny improvement + ugly complexity = DISCARD
7. Git is memory — Every kept change committed. Agent reads history to learn patterns
8. When stuck, think harder — Re-read files, re-read goal, combine near-misses, try radical changes. Don't ask for help unless truly blocked by missing access/permissions
9. Real data only — corpus must be scraped/exported from reality; synthetic cases forbidden
10. Regression gate is absolute — any drop in pre-existing passing tests triggers auto-discard
11. Harness is read-only during iteration — only source files change; harness edits need a separate commit prefixed harness:

Principles Reference

See references/core-principles.md for the 7 generalizable principles from autoresearch.

Adapting to Different Domains

Domain	Metric	Scope	Verify Command	Corpus Source
Backend code	Tests pass + coverage %	src/**/*.ts	npm test	test fixtures
Frontend UI	Lighthouse score	src/components/**	npx lighthouse	staging URLs
ML training	val_bpb / loss	train.py	uv run train.py	training dataset
Blog/content	Word count + readability	content/*.md	Custom script	source manuscripts
Performance	Benchmark time (ms)	Target files	npm run bench	benchmark inputs
Refactoring	Tests pass + LOC reduced	Target module	npm test && wc -l	existing test suite
Security	OWASP + STRIDE coverage + findings	API/auth/middleware	/autoresearch:security	codebase
Real-traffic performance	p95 latency (ms)	hot-path files	python benchmark.py	prod log tail / export

Adapt the loop to your domain. The PRINCIPLES are universal; the METRICS are domain-specific.