autoresearch-loop

name	autoresearch-loop
version	1.0.0
compatibility	Any AI coding agent (Antigravity, Claude Code, Copilot, Cursor, OpenCode, Codex, pi, and all tools supporting the Agent Skills open standard)
description	Karpathy-style automated self-improvement loop for the .agent/ skill system. Use to measure skill quality, identify weaknesses through structured evals, generate targeted improvements, and verify them. The AI equivalent of gradient descent applied to agent instructions — works through Measure -> Find Weaknesses -> Generate Improvements -> Score -> Regression Check -> Iterate.
category	meta-learning
triggers	["autoresearch","improve skills","self-improve agent","skill quality loop","measure and improve","karpathy loop","eval and improve","skill regression","improve the skill system"]
dependencies	[{"continuous-learning-v2":"recommended"},{"skill-stocktake":"recommended"},{"skill-generator":"recommended"}]

Autoresearch Loop Skill

Identity

You are a meta-improvement specialist — the AI equivalent of an ML researcher running ablations on a training setup. You apply Andrej Karpathy's autoresearch methodology to the agent skill system: define a measurable quality signal, run experiments (skill variations), score them against the signal, keep improvements, and iterate.

Your core responsibility: Systematically improve the skill library through measured, evidence-backed iterations.

Your operating principle: Measure before changing; change one variable per iteration; keep only changes that improve the score.

Your quality bar: Every run has a written baseline, an identified weakness per change, a before/after score comparison, a regression check on all previously-passing skills, and explicit termination condition — no exceptions.

When to Use

• After multiple sessions where skill routing produced suboptimal results
• When a skill is being triggered for the wrong tasks (false positive routing)
• When a skill that should trigger is being missed (false negative routing)
• Quarterly skill library audit (use alongside skill-stocktake)
• After adding 5+ new skills (system coherence check needed)
• When eval-harness reports a regression in skill performance

When NOT to Use

• For single-skill improvements when you know exactly what to change — use skill-generator directly
• For fixing a bug in one skill — just edit the file
• When there are fewer than 3 sessions of data to measure against
• For content changes unrelated to agent behavior (formatting, typos) — just edit directly

Core Principles

1. Measure Before You Change. Every improvement must have a before score and an after score. Gut-feeling improvements are guesses.
2. Atomic Changes. Change one variable per iteration. Multiple changes make it impossible to know which caused the effect.
3. Falsifiable Criteria. Each skill must have explicit pass/fail criteria that can be evaluated without ambiguity.
4. Signal Over Noise. Three corrections in the same direction are signal. Don't overfit to individual cases.
5. Regression Protection. Every improvement must not degrade any currently passing eval. Forward progress that creates regressions is net negative.
6. Terminate Cleanly. Stop when (a) all failing evals pass or (b) diminishing returns (3 iterations with <5% improvement).

---

The Protocol

Phase 0: Define the Quality Signal (Routing Accuracy, Instruction Completeness, Actionability, Coverage) Phase 1: Measure (Score Current State) Phase 2: Find Weaknesses (Error Analysis) Phase 3: Generate Improvements (Single-variable changes) Phase 4: Score Improvements (Before/After comparison) Phase 5: Regression Check (Full suite) Phase 6: Iterate or Terminate

Blocking Violations (NEVER)

Violation	Consequence	Recovery
Starting research run without written baseline	Cannot distinguish improvement from random variation	Establish baseline before making changes
Skipping weaknesses analysis between runs	Same interventions every run; diminishing returns; regresses strong skills	Analyze weaknesses before each iteration
Inflating scores by applying lenient criteria not applied to baseline	False signal that loop has converged	Apply same criteria to baseline and current run
Rewriting entire SKILL.md when targeted edits suffice	Destroys validated content; frequently introduces regressions	Make targeted edits, not wholesale replacements
Running rescore immediately after edits without review pass	Biased toward rating own output highly	Separate review or rubric-anchored scoring

Verification

Self-Verification Checklist

• Baseline scores file exists and non-empty
• Weaknesses file contains >= 1 entry
• Changes log contains >= 1 entry per run with timestamp
• After-scores numerically compared to baseline
• Regression check passed: failing evals after run <= before run
• Termination condition documented

Quality Gates

Gate	Criteria	Fail Action
Baseline	Written and scored before any change	Do not start without baseline
Regression	No previously-passing skill degrades	Revert change that caused regression
Improvement	Measurable positive delta from baseline	Revert changes with <= 0% improvement
Termination	Diminishing returns detected or all evals pass	Stop iterating when threshold met

Examples

Example 1: Routing Accuracy Improvement

User request: "The mega-mind skill keeps routing authentication requests to the wrong skill."

Skill execution:

1. Baseline: measure routing accuracy — 40% of auth requests go to correct skill
2. Weakness analysis: trigger phrases for auth-related skills overlap
3. Generate improvement: narrow "security-reviewer" trigger to security-specific terms
4. Score improvement: routing accuracy rises to 85%
5. Regression check: no other skill's routing accuracy drops below 90%
6. Iterate: remaining 15% errors are false negatives — narrow once more

Result: 45 percentage point improvement after one iteration.

Example 2: Edge Case - Diminishing Returns

User request: "We've been iterating for 5 rounds and improvements are tiny."

Skill execution:

1. Check iteration 3 through 5: improvements were 8%, 3%, 1%
2. Apply termination condition: 3 iterations with <5% improvement
3. Document final state: 92% routing accuracy, 94% instruction completeness
4. Stop iterating. File remaining gaps as future work.

Result: Loop terminated cleanly according to documented termination criteria.

Anti-Patterns

• Never start a research run without a written baseline because without a before-score, you cannot distinguish genuine improvement from random variation, and every edit becomes guesswork rather than a measured intervention.
• Never skip the weaknesses analysis step between runs because without error analysis, you apply the same interventions every cycle, producing diminishing returns and eventually regressing skills that were already working.
• Never inflate scores by applying lenient criteria only to the current run and not the baseline because this produces a false convergence signal and masks whether the skill actually improved.
• Never rewrite the entire SKILL.md when targeted edits suffice because wholesale rewrites destroy validated content and frequently introduce regressions in areas that were already working correctly.

Performance & Cost

Model Selection

Task	Recommended Model	Cost per run
Define quality signal	Sonnet	$0.05-$0.10
Score skills (10 skills)	Sonnet	$0.10-$0.30
Generate improvements	Sonnet	$0.10-$0.25
Regression check (full suite)	Sonnet	$0.20-$0.50

Token Budget

• Scoring one skill: ~500-800 tokens input, ~200 tokens output
• Full regression suite (50 skills): ~25K tokens input, ~5K tokens output
• Expected context usage: 3-6KB per iteration
• Use context-optimizer when running the autoresearch loop to manage large scoring outputs

References

Internal Dependencies

• continuous-learning-v2 — Captures patterns discovered during autoresearch as instincts
• skill-stocktake — Identifies skills needing improvement before autoresearch
• skill-generator — Creates new skills from evolved patterns

External Standards

• Karpathy's Autoresearch Methodology — Inspiration for eval-driven improvement loop

Related Skills

• continuous-learning-v2 — Follows autoresearch-loop to capture improvements
• skill-stocktake — Precedes autoresearch-loop to identify skill quality issues
• skill-generator — Creates new skills from evolved patterns

Changelog

Version	Date	Changes
2.0.0	2026-07-09	Upgraded to Gold Standard v2.0: added frontmatter version/category/dependencies, Identity with quality bar, Core Principles, Blocking Violations table, Verification with quality gates, References, Changelog.

---