autoresearch

name	autoresearch
version	2.0
type	meta
description	Teaches other skills to improve themselves using Karpathy's autoresearch pattern
user-invocable	true
agent	NEXUS
agent_model	claude-opus-4-6
mcps	["hermes","paperclip","command-center"]
gstack_skills	[]
eval_metric	skill_improvement_rate
eval_budget	30s
guard	no_skill_regression

/autoresearch — Autonomous Skill Improvement

What it does

The meta-skill. Takes any Skill 2.0 SKILL.md and applies Karpathy's autoresearch loop: Modify -> Eval -> Keep/Discard -> Repeat until the skill's metric improves.

Can run manually or overnight via Hermes cron.

The Pattern (Karpathy, March 2026)

One file.  One metric.  One loop.
Modify → Verify → Keep/Discard → Repeat forever.

Core constraints that make it work:

• ONE file to modify (SKILL.md) — keeps scope manageable
• ONE metric to optimize — eliminates ambiguity
• Fixed eval budget — makes all experiments comparable
• Binary keep/discard — no "maybe later" states
• Git as memory — experiments preserved, failures reverted

Protocol

Step 1: Select target skill

Which Skill 2.0 do you want to improve?

Available targets:
  closer-autopilot   (metric: outreach_response_rate)
  oracle-dd          (metric: dd_checklist_completion)
  pulse-content      (metric: lead_quality_score)
  ledger-models      (metric: model_accuracy)
  atlas-decisions    (metric: decision_confidence_score)
  nexus-router       (metric: routing_accuracy)
  forge-deploy       (metric: deploy_success_rate)
  shield-compliance  (metric: compliance_coverage)
  engine-sprints     (metric: sprint_velocity)

Step 2: Read current state

1. Read the target SKILL.md
2. Read results.tsv for iteration history
3. Identify patterns: what changes improved the metric? What failed?

Step 3: Fetch latest patterns (Paperclip)

Before proposing changes, fetch fresh intelligence:

• Latest Claude Code skill best practices
• Recent gstack updates (new capabilities to leverage)
• Domain-specific improvements (e.g., sales copy research for closer-autopilot)

Step 4: Propose ONE change

Based on history + fresh patterns, propose a single atomic change:

Types of changes (ordered by impact):

1. Add missing coverage — fill gaps in the skill's protocol
2. Sharpen criteria — make scoring/evaluation more precise
3. Add new data source — connect a new MCP or gstack skill
4. Optimize flow — reduce steps or improve ordering
5. Improve examples — better templates or scoring rubrics

Rules:

• ONE change per iteration (atomic, traceable)
• Read before write (understand the full skill first)
• Simpler wins — equal results + less complexity = keep simpler version

Step 5: Apply change

# Edit the target SKILL.md with the proposed change
# Commit with experiment prefix
git add skills-2.0/{{skill_name}}/SKILL.md
git commit -m "experiment: {{description of change}}"

Step 6: Run eval

# Run the skill's eval harness
cd skills-2.0/{{skill_name}}
npx ts-node eval.ts
# or: bun run eval.ts

Check result:

• Metric improved + guard passed → KEEP (commit stands)
• Metric regressed OR guard failed → REVERT

  git revert HEAD --no-edit
  

Step 7: Log result

Append to the skill's results.tsv:

{{iteration}}\t{{commit}}\t{{score}}\t{{delta}}\t{{keep|revert}}

Step 8: Repeat or stop

• Manual mode: Ask "Continue iterating?" after each cycle
• Hermes mode: Continue until max_iterations or time limit

Hermes Overnight Mode

Schedule Hermes to run autoresearch while you sleep:

# Hermes cron config
name: autoresearch-nightly
schedule: "0 2 * * *"  # 2 AM CT
max_duration: 4h        # ~480 iterations at 30s each
targets:
  - skills-2.0/closer-autopilot
  - skills-2.0/oracle-dd
  - skills-2.0/nexus-router
rotation: round-robin    # cycle through targets
report_to: command-center # daily summary on completion

To set up:

wsl -d Ubuntu -- bash -c "hermes cron add autoresearch-nightly '0 2 * * *' 'cd /mnt/c/Users/whitt/Development/henry-ai-company && run autoresearch loop'"

Teaching GStack New Patterns

Autoresearch can also improve gstack skills by targeting their SKILL.md.tmpl files:

Target: ~/.claude/skills/gstack/qa/SKILL.md.tmpl
Metric: bugs_found_per_session
Guard: false_positive_rate < 0.1
After change: bun run gen:skill-docs (regenerate from template)

This lets Hermes teach gstack overnight — improving /qa, /review, /office-hours patterns using real iteration data.

Cross-Pollination

After N iterations of improvement, NEXUS distributes learned patterns:

• Improved /qa patterns → FORGE (better testing methodology)
• Improved /office-hours patterns → ATLAS (better strategic framing)
• Improved /review patterns → SHIELD (better legal doc review)
• Improved /debug patterns → ORACLE (better research methodology)
• Improved /cso patterns → SHIELD (better compliance auditing)

Meta-Eval

This skill itself is autoresearchable:

• Metric: skill_improvement_rate — average metric improvement across all targets per night
• Guard: no skill's metric should regress over a 7-day window

References

• Karpathy's autoresearch — original ML iteration pattern
• Generalized autoresearch skill — Claude Code adaptation
• Pattern deep-dive — implementation guide