autoresearch

name	autoresearch
description	Autonomous goal-directed iteration loop that continuously improves prompts, templates, configs, or code. Two evaluation modes — deterministic (eval.py with proxy heuristics) or AI judge (LLM rubric scoring). Uses four-way separation in both modes. Inspired by Karpathy's autoresearch.

AutoResearch — Autonomous Optimization Loop

USE WHEN the user runs /autoresearch, says "autoresearch", "optimize this prompt", "improve this overnight", "run an optimization loop", "iterate on this", "auto-improve", or wants to autonomously refine a file against measurable criteria.

---

Architecture: Separation of Roles

This skill supports two mutually exclusive evaluation modes. Both maintain four-way separation.

Deterministic Mode (default) — Four-Way Separation

Role	Who	Knows Eval Code?	Knows Prompt History?
Main Agent	You (optimizer)	NO — reads metric number only	Yes — reads logs, plans changes
Eval Agent	autoresearch-eval-agent sub-agent	Yes — writes eval.py	No
Test Runner	autoresearch-test-runner sub-agent	NO — fresh context	NO
Judge Script	eval.py (deterministic Python)	IS the eval	No

Metric: pass_rate. Best for tasks with clear, mechanical quality checks (word count, format, keywords, structure).

AI Judge Mode (opt-in) — Four-Way Separation

Role	Who	Knows Rubric?	Knows Prompt History?
Main Agent	You (optimizer)	Criteria names only	Yes — reads logs, plans changes
Eval Agent	autoresearch-eval-agent sub-agent	Yes — writes rubric.md	No
Test Runner	autoresearch-test-runner sub-agent	NO — fresh context	NO
Judge Agent	autoresearch-judge sub-agent	Follows rubric	NO — fresh context

Metric: quality_score. Best for creative/subjective tasks where mechanical checks miss the point (tone, authenticity, narrative, emotional resonance).

The modes are mutually exclusive. Deterministic mode uses eval.py only. AI judge mode uses the judge agent only. There is no combined score.

Key isolation (both modes):

• The optimizer never writes eval artifacts, never generates outputs
• The eval agent writes the eval system once (eval.py OR rubric.md), then disappears
• The test runner never sees the eval or rubric — fresh context every time
• The judge (script or agent) evaluates without knowing iteration history or optimizer intent

---

User Interaction: Always Use AskUserQuestion

Whenever you need user input or confirmation, use the AskUserQuestion tool. This shows an interactive popup in Cowork.

CRITICAL: Keep popup text SHORT. The popup is small — long text becomes unreadable. Follow this pattern:

1. Write details in chat FIRST (assertions list, eval code, explanations)
2. Then use AskUserQuestion with ONLY a short question — one sentence max

Good:

[In chat]: Here are 7 proposed assertions:
1. Covers all 4 search clusters
2. Each entry has title, channel, views
3. Contains opportunity mapping
...

[AskUserQuestion]: "Do these assertions look right?"
Options: "These look good" / "Adjust some" / [free text]

Bad:

[AskUserQuestion]: "Here are 7 proposed assertions for what a 'good' output looks like: 1. Covers all 4 search clusters (Core, Tools, Niche, Competitors)... [giant paragraph]"

Never put lists, explanations, or details inside AskUserQuestion. The popup is for the QUESTION only.

---

The Three Rules

Every criterion — whether proposed by you or provided by the user — MUST pass these three rules before entering the loop. This is the most important step in the process. Bad criteria produce bad evals, and bad evals waste every iteration.

Rule 1: State the exact condition, not the goal. Don't describe what you want. Describe what you can measure.

Bad	Good
"Make sure the hook is short"	"The first line must be under 136 characters including spaces"
"Should be professional"	"Contains no exclamation marks and no ALL CAPS words (3+ letters)"
"Include relevant data"	"Contains at least one specific number or statistic with a source"

Rule 2: One criterion, one variable. Each criterion tests exactly one thing. If you're tempted to use "and" to connect two checks, split them into two separate criteria.

Bad	Good
"Under 150 words and ends with a question"	Criterion 1: "Under 150 words" / Criterion 2: "Last sentence ends with a question mark"
"Professional tone with no jargon"	Criterion 1: "No words from the banned jargon list" / Criterion 2: "No sentences over 25 words"

Rule 3: Define the test (optional). Describe how to verify the criterion — what to count, what regex to match, what structure to look for. This helps the eval agent write better checks and helps the judge agent score more consistently.

Criterion	Test definition
"First line under 136 characters"	len(lines[0]) <= 136
"Contains at least one statistic"	re.search(r'\d+[%x]?\s', text) returns a match
"Ends with a question"	text.rstrip().endswith("?")

If the user provides criteria that violate The Three Rules, rewrite them — show the user the before/after so they understand the improvement.

---

When the User Pastes Content

1. Save it to a file in the working directory (e.g., target-skill.md)

2. Explain The Three Rules to the user first. Before proposing any criteria, write in chat:

   The Three Rules — every criterion must pass these before we start:

   1. State the exact condition, not the goal. "First line under 136 characters" not "keep the hook short."
   2. One criterion, one variable. If it has "and", split it into two.
   3. Define the test (optional). How to check it — what to count, match, or look for.

3. Propose 5-7 quality criteria. Every criterion MUST pass The Three Rules. Write the list in chat (not in the popup). For each criterion, show it passes the rules — be specific with thresholds, counts, and patterns.

4. Use AskUserQuestion — short text only:

   • Question: "Do these quality criteria look right?"
   • Options: "These look good" / "Adjust some"

5. SEPARATE STEP — always ask this. Use AskUserQuestion:

   • Question: "Which evaluation mode?"
   • Options: "Deterministic" / "AI Judge"
   • In chat before the question, briefly explain: "Deterministic checks things mechanically (word count, keywords, format). AI Judge uses an LLM to score against a rubric — best for creative content where quality is subjective (tone, authenticity, narrative)."

6. SEPARATE STEP — always ask this. Use AskUserQuestion:

   • Question: "How many iterations? (Recommended: 5)"
   • Options: "5 (recommended)" / "10" / "20"
   • In chat before the question, briefly explain: "Each iteration edits the prompt, generates outputs with real tools, and evaluates. 5 iterations is a good starting point — you can always run more later."
   • Store the chosen iteration count for the loop's stop condition.

7. Spawn the autoresearch-eval-agent sub-agent to generate the eval system:

   • Pass it: the target prompt/skill path, the confirmed criteria list, the working directory, and the chosen eval mode (deterministic or ai_judge)
   • Deterministic: it generates eval.py and test_cases.json
   • AI Judge: it generates rubric.md and test_cases.json (no eval.py)
   • You (the optimizer) do NOT read the eval artifacts. You only read their output (metric numbers and criteria names).

8. Show the user the generated eval.py (deterministic) or rubric.md (AI judge) in chat. Then use AskUserQuestion:

   • Question: "Does this eval look right?"
   • Options: "Looks good" / "Adjust"

9. Mark the eval artifacts as READ-ONLY — you MUST NOT modify them during the loop:

   • Deterministic: eval.py + test_cases.json
   • AI Judge: rubric.md + test_cases.json

10. Run baseline (iteration 0): generate outputs via test runner (always live — using real tools and data), then evaluate (run eval.py or spawn judge agent), record the score

11. Start the loop — stop after the chosen iteration count

If the user says "improve this" without criteria, suggest 5 reasonable ones (all passing The Three Rules) and use AskUserQuestion to confirm. If the user says "go" or "start", default to deterministic mode + 5 iterations, spawn the eval agent, and begin immediately.

---

Phase 0: Session Setup

When the user pastes a skill, prompt, or template:

1. Save it to a file (e.g., target-skill.md)
2. Analyze the content and propose 5-7 quality criteria
3. Ask the user to confirm or adjust
4. Confirm the session config:

Field	Description	Example
Goal	What are we optimizing?	"Improve cold email reply-rate signals"
Eval Mode	deterministic or ai_judge	deterministic
Metric	pass_rate (deterministic) or quality_score (AI judge)	pass_rate
Direction	Higher is better	higher
Criteria	The quality checks (confirmed by user)	"Under 150 words", "Contains a question CTA"
Modifiable File	The prompt/skill file ONLY	target-skill.md
Guard	Optional regression check	None

---

Phase 1: Generate the Eval System (via Eval Agent Sub-Agent)

Before any optimization begins, spawn the autoresearch-eval-agent sub-agent:

Deterministic mode:

Design an eval system for the prompt/skill at [path to target skill].
Eval mode: Deterministic.

Assertions (confirmed by user):
1. [assertion 1]
2. [assertion 2]
...

The prompt expects these inputs: [list input fields from the prompt].
Save eval.py and test_cases.json to [working directory path].

AI Judge mode:

Design an eval system for the prompt/skill at [path to target skill].
Eval mode: AI Judge.

Quality criteria (confirmed by user):
1. [criterion 1]
2. [criterion 2]
...

The prompt expects these inputs: [list input fields from the prompt].
Save rubric.md and test_cases.json to [working directory path].

The eval agent will:

• Deterministic: generate test_cases.json + eval.py (deterministic Python with proxy heuristics), verify no syntax errors
• AI Judge: generate test_cases.json + rubric.md (scoring rubric with 1-5 scale per criterion)

IMPORTANT: You (the optimizer) MUST NOT read the eval artifacts in detail. You only interact with them by running the evaluation and reading the output (metric number and criteria names). This prevents you from gaming the evaluation.

After the eval agent finishes:

• Show the user the generated eval.py or rubric.md (read it for them to review) and ask: "Does this eval capture what you mean?"
• If the user wants changes, spawn the eval agent again with adjustments
• Once the user confirms, the eval artifacts are READ-ONLY for the rest of the session

See references/example-eval.py for what a generated eval.py looks like.

---

Phase 2: Baseline (Iteration 0)

1. Create the outputs/ directory
2. Spawn the autoresearch-test-runner sub-agent with this prompt:

   Generate outputs using the prompt at [path to target skill].
   Test cases are at [path to test_cases.json].
   Save each output to outputs/output_00.txt through outputs/output_09.txt (zero-padded index matching test case order).
   Follow the prompt exactly. One output per test case. No commentary — just the raw output in each file.
   

   IMPORTANT: Output file naming must be output_XX.txt (zero-padded index). Both the eval agent (when writing eval.py) and the test runner must use this convention. If eval.py uses a different naming scheme, the files won't be found.
3. Evaluate:
   • Deterministic: Run python eval.py outputs/, parse the METRIC pass_rate=X.XXXX line from stdout
   • AI Judge: Spawn the autoresearch-judge sub-agent against outputs/ + rubric.md, parse quality_score from judge-scores.json
4. Record as the baseline in autoresearch-log.jsonl
5. Create the initial autoresearch-dashboard.html (use template from references/dashboard-template.html)

---

Phase 3: The Loop (Repeat)

Each iteration follows this exact sequence:

Step 1 — Review

• Read the current state of the modifiable file
• Read the last 5 entries from autoresearch-log.jsonl
• Read autoresearch-ideas.md if it exists
• Read the failure details from the last eval run
• Identify patterns: which criteria score lowest? Which test cases are hardest?

Step 2 — Ideate

• Pick ONE idea to try this iteration
• The idea must be atomic — one change, one hypothesis
• Write the hypothesis in plain English before making the change
• Target the weakest area from the last eval

Step 3 — Modify

• Save a backup: copy the modifiable file to [filename].backup
• Make exactly ONE change to the modifiable file

Step 4 — Execute Prompt (via Test Runner Sub-Agent)

• Before spawning, scan the target skill file for any references to other files (e.g., references/, linked files, imported data). List all file paths the skill depends on.
• Spawn the autoresearch-test-runner sub-agent with:

  Execute the prompt at [path to target skill].
  Test cases are at [path to test_cases.json].
  The working project is at [project path].
  
  Reference files the prompt depends on (read these first):
  - [path to references/file1.md]
  - [path to references/file2.md]
  - [... list ALL referenced files]
  
  Use all available tools (web search, file access, APIs) to produce real outputs.
  Save each output to outputs/output_00.txt through outputs/output_[N].txt.
  Follow the prompt exactly. One output per test case. No commentary.
  

• IMPORTANT: Always pass reference file paths explicitly. The test runner has fresh context — it cannot resolve relative paths or guess where files are. Scan the skill for patterns like references/, see also, file paths, [[wikilinks]], or any mentions of other files, and pass their full absolute paths.
• The sub-agent runs the prompt live with real tools — not mocked
• It has fresh context — it does not know:
  • What the eval checks for
  • What iteration we're on
  • What changes were made
  • What the optimization goal is
• You (the optimizer) MUST NOT generate outputs yourself. Always use the sub-agent.

Step 5 — Evaluate

• Deterministic mode: Run python eval.py outputs/, parse the METRIC pass_rate=X.XXXX line. If eval.py crashes, mark this iteration as "crash".
• AI Judge mode: Spawn the autoresearch-judge sub-agent with:

  Score the outputs in [path to outputs/] against the rubric at [path to rubric.md].
  Save your scores to judge-scores.json in the working directory.
  

  The judge agent has fresh context — it does not know iteration count, prompt changes, or optimization goals. Parse quality_score from judge-scores.json.

Step 6 — Guard Check (if guard is set)

• Run the guard check
• If the guard fails, this iteration is automatically "discard"

Step 7 — Decide

Condition	Action
Metric improved (pass_rate or quality_score)	KEEP — the change stays, update .backup to the new version
Metric same or worse	DISCARD — restore from .backup
Eval crashed	CRASH — restore from .backup, note the error

Step 8 — Log

• Append a line to autoresearch-log.jsonl (see JSONL format below)
• Update autoresearch-worklog.md with a human-readable entry
• Update autoresearch-dashboard.html with current stats

Step 9 — Repeat

• Go back to Step 1
• If stuck (3+ discards in a row on similar ideas), try a radically different approach
• If the user sends a message, pause the loop, respond, then resume

---

The Separation Rules

Rule 1: You are the optimizer. You NEVER generate outputs, write eval code, or score quality.

• You edit the prompt, read metrics, decide keep/discard
• You spawn the test runner sub-agent to produce outputs
• You spawn the eval agent sub-agent to write eval.py or rubric.md (Phase 1 only)
• In AI judge mode, you spawn the judge agent each iteration
• You do NOT read eval.py source code or rubric.md scoring details

Rule 2: The eval agent writes the eval system once, then disappears.

• It receives the criteria and the prompt
• Deterministic: it generates eval.py + test_cases.json
• AI Judge: it generates rubric.md + test_cases.json
• It is NEVER called again during the loop
• The optimizer never sees how the heuristics or scoring examples are implemented

Rule 3: The test runner sub-agent NEVER sees the eval or rubric.

• It gets fresh context each time
• It only receives: the prompt file path + test cases path
• It does not know what criteria exist, what the eval checks for, or what the rubric scores on

Rule 4: The judge is READ-ONLY.

• Deterministic: eval.py is the judge. Run it and read the number. Never modify it.
• AI Judge: rubric.md is the judge's instruction. Never modify it.
• test_cases.json is always READ-ONLY.

Rule 5: The judge agent NEVER sees iteration history (AI judge mode).

• It gets fresh context each time
• It only receives: the output files + rubric.md
• It does not know what iteration we're on, what changes were made, or what the optimization goal is
• It scores purely against the rubric

---

State Files

All state lives in the working directory:

File	Modifiable?	Purpose
Target skill/prompt file	YES	The file being optimized
eval.py	NO — READ-ONLY (deterministic only)	The deterministic judge script
rubric.md	NO — READ-ONLY (AI judge only)	LLM judge scoring rubric
test_cases.json	NO — READ-ONLY	Test inputs
outputs/	Overwritten each iteration (by test runner)	Generated outputs for eval
judge-scores.json	Overwritten each iteration (AI judge only)	LLM judge scores
autoresearch-session.md	No	Session config (includes eval mode)
autoresearch-log.jsonl	Append-only	Machine-readable log
autoresearch-worklog.md	Append-only	Human-readable narrative
autoresearch-dashboard.html	Rewritten each iteration	Live visual dashboard
autoresearch-ideas.md	Yes	Backlog of ideas

JSONL Log Format

Deterministic mode:

{"iteration": 1, "timestamp": "2026-04-01T10:30:00Z", "hypothesis": "Add urgency word to subject line", "metric_name": "pass_rate", "metric_value": 0.70, "baseline": 0.50, "best_so_far": 0.70, "delta": "+0.20", "eval_mode": "deterministic", "assertion_breakdown": {"word_count": 10, "no_buzzwords": 9, "has_question": 7}, "guard_pass": true, "status": "keep"}

AI Judge mode:

{"iteration": 1, "timestamp": "2026-04-01T10:30:00Z", "hypothesis": "Add personal story opener", "metric_name": "quality_score", "metric_value": 0.72, "baseline": 0.50, "best_so_far": 0.72, "delta": "+0.22", "eval_mode": "ai_judge", "rubric_breakdown": {"emotional_resonance": 3.8, "authenticity": 3.2, "narrative_arc": 3.5}, "guard_pass": true, "status": "keep"}

Deterministic logs include assertion_breakdown (per-assertion pass counts). AI judge logs include rubric_breakdown (average scores per criterion). Both help target the weakest areas.

Status values: baseline, keep, discard, crash, no-op

---

Dashboard

Use the template at references/dashboard-template.html. On every iteration, rewrite the full HTML file with updated values:

1. Header stats — goal, iterations count, current best, baseline, improvement %
2. Status counts — keep/discard/crash totals
3. Progress bar — visual fill based on current best vs 1.0
4. Criteria breakdown — which assertions are passing/failing (deterministic) or which rubric criteria score lowest (AI judge)
5. Iteration history table — last 10 iterations with hypothesis, metric, delta, status
6. Status badge colors — green for keep, red for discard, orange for crash

---

Git Integration

If Git Is Available

• Create branch autoresearch/[goal-slug]-[date] before starting
• Commit on every "keep" iteration
• git checkout -- [file] on every "discard" iteration

If Git Is NOT Available

• Before each modification, copy the file to [filename].backup
• On discard, restore from .backup
• On keep, update .backup to the new version

---

Stopping

The loop runs until:

• The user says "stop" or "pause"
• The chosen iteration count is reached
• The metric plateaus (no improvement in 10+ consecutive iterations)

When stopping, write a final summary to autoresearch-worklog.md with:

• Total iterations run
• Best metric achieved vs baseline
• Top 3 most impactful changes
• Which criteria improved most
• Ideas that were never tried (saved to autoresearch-ideas.md)

---

Critical Rules

1. ONE change per iteration. Never bundle multiple ideas.
2. NEVER generate outputs yourself. Always use the autoresearch-test-runner sub-agent.
3. NEVER write eval artifacts yourself. Always use the autoresearch-eval-agent sub-agent.
4. NEVER read eval.py source code (deterministic). You only run it and read its stdout.
5. NEVER score quality yourself (AI judge). Always use the autoresearch-judge sub-agent.
6. Eval artifacts are READ-ONLY. Never modify eval.py, rubric.md, or test_cases.json during the loop.
7. Always revert on discard. Never accumulate failed changes.
8. Log everything. Even crashes and no-ops get logged.
9. Don't ask permission mid-loop. Only pause if the user messages you.
10. Respect the modifiable files list. Only the target skill/prompt can be edited.
11. Stuck detection. After 3 consecutive discards on similar ideas, pivot radically.
12. No cherry-picking. Run the full eval every time, not a subset of test cases.
13. Read the failure details. Use assertion breakdown (deterministic) or rubric breakdown (AI judge) to target your next change.