agent-validation-loop

# Agent validation loop A playbook for improving a tool by letting fresh subagents try to use it on a real task, recording where they get stuck, fixing the friction, and running another fresh subagent to confirm. The agents' words become the spec; their failures become the next commits. This skill is for cases where the question "does the tool work?" can't be answered by reading the code or running existing tests — it needs an agent who has never seen the tool before to try to use it. ## When to use - A tool whose UX depends on natural-language output an agent will read (suggestions, error messages, structured signals). - A CLI whose ergonomics are easy to mis-design (flag naming, defaults, output verbosity). - A library where you want to know whether the **abstractions** make sense to a fresh reader, not whether the code compiles. - Any system where the gap between "the code works" and "the user succeeds" is the actual product. ## When NOT to use - For pure algorithmic correctness — a unit test is cheaper. - For one-off scripts with no UX surface. - When you already know what to fix; the subagent overhead isn't worth it for trivial changes. ## The loop, ten steps ### 1. Pick a scenario with a measurable goal The scenario must be **closed-loop**: the agent has a task with a deterministic success criterion (a goal screenshot, expected output, target API response, etc.). Without a measurable outcome the validation turns into vibes. ``` fixtures/<scenario-name>/ brief.md # what the agent should build goal/ # the expected output (artifacts only) attempts/ # one subdir per agent run agent-a/ # filled by the agent agent-b/ ``` The brief and goal artifacts are visible to the agent; the **source** of the goal (the code that produced it) is not. ### 2. Decide what the agent can and cannot read The single most important rule. Each agent must: - Read: the brief, the goal artifacts, the tool's CLI / docs. - NOT read: prior agents' attempts (would bias them), the goal's source code (would short-circuit the test). Put both lists in the prompt explicitly. Mention each forbidden path by name; agents will respect path-level instructions. ### 3. Spawn the first subagent with a clear charter ``` Subagent prompt structure: - Goal in one sentence ("converge on the goal screenshot") - Files you may read (brief / goal artifacts / tool docs) - Files you MUST NOT read (with concrete paths) - Working directory (their own attempts/<name>/ subdir) - Workflow (N rounds budget; tool invocation per round; what to log) - Constraints (tokens / no-magic / etc.) - Deliverables (< 300 words: final metrics + rounds + what worked + what didn't + concrete negative feedback) ``` Run via the `Agent` tool with `subagent_type: "general-purpose"` and `run_in_background: true` so you can do other work while they run. ### 4. Treat the agent's friction as the spec When the agent returns, their negative feedback is more valuable than the convergence number. Quotes like: > "The candidate notation `A → B` looked like 'change from A to B' > but I had B and wanted A — I went the wrong way for two rounds." …are exactly the bug. The fix is implemented **from their words**, not from your imagination of what they meant. File a GitHub issue (or write a draft markdown) per friction point with the agent's quote verbatim. ### 5. Commit per-friction-point, not per-batch One issue, one commit. Commit message includes: - What the agent said (a one-line quote) - What was changed - "closes #N" This makes the history readable as "what the user complained about and what we did" rather than "code churn." ### 6. Write a per-run validation report After each agent run: ``` docs/reports/YYYY-MM-DD-<scenario-name>-vN.md ``` Sections: - **Question** — what hypothesis is this run testing? - **Result** — final metrics + comparison table to prior runs. - **What worked** — direct quotes from the agent confirming what helped. - **What didn't / new gaps** — direct quotes describing friction. - **What's fixed in this commit vs deferred** — explicit boundary. The cumulative folder becomes the curriculum the next maintainer reads to understand "why is the tool shaped this way?" ### 7. Spawn the next agent on the SAME scenario after fixes Re-running on the same scenario lets you compare apples-to-apples. Each agent gets the same brief and budget; what changes is the tool. Convergence-by-version becomes the public metric. | agent | budget | result | what changed in vlmkit since prior | |---|---|---|---| | a | 5 rounds | 10.3% mobile | initial | | d | 5 rounds | 0.2% mobile | post-#22..#29 | | f | 3 rounds | 3.45% mobile | post-v6 fixes | A regression (agent-g got worse than agent-f) is **information**, not failure — record it honestly. ### 8. Tighten the budget as the tool matures Start with a generous round budget (e.g. 5). As the tool's signal quality improves, tighten to 3 rounds. This stress-tests whether the loop genuinely got tighter or whether you're just giving agents more chances. The first time agent-3-rounds beats agent-5-rounds-old, you know the tool's signal quality has materially improved. ### 9. Stop when diminishing returns are visible Signs the loop has converged: - Each new agent run surfaces fewer new gaps than the last. - The remaining variance is **agent-side initial-state**, not tool signal quality. ("Agent picked the wrong initial structure" is not something the tool can fix.) - Per-fix commit size is shrinking — small annotations rather than new affordances. - You're tempted to write tests for hypothetical cases instead of observed ones. Stop, write a release-prep commit summarizing what landed (CHANGELOG + README), and merge. ### 10. Capture the learning After merge: - A CHANGELOG entry per branch that lists every closed issue with a one-line cause. - A docs/reports/ index card listing every validation run. - A skill file (like this one) if the loop is generalizable to other projects. ## File and branch conventions - **Branch**: `claude/<scenario-slug>-<YYYY-MM-DD>`. All loop work on one branch until release-prep. - **Reports**: `docs/reports/YYYY-MM-DD-<scenario>-v<N>.md`. Versioned monotonically per scenario. - **Issue drafts**: `docs/issues-drafts/<NN>-<topic>.md` when GitHub is unavailable; post when MCP / `gh` becomes available. - **Scenario fixture**: `fixtures/<scenario>/{brief.md, goal/, attempts/<agent>/}`. - **Agent letters**: a, b, c, … sequential. Letters are cheap; numbering by date is hard to read at-a-glance. ## Subagent prompt template ``` You are evaluating whether <TOOL> gives an agent enough signal to do <TASK>. The friction you find is the deliverable. # Your task <one paragraph> # Files you may read - <brief> - <goal artifacts> - <tool docs> # Files you MUST NOT read - <goal source> - <prior agents' attempts: explicit paths> # Your working directory <attempts/<your-letter>/> # Workflow Budget: N rounds. Each round: 1. Edit <output files> 2. Run <tool invocation> 3. Read output for <signals> in priority order: ... 4. Log to <attempts/<your-letter>/log.md> 5. Stop when <success criterion> or budget exhausted # Constraints <concrete rules: tokens-only, no peeking, media queries for divergent deltas, etc.> # Deliverables (< 300 words) 1. Final metrics 2. Rounds used, one-line per round 3. What helped — concrete quote-style example 4. What didn't help or was missing — be specific 5. Versus <prior agent>: better / same / worse + why ``` ## Validation report template ```markdown # <Scenario> v<N>: <one-line hypothesis> (<date>) ## Question <what is this run testing?> ## Result | viewport / metric | prior | this run | delta | |---|---|---|---| | ... | <one-sentence honest read> ## What worked — <agent>'s own words > "<verbatim quote>" <short interpretation> ## What didn't / new gaps ### <Gap1> — <one-line summary> > "<verbatim quote>" <root cause analysis> **Fixed in this commit:** <description> ### <Gap2> — <deferred> > "<verbatim quote>" <why it's deferred, what would be needed> ## Files - <fixture paths> - <source files changed> ``` ## Anti-patterns ### Don't write the fix before reading the agent's words Tempting to predict what the agent will say. Resist. When you read their actual quote, the fix is usually different (and smaller) than what you would have written. ### Don't suppress negative results If agent-g converges WORSE than agent-f, that's data. Hiding it as "agent picked wrong initial structure" without noting the friction the new affordance caused (or didn't catch) loses the signal that makes the loop valuable. ### Don't fix multiple agents' findings in one commit Each fix should map to a specific agent's complaint. Bundling makes the commit history harder to read and obscures what was learned where. ### Don't skip the prompt's "MUST NOT" list Agents that read prior attempts converge faster but you learn nothing — they're memorizing, not using the tool. ### Don't grade on the diff metric alone A regression that surfaces a real gap (G7c: cross-suggestion overshoot aggregation) is more valuable than a clean run that surfaces nothing. ### Don't run the same agent prompt verbatim across versions Update the prompt to mention the new affordances. Otherwise the agent won't know they exist. ("As of this version, `[STRUCTURAL]` now names the specific parent property...") A fresh agent on an unchanged prompt won't read the source to discover new tags. ### Don't ship without per-run reports The reports ARE the spec for the next round. Skipping them means the fix's rationale lives only in commit messages and you lose the narrative arc. ## Honest stopping criteria After 7-9 agent runs you'll see: - Average improvement per fix < 1pp. - New surfaced gaps are increasingly specific edge cases. - Agent-side initial state explains > 50% of variance. - The tool's signal hierarchy hasn't gained a new tag in 2 runs. This is the moment to: 1. Tighten the CHANGELOG. 2. Stop the loop. 3. Decide whether the next investment is on this tool (e.g. faster iteration) or on adjacent surfaces. ## Reference run This skill was distilled from the `claude/design-md-scenario-2026-05-15` branch on `mizchi/vrt`: - 9 agent runs (a → i) on a single DESIGN.md → HTML/CSS reproduction scenario. - 18 GitHub issues filed and closed (#22 – #36) + 3 drafts shipped as CLIs (`vlmkit manifest` / `vlmkit watch` / `vlmkit diff-pr`). - Closed-loop floor moved from 10.3% → 0.2% (5-round budget) and to 3.45% (3-round budget). - 38 commits, 183 tests across 32 suites. - Validation reports under `docs/reports/2026-05-15-design-md-scenario-v{1..9}.md`. The story arc: each report quoted the agent's friction in their own words; the next commit implemented the fix from that language; the next agent confirmed (or revealed a new gap). Stopping signal was agent-g regressing on a class of bug (typographic cascade) that the existing signal hierarchy couldn't classify — the new REFLOW tag closed it, and post-REFLOW runs (h, i) converged within the budget. ## Second reference run: an IR judged by its writers (anim-ir v1–v24) `vlmkit-anim` (the `explanatory-animation` skill; moved to [mizchi/explainer](https://github.com/mizchi/explainer) and deleted there on 2026-10-02, so the paths below are in explainer's git history only) was built with this loop from its first day: 24 rounds over six days, 86 writer attempts (`a`…`qb`), reports `docs/reports/2026-09-0{4..9}-anim-ir-v*.md` (still in this repository), fixture `packages/vlmkit-anim/fixtures/anim-scenario/` in explainer. The tool under test is a **format** an agent writes, not a signal it reads, and that changed five things about the loop. They are worth copying whenever the deliverable is "an agent gets it right from the docs alone". - **The docs are the only input.** Each writer gets one brief and one guide (`packages/vlmkit-anim/docs/anim-ir.md` in explainer) and is forbidden the package source, other attempts, the reports and the CHANGELOG — by path, in the prompt. A writer who reads the compiler is measuring their own reading, not the guide. Every JSON block in the guide is compiled by a test, so the guide cannot drift from the tool it describes. - **The success criterion is a fact sheet, not a green check.** Round 13 had five green module maps and two of them were wrong (a true dependency deleted, the wrong edge lit). From round 14 every brief ships `facts/<brief>.expect.json` and success is `vlmkit-anim check` against that sheet exiting 0; from round 18 the tool writes the sheet itself from the code (`vlmkit-anim facts`). Whatever your tool draws or emits: find the ground truth it can be held to and make the check read it, or a clean run proves nothing. - **Two measurements, then compare them.** Geometry (`layout`) said the frames were clean; vision readers on the same frames (round 12, and round 21 on stills) found lines through labels, containers crossing, a corner label read as a caption. Each disagreement was a defect one side could not see, and each became a rule in the cheaper measurement. A second, unlike reader is how you learn what your first one is blind to. - **Ask for "what told you", per round.** From round 24 the log asks the writer to quote the line — a `check` line, a `why` line, the picture, a guess — that made them change what they changed. Two writers quoted the same `why` line before their first edit; both then asked for it in the warning itself, which is where it now is. The quote locates the fix in the tool's output, not in the writer's head. - **Run two model sizes on the same brief.** A smaller model (haiku) beside a larger one (sonnet) on the same brief, from round 14 on: the smaller one hits the gaps in the guide the larger one reads past, and a fix that only the larger model can use is not a fix. When the two converge on the same lever from the same line, the tool said it. Two habits from this run that the ten steps above assume but do not say: a **corpus check** on every compiler change (`layout` over every fixture and attempt, diffed against the previous round's baseline — a fix for one writer's scene has regressed another's more than once), and a report that **names the next round** in its last paragraph, so the list the user gave and the list the evidence gives can be compared before either is started.