weak-agent-test

name	weak-agent-test
description	Run the weak-agent adversarial test harness against pdf-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe) to perform real tasks over real government forms and papers — filling a withholding form, reviewing a legal filing with comments, reading values out of a dense table, answering from a scanned page, extracting figures — rasterizes every result, has opus judge each against a ground-truth rubric, and synthesizes a prioritized ergonomics report. Also runs the same scenarios against competitor arms (Anthropic's official pdf skill; a docling/camelot/markitdown extraction stack) for a head-to-head. Use when the user says 'adversarial review', 'test pdf-cli with weak agents', 'run the haiku harness', 'weak agent test', or 'bake-off'.
allowed-tools	Bash, Read, Write, Glob, Workflow
metadata	{"internal":true}

Adversarial review — weak-agent harness for pdf-cli

This harness answers one question: can weak agents actually use pdf-cli to get real work done, and what should we fix first? A second question rides along: do they do better with it than without it? — the same scenarios run against competitor arms, so the answer is a comparison rather than an assertion.

It runs the weak-agent-test workflow (.claude/workflows/weak-agent-test.js), which fans out one weak exercise agent per scenario (Haiku by default, swappable to Sonnet via args.model), rasterizes the pages that carry the evidence, grades each result against a ground-truth rubric with an opus judge, and has opus synthesize a prioritized report. Exercise agents do NOT self-report tool counts — agents under-count their own calls roughly twofold, so every tool-economy and token number is measured afterwards from the run transcripts.

The corpus is real

Every scenario is a real document — a government form or a published paper, not something invented to be convenient. Fixtures come from the 510-form corpus (docs/forms) and the committed paper fixtures. That matters: a form designed by a state revenue department has furniture no synthetic fixture would think to include, and those are exactly the pages a weak agent fails on.

The corpus is bundled with this skill under scenarios/, one folder per scenario, named after its key. Each folder is self-describing:

• task.md — the AGENT-FACING request, written as a person delegating work: the goal, the data, the intent, and no tool vocabulary (no pdf commands, no locators, no PDF jargon). Discovering which features deliver the outcome is part of what is being measured.
• criteria.md — the JUDGE-ONLY rubric. The stage step withholds it from the agent's workspace and the judge reads it from the pristine source, so the agent never sees the answer key. stage-scenario.ts verifies the deletion and fails the run if it leaked.
• the fixture PDF the task works on.
• assets/ — any additional inputs. Often empty.

The workflow's SCENARIOS manifest holds only ROUTING metadata (key, bucket label, edit/read kind, the doc filename, which pages are worth rasterizing, and the command that proves the change survived a re-read). The request, the rubric and the fixture live in the folder, so the skill travels with its test corpus. To change what a scenario tests, edit the files in its folder.

Run outputs — filled PDFs, renders, reviews, the report — go to ./tmp/pdf-weak-agent-test/<ts>/, never into the repo.

What a run produces

<RUN_DIR>/
  REPORT.md                  ← the synthesized report (+ the measured metrics section)
  exercise-metrics.md/.json  ← measured per-agent tokens, wall-clock, tool split
  <key>/
    task.md, assets/, <fixture>.pdf   ← the staged workspace (no criteria.md)
    answers.md                        ← read scenarios: the agent's product
    renders/output/  *.png, read.md   ← edit scenarios: what the agent produced
    renders/source/  *.png, read.md   ← the pristine document, always
    review.md, verdict.json           ← the judge's grading

Running it

1. Preflight — a fresh binary, gated (not optional)

Always run bun run build:binary, even if dist/pdf exists — never reuse a prior build. A benchmark against a stale binary measures the wrong tree, and you will not notice from the report. Abort the run if any check fails.

REPO="$(git rev-parse --show-toplevel)"
cd "$REPO"
SCENARIOS_DIR="$REPO/.claude/skills/weak-agent-test/scenarios"

# (1) Build the CURRENT working tree into a fresh standalone binary.
bun run build:binary || { echo "BUILD FAILED — abort"; exit 1; }
BINARY="$REPO/dist/pdf"

# (2) Hard gate: the fresh binary must match package.json's version AND carry the
# verbs the scenarios need. `--version` prints "pdf X.Y.Z"; take the 2nd field with
# `cut` — NOT an awk positional, which slash-command argument substitution mangles.
EXPECTED="$(bun -e 'console.log(require("./package.json").version)')"
GOT="$("$BINARY" --version | cut -d' ' -f2)"
echo "built pdf $GOT (package.json: $EXPECTED)"
[ "$GOT" = "$EXPECTED" ] || { echo "VERSION MISMATCH ($GOT != $EXPECTED) — stale build, abort"; exit 1; }
for verb in read render form comments; do
  "$BINARY" "$verb" --help >/dev/null 2>&1 || { echo "$verb MISSING — build stale/broken, abort";  1; }

 

2. Make an isolated run workspace

One empty timestamped dir per run. Do NOT copy the scenarios in — the Stage phase seeds only the active ones, so originals stay untouched and a single-scenario run doesn't drag the whole corpus along.

TS="$(date +%Y.%m.%d-%H%M%S)"
RUN_DIR="$REPO/tmp/pdf-weak-agent-test/$TS"
mkdir -p "$RUN_DIR"
echo "RUN_DIR=$RUN_DIR"

3. Launch

Workflow({
  scriptPath: "<REPO>/.claude/workflows/weak-agent-test.js",
  args: {
    runDir: "<RUN_DIR>",
    binary: "<BINARY>",
    scenariosDir: "<SCENARIOS_DIR>",
    model: "haiku",            // exercise model: "haiku" (default) or "sonnet"
    only: <optional filter: "fill-withholding-choices" or ["a","b"]>
  }
})

Exercise agent type. Exercise agents run as the repo's weak-exercise agent type (.claude/agents/weak-exercise.md): minimal tools, no Skill tool, so the session's skills catalog stays OUT of their context — it is a per-turn token tax and it leaks the tool's name into the "capable-but-fresh agent" premise.

It does not resolve in the session that CREATES or edits it. The registry loads at session start, and the failure is SILENT: the exercise agent never runs, every scenario degrades to lostExerciseAccount, and the judge dutifully grades an empty folder. The symptom from outside is no answers.md and no filled document for EVERY scenario at once. Pass exerciseAgentType: "general-purpose" for that session — and note the base context is then several thousand tokens per turn heavier, so those token numbers are not comparable to a weak-exercise run's.

Never resume a run whose exercise phase failed. Re-run the WHOLE thing in a FRESH run dir. The resume cache is PREFIX-based, not keyed: everything issued after the first missing result re-runs live, so re-run exercise agents would edit ALREADY-EDITED documents — double fills, double comments, unusable verdicts. A dead exercise for a manifest-early scenario re-runs every later exercise against dirty documents. Don't gamble on manifest position. Resume is only safe for failures at or after the render phase, where nothing mutates a document.

4. The competitor arms (the bake-off)

Stage the competitor first — it exits nonzero if anything is missing, because a handicapped competitor would void the comparison and the resulting number would still get quoted:

bun "$SCENARIOS_DIR/../scripts/stage-competitor.ts" skill       "$REPO/tmp/pdf-competitor/skill"
bun "$SCENARIOS_DIR/../scripts/stage-competitor.ts" extractors  "$REPO/tmp/pdf-competitor/extractors"

Then launch with the arm and its dir:

args: { …, arm: "anthropic-pdf-skill", competitorDir: "<REPO>/tmp/pdf-competitor/skill" }
args: { …, arm: "python-extractors",   competitorDir: "<REPO>/tmp/pdf-competitor/extractors" }

The python-extractors arm automatically runs only the READ scenarios — it has no fill or comment story, and racing it on those would be a strawman that flatters us.

5. Metrics (post-run)

The token pass needs the workflow's transcript dir, which is only known after launch, so it is a post-run command:

bun "$SCENARIOS_DIR/../scripts/exercise-metrics.ts" <TRANSCRIPT_DIR> "$RUN_DIR" "$BINARY" haiku --append-report
bun "$SCENARIOS_DIR/../scripts/render-integrity.ts" "$RUN_DIR" "$SCENARIOS_DIR" "$BINARY" --append-report

render-integrity.ts re-opens every document the run produced and compares it byte-for-byte with the pristine fixture. The judge grades rasterized pages and so cannot see that the bytes beneath them were rewritten wholesale — which voids signatures and drops objects nobody modelled — so that is measured here instead, identically for every arm. It exits nonzero if any produced document fails to re-open or lost a page.

6. Writing a run up

One run is the base case. Point the roll-up at a plain run dir and it writes that run's RESULTS.md — grades, cost, render integrity, and the gates a single arm can actually answer:

bun "$SCENARIOS_DIR/../scripts/bakeoff-rollup.ts" "$RUN_DIR" <TRANSCRIPT_DIR> "$BINARY" "$SCENARIOS_DIR" haiku

The document follows docx-cli's format exactly — # Exercise metrics with its Totals / Per scenario / Outliers sections, the compact 2.4M / 525k number scale, the S S p per-scenario grid once a run is replicated, and a Totals table whose last column reads 3.3× fewer rather than leaving the reader to divide. Two projects producing the same tables means a pdf-cli run and a docx-cli run can be read side by side without relearning the layout. The one docx-cli section not ported is ### vs previous run: it finds the prior run as the lexically-previous SIBLING dir, and a bake-off's siblings are arms of the same run, not earlier runs — the anchor would have to change meaning to work here.

No manifest, no <arm>-rN wrapper, no arm columns, and no gates about rivals that were never run — the comparative gates are simply absent, with a line saying how to get them. Add arms and the same command writes COMPARISON.md instead, with a column per arm and the comparative gates scored. The one-arm path is not a degenerate bake-off; the comparative parts just have nothing to say until there is something to compare.

7. The full bake-off (all arms, replicated)

For a freeze — the numbers a README is allowed to quote — drive every arm from one parent workflow instead of launching each by hand. arms defaults to [{ arm: "pdf-cli" }], so omitting it runs our arm alone. Stage both competitors first, then:

Workflow({
  scriptPath: "<REPO>/.claude/workflows/pdf-bakeoff.js",
  args: {
    baseDir: "<REPO>/tmp/pdf-weak-agent-test/<TS>-bakeoff",
    binary: "<BINARY>",
    scenariosDir: "<SCENARIOS_DIR>",
    model: "haiku",
    replicates: 1,          // per invocation; use firstReplicate for later rounds
    arms: [
      { arm: "pdf-cli" },
      { arm: "anthropic-pdf-skill", competitorDir: "<REPO>/tmp/pdf-competitor/skill" },
      { arm: "python-extractors",   competitorDir: "<REPO>/tmp/pdf-competitor/extractors" }
    ]
  }
})

Each run lands in <baseDir>/<arm>-r<N>/. Runs are sequential — a deliberate divergence from docx-cli, which runs replicates three at a time. Output tokens and tool calls are contention-free, but wall-clock is not, and it is a headline number here: arms racing each other on one machine measure the machine, and the seconds stop being comparable to every run recorded before the freeze.

Replicate a round at a time (replicates: 1, then firstReplicate: 2) rather than asking for two up front. A bake-off is hours long; a round is a checkpoint where the results can be read before the next one is paid for, and firstReplicate adds r2 to the same base dir without touching r1.

Then one post-run command measures every run and rolls them up:

bun "$SCENARIOS_DIR/../scripts/bakeoff-rollup.ts" "$BASE_DIR" <TRANSCRIPT_DIR> "$BINARY" "$SCENARIOS_DIR" haiku

It runs exercise-metrics.ts and render-integrity.ts per run, then compare-runs.ts over the lot, writing $BASE_DIR/COMPARISON.md: grades per scenario with every replicate shown, score by arm split read vs edit, the cost/path table as a range, render integrity, and the release gates scored PASS/FAIL/UNMEASURED. A gate nobody could measure never reads as a pass.

All runs share ONE transcript dir — the parent workflow's — so exercise-metrics.ts attributes each transcript by its run dir, not by scenario key alone. Six runs of the same six scenarios otherwise cross-attribute silently.

Reading the result

The scoreboard is in REPORT.md. Per-scenario evidence is in each <key>/review.md with the judge's reasoning, and verdict.json is the machine-readable grade.

Two things to hold onto when reading a run:

• The agent's self-reported completed is not a grade. It describes the run, not the output. The judge grades the rendered pages against the rubric; a confident agent that produced a wrong form is a fail with a cheerful summary.
• UX and correctness are separate verdicts. A correct result reached by a painful path is a task SUCCESS with a UX demerit. Collapsing the two hides the question you actually want answered.

Replication

A single run is an anecdote — models vary run to run. The standard for anything that gets quoted publicly is 3 runs × 2 tiers (haiku, sonnet) × 2 arms, rolled up into a COMPARISON.md with the spread shown, not just the mean. compare-runs.ts reports every replicate's grade and a range for every measured number precisely so a thin run cannot be read as a firm one — start at 2 × haiku and escalate only if the spread is wide enough to need it.