adversarial-verification

name

adversarial-verification

description

Dispatch an independent, skeptical agent to re-derive whether a fix/claim/artifact actually worked, instead of trusting the builder's self-assessment. Use when a subagent, Workflow stage, or parallel candidate reports "fixed it" and the cost of a false green is high — code fixes, pipeline output, generated assets, live-site claims. Do NOT use for verifying your OWN uncommitted work (that is verification-loop — just run the gates), and do NOT use for vetting a plan before anyone builds (that is adversarial-planning).

Independent Re-Check

The problem

An agent that builds a fix and then grades its own fix will round up. "Looks better to me" is not evidence — it is the builder's incentive to report success talking. This is the same failure verification-before-completion names for a single agent's own work; adversarial verification is the version for orchestrated/multi-agent work, where the fix is a separate artifact another agent can independently inspect.

The repo has the receipts. Adversarial review of code whose own TDD suite was green found 3 real defects in the content-evaluator (bug-148), 16 in the deterministic fraud scorer (bug-150), 4 across the Tier-2 scorers (bug-151). In every case the builder's tests passed. They tested that each rule fires; nobody re-derived whether the contract held.

When to use

Use it when:

• a subagent or Workflow stage returns a "done / fixed / verified" report and you are about to relay that to the founder or ship on it
• multiple independent candidates were built in parallel and need honest, comparable grading before one is selected
• the claim carries severity — "production bug fixed", "renders are canon-clean", "the deploy is live" — and the only evidence is the builder's prose
• a gate the builder cites is one that has historically failed open (bug-230)

Do not use it:

• on your own uncommitted edit. Re-reading your own diff is not independence; run the gates (verification-loop) instead.
• before anything is built. A claim with no artifact cannot be re-derived — adversarial-planning is the pre-build loop.
• on a trivially checkable claim you can settle yourself in one command. Dispatching an agent to run git log -1 is ceremony, not verification.

Inputs

Required	Why	If absent
An artifact, not a claim — a diff, an exported file, a rendered image, a raw command transcript, a query result	The verifier must inspect something the builder cannot re-narrate	Stop. Send the builder back for the artifact. Verifying prose produces a verdict about prose.
The original problem statement, verbatim	The verifier grades against the requirement, not against the builder's restatement of it	Stop. Without it the verifier grades "did something change", which always passes.
A re-runnable check the verifier can execute itself (test command, curl, vision read, query)	Independence means re-derivation, not re-reading	Stop and say so. "Cannot be independently checked" is a legitimate verdict; a confident clean is not.
The builder's notes — labeled as unverified claims to check	Useful context, toxic as premise	If they arrive as fact, relabel before handing over.
For a [live] verdict: production access	Severity requires a production probe	Downgrade to [repo]/[repro] and say production is unverified.

Procedure

1. Collect the artifact and the original problem statement. Strip the builder's self-assessment out of the premise; re-attach it as claims under test.
2. Dispatch a fresh agent with no memory of building it. For higher-stakes work make it a different model — a fresh instance of the same model can carry the same blind spot. Pair via agent()'s model option (e.g. builder sonnet, verifier opus).
3. Instruct it to default to skeptical: assume still-broken unless the evidence proves otherwise. A partial improvement is partially-improved, never clean. Ties go to "not fixed".
4. Require re-derivation, not summary. It re-runs the test, re-renders the asset, re-executes the query, re-reads the pixels. If it cannot reproduce the check, it cannot confirm the claim — that is a cannot-verify outcome, not a pass.
5. Force a structured verdict: an enum (clean / partially-improved / no-improvement / regressed) plus a boolean recommend_ship. Free-form prose gives synthesis nothing to compare.
6. Check the verdict's evidence scope covers its claim scope. A clean + "production fixed" backed only by [repo] reading is rejected and sent back for a [live] probe.
7. Synthesis compares verdicts, never builder claims. Only verifier-confirmed artifacts move forward to deploy/merge/ship.

Two-model debate loop (hard-capped at 3 rounds)

Rounds 1-2 are argument; round 3 is not — it is a forcing function that ends debate by making both sides touch ground truth.

1. Round 1 — proposal + challenge. Builder (model A) proposes. Verifier (model B) challenges skeptically and must state specifically what evidence would change its verdict.
2. Round 2 — rebuttal + recheck. Builder addresses that specific challenge, not a full re-argument. Verifier rechecks. Satisfied → stop; do not burn round 3 on agreement.
3. Round 3 — mandatory execution, no more argument. Both sides stop arguing and execute the verifiable task themselves right now — run the test, re-render, re-run the query — and the verdict comes from that raw output. This is the exit condition whether or not the models agree: empirical execution outranks continued debate. Never let round 3 become "argue again, harder."

const MODELS = { builder: 'sonnet', verifier: 'opus' } // deliberately different

let verdict = null
let transcript = []
for (let round = 1; round <= 3; round++) {
  const finalRound = round === 3
  if (verdict?.overall_verdict === 'clean') break // converged early, skip round 3

  const builderTurn = await agent(
    finalRound
      ? `Final round. No more argument. Execute the verifiable check yourself right
         now (run the command/test/render) and report the raw output.
         Debate so far: ${JSON.stringify(transcript)}`
      : `Round ${round}. Defend or refine against this challenge (if any):
         ${JSON.stringify(verdict) || 'none yet — initial proposal'}.`,
    { label: `builder-r${round}`, model: MODELS.builder, schema: BUILD_SCHEMA }
  )

  const verifierTurn = await agent(
    finalRound
      ? `Final round. Do not argue further — independently execute the same
         verifiable check yourself and report raw output. Compare against the
         builder's round-3 execution: ${.stringify(builderTurn)}`
      : ,
    { : , : ., :  }
  )

  transcript.({ round, builderTurn, verifierTurn })
  verdict = verifierTurn
}

Fan the whole loop out per candidate with pipeline() when independent builders compete — each candidate gets its own debate, own pairing, own verdict; only synthesis needs a barrier, since it must compare all candidates at once.

Standalone (non-Workflow) use. The same discipline applies to a single dispatch: after Agent() returns "fixed it", dispatch a second, independent Agent() with the artifact and skeptical instructions before relaying "done".

Verification

The verifier's own output is an artifact and gets checked like any other.

1 — The verdict is structured and in-vocabulary. Free prose fails here.

python3 -c "import json;v=json.load(open('/tmp/verdict.json'));assert v['overall_verdict'] in {'clean','partially-improved','no-improvement','regressed'},v;assert isinstance(v['recommend_ship'],bool),v;assert v.get('evidence'),'no re-derived evidence';print('verdict schema OK')"

PASS: exits 0 and prints verdict schema OK. A verdict with no evidence field is prose wearing a schema. [test]

2 — The verifier re-derived, it did not re-read. Re-run the check yourself and compare to the transcript it reported.

rtk proxy pytest tests/mcp/test_http_mount.py -q

PASS: your run's pass/fail counts equal the counts inside verdict.evidence. A mismatch means the verifier reported a run it did not do — reject the verdict. [repro]

3 — Scope covers severity. A verdict asserting production behaviour needs a production probe of its own.

grep -o '\[live\]\|\[repo\]\|\[repro\]\|\[inferred\]' /tmp/verdict.json | sort -u

PASS: if the verdict text says "production" / "live" / "critical", [live] appears in that output. If only [repo] / [inferred] appear, downgrade the claim before relaying it. [test]

4 — Prove the verifier can return "no" before trusting a clean. Hand it a known-bad artifact once per pairing.

git archive HEAD mcp_tools/http_mount.py | tar -x -C /tmp/pristine-mcp

Then dispatch the same verifier against that pre-fix file with the same problem statement. PASS-OF-THE-PROOF: the verdict is not clean — the pre-fix file compares tokens with !=. A verifier that returns clean against the pristine tree is decoration; replace it. [test]

Never git stash to produce that pristine tree; the stash stack is shared across worktrees and you can pop another session's work.

A verifier run that errored, timed out, or hit a session limit produced an artifact, not a result. Re-run it by hand. Its empty findings list is the fail-open pattern (bug-230), and a verification stage is precisely where it hides.

Worked example

Claim under test (builder): "Hardened the MCP bearer-token check — bug-211 closed."

The verifier was given mcp_tools/http_mount.py, the bug-211 statement from .wolf/buglog.json ("plain string != leaks length/prefix; missing MCP_SERVICE_TOKEN → warn-and-serve-open"), and no builder commentary. It re-derived both halves.

$ grep -n "compare_digest\|MCP_SERVICE_TOKEN" mcp_tools/http_mount.py
7:Auth: a shared Bearer service token (``MCP_SERVICE_TOKEN``). When the token is set
32:_TOKEN_ENV = "MCP_SERVICE_TOKEN"
53:    Enforced only when ``MCP_SERVICE_TOKEN`` is configured. An unset token in a
88:        if not hmac.compare_digest(provided, f"Bearer {token}")

[repo] — the timing-safe comparison is present at line 88.

$ rtk proxy pytest tests/mcp/test_http_mount.py -q
......                                                                   [100%]

[repro] — 6 passed, 0 failed, re-run by the verifier itself, not quoted from the builder.

Verdict issued: partially-improved, recommend_ship: false. Reasoning: half the bug — the timing side-channel — is confirmed closed at [repo] + [repro] scope. The other half, "fail-closed when the token is unset in production", is asserted by http_mount.py:53 and by a unit test, both of which describe the code. No probe of a running production instance was performed, so the fail-open half is [inferred] and carries no severity. clean would have been the round-up.

That is the whole discipline in one verdict: the builder was not wrong, and the verdict still is not clean, because the evidence scope did not reach the claim scope.

Failure modes

Symptom	What is really happening	Bug
Verifier returns clean with prose like "the fix looks correct"	It re-read the builder's diff instead of re-deriving. No command in its evidence = no verification.	—
Green tests cited as proof the contract holds	Tests prove each rule fires; they do not exercise calibration, boundaries, or the never-raises guard. Adversarial review found 16 defects behind a green suite.	bug-150
Verifier and builder are the same model, verdict is always clean	Shared blind spot. Use different weights, not a fresh instance.	—
Debate reaches round 3 and keeps arguing	Round 3 is execution-only by construction. Argument in round 3 means the loop has no exit condition.	—
Verdict says "production bug fixed", evidence is all [repo]	Scope jump. Severity requires [live]. State scope before severity.	bug-287
Verifier stage errored; orchestrator recorded zero findings and moved on	Fail-open. A gate that dies is not a gate that passed.	bug-230 (×6)
Verifier confirms an asset "looks like a plausible Black Rose piece"	Checking against memory of the collection, not side-by-side against the canonical reference. Lenient QC ships hallucinations.	bug-276
Verdict is free-form prose; synthesis picks a winner anyway	No enum, nothing comparable — synthesis is guessing with extra steps.	—