verify-claims-against-execution

name	verify-claims-against-execution
description	Use when about to mark work complete/fixed/passing, accept a subagent's self-reported success or score, or trust any grader/oracle/judge/auto-rater — especially before running a scored batch.

Verify Claims Against Execution

Overview

An agent's claim of success is a hypothesis until external state confirms it — an exit code, a file:line citation that resolves, a delivery receipt, a liveness probe. Most reward-hack episodes ship a confident rationale framing hollow output as legitimate, so reading the claim tells you nothing. Check the world, not the words.

When to use

• Marking any task complete / fixed / passing / "succeeded".
• Accepting a subagent's self-reported result or score.
• Trusting a grader / oracle / judge / auto-rater — before you run it on a real batch.
• A result "looks right" but you haven't watched it run.

The technique

1. Bidirectional oracle gate (do this FIRST). Before trusting any grader, feed it a known-good input (must PASS) and a known-empty/wrong stub (must FAIL). If the empty stub passes, the oracle is broken — halt and fix the oracle, not the agent. A grader you've only ever run against correct answers is unvalidated.
2. Resolve, don't read. Demand evidence that resolves to external state: an exit code from the real test, a file:line that opens to a real path, a delivery-receipt row. A claim with no resolvable citation is unverified.
3. Isolate + recompute. Run the grader OUTSIDE the agent's sandbox; recompute expected output from first principles; give the agent no read/write to fixtures or pass/fail metadata.
4. Hybrid signal. Combine execution-correctness + process-coherence + an independent judge. High coherence + low correctness = a reward-hack flag.

Common mistakes

• Trusting a grader that was never run against a wrong answer (the #1 trap).
• Blending an LLM-judge score INTO the oracle's pass/fail (let it advise, never gate).
• Write-only-at-end evaluation that loses the checkpoint on crash.
• Reading the agent's confident rationale as evidence.

Real-world impact

Live receipt: an agent audit run was marked succeeded while the worker wrote a stub emitting an empty report — the oracle only checked that 4 JSON keys existed, not that they had content. A bidirectional gate (an empty stub MUST fail) catches it instantly. Separately, a UI eval lane read as a capability gap was actually a broken oracle (a subprocess invocation silently failing on one OS) — always validate the oracle before blaming the agent.

Sources: arXiv 2605.02964 (Reward Hacking Benchmark), 2511.21654 (EvilGenie). Eval/CUJ-specific sibling: trustworthy-cuj-scoring.