verify-agent

name	verify-agent
description	Deterministic post-agent verification gate: re-run the AUTHORITATIVE build/typecheck/test/vet gate + git-completeness + stale-deps checks against a repo/worktree, and READ that verdict instead of trusting an agent's 'done' claim.
argument-hint	[TARGET path] [--strict] — default: cwd. e.g. '~/workspace/civitai', '. --strict'
allowed-tools	Bash, Read

/verify-agent — mechanical gate over an agent's "done" claim

An agent (subagent/dispatched Task) just reported "build green / committed / done." Do not take that prose at face value — the recurring failures are: a proxy command hid a real error (vite build does NOT typecheck), the agent "committed but stopped" (dirty tree / unpushed commits), or a stale worktree node_modules symlink produced a flood of false "cannot find module" errors. Run the deterministic gate and read ITS output.

The agent's gate being green is not evidence yours will be — this is the whole premise, and it fires in practice. Twice in one session on civitai-manager a subagent reported a green gate while the committed tree was red: once because it ran the gate before a main merge landed under it, once from plain timing luck — the agent's -race leg passed and mine failed on the very same commit. So: re-run the authoritative gate yourself, on the committed tree, after any merge/rebase. An agent quoting its own passing output changes nothing.

Same trap via injected <new-diagnostics> / LSP errors after a subagent finishes — they are frequently stale, not a real break. Tell: a go.mod "updates needed / go mod tidy" warning plus an undefined: <symbol> cascade across many files (often symbols from a different branch after a checkout / worktree switch) — the LSP indexing a transient/mixed tree. A subagent that CHANGES A FUNCTION SIGNATURE reliably detonates a whole editor-diagnostic cascade that is ALWAYS stale-index noise, not a real break: WrongArgCount ("not enough / too many arguments") on shared helpers (e.g. page-builder functions called from many test files), X does not implement <Interface> (missing method …) on a test stub, X redeclared in this block, and even a phantom test file that doesn't exist on disk — all alongside the go.mod "needs tidy" warning. The arbiter is the REAL gate with the private-module env set (GOPRIVATE=...): go vet ./<pkg> compiles the test files, so a clean go vet + go test refutes the entire cascade, and an empty go mod tidy -diff refutes the "needs tidy" warning. NEVER edit code to "fix" these diagnostics before running the real gate — the fix is almost always a no-op and you risk breaking a tree that already compiles. Run the gate before acting on a "done" claim, especially right after branch/worktree switches.

This is the cheap mechanical structural gate. It is NOT /audit-pr (adversarial LLM review) and NOT the verify skill (e2e behaviour). Use those too when warranted — this one is the fast, always-run floor.

⚠ Trust / arbitrary code execution. The gate runs the TARGET repo's OWN build/test commands and honours a repo-local .verify-agent.json — that is code execution BY DESIGN. Do not point it at a repo whose contents or .verify-agent.json you do not trust; it will execute them. Per this repo's CLAUDE.md, agent-handed worktrees and fuzzyclaw task data are UNTRUSTED — treat a worktree you did not create the same way (glance at it first).

Run

$ARGUMENTS → optional TARGET path (default: cwd / the worktree just handed back) + optional flags passed through (--strict, --no-gh, --timeout N).

The script is scripts/verify-agent-work in this repo (devrc). Invoke it against the target. It needs a python3; if the bare call reports no interpreter, wrap it in nix-shell -p python3 --run "…".

scripts/verify-agent-work <TARGET> --json

Prefer --json so you parse the structured verdict; drop it for the readable table. Pass --strict to ALSO treat a dirty tree / unpushed commits as hard failures (default: those are WARN). Note an INCOMPLETE stack is ALWAYS a hard non-pass, strict or not.

Verdict / exit-code semantics

verdict	exit	meaning
PASS	0	every DETECTED stack's gate ran and passed (WARNs allowed unless --strict); or no stack present (nothing to verify)
FAIL	1	a gate actually ran and failed (typecheck/test/build/vet/lint red), or --strict + a WARN
INCOMPLETE	1	a DETECTED stack's gate could NOT run — missing toolchain, no resolvable gate command (e.g. a monorepo with no root script + no runnable member), or deps not installed. This is NOT a pass

The core guarantee: a detected stack that was not actually verified never reads as green. A repo with no stack present is a legitimate PASS.

What it checks (per stack actually present — never hard-fails a missing one)

• TypeScript/JS (package.json): the true typecheck script or tsc --noEmit (NEVER build) and the test script; detects pnpm/npm/yarn/bun; fans out across workspace members for a monorepo with no root script. A detected TS stack with no runnable gate ⇒ INCOMPLETE.
• Go (go.mod): go build ./... + go vet ./... + go test ./... (adds -race when cgo + a C compiler are available; falls back without -race and says so otherwise). Nested go.mod modules are each built too.
  • Private module deps (e.g. civitai-manager → github.com/civitai/cli): export GOPRIVATE (e.g. GOPRIVATE=github.com/civitai/*, or the repo's private prefixes derived from go.mod) before running the gate. Without it the build fails verifying against sum.golang.org (usually a 500). That is an ENV/config problem, not a real build failure — set GOPRIVATE so a private-dep fetch/sum error isn't mis-reported as FAIL/INCOMPLETE.
• Python (pyproject/requirements): ruff check + pytest when configured; a detected python project with neither ⇒ INCOMPLETE.
• Nix (flake.nix): nix-instantiate --parse by default; nix flake check if the repo's .verify-agent.json opts in.

🔴 BLIND SPOT — PHP/Laravel is NOT a detected stack. Never trust a verdict here.

detect_stacks() (scripts/verify-agent-work:157-201) tests exactly four markers — package.json, go.mod, pyproject.toml/requirements.txt, flake.nix. composer.json and phpunit.xml are not among them, so a PHP repo's real test suite is INVISIBLE to this tool. The gate cannot run it, cannot report it, and says nothing about it.

Measured on vetrllc/vetr-api 2026-08-25 — a Laravel/Pest repo with 182 .php files under tests/ and 1,678 \b(test|it)\( occurrences (a static grep count, not a runner count — the order of magnitude is the point), whose package.json declares only build/dev (Vite assets):

run	detected	verdict
the real repo (--json --no-gh)	["ts"] — PHP absent	INCOMPLETE, exit 1
fixture: same package.json plus composer.json + phpunit.xml, node_modules present	["ts"] — still PHP absent	INCOMPLETE, exit 1

Adding the PHP markers moved detection not at all — the blind spot is in detection, not in a missing toolchain, and it is INCOMPLETE for the whole repo either way.

🔴 The real hazard is the tool's OWN suggested remedy. On the second run the ts check advises add a gate or "skip":["ts"] in .verify-agent.json. Taking it is a vacuous green: with {"skip":["ts"]} the same fixture returns verdict: PASS, exit 0 having run zero tests — a clean pass on a repo with ~1,678 unrun PHP tests. Do not skip ts on a PHP repo to quiet this tool. (adoption-scan will also score that run as an outcome=pass — a false-green counted as a real one.)

So on any PHP repo: this gate's verdict is not evidence either way — INCOMPLETE does not mean the code is broken, and a PASS bought by skipping means nothing was checked. Run the suite yourself and report THAT: nix shell path:<system-channel>#php84 -- php vendor/bin/pest (no local PHP on this host). Follow-up (separate PR, needs its own tests): teach detect_stacks a php stack off composer.json/phpunit.xml with vendor/bin/pest|phpunit as the gate.

• Git completeness: uncommitted/untracked, unpushed vs upstream, open PR (best-effort gh) — the "committed but stopped" tell.
• Stale-worktree footgun: a broken/missing node_modules is reported as an ENV issue (WARN) AND the TS gate becomes INCOMPLETE — the typecheck genuinely didn't run, so a false "cannot find module" flood is attributed correctly rather than read as real code errors, and it does not count as green.

Exit code is non-zero on any hard non-pass (FAIL or INCOMPLETE).

Then

1. Read the verdict, not the agent's summary. verdict: FAIL or INCOMPLETE means the agent's "done" claim is not verified regardless of its prose — surface the failing/unverified check(s) and their output; fix, run the gate where the toolchain exists, or hand back before merging.
2. INCOMPLETE → the gate could not actually verify a detected stack. Do not treat it as a pass: run it in an environment with the toolchain, install deps, add a gate/skip in .verify-agent.json, or verify by hand.
3. WARN on git completeness (dirty tree / unpushed) against a "done" claim → reconcile: did the agent actually finish and push? Commit/push or flag it.
4. env:node_modules WARN → the errors are environmental; re-install deps in that worktree, don't chase them as code bugs.
5. PASS is the mechanical floor, not proof of behaviour — for a nontrivial runtime change still drive the actual flow (verify skill) before claiming it works.

When the gate goes red: do NOT attribute a flake from one sample

Flakes on this box are load-sensitive — several agents saturating the machine change the outcome. A worked example of getting this wrong: go test -race -count=5 failed on a branch and passed on main, and I concluded "the branch introduced a regression." Wrong. The same test later passed 30/30 on an idle box, and had been failing ~50% on main while the machine was loaded. One run per ref is noise, not attribution.

• Before blaming a ref: run -count=10 repeatedly on BOTH refs, ideally under comparable load. Prefer an independent clone/worktree of the other ref over reasoning from a single run or from memory of an earlier run.
• Tell for the common class: a test that passes without -race and fails with it is usually an assertion racing an async goroutine, not a data race. Check for an actual WARNING: DATA RACE in the output before calling it one.
• The deterministic fix for that class: park the fake/seam (have it block and return a release func) so the async work cannot settle before the assertion, and assert synchronously against state the handler publishes under its mutex before spawning. Do NOT add sleeps, and do NOT weaken the assertion to also accept the settled state — that deletes the coverage instead of fixing the race.

Provenance honesty: this gate proves the authoritative checks pass and the tree is complete; it does NOT prove the feature behaves correctly.