| name | judge-ui |
| description | Drive a web UI through a defined flow, capture stage screenshots + a structured read-back, and have an independent vision-judge sub-agent render a PASS / FAIL / UNCERTAIN verdict cross-checked against the read-back — turning "look at the screen" UAT from a human task into an agent-judged one, with a mandatory fall-back to human escalation on low confidence. Use when a UAT step needs someone to drive a browser and judge what renders ("does the screen look right?"), or when user-uat's Visual tier delegates here. Project specifics (server bring-up, auth, testids) come from a project adapter. Invoke as "/judge-ui [flow-or-spec] [--adapter <name>] [--dry-run]". |
| user-invocable | true |
UI Judge
Judging doctrine: invariants, archetypes, and the judge↔advisor spectrum live in _shared/judge-core.md — this skill instantiates them (it is the §10 reference implementation for separate-vision-judge + read-back + escalate).
The generic engine for the visual tier of UAT. It drives a real browser through a flow,
captures a screenshot at each stage, and dispatches an independent vision-judge sub-agent
that views the pixels, cross-checks them against a structured read-back (API JSON / DB query),
and returns a per-stage verdict with evidence.
The point: "did the screen render correctly?" has always been a human's job. A browser driver
plus a vision model can now do most of it — but only safely if the judgment is disciplined.
This skill is that discipline. It is project-agnostic: everything project-specific (how to
start the app, how to reach an authed state, which testids to click) comes from a project
adapter (e.g. toybox's /uat-ui). Without an adapter it runs an inline flow spec you supply.
When to use / not
- Use: a UI flow needs driving + visual judgment — a parent dashboard renders a list, a
wizard advances through screens, a form's result must look right. Or
/user-uat --ui routed a
visual-tier step here. Or a project adapter calls you with a named flow.
- Don't use for things a still frame can't judge: animation/motion, audio/sfx, real-device
input, or subjective "does it feel right" — those stay human (a screenshot is one frame; it
cannot see a countdown tick or hear a sound). Don't use it to refine a fuzzy UAT (that is
/review-uat) or to run shell-only checks (that is /user-uat's mechanical tier).
Invocation
/judge-ui --adapter toybox --flow room-import # run a named flow from a project adapter
/judge-ui <path-to-flow-spec> # run an inline flow spec (no adapter)
/judge-ui --adapter toybox --flow room-import --dry-run # print the plan + the stages; drive nothing
The flow-spec contract (what a flow must provide)
A flow is the parametrized input the engine drives. A project adapter supplies the first three;
the caller or the adapter supplies the rest.
- Bring-up — how to get the app running + reachable: start command(s), a readiness probe,
and the base URL. Skipped if the app is already up. Usually project-specific → adapter.
- Reach state — how to get from cold to the flow's starting UI state: auth (login / PIN),
seed fixtures, feature-flag values. Usually project-specific → adapter.
- Read-back handle — how to read the ground truth out of band (an authed API call, a DB
query) so the judge can cross-check pixels against data. Adapter provides the auth.
- Stages — an ordered list. Each stage = an action (navigate / click / fill), a
screenshot target path, optional mechanical DOM asserts (testid present, heading
text, row count, URL), and an optional read-back call.
- Rubric — per-stage natural-language pass criteria for the vision judge, in observable
terms ("review table shows 7 rows; first row name is editable; a Create button is present").
Borrow these verbatim from a
Verify (human) block if /review-uat produced one.
The run loop
- Ensure the app is up. Run the adapter's bring-up (or verify an already-running instance
with the readiness probe). For Windows / port-collision / parallel-session hazards, defer to
the adapter — it owns the safe bring-up + teardown.
- Drive the flow. Prefer a parametrized Playwright spec (it gives DOM asserts and
screenshots in one pass; see toybox's
frontend/playwright/room-import.spec.ts as the
reference shape). Run the mechanical DOM/API asserts FIRST at each stage — they're cheap,
deterministic, and they ground the screenshot (a green DOM assert + a screenshot beats a
screenshot alone). Capture each stage's screenshot to the agreed path and call the read-back.
A mechanical assert failure stops the flow and is itself a FAIL (no vision call needed).
- Dispatch the vision-judge sub-agent. Hand a separate agent (Agent tool) the stage
screenshots + the read-back JSON + the rubric + the flow's intent. It must (a) view each
image, (b) cross-check what it sees against the read-back, (c) return a per-stage verdict
PASS / FAIL / UNCERTAIN with the concrete evidence it used. The judge is independent eyes —
never the orchestrator grading its own driving.
- Resolve the verdict.
- Every stage PASS and pixels agree with the read-back → PASS.
- Any stage FAIL (with evidence) → FAIL.
- Any stage UNCERTAIN, low confidence, or pixels disagree with the read-back →
escalate to Human — degrade to today's behavior, present the evidence, ask the
one-line question. Never auto-PASS through uncertainty.
- Write the verdict doc (the durable artifact) and return a structured result to the
caller. Doc structure — all six sections always present, even on an early-exit run:
scope, environment, per-stage "what the judge saw" (legible observations, not vibes), the
read-back summary, findings, verdict. On a fail-fast (a mechanical-gate FAIL that never
reached the vision step), still emit every section — mark the unreached ones explicitly
("what the judge saw: not invoked — mechanical gate failed at "; "read-back: not
reached — flow stopped") rather than dropping them, and capture a screenshot of the failing
state to cite as evidence. Last line is exactly
VERDICT: PASS / VERDICT: FAIL /
VERDICT: ESCALATE.
The honesty invariants (do not skip — these are why it's safe)
A vision-judge is an agent grading an agent-rendered surface — the same trap operator UAT
exists to prevent (agents grading agent work codify regressions: toybox G2 / the audit-wire-
shape rule). These five rules keep it honest:
- Read-back cross-check is mandatory. The judge may not PASS on pixels alone — every PASS
must be corroborated by the API/DB read-back. A pretty screen over a wrong DB is a FAIL.
- Low confidence → Human, not PASS. Uncertainty degrades to escalation. The engine's job is
to remove the easy visual checks from the human's plate, not to fabricate confidence.
- Mechanical asserts gate the judge. Deterministic DOM/API checks run first and fail fast;
the vision call only judges what they couldn't.
- The judge is a separate sub-agent. Independent eyes. The orchestrator that drove the
browser does not also render the verdict.
- Evidence on every verdict. Each PASS/FAIL cites the screenshot path and the read-back
value. A verdict with no evidence is a defect (inherited from
/user-uat).
Side-effecting flows (form submits, real writes, sends) inherit /user-uat's safety gate: tag
them side-effectful, confirm before running, prefer an isolated fixture/DB. The adapter should
run against an isolated store, never the operator's real data.
Reuse — don't reimplement
/build-step --ui already owns app lifecycle + capture primitives:
dev/.claude/skills/build-step/scripts/capture_evidence.py (headless Chromium, full-page
screenshots, console/HAR, project-specific --exercise-cmd) plus Windows-aware
readiness-polling and taskkill/port-kill teardown. Reuse those primitives for capture/teardown
rather than rebuilding the Windows signal-handling. What build-step lacks — and what this
skill adds — is the judgment: a structured PASS/FAIL verdict against an acceptance rubric.
Capture there; judge here.
Relationship to other skills
/user-uat — its Visual tier (--ui) delegates here. user-uat owns the
partition + the terse report; judge-ui owns the drive + the vision verdict. The
UNCERTAIN → Human fallback feeds straight into user-uat's Needs you section./review-uat — emits the Verify (visual — vision-judge) blocks this skill consumes;
its rubric is the per-stage pass criteria. Refine → judge./build-step --ui — the capture/lifecycle primitives to reuse (above).- Project adapters (e.g. toybox
/uat-ui) — supply bring-up + auth + the flow library;
this engine is the project-agnostic half they call.
