verify-feature-usability

name	verify-feature-usability
description	Agentic usability gate for a single ARENA feature. Browser-drives the running lab with agent-browser to confirm one feature is reachable from the UI, works end-to-end on real data, and behaves like a real product feature — then writes a verdict the orchestrator feeds back to the feature writer. Triggered by "verify-feature-usability", "ARENA usability judge", or a "Feature under test" block.

verify-feature-usability skill

You are the usability judge for exactly ONE feature — which may be a vuln feature or a supporting (vuln: none) feature; judge both the same way. The verifier has already built the image and started a container; your job is to decide whether this feature is a coherent, reachable product feature — not whether it is exploitable (for vuln features, a separate exploit verifier checks that against the same run; supporting features have no exploit to check).

You do not edit the app and you do not run Docker. You drive the running lab with agent-browser, form a judgement, and write a verdict file. If the feature is broken or incoherent, your verdict bounces it back to the feature writer with specific feedback.

Inputs (in the user prompt)

• The Spec block — theme, credentials.
• The Feature under test block: name, feature_kind, vuln, endpoints, notes.
• $ARENA_LAB_URL — base URL of the running container.
• $ARENA_WORKDIR — write your verdict to $ARENA_WORKDIR/usability/<feature.name>.json.

Process

1. Drive the feature like a real user

Invoke the arena-spawner:verify-browser skill for how to drive the UI with agent-browser (the snapshot/ref loop, sign-in, navigating and navigating back). Drive fast: batch commands per verify-browser (one Bash call per step, chained with &&), aim for ~5–7 Bash turns total, and do not re-read the skill or re-snapshot needlessly. The shape for this feature:

1. Authenticate. If $ARENA_COOKIE_JAR is set, inject it and open the lab in one turn (cookies set --curl … && open … && snapshot -i) — no login form needed. Otherwise sign in with the spec credentials for the role in the prompt (skip entirely if anon). Confirm the post-login state changed.
2. Reach the feature by clicking its nav entry — not by typing endpoint into the URL bar; a feature you can only reach by URL is not reachable.
3. Exercise the primary user flow for the feature_kind (a search accepts a query and shows results; a create form submits and the new item appears; a list shows seeded rows; an admin view shows privileged data).
4. Navigate back to the rest of the app (home / another nav entry, and browser Back) and confirm you return cleanly — a view that traps the user is a real defect.
5. You MUST capture a screenshot to $ARENA_WORKDIR/usability/<feature.name>.png showing the feature's primary flow — this image is surfaced to users beside the feature's solution writeup, so frame it on the working feature, not a blank/login page. Then agent-browser close. Each judge runs with its own AGENT_BROWSER_SESSION, so do not close all sessions.

A typical run is roughly: turn 1 authenticate + snapshot, turn 2 click into the feature + snapshot, turns 3–4 exercise the flow, turn 5 navigate back + snapshot, turn 6 screenshot + close, turn 7 write the verdict. If you're past ~10 turns you're fumbling — re-read the snapshot and act on refs directly.

2. Judge — coherency above all

We currently ship labs whose features are exploitable but don't hang together as a real product. Weigh coherency heavily. Answer each honestly from what the browser actually showed:

1. Reachable? A visible nav entry / affordance takes a user to this feature by clicking?
2. Works end-to-end? The primary flow returns real seeded data (not a placeholder, empty panel, or error), no 500 / blank screen / dead button?
3. Navigable? Can you get back out of the feature to the rest of the app (working back/home link, browser Back doesn't break the shell)? A dead-end view fails.
4. Coherent? Does it behave like a real <feature_kind> in a real product, fit the spec's theme, and sit naturally alongside the rest of the app — not a contrived stub bolted on only to host a vuln?
5. No breakage? No JavaScript errors, console-fatal, or broken layout that blocks the flow?

(For an api_only spec there is no UI — confirm the endpoint responds sensibly and pass on reachability/navigation.)

3. Write the verdict

$ARENA_WORKDIR/usability/<feature.name>.json:

{
  "ok": true,
  "summary": "Signed in as user, clicked the 'Search' nav entry, searched 'alice' and saw 3 seeded results render. Behaves like a real search feature.",
  "issues": [],
  "out_of_scope": [
    "Did not check the logged-out state of the app.",
    "Did not check cross-feature nav consistency or sibling pages."
  ]
}

• Be skeptical, not generous. ok: true only when all five questions are a clear yes from what the browser actually showed you — not what the feature is "supposed" to do. If you are unsure on any question, the answer is false: a wrongly-passed feature ships a broken lab, a wrongly-failed one costs one cheap retry.
• On ok: false, fill issues with specific, actionable problems the feature writer can fix — e.g. "No nav entry for the feature; it is only reachable by typing /api/posts in the URL bar. Add a clickable 'Posts' link per references/ui-wiring.md." or "Clicking 'Search' shows an empty panel — the results area never renders the /api/search response." Vague issues ("doesn't work") are not actionable; be concrete about what you saw and what to change.
• Always fill out_of_scope with what you did NOT verify — you judged ONE feature, so the whole-app picture is unchecked. List the things the later sanity coherence sweep should still confirm: the logged-out state, nav consistency across pages, sibling/decorative pages, the sign-out flow, and anything you noticed but couldn't fully exercise. This list is handed forward to the sanity judge, so be honest about the limits of your single-feature view.

Done when

$ARENA_WORKDIR/usability/<feature.name>.json exists with a well-formed verdict (including out_of_scope) and the browser session is closed. Report done — the orchestrator reads the verdict; a failing verdict bounces the feature back to the writer with your issues (combined with any exploit-verifier feedback from the same run), and your out_of_scope notes are forwarded to the sanity judge.