| name | verify-browser |
| description | Shared how-to for driving an ARENA lab's UI with agent-browser. Invoked by the stages that verify a running lab through a real browser — the feature usability judge (verify-feature-usability) and the final sanity sweep (sanity-check-vuln-app). Covers the snapshot/ref loop, sign-in, navigating between views and back, and what a broken or incoherent UI looks like. Triggered by "verify-browser", "drive the UI", or an agent-browser block.
|
verify-browser skill
How to drive a running ARENA lab's UI with agent-browser. Both the per-feature
usability judge and the final sanity sweep use this; the calling skill says
what to verify, this says how to drive. A lab whose sign-in button does
nothing, or whose feature is only reachable by typing a URL, is broken even if
every API responds. agent-browser is a CLI on the host PATH — invoke it from
Bash. For any lab with a UI the browser drive is required; never substitute
curl for the sign-in and click-through. If agent-browser is not on PATH,
report that in your final response instead of silently skipping.
How agent-browser works
It drives Chrome over CDP and exposes each page as an accessibility-tree
snapshot with element refs, so you act on snapshot output instead of parsing
HTML. The browser stays alive across invocations until you close.
The core loop — snapshot, then act on a ref:
agent-browser open <url>
agent-browser snapshot -i
agent-browser fill e5 "alice"
agent-browser click e6
agent-browser snapshot -i
snapshot -i prints lines like - textbox "Email" [ref=e5] and
- button "Sign in" [ref=e6]. Pass the ref token exactly as shown (e5) to
fill <ref> <text> / click <ref>. Refs are reassigned on every snapshot and
go stale after any navigation, submit, or re-render — always re-snapshot before
the next ref command. Prefer refs; use a semantic locator
(find role button click --name "Sign in") only when a ref is awkward, and
fall back to a fresh snapshot + ref if it returns nothing.
Batch commands to save turns (important for speed)
Every separate Bash call is a full model round-trip, so driving the browser one
command per turn is slow. Chain commands you can run blind — the ones whose
output you don't need to read to choose the next step — into a single Bash call
with &&, ending the chain at the one command whose output you must read (a
snapshot/get). You only need to stop and read before picking a ref.
agent-browser open $ARENA_LAB_URL && agent-browser snapshot -i
agent-browser fill e5 "user@ares.local" && agent-browser fill e6 "ArenaUserPass!1" \
&& agent-browser click e7 && agent-browser wait --text "Sign out" --timeout 5000 \
&& agent-browser snapshot -i
agent-browser click e4 && agent-browser wait --load networkidle && agent-browser snapshot -i
A typical feature judgement should be ~5–7 Bash turns, not ~20. Do NOT use the
agent-browser batch subcommand — it swallows the snapshot/screenshot
output you need; plain shell && chaining is what you want. Wait for the state
you expect rather than sleeping (wait --text, wait --url, wait --load).
Don't re-do work — the browser keeps its state
Two cheap habits cut the drive roughly in half:
open the base URL once. The page persists between Bash calls, so move
between views by clicking nav entries, never by re-open-ing the base URL
you're already on. Re-open only to deliberately reload (e.g. to re-check the
logged-out state after sign-out).- Snapshot only when you need to read something new — a fresh ref, or a
state change you can't already predict. After a click whose destination you
just saw in the previous snapshot (e.g. a nav link whose label you read), chain
click … && wait … and snapshot once at the end of that hop, not after every
command. Don't re-snapshot a view you haven't changed.
Use only real commands — don't invent them
Use exactly these; do NOT guess at command names that "feel" right:
- Navigate to a URL:
agent-browser open <url> — not navigate.
- Read the current URL:
agent-browser get url — not current-url or url.
- Read page title / an element:
agent-browser get title / agent-browser get text <ref>.
Confirm navigation with a fresh snapshot -i, not a URL command. After a
click, the reliable signal that you actually moved is what the new snapshot
shows. If a click doesn't seem to navigate, re-snapshot and click the link's
ref directly (a list/catalog card usually has a real <a> — click that ref,
not the surrounding heading or container), and only conclude "navigation is
broken" after the snapshot still shows the old page. Do not fail a feature on a
URL-introspection command you typed wrong.
Judge a feature only on what that feature owns
You are judging ONE feature. A link to a different, not-yet-built feature
returning 404 (e.g. a catalog judged before the search feature exists) is not
this feature's defect — note it as out-of-scope, don't fail on it. Fail only
when the feature under test is itself broken (its own page errors, its own
primary action does nothing, its own data doesn't render).
Sign in
Fast path — inject a pre-authenticated cookie (use this when $ARENA_COOKIE_JAR
is set). The orchestrator has already logged in as the right role and saved a
cookie jar; inject it and you start authenticated, skipping the whole login form:
agent-browser cookies set --curl "$ARENA_COOKIE_JAR" \
&& agent-browser open $ARENA_LAB_URL && agent-browser snapshot -i
Then confirm the snapshot shows the logged-in state (username / "Sign out"). If
$ARENA_COOKIE_JAR is empty/unset, or the snapshot still shows the login form,
fall back to the form login below.
Form login (fallback, or when there's no cookie jar):
agent-browser open $ARENA_LAB_URL && agent-browser snapshot -i
agent-browser fill e5 "user@ares.local" && agent-browser fill e6 "ArenaUserPass!1" \
&& agent-browser click e7 && agent-browser wait --text "Sign out" --timeout 5000 \
&& agent-browser snapshot -i
Use credentials from spec.credentials for the relevant role. Confirm the
post-login state actually changed (username shown, a "Sign out" affordance, a
redirect to a logged-in view) — don't assume the click worked.
Navigate — including back
Reach a view by clicking its nav entry from the snapshot, not by typing the
URL. After exercising it, confirm you can get back to the rest of the app:
agent-browser click e4 && agent-browser wait --load networkidle && agent-browser snapshot -i
agent-browser click e3 && agent-browser wait --load networkidle && agent-browser snapshot -i
Navigating back is a common breakage — a view that loads but traps the user
(no working back/home link, browser Back throws, or the app shell disappears) is
a real usability defect. Exercise it explicitly: click back/home and, where the
app uses real navigation, also try agent-browser back, then re-snapshot.
What a broken / incoherent UI looks like
- A sign-in or nav affordance that does nothing when clicked.
- A view reachable only by URL — no clickable entry.
- A panel that renders empty / placeholder instead of real seeded data.
- A 500, blank screen, or JavaScript error that blocks the flow.
- Dead-end navigation: you reach a view but can't get back.
- Views that don't hang together — inconsistent shell/nav between features,
broken cross-links, a feature that looks bolted on rather than part of the app.
Always close
agent-browser close when done — even on error. The orchestrator sets a distinct
AGENT_BROWSER_SESSION for concurrent judges, so do not close all sessions.
Don't fake a logged-in state with eval; the test is whether a human can
actually use the UI.
