verify-browser

name

verify-browser

description

Shared how-to for driving an ARENA lab's UI with agent-browser. Invoked by the stages that verify a running lab through a real browser — the feature usability judge (verify-feature-usability) and the final sanity sweep (sanity-check-vuln-app). Covers the snapshot/ref loop, sign-in, navigating between views and back, and what a broken or incoherent UI looks like. Triggered by "verify-browser", "drive the UI", or an agent-browser block.

verify-browser skill

How to drive a running ARENA lab's UI with agent-browser. Both the per-feature usability judge and the final sanity sweep use this; the calling skill says what to verify, this says how to drive. A lab whose sign-in button does nothing, or whose feature is only reachable by typing a URL, is broken even if every API responds. agent-browser is a CLI on the host PATH — invoke it from Bash. For any lab with a UI the browser drive is required; never substitute curl for the sign-in and click-through. If agent-browser is not on PATH, report that in your final response instead of silently skipping.

How agent-browser works

It drives Chrome over CDP and exposes each page as an accessibility-tree snapshot with element refs, so you act on snapshot output instead of parsing HTML. The browser stays alive across invocations until you close.

The core loop — snapshot, then act on a ref:

agent-browser open <url>        # open a page
agent-browser snapshot -i       # list interactive elements + refs (e1, e2, …)
agent-browser fill e5 "alice"   # fill the textbox at ref e5
agent-browser click e6          # click the element at ref e6
agent-browser snapshot -i       # re-snapshot after any page change

snapshot -i prints lines like - textbox "Email" [ref=e5] and - button "Sign in" [ref=e6]. Pass the ref token exactly as shown (e5) to fill <ref> <text> / click <ref>. Refs are reassigned on every snapshot and go stale after any navigation, submit, or re-render — always re-snapshot before the next ref command. Prefer refs; use a semantic locator (find role button click --name "Sign in") only when a ref is awkward, and fall back to a fresh snapshot + ref if it returns nothing.

Batch commands to save turns (important for speed)

Every separate Bash call is a full model round-trip, so driving the browser one command per turn is slow. Chain commands you can run blind — the ones whose output you don't need to read to choose the next step — into a single Bash call with &&, ending the chain at the one command whose output you must read (a snapshot/get). You only need to stop and read before picking a ref.

# one turn: open and read the resulting snapshot
agent-browser open $ARENA_LAB_URL && agent-browser snapshot -i

# one turn: a whole login (refs known from the snapshot above), then read result
agent-browser fill e5 "[e-mail]" && agent-browser fill e6 "ArenaUserPass!1" \
  && agent-browser click e7 && agent-browser wait --text "Sign out" --timeout 5000 \
  && agent-browser snapshot -i

# one turn: click a nav entry, wait, and read where it landed
agent-browser click e4 && agent-browser wait --load networkidle && agent-browser snapshot -i

A typical feature judgement should be ~5–7 Bash turns, not ~20. Do NOT use the agent-browser batch subcommand — it swallows the snapshot/screenshot output you need; plain shell && chaining is what you want. Wait for the state you expect rather than sleeping (wait --text, wait --url, wait --load).

Don't re-do work — the browser keeps its state

Two cheap habits cut the drive roughly in half:

• open the base URL once. The page persists between Bash calls, so move between views by clicking nav entries, never by re-open-ing the base URL you're already on. Re-open only to deliberately reload (e.g. to re-check the logged-out state after sign-out).
• Snapshot only when you need to read something new — a fresh ref, or a state change you can't already predict. After a click whose destination you just saw in the previous snapshot (e.g. a nav link whose label you read), chain click … && wait … and snapshot once at the end of that hop, not after every command. Don't re-snapshot a view you haven't changed.

Use only real commands — don't invent them

Use exactly these; do NOT guess at command names that "feel" right:

• Navigate to a URL: agent-browser open <url> — not navigate.
• Read the current URL: agent-browser get url — not current-url or url.
• Read page title / an element: agent-browser get title / agent-browser get text <ref>.

Confirm navigation with a fresh snapshot -i, not a URL command. After a click, the reliable signal that you actually moved is what the new snapshot shows. If a click doesn't seem to navigate, re-snapshot and click the link's ref directly (a list/catalog card usually has a real <a> — click that ref, not the surrounding heading or container), and only conclude "navigation is broken" after the snapshot still shows the old page. Do not fail a feature on a URL-introspection command you typed wrong.

Judge a feature only on what that feature owns

You are judging ONE feature. A link to a different, not-yet-built feature returning 404 (e.g. a catalog judged before the search feature exists) is not this feature's defect — note it as out-of-scope, don't fail on it. Fail only when the feature under test is itself broken (its own page errors, its own primary action does nothing, its own data doesn't render).

Sign in

Fast path — inject a pre-authenticated cookie (use this when $ARENA_COOKIE_JAR is set). The orchestrator has already logged in as the right role and saved a cookie jar; inject it and you start authenticated, skipping the whole login form:

agent-browser cookies set --curl "$ARENA_COOKIE_JAR" \
  && agent-browser open $ARENA_LAB_URL && agent-browser snapshot -i

Then confirm the snapshot shows the logged-in state (username / "Sign out"). If $ARENA_COOKIE_JAR is empty/unset, or the snapshot still shows the login form, fall back to the form login below.

Form login (fallback, or when there's no cookie jar):

agent-browser open $ARENA_LAB_URL && agent-browser snapshot -i   # find email/password/submit refs
# then, in one turn:
agent-browser fill e5 "[e-mail]" && agent-browser fill e6 "ArenaUserPass!1" \
  && agent-browser click e7 && agent-browser wait --text "Sign out" --timeout 5000 \
  && agent-browser snapshot -i

Use credentials from spec.credentials for the relevant role. Confirm the post-login state actually changed (username shown, a "Sign out" affordance, a redirect to a logged-in view) — don't assume the click worked.

Navigate — including back

Reach a view by clicking its nav entry from the snapshot, not by typing the URL. After exercising it, confirm you can get back to the rest of the app:

# one turn: click the feature's nav entry and read the view it rendered
agent-browser click e4 && agent-browser wait --load networkidle && agent-browser snapshot -i
# … exercise the view …
# one turn: go back home and confirm you returned cleanly
agent-browser click e3 && agent-browser wait --load networkidle && agent-browser snapshot -i

Navigating back is a common breakage — a view that loads but traps the user (no working back/home link, browser Back throws, or the app shell disappears) is a real usability defect. Exercise it explicitly: click back/home and, where the app uses real navigation, also try agent-browser back, then re-snapshot.

What a broken / incoherent UI looks like

• A sign-in or nav affordance that does nothing when clicked.
• A view reachable only by URL — no clickable entry.
• A panel that renders empty / placeholder instead of real seeded data.
• A 500, blank screen, or JavaScript error that blocks the flow.
• Dead-end navigation: you reach a view but can't get back.
• Views that don't hang together — inconsistent shell/nav between features, broken cross-links, a feature that looks bolted on rather than part of the app.

Always close

agent-browser close when done — even on error. The orchestrator sets a distinct AGENT_BROWSER_SESSION for concurrent judges, so do not close all sessions. Don't fake a logged-in state with eval; the test is whether a human can actually use the UI.