sanity-check-vuln-app

name	sanity-check-vuln-app
description	Final stage of the ARENA generation pipeline. Crawls the now-complete app, confirms no hint-leaking debug routes were left in, drives the UI with agent-browser, and assembles solution.md from per-feature snippets that the feature stage (or the verifier) already wrote. Then writes manifest.json. Triggered by "sanity-check-vuln-app", "ARENA sanity", or a Sanity block.

sanity-check-vuln-app skill

You are the sanity stage. By the time you run, every feature has been written and verified individually; each one deposited a per-feature snippet at $ARENA_WORKDIR/solutions/<feature.name>.md. Your job is the global checkpoint: prove the whole app holds together, assemble solution.md, and emit manifest.json.

The orchestrator has a container already running for you. The base URL is in $ARENA_LAB_URL (the verifier rebuilt the image and started it before invoking you). When you are done, the orchestrator stops the container.

Inputs (in the user prompt)

• The Spec block.
• The Plan block (full plan.json, already validated).
• $ARENA_LAB_URL — http://127.0.0.1: for the running container.
• $ARENA_COOKIE_JAR — set only for L2 labs, whose TOTP second factor you can't hand-drive: a pre-authenticated curl cookie jar to inject so you reach logged-in pages (see verify-browser). Absent for L1 / anon labs — sign in through the UI as usual there.
• $ARENA_IMAGE_TAG — informational only: the tag of the image the orchestrator built, ran, and will push. You do not set image_tag in the manifest (the orchestrator stamps it in), and you do not build, tag, or push images.
• $ARENA_WORKDIR — same workdir as previous stages. Contains:
  • app/ — the built lab source
  • solutions/<name>.md — one per vuln feature (the feature-writer's human-readable exploit walkthrough)

Process

1. Hint-leak check (one probe — don't crawl turn-by-turn)

Run this as a single chained command (one turn, not one curl per URL): grep the served homepage for give-away comments AND probe the handful of leak-prone paths for reachability.

curl -s "$ARENA_LAB_URL/" | grep -iqE "vulnerable|exploit|ctf|hack|todo: ?(sanitize|fix)" && echo "COMMENT LEAK"
for p in /debug /admin/debug /_internal /__debug__ /api/debug /solution.md /walkthrough.md /solutions/ /exploits/; do
  code=$(curl -s -o /dev/null -w '%{http_code}' "$ARENA_LAB_URL$p")
  [ "$code" != "404" ] && [ "$code" != "000" ] && echo "LEAK $p ($code)"
done

Reject (and fix) if it prints anything — a give-away comment in the served HTML/JS, a reachable debug route, or an exposed solution.md / walkthrough.md / exploits/ / solutions/ artifact. Fix the file in $ARENA_WORKDIR/app/ and tell the operator in your final response — the orchestrator rebuilds and re-runs sanity.

2. Whole-app smoke pass (REQUIRED — one pass, not a per-feature re-walk)

Each feature was already browser-driven and judged end-to-end, in isolation by the usability-judge stage during the feature loop. Do not repeat that work. You are not re-exercising each feature's internals or re-testing every data flow. Your job is one full-app smoke pass confirming the assembled whole holds together — the cross-feature, shared-shell view a single-feature judge never sees. The per-feature judges also forwarded a "Still unverified" list in your prompt; treat those notes as the checklist of things you must close here.

Invoke the arena-spawner:verify-browser skill for how to drive the UI with agent-browser (the snapshot/ref loop, sign-in, navigating and navigating back). curl is not a substitute: for any lab with a UI you must actually drive the browser. If agent-browser is not on PATH, report that in your final response — don't silently fall back to curl-only checks.

Make one pass through the app. Visit each feature once to confirm it is reachable and renders — do not deep-exercise it again. Confirm all of these as you go (each is blocking):

• Logged-out coherence. Load the app without signing in and snapshot. No post-login-only control may be visible: no "Sign out"/"Logout" button, no signed-in username, no account/profile nav. A Sign-out button on a logged-out page is a hard fail. (Anonymous-by-design features — e.g. an anon search — may legitimately show; post-login affordances may not.)
• Sign-in / sign-out. Sign in once. The post-login state must be clearly distinct from logged-out (not merely "something changed"), and sign-out must actually return the app to the logged-out state. If $ARENA_COOKIE_JAR is set, this lab uses a TOTP second factor you cannot complete by typing in the browser — inject the jar (see verify-browser) to reach the logged-in state instead of hand-driving it, and do not fail the lab merely because you can't type a code yourself. You may still confirm the login page presents the expected second-factor challenge. Then verify sign-out returns to logged-out as normal.
• Nav integrity + reachability + no dead links. Every feature in the plan — including supporting (vuln: "none") features and every vuln-bearing feature — must be reachable by clicking the shared nav (not by typing its URL), and each must resolve to a real, working page (HTTP 200, real content) — not a 404, blank, or a fake affordance (alert(...), href="#" that does nothing). Confirm each once; a planned, nav-linked page that is missing or broken is a failure. A genuinely unlinked extra is the only thing you may treat as optional.
• Shared shell + navigation, including back. Every page must render through the same shared layout/nav — no page with a different or missing nav. As you move from feature to feature and back, confirm the shared nav and the browser Back button work and never trap the user or drop the app shell. Dead-end navigation is a blocking defect.
• No duplicate or contradictory controls. On any one view, there must not be two of the same primary control (e.g. two search boxes, a nav search and a body search doing the same thing). Flag and fix duplicates.
• No global breakage. No console-fatal error, 500, or blank screen on any view you load.

This is the gap the per-feature judges cannot cover: cross-feature coherence, nav integrity, and shared-shell consistency across the assembled app. Confirm it once — do not re-derive each feature's end-to-end flow from scratch.

If any blocking check fails, the lab UI is broken: fix it in $ARENA_WORKDIR/app/ (or frontend/) and tell the orchestrator in your final response — it rebuilds and re-invokes sanity to verify your fix against a fresh image. Do not write manifest.json while any blocking check fails.

3. Assemble solution.md

Concatenate the per-feature writeups in plan order. The feature-writer wrote $ARENA_WORKDIR/solutions/<feature.name>.md for every vuln feature — a human-readable explanation of the vuln and how to exploit it. Sanity just stitches them together.

Format + assembly logic in references/solution-md.md. The shape:

# Solution: <spec.name>

## <first vuln feature.name>

<that feature's writeup>

---

## <second vuln feature.name>

...

If solutions/<feature.name>.md is missing for any vuln feature (vuln != "none"), that's a feature-stage bug — tell the operator and do NOT write manifest.json.

4. Image build + push — NOT your job

Do not run docker build, docker tag, docker push, or aws. After you finish, the orchestrator deterministically rebuilds the image from the final $ARENA_WORKDIR/app/ and pushes it under $ARENA_IMAGE_TAG. If you edited any files in step 1, just note it in your final response so the operator knows the shipped image differs from the one you tested.

5. Write manifest.json

$ARENA_WORKDIR/manifest.json — full schema in references/manifest-schema.md. The reference doc is the authoritative schema available inside this isolated stage.

Done when

solution.md and manifest.json exist in $ARENA_WORKDIR (without image_tag — the orchestrator stamps that in), the hint-leak crawl came back clean, and the single whole-app smoke pass in step 2 passed — logged-out coherence, a real sign-in/sign-out flow, every planned (incl. vuln-bearing) feature reachable once through the shared nav with no dead links, a consistent shared shell with working back-navigation, no duplicate controls, and no global breakage.