Run vercel-plugin eval scenarios in Vercel Sandboxes instead of local WezTerm panels. Provisions ephemeral microVMs with Claude Code + plugin pre-installed, runs benchmark prompts, extracts hook artifacts, and produces coverage reports.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Run vercel-plugin eval scenarios in Vercel Sandboxes instead of local WezTerm panels. Provisions ephemeral microVMs with Claude Code + plugin pre-installed, runs benchmark prompts, extracts hook artifacts, and produces coverage reports.
Benchmark Sandbox — Remote Eval via Vercel Sandboxes
Run benchmark scenarios inside Vercel Sandboxes — ephemeral Firecracker microVMs with node24. Each sandbox gets a fresh Claude Code + Vercel CLI + agent-browser install, the local vercel-plugin uploaded, and runs a 3-phase eval pipeline:
Phase 1 (BUILD): Claude Code builds the app with --dangerously-skip-permissions --debug
Phase 2 (VERIFY): A follow-up Claude Code session uses agent-browser to walk through user stories, fixing issues until all pass (20 min timeout)
Phase 3 (DEPLOY): A third Claude Code session links to vercel-labs, runs vercel deploy, and fixes build errors (up to 3 retries). Deployed apps have deployment protection enabled by default.
Skills are tracked across all 3 phases — each phase may trigger additional skill injections as new files/patterns are created. After each phase, a haiku structured scoring step (claude -p --json-schema --model haiku) evaluates the results as structured JSON.
Proven Working Script
Use run-eval.ts — the proven eval runner:
# Run default scenarios with full 3-phase pipeline
bun run .claude/skills/benchmark-sandbox/run-eval.ts
# With dynamic scenarios from a JSON file (recommended — see "Dynamic Scenarios" below)
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios-file /tmp/my-scenarios.json
# Keep sandboxes alive overnight with public URLs
bun run .claude/skills/benchmark-sandbox/run-eval.ts --keep-alive --keep-hours 8
bun run .claude/skills/benchmark-sandbox/run-eval.ts --skip-verify --skip-deploy
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios splitwise-clone,calendly-clone
# Build-only (skip verification and deploy)
# Run specific scenarios by slug
CLI Flags
Flag
Default
Description
--concurrency N
5
Max parallel sandboxes (max 10)
--timeout MS
1800000 (30 min)
Per-phase timeout in ms
--keep-alive
off
Keep sandboxes running after eval
--keep-hours N
8
Hours to keep alive (with --keep-alive)
--skip-verify
off
Skip the agent-browser verification phase
--skip-deploy
off
Skip the Vercel deploy phase
--scenarios a,b,c
all
Only run specific scenarios by slug
--scenarios-file path
—
Load scenarios from a JSON file instead of built-in defaults
Dynamic Scenarios (Recommended Approach)
Instead of hardcoding tech-specific prompts, generate scenarios dynamically as a JSON file. Prompts should describe real-world apps people want to build using user stories — no tech name-dropping. Let the plugin figure out what Vercel tech to inject.
Scenario JSON Format
[{"slug":"pet-adoption-board","prompt":"Build me a pet adoption listing board where shelters can post animals...","expectedSkills":["ai-sdk","nextjs","shadcn","vercel-functions"],"userStories":["As a visitor, I can see a grid of pet listings with photos and names","As a visitor, I can click a pet card to see a detail page","As a visitor, I can filter pets by type"]}]
Each scenario needs: slug (string), prompt (string), expectedSkills (string[]), userStories (tuple of exactly 3 strings).
Prompt Design Guidelines
Focus on what the user wants, not what tech to use
Describe real-world apps that solve real problems with friendly, stylish UX
Include AI features naturally (recommendations, analysis, generation)
Always end with: "Link the project to my vercel-labs team. After building all files, start the dev server on port 3000 with \npx next dev --port 3000`."`
Include storage needs (photos, uploads) to trigger vercel-storage
Include scheduled tasks (reminders, cleanup) to trigger cron-jobs
Include auth/middleware to trigger routing-middleware
Structured Scoring (Haiku)
Each phase gets a structured JSON score via claude -p --json-schema --model haiku --setting-sources "" running inside the sandbox. This is a separate quick pass — no tools, no hooks — just reads the phase output and returns structured data.
Important: The claude -p --output-format json response wraps results — the actual schema data is in parsed.structured_output, not the top-level object.
Critical Sandbox Environment Facts
Property
Value
Home directory
/home/vercel-sandbox (NOT /home/user/ or /root/)
User
vercel-sandbox (NOT root)
Claude binary
/home/vercel-sandbox/.global/npm/bin/claude
PATH (via sh -c)
Includes ~/.global/npm/bin — claude findable by name
Files AND npm globals survive snapshot restore — use sandbox.snapshot() → Sandbox.create({ source: { type: "snapshot", snapshotId } })
SDK version
@vercel/sandbox@1.8.0 (v2 beta's named sandbox endpoint returns 404 for this team)
Team tier
Enterprise (vercel-labs) — no known sandbox time cap
Key Discoveries (Hard-Won)
Snapshots work: sandbox.snapshot() preserves files AND npm globals. Use it after build to create a restore point before verify/deploy. Note: snapshotting stops the source sandbox — create a new one from the snapshot to continue.
Plugin install: Use npx add-plugin <path> -s project -y --target claude-code — works because claude is in PATH after npm install -g. The --target claude-code flag is required because add-plugin can't auto-detect Claude Code without an initialized ~/.claude/ dir.
File uploads: Use sandbox.writeFiles([{ path, content: Buffer }]) — NOT runCommand heredocs. Heredocs with special characters cause 400 errors from the sandbox API.
Claude flags: Always use --dangerously-skip-permissions --debug. The --debug flag writes to ~/.claude/debug/.
Auth: API key from macOS Keychain (ANTHROPIC_AUTH_TOKEN — a vck_* Vercel Claude Key for AI Gateway), Vercel token from ~/.local/share/com.vercel.cli/auth.json (a vca_* token).
OIDC for sandbox SDK: Run npx vercel link --scope vercel-labs -y + npx vercel env pull once before first use.
Port exposure: Pass ports: [3000] in Sandbox.create() to get a public URL immediately via sandbox.domain(3000). Works on v1.8.0 — URL is assigned at creation time, before anything listens.
extendTimeout: Use sandbox.extendTimeout(ms) to keep sandboxes alive past their initial timeout. Verified working — extends by the requested duration. Use this for overnight keep-alive.
Background commands: runCommand with backgrounded processes (& or nohup) may throw ZodError on v1. Write a script file first, then execute it.
Session cleanup race: The session-end-cleanup.mjs hook deletes /tmp/vercel-plugin-*-seen-skills.d/ on session end. Extract artifacts BEFORE the session completes, or rely on poll history data.
agent-browser works in sandboxes: Install via npm install -g agent-browser. Claude Code can use it for browser-based verification inside the sandbox.
No hobby tier cap: Early 301s timeouts were from lower default timeout values in earlier script iterations, not a tier limitation. Enterprise (vercel-labs) has no known sandbox time cap — sandboxes ran 10+ minutes successfully.
claude -p works inside sandboxes: claude -p --json-schema --output-format json --model haiku works for structured scoring passes. No nesting issue when running inside a sandbox (only fails when running Claude inside Claude on the same machine).
Deploy project naming: ALWAYS use timestamped slugs with minute precision (e.g., pet-adoption-board-202603101853) to avoid collisions when linking to vercel-labs team projects. These are demo projects — we generate many per day. Format: <slug>-<YYYYMMDDHHMM>.
Results scored by haiku — no more parsing STORY_1: PASS from free text
Deploy Phase Details
The deploy phase uses a full Claude Code session (for skill tracking) to:
Run vercel link --yes --scope vercel-labs --project <slug>-YYYYMMDD
Run vercel deploy --yes
If build fails, fix code and retry (up to 3 attempts)
Important: unsets VERCEL_TOKEN env var so CLI falls back to ~/.local/share/com.vercel.cli/auth.json
Deployment protection is enabled by default on vercel-labs team
Deploy URL is extracted by regex from Claude's output, with haiku as fallback URL extractor.
DO NOT (Hard Rules)
Same rules as benchmark-agents, plus sandbox-specific:
DO NOT use claude --print or -p flag for BUILD/VERIFY/DEPLOY phases — hooks don't fire without tool-calling sessions (use -p only for haiku scoring passes)
DO NOT let sandboxes run without extracting artifacts — ephemeral filesystem is lost on stop
DO NOT pass API keys via writeFiles() — use Sandbox.create({ env: { ... } })
DO NOT skip snapshotting after build — it's your safety net if verify/deploy kills the sandbox
DO NOT use v2 beta SDK — named sandbox endpoint returns 404 for this team; use v1.8.0
DO NOT use runCommand heredocs to write file content — use sandbox.writeFiles() instead
DO NOT assume /home/user/ exists — the home dir is /home/vercel-sandbox/
DO NOT use simple project names without timestamps — always append -YYYYMMDDHHMM to avoid collisions across runs
Prerequisites
# One-time setup: link project for OIDC sandbox auth
npx vercel link --scope vercel-labs -y
npx vercel env pull .env.local
# Auth (auto-resolved from macOS Keychain + Vercel CLI auth):# - ANTHROPIC_API_KEY: from Keychain "ANTHROPIC_AUTH_TOKEN" (vck_* key) or env var# - VERCEL_TOKEN: from ~/.local/share/com.vercel.cli/auth.json (vca_* token) or env var# - ANTHROPIC_BASE_URL: defaults to https://ai-gateway.vercel.sh
Commands
Run eval with dynamic scenarios (recommended)
# Generate scenarios as JSON, then run
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios-file /tmp/my-scenarios.json
# With all phases + keep-alive for overnight
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios-file /tmp/scenarios.json --keep-alive --keep-hours 8
# Build-only, no verification or deploy
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios-file /tmp/scenarios.json --skip-verify --skip-deploy
# Filter to specific slugs from file or defaults
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios splitwise-clone,calendly-clone
Monitoring While Running
The orchestrator prints live status. For manual checks on a running sandbox:
User-story-focused prompts (no tech name-dropping) work — plugin detects patterns from actual code
ai-sdk, shadcn, nextjs, vercel-functions are the most consistently detected skills
cron-jobs, routing-middleware need Claude to write specific file patterns to trigger
Lexical prompt inject (UserPromptSubmit) working — skills injected before any files written
session-end-cleanup deletes claim dirs — use poll history for final skill counts
Enterprise tier (vercel-labs) — no sandbox time cap; builds ran 10+ minutes
Known Limitations
Snapshot stops the source sandbox: sandbox.snapshot() stops the original sandbox. Create a new sandbox from the snapshot to continue. Files and npm globals DO survive.
v2 beta incompatible: @vercel/sandbox@2.0.0-beta.3's named sandbox endpoint returns 404 for this team. Stick with v1.8.0.
Artifact window: Must extract before sandbox.stop() — filesystem is ephemeral. Session cleanup hook may delete claim dirs before extraction.
Amazon Linux paths: User is vercel-sandbox (home at /home/vercel-sandbox/). NOT /home/user/ or /root/.
--dangerously-skip-permissions parity: Sandbox evals auto-approve all tool calls. WezTerm evals use normal permission flow. Coverage results may differ.
runCommand timeout: Use { signal: AbortSignal.timeout(ms) } — the { timeout } option is silently ignored.
BrotliDecompressionError: Transient Vercel API errors can kill sandbox creation. Retry logic recommended for production runs.
Deploy reliability: Claude Code deploy sessions sometimes fail to output a parseable *.vercel.app URL. The haiku scoring step provides a fallback URL extraction attempt.
Verify timeout: Complex apps may need the full 20 minutes for agent-browser to test all stories. Simpler apps finish in 2-5 minutes.
