headless-capture

# Headless Capture Capture screenshots and recordings of desktop or web applications for visual verification. This skill helps remote agents show their UI work to humans who aren't watching the screen live. This skill is about delivering visual artifacts to a human for verification. It is not an agent-vision feedback loop; for browser automation and self-correction, prefer Playwright/CDP-style tooling directly. ## When to use this skill **Proactively suggest captures when:** - You just modified UI code (React, Vue, Svelte, HTML/CSS, Tauri frontend, Electron renderer) - You implemented a visual feature or fixed a UI bug - The user asks "what does it look like?" or "can you show me?" - You're working in a remote session and the human can't see the app **Don't use this skill when:** - Pure backend/API changes with no UI impact - The user is sitting in front of the same screen (just tell them to look) - The app doesn't have a visual interface ## Supported platforms and app types | App Type | Linux | macOS | Windows | |----------|-------|-------|---------| | **Web apps** (Vite, Next.js, etc.) | Playwright | Playwright | Playwright | | **Tauri** | Xvfb + import/ffmpeg | screencapture + ffmpeg | nircmd + ffmpeg | | **Electron** | Xvfb + import/ffmpeg | screencapture + ffmpeg | nircmd + ffmpeg | | **Native GTK/Qt/etc.** | Xvfb + import/ffmpeg | screencapture + ffmpeg | nircmd + ffmpeg | | **VS Code extension** | Xvfb + downloaded VS Code | adapt (Xvfb shape) | adapt (Xvfb shape) | Playwright is the preferred path for web apps — it's simpler, cross-platform, and doesn't need a virtual display. ### Templates and recipes | App type | Template | Recipe | |----------|----------|--------| | Tauri / Electron / GTK / Qt / generic X11 (own binary) | `templates/capture.mjs` | `recipes/tauri.md` | | VS Code extension (hosted by a downloaded VS Code) | `templates/capture-vscode-extension.mjs` | `recipes/vscode-extension.md` | | Web apps | `templates/capture.mjs` (Playwright path) | — | Own-binary apps (Tauri/Electron/GTK/Qt) differ only by **values** (binary path, window name, env hints, build command) → one generic template. A VS Code extension diverges **structurally** (download the editor, run in Extension Development Host mode, seed a user-data-dir to auto-open the panel) → its own template. **A new variant earns its own template only when it adds lifecycle steps, not when it only changes values.** ## What you're building 1. **Capture script** (`scripts/capture.mjs`) — unified script that: - Detects OS and app type - Uses Playwright for web apps, platform-native tools for desktop apps - Outputs to `artifacts/captures/` 2. **SessionStart hook** (Linux/Claude Code on the web) — installs toolchain automatically 3. **Wrapper scripts** in `package.json` for easy invocation 4. **`.gitignore`** entry for artifacts ## Workflow ### 1. Detect environment First, determine OS and app type: ```javascript // OS detection const platform = process.platform; // 'linux', 'darwin', 'win32' // App type detection - check in order: // 1. Web app: package.json has vite/next/nuxt/astro/remix OR user specifies URL // 2. VS Code extension: package.json has engines.vscode + contributes // 3. Tauri: src-tauri/Cargo.toml exists // 4. Electron: package.json has "electron" dependency // 5. Native: ask user for binary path ``` A **VS Code extension** has no URL and isn't your own binary — the "app" is a real VS Code (Electron) editor hosting your extension. It needs the dedicated `templates/capture-vscode-extension.mjs` + `recipes/vscode-extension.md`, not the generic `capture.mjs`. If you can't tell the app type, ask the user — don't guess. ### 2. Choose capture strategy **Web apps (Playwright path):** - Install: `npm install -D playwright @playwright/test` - No Xvfb needed — Playwright runs headless Chromium - Works identically on all platforms - Can capture at specific viewport sizes, wait for selectors, etc. **Desktop apps (platform-native path):** | OS | Display | Screenshot | Recording | Input | |----|---------|------------|-----------|-------| | Linux | Xvfb (virtual) | `import -window root` | `ffmpeg -f x11grab` | `xdotool` | | macOS | Real display | `screencapture -x` | `ffmpeg -f avfoundation` | `cliclick` / AppleScript | | Windows | Real display | PowerShell or `nircmd` | `ffmpeg -f gdigrab` | `nircmd` / AutoHotkey | ### 3. Confirm scope with user Use `AskUserQuestion` for: - **App type** if ambiguous (web vs desktop, which URL/binary) - **Build behavior**: auto-build when missing vs. fail with hint - **Capture trigger**: one-shot vs. watching for changes Don't ask about things you can decide yourself. ### 4. Set up the SessionStart hook (Linux only) Copy `templates/session-start.sh` to `.claude/hooks/session-start.sh`. Register in `.claude/settings.json`: ```json { "hooks": { "SessionStart": [ { "hooks": [ { "type": "command", "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/session-start.sh" } ] } ] } } ``` The hook is gated on `$CLAUDE_CODE_REMOTE` / `$CURSOR_AGENT` so it only runs in remote agent sessions, not on a local Linux dev machine. It falls back to `sudo` when the container runs as a non-root user. **Cursor Cloud Agent caveat:** Cursor does *not* run SessionStart hooks. Install the toolchain explicitly there — run `session-start.sh` directly with `CURSOR_AGENT=1`, or call it from a build/test step. For macOS/Windows, document manual install in README instead of a hook. ### 5. Drop in the capture script **Web / own-binary apps (Tauri/Electron/GTK/Qt)** — copy `templates/capture.mjs` to `scripts/capture.mjs` and customize (see `recipes/tauri.md` for the own-binary swap-ins): - `APP_TYPE`: `'web'`, `'tauri'`, `'electron'`, or `'native'` - `WEB_URL`: dev server URL for web apps (e.g., `http://localhost:5173`) - `APP_BINARY`: path to desktop app binary - `WINDOW_NAME`: for desktop app window detection - Build command in `ensureBinary()` for desktop apps On Linux the desktop path defaults to `1920x1080`, resizes the window to fill the virtual screen (no window manager under Xvfb to maximize it), and captures the **window region** rather than the whole screen — so the screen background doesn't bleed in as a black strip. It falls back to root capture when `xdotool` is absent. **VS Code extension** — copy `templates/capture-vscode-extension.mjs` to `scripts/capture-vscode.mjs` and fill in `EXTENSION_MAIN`, `BUILD_CMD`, `AUTO_OPEN_SETTINGS`, and (optionally) the tour coords. See `recipes/vscode-extension.md`. If you only changed the extension's **webview** content (not the editor chrome), prefer Playwright against the webview HTML or over CDP — it's faster and more reliable than full-editor capture. ### 6. Add wrapper scripts ```json { "scripts": { "capture": "node scripts/capture.mjs screenshot", "capture:record": "node scripts/capture.mjs record", "capture:web": "node scripts/capture.mjs screenshot --web", "capture:web:record": "node scripts/capture.mjs record --web" } } ``` ### 7. Ignore output directory Add to `.gitignore`: ``` artifacts/captures/ ``` ### 8. Validate end-to-end **For web apps:** 1. Start dev server: `npm run dev` 2. Run: `npm run capture:web` 3. Verify PNG shows the app, not a blank page **For desktop apps:** 1. Build the app if needed — captures only matter if they show the real app, not a placeholder. 2. Run: `npm run capture` 3. Inspect the output; don't just trust the exit code: - Tiny PNG (~hundreds of bytes, 1-bit grayscale) → window appeared but hadn't painted. Bump `--settle`. - Black/empty frame → check `WINDOW_NAME` matches, or the app isn't launching on the capture display. - Recording shows nothing changing → app may be off-display, **or** (during a tour) `xdotool` coords are hitting title-bar chrome instead of the UI. 4. Run a short recording too if you'll deliver video. ### 9. Attach capture to user Getting the artifact in front of the human proves the loop works end to end. - **Claude Code on the web:** call `SendUserFile` with the PNG/MP4 and a one-line caption. Paths under `artifacts/captures/` work even though they are gitignored. - **Cursor Cloud Agent / other harnesses:** use the **Read** tool on the artifact path — but only after copying out of the gitignored `artifacts/captures/` (see below). PNG for stills; **MP4 for recordings** (inlines reliably). Avoid GIF in chat — it often hangs while loading. If the artifact is broken (blank/wrong) after the obvious fixes, send what you have anyway with a note on what went wrong rather than reporting success. #### Cursor Cloud Agent: copy before Read Capture scripts write to `artifacts/captures/`, which is **gitignored**. That's fine for generation, but **do not Read or reference a gitignored path directly** — Cursor chat shows **"Waiting for upload…"** forever. Copy it to the harness's public artifact dir first, then Read *that* path: ```bash mkdir -p /opt/cursor/artifacts/screenshots cp artifacts/captures/my-shot.png /opt/cursor/artifacts/screenshots/ # then Read /opt/cursor/artifacts/screenshots/my-shot.png ``` Don't reference `artifacts/captures/…` in markdown image syntax — those URLs aren't reachable from chat. Optional: bake the copy step into the capture script via a `PUBLIC_ARTIFACT_DIR` env var pointing at the public dir. ### 10. Document in README/AGENTS.md Add a section explaining: - How to run captures - Env knobs available - Platform-specific setup (macOS: `brew install ffmpeg cliclick`, Windows: `choco install ffmpeg nircmd`) ## Proactive usage After modifying UI files, proactively offer to capture: ``` I've updated the login form styling. Would you like me to capture a screenshot so you can verify the changes? ``` If the user has previously accepted captures in the session, just do it: ``` I've updated the button colors. Here's a screenshot of the result: [attach screenshot] ``` ## Platform-specific notes ### Linux (Claude Code on the web) - SessionStart hook handles all deps - Xvfb creates a virtual display — no real monitor needed - Default display `:99`, override with `CAPTURE_DISPLAY` ### macOS - Uses real display — works even without a logged-in GUI session if you enable screen recording permissions - Install: `brew install ffmpeg cliclick` - `screencapture -x` for silent screenshots - For GUI-less sessions (SSH), need to enable headless screen capture (complex, prefer Playwright for web apps) ### Windows - Uses real display - Install: `choco install ffmpeg nircmd` or `winget install ffmpeg` - PowerShell `Add-Type` can capture screen without external tools - For headless/CI, use virtual display drivers or prefer Playwright ### All platforms (Playwright path) - `npm install -D playwright` - `npx playwright install chromium` - Works headless everywhere, no platform-specific setup ## Mouse-driven tours (optional, Linux/Xvfb) The default `record` mode grabs a settled, idle window — no input during the clip. Validate that first; only reach for tours if the user wants navigation in the deliverable (clicking tabs, opening menus, etc.). - Start `ffmpeg -f x11grab`, then drive clicks with `xdotool` during the grab. - Pin the window first (`xdotool windowmove <id> 0 0`) and **calibrate Y**: `mousemove --window` coords are relative to the *decorated* window, not the webview. Low Y often lands on the title bar / app logo, so the tour looks frozen on one view. Take a calibration screenshot before scripting clicks. - Mis-clicks on the title bar or resize edges can drag the window off-screen — keep tours session-local; don't commit machine-specific coordinate scripts to the repo. - Robust long-term alternative: switch views from inside the app (by element id / a test hook), or use CDP / a Tauri MCP plugin instead of pixel coordinates. ## Common issues **Cursor chat stuck on "Waiting for upload…":** - You attached a gitignored path (`artifacts/captures/…`) or used markdown ``. Copy to `/opt/cursor/artifacts/screenshots/` and **Read** that path instead (see step 9). **Blank captures (desktop apps):** - Window exists but hasn't painted yet - Increase `--settle` time (default 3s) - For Tauri, ensure WebKit env hints are set **Playwright can't connect:** - Dev server not running — start it first - Wrong URL — check port number - Page not ready — add `waitForSelector` or increase timeout **Permission denied (macOS):** - System Preferences → Security & Privacy → Screen Recording - Add Terminal/iTerm/your IDE **Display collision (Linux):** - Another Xvfb on `:99` - Set `CAPTURE_DISPLAY=:98` or another number **Recording shows nothing:** - App launched off-screen or on wrong display - Check `DISPLAY` env var matches Xvfb **xdotool coords on decorated windows (Linux tours):** - Low Y hits title-bar chrome, not the app — the tour looks "stuck" on one view. Calibrate from a screenshot. - Bad edge clicks can slide the window off-frame on Xvfb. **Window-name mismatch (Linux):** - Window titles vary by app and locale. If `--name` finds nothing, the script falls back to capturing the root display, which is fine. **Tauri build deps (Linux):** - If `cargo build` fails with `gdk-3.0 not found` or similar, install: `libgtk-3-dev libwebkit2gtk-4.1-dev libayatana-appindicator3-dev librsvg2-dev libcurl4-openssl-dev libssl-dev pkg-config`. - The SessionStart hook intentionally does *not* pull these in (heavy, only needed when building) — flag them in your docs. **EGL / DRI3 warnings (Linux):** - Harmless under Xvfb — neither is available. Don't chase them. **Lingering processes:** - Trap SIGINT/SIGTERM in the capture script and kill the app + Xvfb in cleanup, otherwise repeat runs collide on the display. **`apt-get` without root:** - The SessionStart hook assumes root or passwordless sudo in the remote container. If neither is available, install manually: `apt-get install -y ffmpeg imagemagick xdotool x11-utils xvfb`. ## Templates and recipes - `templates/capture.mjs` — unified capture script (web + own-binary desktop) - `templates/capture-vscode-extension.mjs` — VS Code extension capture (downloads VS Code, EDH mode) - `templates/session-start.sh` — Linux SessionStart hook - `recipes/tauri.md` — own-binary (Tauri/Electron/GTK/Qt) swap-ins - `recipes/vscode-extension.md` — VS Code extension lifecycle + calibration