Menü
Drive the NixOS + Hyprland desktop programmatically — window focus, keyboard input, screenshots, clipboard, VNC, and Chromium via CDP. Use when a task requires headful UI interaction, visual verification, or browser automation.
| name | desktop-automation |
| description | Drive the NixOS + Hyprland desktop programmatically — window focus, keyboard input, screenshots, clipboard, VNC, and Chromium via CDP. Use when a task requires headful UI interaction, visual verification, or browser automation. |
What: Tools and patterns for driving the NixOS + Hyprland desktop without human input. Why: Some tasks genuinely need a real UI — testing click flows, verifying visual output, pasting into a running TUI. This is a yolo box; treat the desktop like any other tool.
WAYLAND_DISPLAY and HYPRLAND_INSTANCE_SIGNATURE are unset in SSH sessions. Everything below fails silently without them.
export WAYLAND_DISPLAY="$(ls /run/user/$(id -u)/ | grep -E '^wayland-[0-9]+$' | head -1)"
export HYPRLAND_INSTANCE_SIGNATURE="$(ls /run/user/$(id -u)/hypr/ | head -1)"
Fire-and-forget doesn't work. Every desktop action follows this cycle:
hyprctl activewindow -j or grim /tmp/out.png + Read the PNGIf step 3 doesn't show the expected change, re-observe before retrying. Don't blindly retry — the window focus, a modal, or the element position may have changed.
| Tool | Use for | Notes |
|---|---|---|
hyprctl dispatch sendshortcut KEY,class:^(app)$ | Key chord to a specific window — bypasses focus | Preferred over wtype when you know the target class |
wtype | Free-form text or chords to focused window | wtype -M ctrl -p v -m ctrl = Ctrl+V |
ydotool | Mouse clicks and moves | Needs ydotoold running + user in ydotool group |
hyprctl clients -j | List all windows with class, title, geometry | Always locate before acting |
grim -g "X,Y WxH" /tmp/out.png | Screenshot a region | Pull geometry from hyprctl clients -j |
hyprctl dispatch exec "[float; size W H; center 1] cmd" | Launch floating window centred | Good for browser UI tests |
If ydotool fails with permission denied: check id for ydotool group membership. For keys-only, fall back to wtype or sendshortcut instead.
Multiple full-screen screenshots accumulate in context and Claude rejects requests once cumulative dimensions exceed ~2000px.
grim -g "X,Y WxH" — pull the rect from hyprctl clients -jPage.captureScreenshot with a clip region instead of grimwl-copy --type image/png < file.png # load image with correct MIME type
wl-paste -l # verify what's on clipboard
wl-copy --clear # teardown
WAYLAND_DISPLAY must be set (see above).
wayvnc does not autostart. Check first:
ss -tlnp | grep 5900
If nothing listening, start in a tmux pane:
export WAYLAND_DISPLAY=wayland-1 && export XDG_RUNTIME_DIR=/run/user/$(id -u)
wayvnc -L info 0.0.0.0 5900
Auth gotchas — config at ~/.config/wayvnc/config:
enable_auth=falsevncviewer host::5900) or switch Remmina to GTK-VNC backendCDP (Chrome DevTools Protocol) beats wtype + screenshots for browser testing — no focus handling, no pixel coordinates, direct DOM access.
Launch:
brave --remote-debugging-port=9998 --user-data-dir=/tmp/brave-test &
# verify: curl -s http://localhost:9998/json/version
Use a fresh --user-data-dir if a normal instance is already running — otherwise the flag is silently ignored.
Drive (Node 22+ stdlib WebSocket):
const targets = await fetch('http://localhost:9998/json/list').then(r => r.json());
const ws = new WebSocket(targets.find(t => t.type === 'page').webSocketDebuggerUrl);
// send({method, params}) helper — see unattended skill for full pattern
await send('Page.navigate', { url: 'http://localhost:9585' });
const { result } = await send('Runtime.evaluate', { expression: 'document.title' });
Methods you'll actually use:
Page.navigate — go to URLRuntime.evaluate — run JS, get result back (covers 90% of clicking/scraping)Page.captureScreenshot with clip — region screenshot without grimDon't connect to the user's logged-in browser profile.
Switch to unattended/solo mode for research and long-running tasks. Push through friction instead of pausing. Only invoked on explicit /unattended — never auto-load.
Writing, structuring, and maintaining project documentation. Auto-loaded when creating or updating docs, mkdocs sites, architecture pages, or any markdown documentation.
Visual and structural documentation review. Screenshots live rendered pages, passes to Gemini vision for readability/colour/diagram analysis. Separate pass for structure, acronyms, assumed knowledge, and factual accuracy (broken links, stale code refs, image/claim mismatches). Persona is a junior researcher with no domain context.
Junior-dev comprehension check on changed code. Per-file design review (role, assumed knowledge, wheel-reinvention), per-unit logic review (WHAT/WHY/LOGIC/SIMPLER), then a holistic diff review. Does NOT modify code.
Generate visual diagrams (flowcharts, matrices, architecture, org charts, quadrants, mind maps, etc.) with a render-and-review feedback loop. Use when the user asks for a diagram, chart, visualization, flowchart, or "visual". Picks the right tool for the data shape (Mermaid / D2 / raw SVG / Graphviz / quadrantChart) and iterates until the rendered output is actually readable.
Run long-running or interactive commands in a tmux session the user can watch and interact with. Use this for any command expected to take >30s, dev servers, watch modes, builds, tests, REPLs, migrations, or anything the user might need to Ctrl+C or send input to. Skip for quick one-shot commands (<30s, no prompts).