macos-background-app-control

name

macos-background-app-control

description

Screenshot or click a macOS app that's behind your foreground window without stealing focus, moving the cursor, or interrupting the user. Covers occluded-window screenshots via CGWindowListCreateImage, native-control clicks via Accessibility (AX) and CGEventPostToPid, the briefly-raise-then-restore fallback for WKWebView/Tauri/Electron content, locked/display-state limits, and the honest limits of each. Use when verifying a GUI end-to-end while sharing a Mac with the user.

macos-background-app-control

Verified techniques for screenshotting and controlling a macOS app while it sits occluded behind the user's foreground window, on the same machine the user is working on. The goal: catch the app's state without the cursor moving and without the user's frontmost app losing focus.

What works, what doesn't, and the honest fallback when the app is a WebView host (Tauri/Electron/Safari).

When to use

• You need to verify a GUI end-to-end while sharing a Mac with the user.
• You want to capture screenshots of a window that is not frontmost (behind Chrome, another Terminal, etc.).
• You're driving a native AppKit/SwiftUI app and want zero visible cursor jumps.
• You're driving a Tauri/Electron app and need to know which automation paths actually work versus which only appear to work.

Do not use when:

• You can ask the user to click. That's the lowest-cost verification.
• You're driving inside a webview and the app exposes a debug IPC. Use the app's own eval_js command instead; see Pitfalls.
• A remote session or KVM is black, locked, or cannot show the desktop. Use macos-remote-access-troubleshooting for the transport/display problem, then return here once the local capture path is healthy.

Prerequisites

• macOS (verified on macOS 15+).
• cliclick (brew install cliclick) — used only for reading cursor pos as a no-op sanity check.
• uv and Python 3.13 (brew install uv) for the pyobjc venv. Path: /tmp/bgctl/.venv.
• The current Terminal/iTerm has Accessibility permission for the agent shell (System Settings → Privacy → Accessibility). Without this, AX returns empty and CGEventPost* silently no-ops. Important — the grant is scoped to the responsible process, not to the literal terminal app you can see. If your agent shell is parented to a long-lived tmux server started by a LaunchAgent (common for headless/SSH-driven agent setups), the responsible process is tmux (or whatever LaunchAgent launched it) — granting Terminal.app/iTerm/Termius does nothing. Verify with ps -o ppid= -p $$ repeatedly until you hit launchd; the topmost non-launchd ancestor is the binary that needs the grant. Empirically: run the Verification block below — if set frontmost of process "Finder" actually brings Finder forward, your grants are working regardless of which app appears in System Settings.
• Screen-recording permission for screencapture of other apps' windows.
• Accessibility and Automation (Apple Events) are two separate TCC grants — don't conflate them. Direct AX (AXUIElementCopyAttributeValue, AXIsProcessTrusted) needs only kTCCServiceAccessibility. Driving System Events over AppleScript (tell app "System Events" to …) ALSO needs kTCCServiceAppleEvents for the responsible process. A tmux-LaunchAgent commonly has Accessibility granted but Automation denied with no interactive prompt to approve it — and in that state the System Events route hangs (it does not error). So verify grants with a direct AX read (below), and prefer direct AX over System Events for foregrounding. If an osascript "System Events" call hangs, suspect missing Automation, not missing Accessibility.

Bootstrap once per machine:

mkdir -p /tmp/bgctl && cd /tmp/bgctl
uv venv --python 3.13 --quiet
uv pip install --quiet pyobjc-framework-Quartz pyobjc-framework-Cocoa pillow

Quick start

The capture_app.sh wrapper is the right entry point for the common case ("screenshot an app window autonomously"). The default flow does NOT raise the target — verified empirically that occluded windows (Tauri/Electron and native AppKit) capture cleanly via CGWindowListCreateImage without stealing focus, AND that this works even when the screen is locked, as long as the window was composited before lock. The --raise flag is for the unlocked-screen + AX-hidden-window fallback.

# Default — no focus theft. Works for occluded + composited-but-locked.
scripts/capture_app.sh "MyApp dev - feature-x" /tmp/out.png

# Fallback when the default returns blank AND screen is unlocked. Usually
# means the window is AX-hidden (Cmd-W on Tauri leaves the process alive
# but closes the window).
scripts/capture_app.sh --raise "MyApp dev - feature-x" /tmp/out.png

# Lower-level building blocks (for clicks, AX traversal, or custom flows):

# Find target window. Returns pid, winid, bounds for every match.
.venv/bin/python3 scripts/find_window.py "MyApp dev - feature-x"

# Capture by winid (CGWindowListCreateImage path; works occluded + locked).
.venv/bin/python3 scripts/capture_window.py <WINID> /tmp/out.png

# Click a NATIVE button at screen-point (x,y) without moving cursor.
.venv/bin/python3 scripts/click_pid.py <PID> <X> <Y>

When a needle matches multiple windows (e.g., two dev instances are running), capture_app.sh iterates candidates in largest-first order and returns the first one that passes pixel-validation — so a stale/uncomposited sibling won't poison the result.

Exit codes: 0 success, 2 --raise requested under lock, 3 no matching window, 4 CGWindowListCreateImage / screencapture failure, 5 all candidates returned blank/uniform pixels (try --raise if unlocked, or unhide the window), 6 pyobjc venv missing.

Scripts live in this skill's scripts/ dir; copy them to /tmp/bgctl/ or run from wherever.

What works (verified)

Screenshot a background/occluded window

CGWindowListCreateImage with kCGWindowListOptionIncludingWindow pulls from the WindowServer's backing store. No matter where the window is in z-order — even fully occluded — you get a clean capture.

img = Quartz.CGWindowListCreateImage(
    Quartz.CGRectNull,
    Quartz.kCGWindowListOptionIncludingWindow,
    winid,
    Quartz.kCGWindowImageBoundsIgnoreFraming | Quartz.kCGWindowImageNominalResolution,
)

• kCGWindowImageNominalResolution gives a 1x (points-sized) PNG. Skip it for native 2x Retina.
• kCGWindowImageBoundsIgnoreFraming strips drop shadow / NSWindow chrome around the bitmap.

CLI alternative: screencapture -x -l<WINID> file.png. Same source bitmap, includes Retina scaling and drop shadow by default.

macOS 26 (Tahoe) regression: CGWindowListCreateImage now nil-returns even with a Screen Recording grant — Apple is retiring the legacy CGWindowList capture path for ScreenCaptureKit. screencapture -l<WINID> (SCK-backed) still works and captures occluded windows. capture_window.py falls back to it automatically when the Quartz call returns None; on Tahoe, prefer screencapture -x -o -l<WINID> file.png directly.

Caveat — the SCK fallback is inconsistent for occluded windows on Tahoe. The same machine/session produced both real content (e.g. a real Finder window) and blank, zero-spread, wrong-sized frames (e.g. kilo → 5120×60, a Finder helper → 128×128, both extrema spread 0) from screencapture -l on different occluded windows. Don't trust the exit code — always validate the PNG's per-channel extrema spread (max-min > 30 on some channel), and if it's blank, briefly raise the target and re-capture. Treat "silent occluded capture" as best-effort on macOS 26, not guaranteed.

Locked-screen rule of thumb

A locked Mac is capture-only. CGWindowListCreateImage can capture an already-composited window backing store while locked, but clicks/AX/HID do not reliably navigate or change app UI, and fresh Tauri/WKWebView windows may be black or stale if they have no valid backing store yet. Headless Playwright/Chrome screenshots are different: they render offscreen inside the browser process, so they can still validate browser-renderable UI while the Mac is locked.

Detect lock state deterministically — don't guess

Before assuming "the screen is probably locked," read the session state. The dependency-free check is ioreg (no pyobjc needed):

ioreg -n Root -d1 -a | grep -q CGSSessionScreenIsLocked && echo LOCKED || echo UNLOCKED

CGSSessionScreenIsLocked=<true/> present ⇒ locked; its absence (with kCGSSessionOnConsoleKey=<true/>) ⇒ unlocked. Confirm with a real capture: screencapture -x /tmp/x.png yields actual pixels only when unlocked (a locked screen gives a black/blank frame). The pyobjc equivalent (Quartz.CGSessionCopyCurrentDictionary().get('CGSSessionScreenIsLocked')) is cleaner but Quartz is often not installed in the system/homebrew python3, so prefer ioreg. Run this check before telling the user you can't verify visually — locked is a fact you can read, not a state you have to assume.

Blank capture while the app is live

If a DOM/debug bridge, app log, or backend probe proves the app is live but every WindowServer capture is blank, white, black, or stale, stop retrying capture variants first. That signature usually means the display/session layer is not providing current pixels: the Mac may be locked, the window may not have a valid backing store yet, Screen Recording may be missing for the running bundle id, or the VNC/KVM path may be unavailable. Use macos-remote-access-troubleshooting for the remote/display branch before adding WebView screenshot hooks.

Find a window's CGWindowID and PID

opts = Quartz.kCGWindowListOptionAll | Quartz.kCGWindowListExcludeDesktopElements
wl = Quartz.CGWindowListCopyWindowInfo(opts, Quartz.kCGNullWindowID)
for w in wl:
    if owner_name_matches and w.get('kCGWindowLayer') == 0 and w.get('kCGWindowIsOnscreen'):
        winid, pid, bounds = w['kCGWindowNumber'], w['kCGWindowOwnerPID'], w['kCGWindowBounds']

Filter on kCGWindowLayer == 0 (real app windows) and prefer kCGWindowIsOnscreen true. Tauri apps emit many off-screen helper windows that pass the name filter.

Read window pose & native chrome via Accessibility (AX)

osascript -e 'tell app "System Events" to tell process "MyApp dev - feature-x" to ¬
  return (get position of window 1) & (get size of window 1) & (title of window 1)'

AX exposes: window position, size, title, the menu bar (clickable), traffic-light buttons, and AppKit-native UI elements (AXButton, AXTextField, AXMenuItem).

Click on native AppKit content without moving the cursor

CGEventPostToPid(pid, event) delivers the event directly to the target process's event queue. The system cursor does not move; the frontmost app keeps focus.

src = Quartz.CGEventSourceCreate(Quartz.kCGEventSourceStateHIDSystemState)
pt = (x, y)
for kind in (Quartz.kCGEventMouseMoved,
             Quartz.kCGEventLeftMouseDown,
             Quartz.kCGEventLeftMouseUp):
    ev = Quartz.CGEventCreateMouseEvent(src, kind, pt, Quartz.kCGMouseButtonLeft)
    Quartz.CGEventPostToPid(pid, ev)
    time.sleep(0.03)

Same pattern with CGEventCreateKeyboardEvent(src, keycode, True/False) for keystrokes — works for any control that has key focus inside the target process.

What doesn't work (and why)

CGEventPostToPid clicks on WKWebView / Tauri / Electron content

The event reaches the target process, but WebKit (and Electron's Chromium) discards synthetic mouse events that didn't come through the HID dispatch path. Result: the click is silently dropped. Verified empirically with a Tauri 2 app — events delivered, no hit-test fires, no React handler invoked.

Symptom: scripts report clicked pid=N at (x,y), but the UI state is unchanged in the next screenshot.

CGEventPost (HID tap) while window is occluded

CGEventPost(kCGHIDEventTap, ev) does go through normal hit-testing and reaches WebKit fine — but only against the frontmost window at that screen coordinate. If the target window is behind Chrome at (2039, 298), the click goes to Chrome, not to the target.

You can't have "real" hit-testing AND occluded targeting at the same time on macOS. Pick one.

AX on Tauri/Electron webview content

System Events traversal of a Tauri window yields a single opaque AXGroup child. The WKWebView's DOM is invisible to AX. Buttons, text fields, menus inside the React UI are unreachable via AppleScript. Same for Electron (Chromium WebContents is not AX-exposed by default).

WebKit Remote Inspector via TCP port

Tauri (wry) does not start a remote-inspector TCP server. The WebView is inspectable via Safari's Develop menu (Bonjour-discovered XPC), but there is no localhost port to connect to with puppeteer/safaridriver/CDP clients. Don't go looking for one — verified by lsof -i -P -p <tauri_pid>, no inspector port.

How to drive webview content anyway

Three options, in order of preference. Note: option 2 is only needed when the app doesn't expose option 1 — for screenshots of Tauri/Electron windows the default no-raise path in capture_app.sh works regardless of occlusion.

1. Ask the app to expose an eval-JS bridge. For a Tauri/Electron app you control, add a debug-only path that takes a JS string and runs it inside the WebView via webview.eval (Tauri) / webContents.executeJavaScript (Electron). The cleanest shape is a loopback TCP listener bound to 127.0.0.1:<auto> at startup, with the chosen port + a per-instance random auth token written to a discovery file the agent can find. Gate the entire module on cfg(debug_assertions) (or the equivalent dev-only build flag) so release binaries can't expose it. Three security details matter and they're non-obvious enough to be worth listing:

   • Loopback is cross-user reachable on macOS — any local user can hit 127.0.0.1:<port>. The bridge needs a per-instance auth token (compared in constant time, not ==), and the discovery file/dir must be mode 0600/0700.
   • Bound each request read so a hung peer can't OOM the app (cap bytes AND wrap the read in a timeout — slowloris defeats a byte cap alone).
   • Per-PID discovery file (e.g. ~/.<app>/run/debug-eval-<pid>.json) so multiple dev instances don't clobber each other; the agent globs and kill -0s to pick a live one. Unlink-then-O_EXCL create the file so a stale loose-perm file from a recycled PID can't keep its old mode.

   Once the bridge exists, one IPC command covers every button and read-back — there's no per-button glue. Ask the frontend team to add data-testid="..." attributes to anything you want to drive so selectors stay stable across CSS/layout changes (same convention Playwright/Cypress use). The wired-in agent-side client (the script that reads the discovery file and sends {token, js}) is project-specific — see your repo's AGENTS.md for its path and usage.

   # Generic shape — substitute your project's eval client:
   eval.sh 'document.querySelector("[data-testid=share]").click()'
   eval.sh 'JSON.stringify({url: location.href, title: document.title})'
   

2. Briefly bring the window to front, click via HID, return focus. Necessary for clicks into WebView content when option 1 isn't available (CGEventPostToPid is silently dropped by WebKit, AX can't see DOM elements). Visual disruption is short (~500ms) but real. Critical: always capture $PREV before raising the target, and restore to $PREV — never hardcode a process name like "Termius" or "iTerm" as the restore target. Hardcoded restore targets silently fail when the previously-frontmost app is something else (System Settings, Chrome, the user's editor), leaving the target window stuck in front and creating a false signal that AX permissions are broken when they aren't.

   PREV=$(osascript -e 'tell app "System Events" to get name of first application process whose frontmost is true')
   osascript -e 'tell app "System Events" to set frontmost of process "MyApp dev - feature-x" to true'
   sleep 0.3
   scripts/click_hid.py <X> <Y>
   osascript -e "tell application \"$PREV\" to activate"
   # Verify restore landed:
   osascript -e 'tell app "System Events" to get name of first application process whose frontmost is true'
   

3. Don't drive — verify visually. Capture the background window (no raise needed), ask the user to click. Most automated verification only needs the screenshot half of the loop.

Pitfalls

• Accessibility permission required — and attaches to the responsible process, not the visible terminal. Without it, AX queries return nothing and CGEventPost* does nothing — silently. The grant target is the binary at the top of your process ancestry (excluding launchd), which for tmux-LaunchAgent setups is tmux itself — NOT the terminal app you SSH'd in through. Always confirm grants empirically by running the Verification block, not by trusting which app appears toggled in System Settings.
• Never hardcode the restore-focus target. osascript ... tell app "Termius" to activate silently no-ops when Termius isn't the previously-frontmost app, and the failure is invisible — leading to false conclusions like "AX permission must be missing." Always capture $PREV first, restore to $PREV, and verify the restore landed by re-reading frontmost. Same goes for assuming the agent shell is running under a specific terminal: trace ps -o ppid= to be sure.
• Tauri/Electron WebView windows DO capture cleanly while occluded — the earlier "WebView occluded-paint suspension" claim was wrong. Verified empirically (June 2026): CGWindowListCreateImage returns real content for occluded Tauri 2 / wry windows without raising. The actual failure mode that produces blank/black PNGs is the window being AX-hidden (tell process X to get count of windows returns 0) — usually because the user pressed Cmd-W, which destroys the Tauri window but leaves the process alive. Distinguish via AX windows count before assuming occlusion is the problem. macOS 26 (Tahoe) caveat: CGWindowListCreateImage now nil-returns regardless (see the Tahoe regression note above), and the screencapture -l fallback is inconsistent for occluded windows — so on Tahoe, "captures cleanly while occluded" no longer holds; validate pixel spread and be ready to briefly raise.
• activate / set frontmost can silently no-op under modern focus-steal prevention. On macOS 26, NSRunningApplication.activateWithOptions_(NSApplicationActivateIgnoringOtherApps) was observed NOT to foreground the target (frontmost stayed on the prior app). Prefer setting kAXFrontmostAttribute = True on the app's AX element, then verify with NSWorkspace.frontmostApplication() and retry. After a raise, give the window a settle delay before a HID click — an immediate MouseMoved+Down+Up may register only as a hover (tooltip) if the raise hasn't landed; verify frontmost before posting the click.
• A child webview breaks the debug-eval bridge's window lookup. Opening a second webview (e.g. a browser/canvas panel) demotes a Tauri WebviewWindow to a multi-webview Window, so app.get_webview_window("main") returns None and the bridge goes dark exactly when you're driving an artifact. Fall back to app.webviews() (a HashMap<label, Webview>) and select by label or first entry.
• Locked screen is partial NO — CGWindowListCreateImage still works for windows composited before lock. Verified empirically (June 2026) with help from a parallel agent: CGWindowListCreateImage retrieves the WindowServer backing store, which persists for any window that was actually drawn while the screen was unlocked — including Tauri/Electron WebViews. screencapture -l <winid> fails outright with "could not create image from window" under lock, and screencapture -x full-screen returns a valid-shape PNG with all-zero pixels (silent failure). For locked-screen captures, always use the pyobjc CGWindowListCreateImage path, never the screencapture CLI. Windows that were never composited (AX-hidden, freshly opened invisible, etc.) still return blank — there's no way around that without unlocking and raising.
• Foregrounding, clicking, and typing do not work while the screen is locked — set frontmost is a no-op, CGEventPostToPid is dropped, AX queries return stale state. The --raise flag of capture_app.sh refuses fast under lock for this reason.
• unique_sample_count > N is a bad blank-frame detector — dark UIs (dark-mode Tauri/Electron) trip it. Use channel extrema spread instead: a real frame will have at least one RGB channel where max - min > 30; a blank frame has all channels at (0, 0). capture_app.sh uses extrema-based detection.
• Tauri has many helper windows. Filter find_window.py results by kCGWindowLayer == 0, onscreen == true, and largest bounds. The main React window is usually the only one matching all three.
• CGWindowID is not stable across app restarts. Re-resolve each session. PIDs change too.
• screencapture -l includes drop shadow + Retina 2x. Use the pyobjc route when you want exact window-content pixels (1x, no shadow).
• Cursor restore via CGWarpMouseCursorPosition is visible for ~1 frame. If you need truly invisible operation, use CGEventPostToPid and accept the WebView limitation.
• AppleScript set frontmost doesn't always work first-call. Some apps need two calls or activate instead. Sleep 100ms between calls.

Verification

Quick smoke test once you've installed the venv:

cd /tmp/bgctl
# 1. Find any app's main window
.venv/bin/python3 scripts/find_window.py Finder
# 2. Capture without bringing Finder forward
.venv/bin/python3 scripts/capture_window.py <WINID> /tmp/finder.png
file /tmp/finder.png   # should be a non-trivial PNG
# 3. Confirm cursor and frontmost are unchanged
cliclick p; osascript -e 'tell app "System Events" to get name of first application process whose frontmost is true'

References

• Apple: CGWindowListCreateImage
• Apple: CGEventPostToPid
• WebKit Remote Inspector protocol notes: WebKit/Source/WebInspectorUI (no public macOS API for external connections without XPC entitlements)