build-loop-native-ax-driver

name	build-loop:native-ax-driver
description	Use when the build needs to automate a macOS .app without touching the hardware cursor, or the user asks to "click through the app" or "test the UI headlessly". Drives running apps via Accessibility API; self-contained Swift binary — no IBR, Playwright, or Appium required.
version	1.1.0
user-invocable	false

Native AX Driver

Built-in capability, not a bridge. Build-loop ships its own Swift binary and Python launcher for navigating macOS apps via the Accessibility API. When this skill loads, the orchestrator can read the AX tree of any running .app and dispatch actions on individual elements while the user keeps using their cursor for something else.

Why this exists

Build-loop's Iterate phase needs to verify native macOS UI fixes the same way it verifies web routes. Previously, build-loop deferred to IBR's MCP for native automation, which made native verification IBR-dependent. This skill lifts the needed capability into build-loop directly: same Swift code lineage, same AX actions, no MCP hop, no plugin requirement.

Build-loop still routes to IBR as the primary verifier when skills/ibr-bridge/SKILL.md detects it, because IBR can add session management, baselines, and screenshots as auxiliary evidence. This skill is the native fallback path when IBR is absent.

What "cursor-free" means

The Swift binary calls AXUIElementPerformAction and AXUIElementSetAttributeValue on the resolved element directly. It does not invoke CGEventCreateMouseEvent, CGEventPost, or IOHIDPostEvent. Concretely:

Path	API	Cursor moves?
This skill	macOS Accessibility (AXUIElement*)	❌ Never
Virtual HID fallback (NOT bundled)	Quartz Event Services / IOKit HID	✅ Yes
AppleScript "click at {x,y}"	CGEvent under the hood	✅ Yes

When an element has no AX action handler (rare in SwiftUI/AppKit; common in custom Metal/canvas surfaces), the driver returns Element not found at path or an action-specific error rather than secretly falling through to mouse synthesis. This is intentional — a missing AX handler is usually a real accessibility bug in the app being tested, and papering over it hides the defect.

What the driver supports

Actions exposed by python3 scripts/native_driver.py action:

Action	What it does	macOS AX constant
press	Click a button, activate a control	kAXPressAction
setValue	Set a text-field / slider / picker value (requires --value)	AXSetAttributeValue(kAXValueAttribute, …)
increment	Step a stepper / slider up	kAXIncrementAction
decrement	Step a stepper / slider down	kAXDecrementAction
showMenu	Open a contextual / popup menu	kAXShowMenuAction
confirm	Default button activation in a dialog	kAXConfirmAction
cancel	Cancel-button activation in a dialog	kAXCancelAction
focus	Move keyboard focus to the element	AXSetAttributeValue(kAXFocusedAttribute, true)
scrollToVisible	Scroll an element into the visible viewport	AXScrollToVisible

Element targeting uses an integer index path from the main window root (e.g. 0,2,1 = first child → third child → second child). The path is returned by every scan element under the path key, so the typical loop is scan → match by identifier / title → use that element's path for action.

python3 scripts/native_driver.py launch is the deterministic launch + pid-capture step: it starts an isolated instance of a .app/bundle id and returns the new pid to scope every subsequent scan/action call to. See "Single-instance PID-scoped verification mode" below.

Files in this skill

skills/native-ax-driver/
├─ SKILL.md                                 (this file)
├─ scripts/
│  ├─ layout_fill.py                        (layout-fill / gap analyzer; stdlib only)
│  ├─ native_driver.py                      (Python launcher; stdlib only)
│  └─ test_native_driver.py                 (pure-helper tests; run with pytest)
└─ swift/bl-ax-driver/
   ├─ Package.swift                         (Swift 5.9, macOS 13+)
   └─ Sources/main.swift                    (~535 LOC, AX implementation)

The Swift binary compiles on first use to the consumer project's .build-loop/bin/bl-ax-driver — never inside the plugin tree. Subsequent runs reuse the cached binary; rebuild fires only if Sources/main.swift or Package.swift is newer than the cached binary.

Prerequisites

Prereq	How to check	Failure mode
swift on PATH	command -v swift	RuntimeError: \swift` not found on PATHfromensure_binary()`
Xcode CLT installed	xcode-select -p	First swift build fails with missing-SDK error
AX permission for parent process	python3 native_driver.py preflight	All AX calls return kAXErrorAPIDisabled; binary exits with the canonical "Accessibility permission required" message
Target app running	python3 native_driver.py apps	findMainWindow returns nil; binary exits with "No windows found for pid …"

The parent process needing AX permission is whichever process invoked Claude Code (Terminal, iTerm, VS Code, etc.). The driver binary itself does not need to be in the AX list — permission inherits from the parent.

Operational protocol

Pre-flight (run once per build-loop run with a native target)

python3 ${CLAUDE_PLUGIN_ROOT}/skills/native-ax-driver/scripts/native_driver.py preflight

Exit codes: 0 AX granted · 2 AX missing · 1 osascript missing.

If 2, surface to Iterate as a blocker rather than retrying — the user has to grant permission once in System Settings; build-loop cannot do that itself.

Single-instance PID-scoped verification mode

This is the documented standard for native UI verification whenever other instances of the target app may be running — including the user's own. Do not skip verification in that situation, and do not target by --app <name> — --app matches by substring against every running process's name, so it can resolve to (and drive) the user's window instead of the isolated one being verified.

The Swift driver already scopes every AX call to one process: it resolves the target via AXUIElementCreateApplication(pid) (swift/bl-ax-driver/Sources/main.swift:264) and only ever walks that process's AX tree. scan --pid <pid> and action --pid <pid> were always safe to run alongside ambient instances — the missing piece was a deterministic way to launch a fresh, isolated instance and know for certain which pid is it. launch closes that gap:

1. Launch an isolated instance under a private state dir, forcing a brand-new process (open -n) and (optionally) ignoring any saved window/scene state (-F via --fresh) so it never resumes into the user's prior session:

   python3 .../native_driver.py launch --app-path /Applications/MyApp.app \
       --state-env-var ET_STATE_DIR --state-dir /tmp/myapp-verify-$$  --fresh
   

2. Read the captured pid from the JSON result — launch snapshots the running GUI pid set before launch and diffs it against the pid set after, so the returned pid is provably the new instance, never a guess. If the diff is ambiguous (zero or more than one new pid appeared), success is false and no pid is reported — treat that as a hard stop, not a fallback to name-matching:

   {"success": true, "pid": 55210, "bundle_id": "com.example.myapp", "state_dir": "/tmp/myapp-verify-1234", "fresh": true, "error": null, "next": "drive AX scoped to this pid: native_driver.py scan --pid 55210 / action --pid 55210 ..."}
   

3. Scan and drive using ONLY --pid <captured> for the remainder of the verification pass — never --app:

   python3 .../native_driver.py scan --pid 55210
   python3 .../native_driver.py action --pid 55210 --element-path 0,2,1 --action press
   

Because AX targeting is pid-scoped end-to-end, every action in step 3 is confined to the launched instance's AX tree — the user's other windows, and any other ambient instances of the same app, are never touched. This is safe to run with ambient instances live.

launch flags: --app-path (path to a .app) or --bundle-id (mutually exclusive, one required); --state-dir + --state-env-var (set an env var to a private state directory before launch, e.g. an app-specific ET_STATE_DIR); --fresh (pass -F to open); --arg (repeatable, extra argv passed to the launched app after --args); --timeout (seconds to wait for the new pid to appear, default 10.0). Exit codes: 0 success · 1 launch or pid-capture failed · 2 bad arguments.

Scan a running app

python3 .../native_driver.py scan --app "MyApp"      # by name (substring, case-insensitive)
python3 .../native_driver.py scan --pid 44330                # by pid

Stdout: WINDOW:<id>:<WxH>:<title> header followed by the window root's children as a JSON array of AXExtractedElement (role, subrole, title, identifier, value, enabled, focused, actions[], position, size, children[], path[]).

Analyze layout-fill / gap findings

# Analyze a saved Swift scan JSON array
python3 .../native_driver.py analyze-layout --from-file /tmp/ax-tree.json

# Accept raw scan stdout with the leading WINDOW:<id>:<WxH>:<title> header
python3 .../native_driver.py scan --app "Easy Terminal" \
  | python3 .../native_driver.py analyze-layout --stdin

# Re-scan a running app or pid and analyze the result
python3 .../native_driver.py analyze-layout --app "Easy Terminal"
python3 .../native_driver.py analyze-layout --pid 44330

analyze-layout catches the layout bug class where a content element renders narrow and centered inside a larger container, leaving large empty gutters that can be invisible in screenshots. It reads the existing Swift scan JSON only; no Swift change or rebuild is required.

Inputs:

• --from-file, --stdin, --pid, or --app are mutually exclusive.
• --threshold defaults to 0.12.
• --min-container-px defaults to 50.

The analyzer returns the bridge envelope shape from skills/ibr-bridge/SKILL.md:

{
  "status": "ran",
  "route": "native",
  "verifier": "native-ax-driver",
  "artifacts": ["stdin"],
  "verification": "native-ax-driver analyze-layout ran; found 1 layout-fill finding.",
  "findings": [
    {
      "severity": "warning",
      "category": "structure",
      "message": "layout-fill: AXSplitGroup [Main]: leading empty band 317px = 30% of container width 1074px (horizontal)",
      "finding": {
        "containerRole": "AXSplitGroup",
        "containerLabel": "Main",
        "axis": "horizontal",
        "emptyPx": 317.0,
        "emptyPct": 0.2951582867783985,
        "position": "leading",
        "containerWidth": 1074.0,
        "containerHeight": 700.0,
        "detail": "AXSplitGroup [Main]: leading empty band 317px = 30% of container width 1074px (horizontal)"
      }
    }
  ]
}

The Swift extractor emits absolute screen coordinates, which are correct for this analysis. Each computed value is an intra-container delta, so a constant origin offset cancels out: firstChild.min - container.min, container.max - lastChild.max, and band / container.extent.

Drive an element

# Press a button at AX index path 0,2,1 in pid 44330
python3 .../native_driver.py action --pid 44330 --element-path 0,2,1 --action press

# Type into a text field
python3 .../native_driver.py action --pid 44330 --element-path 0,4,0 \
    --action setValue --value "BUILD_LOOP_TEST"

# Open the popup menu of a NSPopUpButton
python3 .../native_driver.py action --pid 44330 --element-path 0,1,3 --action showMenu

Stdout JSON shape: {"success": bool, "action": "press", "error": "AXPress failed" | null}. Exit 0 on success, 1 on AX failure, 2 on bad arguments.

Resolve a name without AX permission

resolve and apps work without AX permission — useful for the orchestrator to confirm a freshly-launched app has actually started before the AX-gated operations.

python3 .../native_driver.py resolve --app "MyApp"
# {"pid": 44330, "name": "MyApp", "bundleIdentifier": "com.example.myapp"}

python3 .../native_driver.py apps
# [{"name": "Finder", "pid": 612, "bundleIdentifier": "com.apple.finder"}, ...]

Integration with build-loop phases

Phase	Use
Sub-step B Validate (Review) — when uiTarget kind = native-macos	Run: preflight → scan → drive critical-path actions → re-scan → diff.
Sub-step D Coverage gaps	Enumerate scan results; for any element with actions != [] and no corresponding repo-native render/interaction coverage, write a .build-loop/ux-queue/ entry with a proposed test step pinned by identifier (preferred) or title.
Phase 5 Iterate — .swift files under a macOS target	Re-launch the rebuilt .app (open -b <bundleId>); replay the failing element-path + action; if two consecutive iterations fail on the same element, escalate to root-cause-investigator with the AX path — common cause is a missing .accessibilityIdentifier(...) modifier.

When not to use this skill

• Web targets — use ui-validator and the host browser/screenshot tooling; this skill won't help.
• iOS simulator — the simulator runs on macOS, but interaction goes through idb ui tap, not direct AX (the simulator's AX surface is too noisy for path stability). See reference_idb_sim_tap.md.
• Drag-and-drop, hover-only effects, NSTrackingArea-driven UI — these need real CGEvent mouse events. Out of scope. If the feature is critical, fix the AX surface in the app under test (add .accessibilityAction { … }) rather than synthesizing mouse events.
• App not yet running — use launch (see "Single-instance PID-scoped verification mode" above) to start an isolated instance and capture its pid before driving, especially when other instances of the app may already be running. For a single-instance context where no ambient collision risk exists, a plain open -b <bundleId> (or open <path/to/.app>) followed by resolve also works.

Failure modes & recovery

Symptom	Likely cause	Fix
Binary missing after first install	swift build failed silently (sandbox rejection)	ensure_binary() retries with --disable-sandbox automatically; if both fail, install Xcode Command Line Tools
All AX calls return kAXErrorAPIDisabled	Parent process not in System Settings → Privacy & Security → Accessibility	Add the process; macOS 13+ requires a fresh launch after granting
Element not found at path after a UI change	Path indices shifted	Re-scan and look up the element by identifier again — paths are not stable across UI changes
AXPress failed on a clearly-clickable button	SwiftUI view missing .accessibilityAction	Treat as a real bug in the app under test, not a driver bug
iOS simulator AX tree is empty	Simulator was scanned in macOS mode	Pass --device-name to scope to the simulator window

Self-test

python3 ${CLAUDE_PLUGIN_ROOT}/skills/native-ax-driver/scripts/native_driver.py preflight
python3 ${CLAUDE_PLUGIN_ROOT}/skills/native-ax-driver/scripts/native_driver.py apps | head
# pick a running app, then:
python3 ${CLAUDE_PLUGIN_ROOT}/skills/native-ax-driver/scripts/native_driver.py resolve --app "Finder"
python3 ${CLAUDE_PLUGIN_ROOT}/skills/native-ax-driver/scripts/native_driver.py scan --app "Finder" | head -40

If all four print sensible JSON, the skill is healthy.

Provenance

Swift extractor ported from interface-built-right/src/native/swift/ibr-ax-extract/Sources/main.swift. The two copies started identical; build-loop's copy can drift independently and is not auto-synced. If a future bug is fixed in IBR's copy, port it manually and bump this skill's version.

Layout-fill / gap analysis in scripts/layout_fill.py is ported from interface-built-right/src/native/layout-fill.ts v1.4.0. Its fixtures mirror interface-built-right/src/native/layout-fill.test.ts, especially the Easy Terminal regression case: a 440px terminal centered in a 1074px container yields emptyPx == 317, emptyPct ~= 0.2952, and position == "leading". The analyzer is pure Python and has no live AX dependency.