| name | antigravity-computer-use |
| description | Use Claude Code to control Google Antigravity IDE on macOS via Peekaboo. Explicitly supports targeting either the Agent Manager window or the Editor window (worktree/editor titles), plus auto routing. Covers workspaces, conversations, Knowledge/Browser/Settings panels, keyboard shortcuts, model selection, planning modes, artifacts, and menu bar. Trigger when the user asks to automate, click, send messages to, or interact with the Antigravity app, or to choose Manager vs Editor. |
Antigravity Computer Use (macOS via Peekaboo)
Google Antigravity is an AI-powered, agent-first IDE (a modified VS Code fork) by Google. It has two primary views: Agent Manager ("Mission Control") and Editor (VS Code-like). This skill controls it via Peekaboo's native macOS accessibility API.
For the core screenshot-decide-act loop, see the claude-code-computer-use skill.
Official docs: https://antigravity.google/docs
CLI tool: agy
MANDATORY: Allow Dialog Check — After EVERY Screenshot
Mandatory lesson log (most recent first)
- 2026-04-05: Sidebar spinner is UNRELIABLE for detecting active conversations — spinner ◌ persists even when conversation is dead/idle; MUST cross-check with red stop square in bottom-right of chat input area AND the status bar model indicator; three failure modes: (1) zombie spinner = spinner + no red square + send arrow → conversation dead, send "Continue"; (2) frozen-active = spinner + red square + timestamp >1h → stuck in loop, interrupt and redirect; (3) actually working = spinner + red square + timestamp <30m → leave alone; status bar shows "Claude Opus 4.6 (Thinking)" with red ■ = active, "Gemini 3.1 Pro (High)" with send → = idle
- 2026-04-05: VS Code "Remote Window" dialog can open in editor windows (shows SSH/container options) — blocks agent silently; press Escape to close, then verify red-stop still present to confirm agent resumes
- 2026-04-05: win 902 gutter FP — x≈2 (retina x=4) at left edge is always a VS Code diff/edit gutter indicator; add
cx//2 > 30 filter (logical x > 30) to PIL scan to skip all left-edge detections in editor windows
- 2026-04-05: Manager window IDs change every Antigravity relaunch — ALWAYS re-run
peekaboo window list at session start; never cache window IDs across restarts; current session IDs: Manager=1191, worldai_claw editor=902
- 2026-04-05: Clicking conversation sidebar items by elem_N sometimes doesn't switch view when another conversation is actively generating — use coordinate click at approximate sidebar y position instead; if coord also fails, try a different y offset (items are ~25px apart logical)
- 2026-04-05: "New Conversation" (elem_22) click in Manager sometimes routes message to wrong conversation (the one already selected, not the new one) — after clicking New Conversation, verify the title bar changed before pasting; use
agy chat --mode agent --reuse-window "prompt" from the target workspace directory as alternative
- 2026-04-05: Opening a workspace folder from Manager (clicking "Open Folder" or workspace item) closes the Manager floating window and opens an editor window with inline Agent panel — the Manager window_id becomes invalid; use "Open Agent Manager" button or "Switch to Agent Manager" in the editor to reopen
- 2026-04-05: Editor Agent panel text field coords: window at x=-28,y=-998,w=1189,h=857 → text input at logical x≈987, y≈390 → screen (959,-608); use these as starting coords, adjust if click misses
- 2026-04-05: Manager conversation input at screen (860,-230) when Manager is at x=429,y=-1005,w=1437,h=849 — reliable across sessions as long as Manager hasn't been resized
- 2026-04-05: Red stop square (active generation indicator) in editor Agent panel appears at retina x≈2312-2331, y≈1608-1627 (for win 902 1189x857) = logical x≈1156, y≈804 — bottom-right of Agent panel; filter right-side only (x > 80% of image width) to avoid false positives from left-side gutter
- 2026-04-05: win313/319/334 FPs — light-blue code highlights [136,201,245] and blue UI elements in non-Manager editor windows persist after click; muted blue with G>130 is NOT a button; add all persistent coords to FP_SET after first click-persist confirmation; check G<165 threshold before counting as real Allow
- 2026-04-05: win319 ($USER—list window at 186,-990) generates multiple stacked Allow dialogs — each click reveals next one; scan persists showing new coords; click each until CLEAN; FP after 2nd persistence
- 2026-04-04: Tab bar FP — Antigravity's blue active-tab indicator spans the full window width at retina y=0-120 (logical y=0-60); skip rows < 120 retina to avoid detecting tab bar blue as Allow dialogs; bottom status bar also fires at retina y>1600 with muted blue [70,127,161] — skip if B<165 or if G>130 (muted blue is not Allow button)
- 2026-04-04: PIL scan thresholds were too strict — B>150, R<150, B>R+50 missed Allow button with edge pixels having R up to 197; real Allow button had B
203, R39 mean but gradients/edges up to R=197; CORRECT thresholds: B>130, R<180, B>R+30, G<B; also scan from row 0 (not h//2) since dialogs can appear anywhere; min pixel threshold raised to 200 to compensate for looser thresholds
- 2026-04-04: MANDATORY — keep 3 conversations active at all times; check at every cycle end; if <3 active, click idle 24h conversations in reverse-cron order and send continuation tasks until 3 are generating (confirmed via red stop square, not just A11y spinner which lags)
- 2026-04-04: PIL scan MUST cover ALL Antigravity editor windows dynamically — hardcoding only wt-task2 (WIN_X=186) missed Allow dialog in worldai_claw editor (WIN_X=-28, WIN_Y=-998) for >10 min; fix: run
osascript to get ALL window positions each cycle, then scan each one; never hardcode a single window's bounds as the only scan target; the worldai_claw conversation is a SEPARATE editor window from wt-task2
- 2026-04-04: "Allow directory access to /Users/.../worldai_claw/.github/workflows?" dialog has TWO blue buttons ("Allow Once" + "Allow This Conversation") — both are valid click targets; the PIL scan still works because it finds the rightmost blue cluster; click the rightmost detected button (Allow This Conversation) for persistent permission
- 2026-04-03: FP at (1458,846) — persists after click; blue [99,147,199] "Open Editor" link text in Manager top bar; add to FP set
- 2026-04-03: Teal/cyan VS Code filter — add
G > B rejection to PIL scan; [114,199,177] and similar teal pixels pass the B>R+50 filter but are VS Code editor syntax colors, not Allow buttons; check center pixel: if arr[cy2,cx2][1] > arr[cy2,cx2][2] (G>B) then skip; also add FP coords (1129,672) and (1115,674) to FP set for this region
- 2026-04-03: FPs at (1322,695), (1127,897), (1132,934) — persistent after click; (1322,695) in right-panel conversation text area, (1127,897) and (1132,934) near bottom ~y=900-940 which is the conversation input edge; add all to FP set
- 2026-04-03: FP at (968,887) — near-bottom of Manager window, persists after click; likely blue text in conversation input area or VS Code bleed-through; add
abs(scr_cx-968)<30 and abs(scr_cy-887)<30: continue to FP filter
- 2026-04-03: FP at (1084,544) — VS Code editor blue Python import/type keywords bleeding through Manager window at mid-height; persists after multiple clicks; add
abs(scr_cx-1084)<30 and abs(scr_cy-544)<30: continue to FP filter
- 2026-04-03: VS Code editor window overlapping Manager causes FP loop — editor Python syntax highlighting (blue keywords: import, from, def) triggers persistent detections at (1093,706), (1226,761), (1342,634); all in x=1050-1400, y=630-780 zone; add FP guards for each and raise min x to 1400+ when editor is overlapping, OR close editor before PIL scan
- 2026-04-03: blue hyperlink in Manager conversation panel at (964,626) — PR URL in text, NOT a dialog; add
abs(scr_cx-964)<50 and 580<scr_cy<680: continue to FP filter; or better: raise minimum x for Manager to 1050 since real Allow dialogs appear in right portion of conversation panel
- 2026-04-03: CRITICAL coordinate bug —
screencapture -x captures FULL SCREEN at retina 2x resolution; correct screen coord is scr_cx = cx//2 and scr_cy = cy//2 (NO WIN_X/WIN_Y offset); WIN offset is wrong because cx,cy are already full-screen pixel positions; the old formula scr_cx = WIN_X + cx//2 was off by 100px and caused all Allow clicks to miss; update all PIL scans to use bare cx//2, cy//2; window bounds check still valid with logical coords: if not (WIN_X < cx//2 < WIN_X+WIN_W and WIN_Y < cy//2 < WIN_Y+WIN_H): continue
- 2026-04-03: PIL scan threshold should be 200+ pixels (not 50) to avoid artifact icon badges (small ~50px blue circles on file/code action icons in conversation); add window bounds check:
if not (WIN_X < scr_cx < WIN_X+WIN_W and WIN_Y < scr_cy < WIN_Y+WIN_H): continue to exclude detections outside Antigravity window
- 2026-04-03: textField LABEL in peekaboo A11y shows the last SENT message content (aria-label pattern) — not current input. Check VALUE field to see what's currently typed. Empty label="text entry area" = field is empty.
- 2026-04-03: test keystrokes sent via
osascript -e 'tell application "System Events" to keystroke "..."' ACCUMULATE in text field — always run cmd+a+delete FIRST, then paste, to ensure field is clean; osascript with colons in strings can fail with syntax errors — break message into pieces without colons
- 2026-04-03: peekaboo paste does NOT replace selected text in Antigravity's web-rendered textarea — use osascript to clear field:
osascript -e 'tell application "System Events" to tell process "Antigravity" to keystroke "a" using {command down}' && osascript -e 'tell application "System Events" to tell process "Antigravity" to key code 51' then peekaboo paste; this reliably clears and replaces field content
- 2026-04-03: blue file-path hyperlinks in agent output appear as FPs at x≈1067, y≈660-720 — add
abs(scr_cx-1067)<50 and 660<scr_cy<720 to FP filter; prefer bottom-strip scan (y > 85% of h) for real Accept All buttons which have 500+ px
- 2026-04-03: persistent blue FP at (1247,689) — blue code/GraphQL text in conversation panel, same coord after click → add
abs(scr_cx-1247)<30 and abs(scr_cy-689)<30 to FP filter; consider raising x threshold from 1200→1250 to eliminate this zone
- 2026-04-03: persistent blue FP at (1579,782) — same coord fires after click, does not dismiss → add
abs(scr_cx-1579)<30 and abs(scr_cy-782)<30 to FP filter
- 2026-04-03: blue diff text / code in conversation panel triggers persistent FP at (1380,537) — add
abs(scr_cx-1380)<30 and abs(scr_cy-537)<30 to FP filter; if same coord fires 2+ times after clicking, it's a FP
- 2026-04-03: blue terminal text (worktree paths, highlighted lines) triggers FP at (1256,546-570) — add
abs(scr_cx-1256)<30 and 540<scr_cy<580 to FP filter; real Allow dialogs have 500+ blue pixels, not terminal text
- 2026-04-03: Manager conversation textField (elem_68) only appears in A11y after clicking at screen coords (480,905) — NOT at y=940+ (which hits Add context or model selector popups); click (480,905) first, THEN get fresh snapshot and click elem_68 to focus; TF label shows current text content (updated as you type)
- 2026-04-03: elem_30 input field accumulates unset text from multiple paste attempts across monitoring cycles → always click elem_30, osascript cmd+a+delete, THEN paste new message; inspect elem_30 label to detect residual text from prior sends
- 2026-04-03: (1746,849) FP — just outside abs(x-1707)<40 and y<840 guard (y=849>840); extend y threshold to y<860 to catch this zone:
if abs(scr_cx-1707)<40 and scr_cy<860: continue
- 2026-04-03: x≈1707 full-column FP zone filtered real Accept All/Allow button at (1697,856) → FP zone must include y<840 guard; real button appears at y≈856
This is the #1 cause of stuck agents. Check EVERY time you take a screenshot.
After any screencapture, immediately run this PIL scan before doing anything else:
check_allow_dialog() {
local IMG="$1" WIN_X="$2" WIN_Y="$3"
python3 -c "
from PIL import Image
import numpy as np, os, sys
img = Image.open('$IMG')
arr = np.array(img)
h, w = arr.shape[:2]
for row_start in range(0, h, 20):
region = arr[row_start:row_start+40, :]
blue_mask = (region[:,:,2] > 130) & (region[:,:,0] < 180) & (region[:,:,2] > region[:,:,0] + 30) & (region[:,:,1] < region[:,:,2])
if np.sum(blue_mask) > 200:
coords = np.where(blue_mask)
xs = coords[1]
xs_sorted = np.sort(np.unique(xs))
gaps = np.diff(xs_sorted)
big_gaps = np.where(gaps > 15)[0]
right_start = xs_sorted[big_gaps[-1]+1] if len(big_gaps) > 0 else xs_sorted[int(len(xs_sorted)*0.6)]
cx = int((right_start + xs_sorted[-1]) / 2)
cy = int(np.mean(coords[0][xs >= right_start])) + row_start
scr_cx = int(os.environ.get('WIN_X',0)) + cx//2
scr_cy = int(os.environ.get('WIN_Y',0)) + cy//2
wx = int(os.environ.get('WIN_X',0))
# FP zones (worldai_claw window x=70,y=81,w=1684,h=900):
if scr_cx < wx+150: continue # gutter FP
if abs(scr_cx-1434)<40 and abs(scr_cy-629)<40: continue # Review Changes link FP
if abs(scr_cx-1707)<40 and scr_cy < 840: continue # sidebar FP (y<840 only — real Allow/Accept button appears at y≈856)
if abs(scr_cx-1504)<30 and abs(scr_cy-769)<30: continue # PR URL link FP
print(f'ALLOW:{scr_cx},{scr_cy}')
break
" WIN_X=$WIN_X WIN_Y=$WIN_Y 2>/dev/null
}
ALLOW=$(check_allow_dialog /tmp/antig_screenshot.png $WIN_X $WIN_Y)
if [[ "$ALLOW" == ALLOW:* ]]; then
COORDS="${ALLOW#ALLOW:}"
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --coords "$COORDS"
sleep 1
screencapture -x -R${WIN_X},,, /tmp/antig_screenshot.png
When to run this check:
- After initial screenshot at start of any task
- After sending any message (wait 3s, re-screenshot, check)
- In every monitoring loop cycle
CRITICAL — Scan ALL windows, not just one: Every monitoring cycle MUST enumerate ALL Antigravity editor windows dynamically and scan each one. Hardcoding a single window's bounds is a bug — different workspaces open in different editor windows with different coordinates. Example:
ALL_WINS=$(osascript -e '
tell application "System Events"
tell process "Antigravity"
set wins to every window
repeat with w in wins
log (name of w) & "|" & (position of w as string) & "|" & (size of w as string)
end repeat
end tell
end tell
' 2>&1)
echo "$ALL_WINS" | while IFS='|' read title pos size; do
x=$(echo "$pos" | grep -oE '^-?[0-9]+')
y=$(echo "$pos" | grep -oE -- '-?[0-9]+$')
w=$(echo "$size" | grep -oE '^[0-9]+')
h=$(echo "$size" | grep -oE '[0-9]+$')
screencapture -x -R${x},${y},${w},${h} /tmp/antig_scan_${x}_${y}.png 2>/dev/null
done
This was the root cause of the 2026-04-04 miss: scan was targeting only wt-task2 (x=186, y=-990) but the Allow dialog appeared in the worldai_claw editor window (x=-28, y=-998) — a completely different window never scanned.
Why: Allow dialogs are web-rendered (not in A11y tree). They block ALL agent progress silently. The agent shows progress_activity spinner but never executes commands. This was a real miss on 2026-04-03 — the agent was stalled until a human flagged it. It was also missed on 2026-04-04 for the same multi-window reason.
Known False Positives — Add to FP dict
When using the PIL scan, these coordinates are recurring false positives (blue pixels from UI elements, not real Allow dialogs). Add them to a FP set and skip with SKIP_FP: prefix:
FP = {
'1702,-514', '1543,-479', '1447,-493',
'1000,672', '640,626', '726,628',
'454,715',
'928,595',
'83,623', '83,586', '91,585',
'556,-742', '560,-764',
'547,-682', '579,-767',
'713,781',
}
Gutter FP rule: any coord with screen x < 100 in the fullscreen window (0,37,1728,1080) is a VS Code diff/edit gutter indicator — always skip. Real Allow dialogs appear in the center-right of the window (x > 300).
Editor window left-edge rule (2026-04-05): in any editor window, cx//2 < 30 (logical x < 30) is always a VS Code diff/edit gutter indicator — add cx//2 > 30 to the PIL scan filter. This catches the recurring x≈2 FP in win 902.
Detection rule: if click does NOT dismiss the dialog (same coord detected again after click), add to FP. Real Allow dialogs disappear after click.
Editor window bounds drift: When the Agent Manager window closes (agent switches to editor-only mode), PIL scan bounds must switch from Manager bounds to editor window bounds. Re-fetch with peekaboo window list each cycle — do NOT hardcode Manager bounds.
Conversation Activity Detection — Busy vs Idle
Two reliable indicators that a conversation is currently generating:
1. A11y label: progress_activity prefix (sidebar)
SNAPSHOT=$(peekaboo see --app Antigravity --window-id $MANAGER_ID 2>/dev/null)
ACTIVE=$(echo "$SNAPSHOT" | grep -c 'progress_activity.*button' || echo 0)
2. Red stop square in the input area (currently-viewed conversation only)
When a conversation is actively generating, a red stop square appears in the bottom-right of the input box (next to the microphone icon). Detect with PIL:
def is_busy_red_square(img_path, win_x, win_y):
"""Returns True if red stop button is visible in input area."""
from PIL import Image
import numpy as np
img = Image.open(img_path)
arr = np.array(img)
h, w = arr.shape[:2]
bottom = arr[int(h*0.85):, :]
red_mask = (bottom[:,:,0] > 160) & (bottom[:,:,1] < 100) & (bottom[:,:,2] < 100)
right_mask = np.zeros_like(red_mask)
right_mask[:, int(w*0.8):] = red_mask[:, int(w*0.8):]
return np.sum(right_mask) > 30
The red square appears at approximately screen (win_x + w*0.54, win_y + h*0.93) in the Manager window (right side of input box).
Known red-stop location (2026-04-05): in win 902 (x=-28, y=-998, w=1189, h=857), the red stop is at retina pixels x≈2312-2331, y≈1608-1627 (logical x≈1156, y≈804). Always filter x > 80% of image width to exclude gutter noise.
Use A11y method first (no screenshot needed). Use PIL red-square only to verify the currently-displayed conversation's state when A11y is ambiguous.
3. Conversation Health Classification (MANDATORY — use this, not spinner alone)
The sidebar spinner icon (◌) is UNRELIABLE. It persists after conversations die. Always cross-check with TWO additional signals:
- Red stop square (■) in bottom-right of chat input area — PIL detection
- Status bar indicator — bottom of window shows model name + button type
Four conversation states:
| State | Sidebar | Red ■ | Status bar | Timestamp | Action |
|---|
| Actually working | spinner ◌ | YES | "Claude Opus 4.6 (Thinking)" + red ■ | < 30m | Leave alone |
| Frozen-active | spinner ◌ | YES | "...Thinking" + red ■ | > 1h | Interrupt: send "why did you freeze? Use a fresh terminal" |
| Zombie spinner | spinner ◌ | NO | "Gemini 3.1 Pro (High)" + send → | any | Dead conversation. Send "Continue with the next task" |
| Cleanly idle | no spinner | NO | model name + send → | any | Send new task if slot needed |
Detection procedure (for each conversation):
- Click conversation in sidebar to select it
- Screenshot the bottom 15% of the Manager window
- Check for red stop square with
is_busy_red_square()
- Check status bar text: look for "Thinking" + red ■ vs model name + send arrow →
- Read sidebar timestamp (Xm, Xh, Xd)
- Classify using table above
Status bar PIL detection — send arrow (→) vs red stop (■):
def classify_status_bar(img_path):
"""Returns 'generating' or 'idle' based on bottom status bar."""
from PIL import Image
import numpy as np
img = Image.open(img_path)
arr = np.array(img)
h, w = arr.shape[:2]
bar = arr[int(h*0.95):, int(w*0.7):]
red = (bar[:,:,0] > 160) & (bar[:,:,1] < 100) & (bar[:,:,2] < 100)
if np.sum(red) > 20:
return 'generating'
return 'idle'
CRITICAL: A conversation showing "Gemini 3.1 Pro (High)" with a send arrow (→) in the status bar is IDLE regardless of what the sidebar spinner shows. The spinner is a stale UI artifact.
24h Conversation Scanner — Keep 3 Active AT ALL TIMES
HARD RULE: Always maintain exactly 3 conversations generating in parallel. This is not optional.
At every cycle end, check the active count. If < 3:
- Enumerate 24h conversations in reverse-cron order (most recent first)
- Skip any with
progress_activity spinner (already counting toward 3)
- For each idle conversation until
active_count == 3:
- Click it
- Verify busy with red stop square (A11y spinner lags by 5-10s)
- If idle, send a continuation prompt
- Report at cycle end: "Active: N/3 — [titles of active convos]"
Detecting active vs idle (use health classification above, NOT spinner alone):
- DO NOT trust sidebar spinner (◌) alone — it persists on dead conversations
- For each conversation, click it to select, then check:
- Red stop square in bottom-right of input area (PIL
is_busy_red_square())
- Status bar: "...Thinking" + red ■ = generating; model name + send → = idle
- Sidebar timestamp: if > 1h with red square = frozen-active (stuck in loop)
- Classify as: actually-working / frozen-active / zombie-spinner / cleanly-idle (see §3 above)
- Zombie spinners (spinner + no red square) are the most common failure — treat as idle, send "Continue"
- Frozen-active (spinner + red square + >1h) — interrupt with "why did you freeze? Use a fresh terminal"
Sidebar click reliability (2026-04-05):
- Clicking a conversation's
elem_N in the Manager may NOT switch the view if another conversation is actively generating — the generating convo stays displayed
- Fallback: use coordinate click at the sidebar item's approximate y position; items are ~25px apart in logical coords
- If elem click and coord click both fail to switch, wait for the active convo to finish (or send it a message) then retry
Reliable input field coordinates (2026-04-05):
- Manager (x=429, y=-1005, w=1437, h=849): input at screen
(860, -230) — reliable across sessions if Manager not resized
- Editor Agent panel win 902 (x=-28, y=-998, w=1189, h=857): input at screen
(959, -608)
- Always cmd+a → delete to clear before pasting to avoid accumulated text
Opening a workspace folder closes the Manager (2026-04-05):
- Clicking a workspace item in the startup picker or using
agy chat --new-window can open a new editor window and close the floating Manager window
- After opening a workspace, use "Switch to Agent Manager" (visible in left panel) or "Open Agent Manager" button in the title bar to restore the Manager
- Re-enumerate windows after any workspace open:
peekaboo window list --app Antigravity --json
Scan conversations active in the last 24 hours, continue idle ones, keep 3 running at once.
scan_and_continue_convos() {
local MANAGER_ID="$1"
local MAX_ACTIVE="${2:-3}"
local SNAP
SNAP=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" 2>/dev/null)
local CONVOS
CONVOS=$(echo "$SNAP" | grep -E '\(button\)' | grep -E '(now$|[0-9]+m$|[1-9]h$|1[0-9]h$|2[0-3]h$)' \
| grep -v 'arrow_back\|arrow_forward\|New Conversation\|Conversation History\|Open Workspace\|See all\|settings\|Knowledge\|Feedback')
local ACTIVE_COUNT
ACTIVE_COUNT=$(echo "$CONVOS" | grep -c 'progress_activity' 2>/dev/null || echo 0)
local AVAILABLE=$(( MAX_ACTIVE - ACTIVE_COUNT ))
echo "Convos in 24h: $(echo "$CONVOS" | wc -l | tr -d ' '), Active: $ACTIVE_COUNT, Slots: $AVAILABLE"
if [ "$AVAILABLE" -le 0 ]; then
echo "All $MAX_ACTIVE slots busy — skipping"
return
fi
IDLE_CONVOS
IDLE_CONVOS=$( | grep -v )
STARTED=0
IFS= -r line && [ -lt ];
ELEM
ELEM=$( | grep -oE )
TITLE
TITLE=$( | sed | sed | sed )
peekaboo click --app Antigravity --window-id --element 2>/dev/null
2
screencapture -x -R,,, /tmp/antig_convo_check.png
python3 -c 2>/dev/null | grep -q ;
LAST_MSG
LAST_MSG=$(peekaboo see --app Antigravity --window-id 2>/dev/null \
| grep -E | -5)
osascript -e 2>/dev/null
0.2
osascript -e 2>/dev/null
0.2
peekaboo --app Antigravity --text 2>/dev/null
0.3
osascript -e 2>/dev/null
STARTED=$((STARTED+))
3
<<<
}
MANAGER_ID=$(peekaboo window list --app Antigravity --json 2>/dev/null \
| python3 -c )
WIN_X=429; WIN_Y=-1005; WIN_W=1437; WIN_H=849
scan_and_continue_convos 3
Native CDP Execution (Preferred Approach)
The Antigravity runtime plugin now uses a direct Chrome DevTools Protocol (CDP) WebSocket connection as the primary driver, falling back to Peekaboo A11y only if CDP is unavailable.
Why CDP?
- Bypasses fragile A11y tree limits and unreliable web-rendered UI coordinate clicks.
- Connects directly to the electron renderer via
ws://localhost:9222.
- Reliable UI manipulation and state reading using injected JS evaluating directly in the Antigravity context.
How it works under the hood (cdp-client.ts):
- Connects to the active page target via
http://localhost:9222/json.
- Finding Elements: Uses DOM queries to find precise elements, ignoring accessibility tree limits.
- Sending Messages: Directly sets input values and fires
input events, avoiding OS-level paste quirks.
- Extracting Text: Reads conversation threads precisely via DOM
.innerText, much more reliable than snapshot OCR or A11y string joins.
Architecture overview
| View | Purpose | Window title |
|---|
| Agent Manager | Orchestrate AI agents, manage workspaces, monitor conversations | Manager |
| Editor | VS Code-like coding with AI side panel | Antigravity or worktree_* |
| Launchpad | Background/launcher window | Launchpad |
| hidden-nova | Internal background window | hidden-nova |
Toggle between views: Cmd+E or Cmd+Shift+M
Window target: Agent Manager vs Editor (explicit control)
Antigravity exposes two automation surfaces. Pick one before driving UI:
| Target | peekaboo window title(s) | Best for |
|---|
| Agent Manager | Manager | New conversations, Chat History, multi-workspace list, Settings from sidebar, model/planning in new-convo view (A11y-rich) |
| Editor | worktree_*, project_*, or often Antigravity | Coding with Cmd+L side panel, Cmd+I inline AI, terminal ↔ agent, current workspace only |
Environment variable (recommended):
ANTIGRAVITY_WINDOW | Behavior |
|---|
manager | Use only the Manager window. If missing, try to surface it (see below) or fail with a clear message — do not silently use Editor. |
editor | Use only an Editor window. Prefer worktree_* / project_*, then first window whose title is not Manager, Launchpad, or hidden-nova. |
auto or unset | Default: prefer Manager; if not found, set USE_EDITOR=1 and use Editor (same as legacy Step 0). |
Optional: ANTIGRAVITY_EDITOR_TITLE — substring or exact title to pick a specific editor window when multiple exist (e.g. worktree_myrepo).
Single resolver — run once per task; exports ANTIGRAVITY_TARGET, ANTIGRAVITY_WINDOW_ID, USE_EDITOR:
antigravity_resolve_window() {
local mode="${ANTIGRAVITY_WINDOW:-auto}"
local json
json=$(peekaboo window list --app Antigravity --json 2>/dev/null) || true
ANTIGRAVITY_TARGET=""
ANTIGRAVITY_WINDOW_ID=""
USE_EDITOR=0
_pick_manager() {
echo "$json" | python3 -c "import json,sys; ws=json.load(sys.stdin).get('data',{}).get('windows',[]); m=[w for w in ws if w.get('window_title')=='Manager']; print(m[0]['window_id'] if m else '')"
}
_pick_editor() {
export ANTIGRAVITY_EDITOR_TITLE="${ANTIGRAVITY_EDITOR_TITLE:-}"
echo "$json" | python3 -c "
import json, os, sys
hint = (os.environ.get('ANTIGRAVITY_EDITOR_TITLE') or '').strip()
ws = json.load(sys.stdin).get('data',{}).get('windows',[])
if hint:
for w in ws:
t = w.get('window_title') or ''
if hint in t or t == hint:
print(w['window_id']); raise SystemExit
skip = {'Manager','Launchpad','hidden-nova'}
for w in ws:
t = w.get('window_title') or ''
if t.startswith('worktree_') or t.startswith('project_'):
print(w['window_id']); raise SystemExit
for w in ws:
t = w.get('window_title') or ''
if t not in skip:
print(w['window_id']); raise SystemExit
"
}
case "$mode" in
manager)
ANTIGRAVITY_WINDOW_ID="$(_pick_manager)"
if [ -n "$ANTIGRAVITY_WINDOW_ID" ]; then
ANTIGRAVITY_TARGET=manager
osascript -e 2>/dev/null
1
peekaboo press --app Antigravity --keys 2>/dev/null
2
json=$(peekaboo window list --app Antigravity --json 2>/dev/null) ||
ANTIGRAVITY_WINDOW_ID=
[ -n ] && ANTIGRAVITY_TARGET=manager
USE_EDITOR=0
;;
editor)
ANTIGRAVITY_WINDOW_ID=
[ -n ] && ANTIGRAVITY_TARGET=editor && USE_EDITOR=1
;;
auto|*)
ANTIGRAVITY_WINDOW_ID=
[ -n ];
ANTIGRAVITY_TARGET=manager
USE_EDITOR=0
ANTIGRAVITY_WINDOW_ID=
[ -n ] && ANTIGRAVITY_TARGET=editor && USE_EDITOR=1
;;
}
antigravity_resolve_window
Routing after resolve:
ANTIGRAVITY_TARGET=manager — follow all Manager window sections below (peekaboo see on Manager, sidebar, new conversation, etc.). Use MANAGER_ID=$ANTIGRAVITY_WINDOW_ID.
ANTIGRAVITY_TARGET=editor — follow Editor fallback — sending messages via Cmd+L (and Editor UI map). Use EDITOR_ID=$ANTIGRAVITY_WINDOW_ID. Do not assume textField in Manager A11y; use Cmd+L + paste + Return.
- Failure — if
ANTIGRAVITY_WINDOW_ID is empty, report: Antigravity not running, wrong ANTIGRAVITY_EDITOR_TITLE, or no Editor window; suggest agy /path/to/workspace or opening Manager with Cmd+Shift+M.
Step 0 — Routing: Agent Manager (preferred) → Editor (fallback)
Auto mode (ANTIGRAVITY_WINDOW=auto or unset): Prefer the Agent Manager for orchestration (new conversations, monitoring, steering). Use the Editor's Cmd+L panel when:
- The Manager window is not found in
peekaboo window list
- The Manager A11y tree is hanging/timing out after 3 retries
- The user explicitly requests editor-mode interaction
Explicit modes: Use the Window target resolver above instead of this block when ANTIGRAVITY_WINDOW is manager or editor.
MANAGER_ID=$(peekaboo window list --app Antigravity --json 2>/dev/null \
| python3 -c "import json,sys; ws=[w for w in json.load(sys.stdin)['data']['windows'] if w['window_title']=='Manager']; print(ws[0]['window_id'] if ws else '')" 2>/dev/null)
if [ -z "$MANAGER_ID" ]; then
echo "Manager window not found — falling back to Editor path"
USE_EDITOR=1
else
echo "Manager found: window $MANAGER_ID"
USE_EDITOR=0
fi
if [ "$USE_EDITOR" = "1" ]; then
EDITOR_ID=$(peekaboo window list --app Antigravity --json 2>/dev/null \
| python3 -c "
import json,sys
ws=json.load(sys.stdin)['data']['windows']
# Prefer worktree windows, then any non-Manager/Launchpad window
for w in ws:
t=w['window_title']
if t.startswith('worktree_') or t.startswith('project_'):
print(w['window_id']); exit()
for w in ws:
t=w['window_title']
if t not in ('Manager','Launchpad','hidden-nova'):
print(w['window_id']); exit()
")
fi
[antig] Commit Prefix and PR Claim Convention
All commits made by Antigravity agents MUST use the [antig] prefix — analogous to [agento] for AO workers and [worldai_claw] for wc workers.
[antig] feat(parties): add name field and owner_id schema
[antig] fix(doctor): tighten Discord guard
PR claim: Include the PR number and worktree path in every Antigravity prompt so the agent knows what it owns:
You are working in /path/to/worktree on branch feat/xxx (PR #NNN).
Use [antig] prefix for ALL commits.
Push to origin feat/xxx after each logical unit of work.
Zero-touch tracking: [antig]-prefixed merged PRs count toward the Antigravity zero-touch rate, separate from [agento] (AO workers). This lets the eloop measure Antigravity's contribution independently.
Fallback prefix: If Antigravity agent doesn't use [antig], the commit author OpenClaw <openclaw@openclaw.ai> identifies it as an agent commit — but the prefix is required for zero-touch counting.
Manager window — complete UI element map (preferred path)
Use this section when ANTIGRAVITY_TARGET=manager (or auto resolved to Manager). Set MANAGER_ID="${ANTIGRAVITY_WINDOW_ID}" from the resolver; all --window-id examples below assume that ID.
Top bar
| Element label | Role | Description |
|---|
dock_to_right / dock_to_left | text | Dock position toggle |
Agent Manager | text | Title |
Open Editor | text | Switch to Editor view |
settings | text | Settings icon (top bar) |
Sidebar navigation buttons
| Element label | Role | Description |
|---|
add Start new conversation | button | Creates new conversation (global) |
history Chat History | button | Opens chat history panel |
Open Workspace | button | Opens workspace picker/folder browser |
import_contacts Knowledge | button | Opens Knowledge panel |
chrome_product Browser | button | Opens Browser integration panel |
settings Settings | button | Opens Settings panel |
lightbulb Provide Feedback | button | Opens Feedback panel |
Conversation list (dynamic)
| Pattern | Meaning |
|---|
progress_activity <Title> now | Active/running conversation (status: Running) |
<Title> <N>m / <N>h / <N>d | Completed/idle conversation with time-ago (status: Idle) |
See all (N) | Expand to show all conversations |
See less | Collapse conversation list |
Conversation status indicators
Conversations have three states visible in the Manager:
| Status | Visual indicator | Meaning |
|---|
| Running | progress_activity spinner prefix + now | Agent is actively working |
| Idle | Time-ago suffix (5m, 1h, 12h) | Agent finished or waiting |
| Blocked | May show warning icon or stalled state | Agent needs human input |
Sidebar icon prefixes (Material Icons as text)
| Prefix | Icon |
|---|
add | Plus (new conversation) |
history | Clock (chat history) |
import_contacts | Contacts (knowledge) |
chrome_product | Browser |
settings | Gear |
lightbulb | Bulb (feedback) |
progress_activity | Spinner (active task) |
more_horiz | Three dots (context menu) |
keyboard_arrow_down / keyboard_arrow_right | Expand/collapse workspace |
Workspace section elements
| Element label | Role | Description |
|---|
worktree_<name> | other/text | Workspace header label |
project_<name> | other/text | Project workspace header |
Playground | other/text | Playground section header (scratch/test conversations) |
more_horiz | text | Workspace options menu |
add | button | Workspace-specific "+" button for new conversations (only for Playground section — workspace + icons are web-rendered) |
info | other | Info icon (appears in Playground section) |
Conversation view controls (appear when conversation is open)
| Element label | Role | Description |
|---|
| text entry area (textField) | textField | Message input area |
Planning / Fast | button/text | Planning mode toggle |
Claude Opus 4.6 (Thinking) etc. | button/text | Model selector |
Record voice memo | button | Voice input |
Send | button | Send message |
code Open editor | button | Open in editor |
Use Playground | button | Switch to playground mode |
Review | button | Review mode toggle (top-right of conversation) |
Walkthrough | tab | Walkthrough artifact view |
Editor window — complete UI element map
Use this section when ANTIGRAVITY_TARGET=editor (or auto fell back to Editor). Set EDITOR_ID="${ANTIGRAVITY_WINDOW_ID}"; drive the agent via Cmd+L (see Editor fallback section) rather than Manager sidebar A11y.
All editor windows (Antigravity, worktree_*) share the same VS Code structure.
Sidebar tabs
| Tab label | Keyboard shortcut |
|---|
| Explorer | Shift+Cmd+E |
| Code Search | Shift+Cmd+F |
| Source Control | Ctrl+Shift+G |
| Run and Debug | Shift+Cmd+D |
| Remote Explorer | — |
| Extensions | Shift+Cmd+X |
| Containers | — |
| GitHub Actions | — |
AI-specific controls
| Control | How to access |
|---|
| Agent side panel | Cmd+L (toggle/focus) |
| Inline AI command | Cmd+I (in editor and terminal) |
@ context inclusion | Type @ in agent panel → files, dirs, MCP servers |
/ workflow invocation | Type / in agent panel → saved workflows |
| Artifacts button | Bottom-right of editor |
| "Explain and Fix" | Hover over problems |
| "Send all to Agent" | Problems panel → batch error resolution |
| Terminal output → agent | Select terminal output, press Cmd+L to send to agent |
| Tab-to-import | Auto-suggests missing dependency imports |
| Tab-to-jump | Navigates cursor to next logical code location |
Menu bar items (Antigravity focused)
Antigravity menu
- About Antigravity
- Quit Antigravity
File menu
- Start Conversation — new conversation (Cmd+N equivalent)
- New Editor — new editor window
- Open Folder — open a workspace folder
Edit menu
- Standard: Undo, Redo, Cut, Copy, Paste, Select All
- Writing Tools submenu (macOS): Proofread, Rewrite, Make Friendly/Professional/Concise, Summarize, Create Key Points, Make List/Table, Compose
- AutoFill submenu: Contact, Passwords
View menu
Keyboard shortcuts
Antigravity-specific
| Shortcut | Action |
|---|
| Cmd+E | Toggle Agent Manager ↔ Editor |
| Cmd+Shift+M | Toggle Agent Manager ↔ Editor (alt) |
| Cmd+L | Toggle/focus agent side panel |
| Cmd+I | Inline AI command (editor + terminal) |
| Cmd+B | Toggle sidebar |
| Tab | Accept AI code completion |
| Esc | Dismiss AI suggestion |
VS Code inherited (selection)
| Shortcut | Action |
|---|
| Cmd+P | Quick Open (file search) |
| Cmd+Shift+P | Command Palette |
| Cmd+Shift+F | Search across project |
| Ctrl+` | Toggle integrated terminal |
| Cmd+D | Select next occurrence |
| Cmd+Shift+L | Select all occurrences |
| Opt+Up/Down | Move current line |
| Cmd+Shift+K | Delete current line |
| Cmd+/ | Toggle line comment |
| Cmd+\ | Split editor vertically |
| Cmd+1/2/3 | Focus split editor groups |
| Ctrl+G | Go to line number |
| Cmd+Click | Go to definition |
| Ctrl+- | Navigate back |
Development modes
Antigravity supports these execution modes (configured during onboarding or in Settings):
| Mode | Description |
|---|
| Agent-driven (Autopilot) | AI writes code, creates files, runs commands automatically |
| Review-driven (Recommended) | AI asks permission before actions |
| Agent-assisted | User stays in control; AI helps with safe automations |
| Secure Mode | Enhanced security restrictions |
| Custom Configuration | User-defined policy per action type |
Execution policies
| Policy | Options |
|---|
| Terminal execution | Always Proceed (with deny list) / Request Review (with allowlist) |
| Review policy (artifacts) | Always Proceed / Agent Decides (default) / Request Review |
| JavaScript execution (browser) | Always Proceed / Request Review / Disabled |
Planning modes
| Mode | Description |
|---|
| Planning | Agent builds task lists and implementation plans before executing. Use for complex tasks. |
| Fast | Agent executes directly without planning. Use for simple tasks. |
Toggle via the dropdown in the conversation input area (shows as Planning or Fast button).
A11y access: In the new conversation creation view, Planning/Fast buttons appear as A11y button elements. In active conversations, they are web-rendered. When clicking "Start new conversation", a conversation mode picker popover appears with Planning/Fast options — this popover is entirely web-rendered (NOT in A11y tree).
Model selection
Priority order (best first)
| Priority | Model | Notes |
|---|
| 1 | Claude Opus 4.6 (Thinking) | Best code quality |
| 2 | Gemini 3.1 Pro (High) | Strong general purpose |
| 3 | Gemini 3.1 Pro (Low) | Same model, lower priority tier |
| 4 | Gemini 3 Flash | Fast but lower quality |
| 5 | Claude Sonnet 4.6 (Thinking) | Good quality, use when Opus exhausted |
| 6 | GPT-OSS 120B (Medium) | Last resort |
Checking quota before starting conversations
Before starting a new conversation, check model quota via Settings > Models:
MANAGER_ID=<MANAGER_ID>
SNAP=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
SETTINGS_ID=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if 'Settings' in (e.get('label','') or '') and e.get('role') == 'button':
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$SETTINGS_ID" --snapshot "$SNAP"
sleep 1
SNAP2=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
MODELS_ID=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if e.get('label','') == 'Models' and e.get('role') == 'button':
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$MODELS_ID" --snapshot "$SNAP2"
sleep 1
BOUNDS=$(peekaboo window list --app Antigravity --json 2>/dev/null | python3 -c "
import json, sys
for w in json.load(sys.stdin)['data']['windows']:
if w['window_id'] == $MANAGER_ID:
b = w['bounds']
print(f\"{b['x']},{b['y']},{b['width']},{b['height']}\")")
screencapture -x -R${BOUNDS} /tmp/antig_models.png
Quota bar colors: Gray filled = quota available. Yellow/orange filled = quota used. Empty gray segments = quota remaining.
Model selector in new conversation view
A11y access: In the new conversation creation view, the model selector appears as an A11y button (e.g., Select model, current: Claude Opus 4.6 (Thinking)). Click it to open the model dropdown — the dropdown options ARE A11y accessible as buttons.
PROVEN WORKING FLOW (2026-03-31):
- Click "New Conversation" button (A11y:
add New Conversation)
- After view loads, find model selector:
elem_66 or search for 'Select model' in label
- Click it — dropdown opens immediately with all models as A11y buttons
- Click desired model (e.g.,
Gemini 3.1 Pro (High) New)
- Verify: re-snapshot and check
'Select model, current: Gemini 3.1 Pro (High)'
MANAGER_ID=<MANAGER_ID>
SNAP_JSON=$(timeout 12 peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null)
SNAP=$(echo "$SNAP_JSON" | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
NEW_ID=$(echo "$SNAP_JSON" | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if 'New Conversation' in (e.get('label','') or '') and e.get('role') == 'button':
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$NEW_ID" --snapshot "$SNAP"
sleep 2
SNAP_JSON2=$(timeout 12 peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null)
SNAP2=$(echo "$SNAP_JSON2" | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
MODEL_BTN=$(echo "$SNAP_JSON2" | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if 'Select model' in (e.get('label','') or '') and e.get('role') == 'button':
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$MODEL_BTN" --snapshot "$SNAP2"
sleep 1.5
SNAP_JSON3=$(timeout 12 peekaboo see --app Antigravity --window-id --json 2>/dev/null)
SNAP3=$( | python3 -c )
GEMINI_BTN=$( | python3 -c )
peekaboo click --app Antigravity --window-id --on --snapshot
1
SNAP_JSON4=$( 12 peekaboo see --app Antigravity --window-id --json 2>/dev/null)
| python3 -c
Quota exhaustion recovery
When a conversation stalls with capacity errors ("Error You have exhausted your capacity on this model"):
- Do NOT try to change model in the stalled conversation — model selector is web-rendered in active/idle convos
- Start a new conversation using the flow above, select next model in priority order
- Include context from the stalled conversation in the new prompt
In active/idle conversations, the model selector is web-rendered and cannot be changed via A11y.
"Too much traffic" rate limiting
When Antigravity shows "Too much traffic — please try again in a few minutes.":
- Retry with backoff — wait 30s, retry the same conversation (click Retry button)
- After 10 retries — if still failing, switch to a different model (start new conversation with next model in priority)
- Max 3 NEW windows — if workspace window count > 3, close excess Editor windows (keep Manager + max 2 Editor). NOT 3 total.
- Do NOT just stop — the work items still need completion, try again when traffic subsides
- Switch workspace context — try starting conversation in a different workspace to route to different backend
Priority order: Claude Opus 4.6 → Gemini 3.1 Pro (High) → Gemini 3.1 Pro (Low) → Gemini 3 Flash → Claude Sonnet 4.6
Conversation monitoring loop (for /eloop)
When monitoring active Antigravity conversations:
- Scan all conversations — capture Manager screenshot, identify active (spinner) vs idle
- Check window count — ensure workspace windows ≤ 3 NEW windows (Editor + Manager). Count via
peekaboo window list excluding Launchpad and empty. If >3, close excess Editor windows (keep Manager + max 2 Editor) until ≤3.
- Check for errors — look for "Too much traffic", "Error", "Failed" in conversation content
- Retry on error — if error detected, click Retry or restart conversation with same task
- After 10 retries — if same error persists, switch model (start new conversation with different model)
- Max 3 parallel — ensure no more than 3 active conversations at once
- Report status — document which conversations are running, which PRs they're working on
- PR-Target verification (MANDATORY before sending messages):
- User specifies "work on PR #177" → must send message to PR #177 conversation
- If agent is on wrong PR (e.g., stuck on #178 when asked for #177), explicitly close wrong conversation and open new one for target PR
- Before typing message: verify conversation title matches target PR number
- Health check for stuck agents:
- If
progress_activity persists > 30 minutes with no Git commits pushed → agent is hung
- Force restart:
osascript -e 'tell application "Antigravity" to quit' then open -a Antigravity
- Do NOT keep polling stuck agents — restart and start fresh conversation
Clearing the text input field
peekaboo press does NOT support modifier key combos like cmd+a or Command+a — those throw "Unknown key" errors. To clear the input field before pasting:
- Click the text field by element ID or coordinates
- If field has existing text: paste the new content directly (it will append) — avoid trying to select-all with keyboard shortcuts
- Alternatively: open a fresh "New Conversation" which starts with an empty field
Artifacts system
Types: Task Lists, Implementation Plans, Code Diffs, Walkthroughs, Screenshots, Browser Recordings.
- Agent Manager: toggle artifacts button (top-right)
- Editor: click "Artifacts" button (bottom-right)
- Supports Google Docs-style comments — agents ingest feedback and iterate
Walkthrough mode
Post-completion artifact with: summary, task list, file changes, screenshots, testing instructions. Appears as a tab in conversation view.
Configuration and customization
Rules (agent guidelines)
| Scope | Path |
|---|
| Global | ~/.gemini/GEMINI.md |
| Workspace | <workspace>/.agents/rules/ |
Workflows (saved prompts, invoked with /)
| Scope | Path |
|---|
| Global | ~/.gemini/antigravity/global_workflows/<NAME>.md |
| Workspace | <workspace>/.agents/workflows/ |
Skills
| Scope | Path |
|---|
| Global | ~/.gemini/antigravity/skills/ |
| Workspace | <workspace>/.agents/skills/ |
Skill structure: SKILL.md (required, YAML frontmatter), optional scripts/, references/, assets/.
Customization access
Editor mode: (...) > Customizations
"Allow this conversation" dialog — always click Allow
Antigravity shows "Allow directory access to /path?" prompts with Deny / Allow Once / Allow This Conversation buttons when agents try to access workspace files. Always click "Allow This Conversation" without asking the user — pre-authorized.
CRITICAL: These buttons are web-rendered
The Allow/Deny buttons are web-rendered inside the conversation panel and do NOT appear in the Peekaboo accessibility tree. peekaboo see only returns native macOS accessibility elements. The Allow prompt appears every new conversation for every workspace — not just the first time.
Solution: Coordinate-based clicking via annotated screenshot
Method 0: Simplified blue-button detection (FASTEST — proven 2026-03-27)
WIN_X=832; WIN_Y=-1013; WIN_W=1025; WIN_H=904
screencapture -x -R${WIN_X},${WIN_Y},${WIN_W},${WIN_H} /tmp/antig_allow.png
COORDS=$(python3 -c "
from PIL import Image; import numpy as np
img = np.array(Image.open('/tmp/antig_allow.png'))
blue_mask = (img[:,:,2] > 150) & (img[:,:,0] < 120) & (img[:,:,2] > img[:,:,1] + 30)
ys, xs = np.where(blue_mask)
if len(xs) > 0:
right_mask = xs > (xs.max() - 250)
cx, cy = int(np.mean(xs[right_mask])), int(np.mean(ys[right_mask]))
print(f'${WIN_X}+{cx//2},${WIN_Y}+{cy//2}')
" 2>/dev/null | python3 -c "import sys; parts=sys.stdin.read().strip().split(','); print(f'{eval(parts[0])},{eval(parts[1])}')")
[ -n "$COORDS" ] && peekaboo click --app Antigravity --window-id MANAGER_ID --coords "$COORDS"
Method 1: Programmatic blue-button detection (PROVEN — use this)
Uses screencapture + PIL pixel analysis to find blue button coordinates automatically:
MANAGER_ID=<MANAGER_ID>
BOUNDS=$(peekaboo window list --app Antigravity --json 2>/dev/null | python3 -c "
import json, sys
for w in json.load(sys.stdin)['data']['windows']:
if w['window_id'] == $MANAGER_ID:
b = w['bounds']
print(f\"{b['x']},{b['y']},{b['width']},{b['height']}\")
")
WIN_X=$(echo "$BOUNDS" | cut -d, -f1)
WIN_Y=$(echo "$BOUNDS" | cut -d, -f2)
WIN_W=$(echo "$BOUNDS" | cut -d, -f3)
WIN_H=$(echo "$BOUNDS" | cut -d, -f4)
screencapture -x -R${WIN_X},${WIN_Y},${WIN_W},${WIN_H} /tmp/allow_check.png
python3 -c "
from PIL import Image
import numpy as np
img = Image.open('/tmp/allow_check.png')
arr = np.array(img)
h, w = arr.shape[:2]
# Find blue button pixels (bottom half of image)
for row_start in range(h//2, h, 30):
region = arr[row_start:row_start+30, :]
blue_mask = (region[:,:,2] > 150) & (region[:,:,0] < 150) & (region[:,:,2] > region[:,:,0] + 50)
if np.sum(blue_mask) > 100:
coords = np.where(blue_mask)
xs = coords[1]
xs_sorted = np.sort(np.unique(xs))
# Find gap between buttons
gaps = np.diff(xs_sorted)
big_gaps = np.where(gaps > 15)[0]
if len(big_gaps) > 0:
# Rightmost button = 'Allow This Conversation'
right_start = xs_sorted[big_gaps[-1] + 1]
right_end = xs_sorted[-1]
else:
# Single button region — rightmost 40%
right_start = xs_sorted[int(len(xs_sorted) * 0.6)]
right_end = xs_sorted[-1]
cx = int((right_start + right_end) / 2)
cy = int(np.mean(coords[0][xs >= right_start])) + row_start
# Convert 2x retina pixels to points
print(f'{$WIN_X + cx//2},{$WIN_Y + cy//2}')
break
"
CLICK_COORDS=$(python3 ...)
peekaboo click --app Antigravity --window-id --coords
1
screencapture -x -R,,, /tmp/allow_verify.png
Method 2: Visual inspection (fallback)
peekaboo see --app Antigravity --window-id "$MANAGER_ID" --annotate
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --coords <x>,<y>
Key facts about coordinate clicking
peekaboo click --coords uses screen-absolute coordinates (not window-relative)
- macOS Retina displays use point coordinates (not pixel coordinates) — divide retina pixels by 2
screencapture -x -R uses point coordinates for the region but captures at 2x pixel resolution
- External displays may have negative y coordinates (e.g., y=-1013 for a monitor above the primary)
- Window bounds from
peekaboo window list are in point coordinates
Alternative: Antigravity Settings
Configure execution policy to avoid Allow prompts entirely:
- Open Settings (sidebar gear icon)
- Set terminal execution to "Always Proceed" for known workspaces
- IMPORTANT: "Always Proceed" does NOT eliminate per-conversation Allow prompts — those are separate (workspace directory access vs terminal command approval)
- "Always Proceed" only applies to new messages. In-progress conversations keep the policy that was active when they started. If an agent is stuck on "Waiting for command completion", start a NEW conversation rather than trying to approve the stuck one.
Terminal approval stuck state — start new conversation
If you see "The commit command is waiting for user approval" or "Waiting for command completion (up to 300 seconds)", the conversation started under "Request Review" mode. The "Always run ^" dropdown buttons are web-rendered and coordinate clicks rarely work reliably on them. Recovery:
- Do NOT try to click "Always run" — it's unreliable
- Verify Settings > Terminal > "Always Proceed" is set
- Start a new conversation — it will use the updated policy
- Include results from the stuck conversation's work (skip completed items)
Agent crash recovery — start fresh
When "Agent terminated due to error" appears with Dismiss/Copy debug info/Retry buttons:
- These buttons are web-rendered — Dismiss clicks often land on wrong elements (Knowledge panel, etc.)
- Do NOT try Retry — it may restart with stale context (e.g., trying to fix already-merged PRs)
- Instead: start a new conversation with updated task description
- Check what the crashed agent accomplished:
git log, git branch -r, gh pr list
- Skip completed work in the new task prompt
- Two conversations CAN run simultaneously — the crashed one becomes IDLE and won't block
Detection pattern
After starting any conversation, check for Allow prompts:
- Capture the Manager window with
screencapture -x -R<bounds>
- Use PIL to search for blue button pixels in the bottom half
- If blue buttons found → run the programmatic click flow above
- Re-capture to confirm the prompt dismissed
Web-rendered content interaction — general technique
Antigravity uses web-rendered UI elements that don't appear in the A11y tree. For ANY web-rendered button/control:
- Capture:
screencapture -x -R<window_bounds> /tmp/element.png
- Analyze: Use PIL/numpy to find the element by color, shape, or position
- Convert: Divide retina pixels by 2, add window origin for screen coordinates
- Click:
peekaboo click --app Antigravity --window-id <ID> --coords <screen_x>,<screen_y>
- Verify: Re-capture and confirm the element is gone/changed
This technique works for: Allow prompts, Review buttons, walkthrough controls, artifact interactions, and any other web-rendered UI that peekaboo see cannot expose.
Note: peekaboo screenshot does NOT exist — use screencapture (macOS built-in) instead.
Starting a new conversation in a workspace
Method 0: agy CLI (FASTEST — use for new workspaces)
The agy CLI (~/.antigravity/antigravity/bin/agy) opens a workspace directly and registers it with the Manager. This is the best method for workspaces not yet visible in the Manager sidebar.
agy /path/to/workspace
peekaboo window list --app Antigravity --json 2>/dev/null | python3 -c "
import json, sys
for w in json.load(sys.stdin)['data']['windows']:
print(f'{w[\"window_id\"]:6} {w[\"window_title\"]}')"
After agy, the workspace appears in peekaboo window list but may require scrolling down in the Manager sidebar to become visible as an other element. The Manager sidebar only shows a viewport — new workspaces appear at the bottom.
Method 1: "Start new conversation" button (RECOMMENDED — tested and proven)
The global "add Start new conversation" button in the sidebar opens a new conversation creation view with full A11y support. This is the most reliable method.
How to find it: Scroll to the top of the sidebar — the button may be hidden below the fold. Use peekaboo scroll --direction up --amount 10 then check for add Start new conversation in the A11y tree.
MANAGER_ID=<MANAGER_ID>
peekaboo scroll --direction up --amount 10 --app Antigravity --window-id "$MANAGER_ID"
SNAP=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; d=json.load(sys.stdin); print(d['data']['snapshot_id'])")
peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if 'Start new conversation' in (e.get('label','') or ''):
print(e['id'])"
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on <ADD_BUTTON_ID> --snapshot "$SNAP"
The new conversation view exposes these A11y elements:
| Element label | Role | Description |
|---|
keyboard_arrow_down <workspace> | button | Workspace selector dropdown |
text entry area | textField | Message input |
Send | button | Send button (visible here, unlike in active conversations) |
Planning / Fast | button | Planning mode toggle |
Claude Opus 4.6 (Thinking) etc. | button | Model selector |
Record voice memo | button | Voice input |
code Open editor | button | Switch to editor |
Use Playground | button | Switch to playground mode |
New conversation in | other | Label text |
Important: The workspace selector dropdown (keyboard_arrow_down) is accessible in A11y but clicking it does NOT reliably open the web-rendered dropdown menu. The dropdown is web-rendered. Workaround: include the explicit workspace path in your prompt.
SNAP2=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
label = e.get('label','') or ''
role = e.get('role','')
if label in ('text entry area','Send') or role == 'textField':
print(e['id'], role, label)"
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on <TEXT_FIELD_ID> --snapshot "$SNAP2"
peekaboo paste --app Antigravity --text "You are working in /path/to/worktree on branch X. <your task>"
peekaboo press "Return" --app Antigravity
sleep 3
peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
label = e.get('label','') or ''
if 'progress_activity' in label:
print('ACTIVE:', label[:100])"
Method 2: Click workspace label in Manager sidebar
Clicking a workspace other element or "No chats yet" text expands the workspace section. After expanding, a + icon appears next to the workspace name (web-rendered, requires coordinate clicking).
Limitations: Only the currently focused workspace label appears as an other element in the A11y tree. Other workspace labels are web-rendered and not directly clickable via A11y.
Method 3: Click existing conversation to switch
Click any conversation button in the sidebar to switch to it. The conversation buttons ARE in the A11y tree (role=button, label includes title + time-ago).
⚠️ Known issues
- "Open Workspace" dropdown scoping: The
Open Workspace sidebar button does NOT reliably scope to the selected workspace.
- Workspace dropdown in new conversation view: The
keyboard_arrow_down button doesn't open its web-rendered dropdown via A11y click. Include workspace path in prompt as workaround.
- Workspace-specific
+ icon: Web-rendered, requires coordinate clicking which is unreliable due to dynamic sidebar layout.
Full flow (Method 0 + Method 1 combined — most reliable)
MANAGER_ID=<MANAGER_ID>
agy /path/to/workspace
sleep 2
peekaboo scroll --direction up --amount 10 --app Antigravity --window-id "$MANAGER_ID"
SNAP=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
ADD_ID=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if 'Start new conversation' in (e.get('label','') or ''):
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$ADD_ID" --snapshot "$SNAP"
sleep 1
SNAP2=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
TF_ID=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if e.get('role','') == 'textField' and 'text entry' in (e.get('label','') or ''):
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$TF_ID" --snapshot "$SNAP2"
peekaboo paste --app Antigravity --text "You are working in /path/to/worktree on branch X. <task>"
peekaboo press --app Antigravity
Important notes
- Use
peekaboo paste not peekaboo type — paste avoids character drift in Antigravity's input field.
- Both Return and Send button work for sending messages. In active conversations, the Send button may NOT be in the A11y tree. In the new conversation view, it IS accessible.
- Always include the explicit workspace path in your prompt as a safety net for workspace scoping: "You are working in /path/to/worktree on branch X"
- New workspaces need scrolling — after
agy <path>, the Manager sidebar may not show the workspace without scrolling down.
- Conversations auto-rename — after the first message, Antigravity renames the conversation based on content.
Workspace scoping workaround — CRITICAL
The "New conversation in > [workspace]" dropdown is web-rendered and cannot be switched via A11y or coordinate clicking reliably. After 3 failed attempts to switch it, use this workaround:
Don't fight the dropdown. Include the workspace path in your prompt text instead:
peekaboo paste --app Antigravity --text "You are working in $HOME/project_worldaiclaw/worldai_claw on branch main. <your task here>"
This works because Antigravity agents resolve the workspace from the path in the prompt, regardless of which workspace the dropdown shows. The dropdown only sets the default context — explicit paths override it.
UI interaction retry cap
After 3 failed attempts at the same UI interaction (coordinate clicks, element clicks, keyboard shortcuts), STOP. Do not continue guessing. Instead:
- Disclose to user: "I tried X 3 times and it's not working because Y"
- Use a documented workaround (e.g., workspace path in prompt text)
- If no workaround exists, ask user for help
This prevents the failure mode where the agent spends 15+ tool calls trying coordinate variations on web-rendered elements that are invisible to A11y.
Manager window startup
The Manager window may not appear immediately after keyboard shortcuts. Reliable procedure:
osascript -e 'tell application "Antigravity" to activate' — ensure Antigravity is focused
peekaboo press --app Antigravity --keys "cmd shift m" — toggle Manager
sleep 3 — Manager takes 2-3 seconds to initialize
- Check
peekaboo window list for a window titled "Manager"
- If not found after ONE retry (total wait 6s), disclose to user: "Manager window isn't opening"
Do NOT loop Cmd+Shift+M or Cmd+E repeatedly — it toggles the window on/off.
Sending messages to active conversations
To send a follow-up message to a conversation already open in the Manager:
MANAGER_ID=<MANAGER_ID>
SNAP=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
CONVO_ID=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if e.get('role','') == 'button' and 'your search term' in (e.get('label','') or '').lower():
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$CONVO_ID" --snapshot "$SNAP"
sleep 1
SNAP2=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
TF_ID=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null | python3 -c "
import json,sys; data=json.load(sys.stdin)
for e in data['data']['ui_elements']:
if e.get('role','') == 'textField' and 'text entry' in (e.get('label','') or ''):
print(e['id']); break")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on "$TF_ID" --snapshot "$SNAP2"
peekaboo paste --app Antigravity --text "your follow-up message"
peekaboo press "Return" --app Antigravity
Tested behavior: Messages sent via paste+Return are received and processed immediately. The conversation auto-renames based on the first message content.
Note: In active/idle conversations, the Send button does NOT appear in A11y. Use peekaboo press "Return" instead — confirmed working.
Important: The textField may NOT appear in the A11y tree initially when switching to a conversation. It becomes accessible after clicking the input area (either via coordinates or by clicking the conversation content area first). If textField is not in the A11y tree:
- Click the input area coordinates (bottom of content pane, roughly
y=-140 relative to Manager window)
- Re-snapshot —
textField with label "text entry area" should now appear
- Then paste + Return as usual
Chat History panel
The Chat History panel shows a flat list of ALL conversations across all workspaces, with search and timestamps.
SNAP=$(peekaboo see --app Antigravity --window-id "$MANAGER_ID" --json 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['snapshot_id'])")
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --on <CHAT_HISTORY_BUTTON> --snapshot "$SNAP"
The Chat History button is labeled history Chat History in A11y. It shows:
- Conversation title and workspace name
- Time-ago timestamps
- Blue dot for active/recent conversations
- Spinner icon for running conversations
- Search bar for filtering
Use case: When you need to find a specific conversation across workspaces, Chat History is faster than scrolling through each workspace section in the sidebar.
MANDATORY: Monitor, read, and assess active conversations
This is the most important part of the skill. Starting a conversation is useless if you don't read what Gemini is actually producing and respond to it. Every interaction with Antigravity MUST include a read-assess-respond cycle.
After starting or checking a conversation, ALWAYS:
- Read the conversation content — conversation text is web-rendered and NOT in the A11y tree. Use
screencapture + vision to read what Gemini wrote:
MANAGER_ID=<MANAGER_ID>
BOUNDS=$(peekaboo window list --app Antigravity --json 2>/dev/null | python3 -c "
import json, sys
for w in json.load(sys.stdin)['data']['windows']:
if w['window_id'] == $MANAGER_ID:
b = w['bounds']
print(f\"{b['x']},{b['y']},{b['width']},{b['height']}\")")
screencapture -x -R${BOUNDS} /tmp/convo_read.png
peekaboo scroll --app Antigravity --window-id "$MANAGER_ID" --direction down --amount 10
peekaboo click --app Antigravity --window-id "$MANAGER_ID" --coords <center_x>,<center_y>
peekaboo press "pagedown" --app Antigravity
IMPORTANT: peekaboo see --json on the Manager window returns sidebar elements (conversation titles, workspace labels, menu items) but NOT the web-rendered conversation content (chat messages, code blocks, progress updates). You MUST use screenshots to read what the agent is actually doing.
-
Assess the agent's state — determine what's happening:
- Is it making progress on the task?
- Is it stuck, asking a question, or hitting an error?
- Has it drifted to side work (lint, formatting) while the primary task is undone?
- Did it produce real output (commits, files) or just text?
-
Check for real artifacts — verify the agent actually did work:
cd /path/to/workspace && git log --oneline -5
git diff --stat HEAD~1
- Respond or redirect — if the agent is stuck or drifting:
- Paste a follow-up message with specific instructions
- If idle, start a new message to continue the work
- If erroring, paste the fix or adjust the prompt
When monitoring for someone else (e.g., a subagent or /loop):
Report these fields:
- Conversation title (from Manager)
- Status (Running/Idle/Blocked — from Manager indicators)
- Last visible output (from conversation window
peekaboo see)
- Real artifacts (commits, files, PRs — from git/gh)
- Assessment (on-track / drifting / stuck / done)
- Recommended action (continue / redirect / kill)
Anti-pattern: "conversation started" without reading it
NEVER declare Exit Criterion B or similar "produces output" criteria PASS based solely on:
- Manager showing
progress_activity (that just means Gemini is responding)
- A window existing with the workspace name
- A fibonacci test prompt getting a response
You MUST read the actual conversation content and verify it relates to the assigned task.
Listing all workspaces and conversations
Use when the user asks "what's happening in Antigravity", "list conversations", "what are agents doing".
Complete workspace inventory
peekaboo window list is the authoritative source for ALL workspaces (the A11y tree only shows the currently expanded workspace):
peekaboo window list --app Antigravity --json 2>/dev/null | python3 -c "
import json, sys
wins = json.load(sys.stdin)['data']['windows']
print('All Antigravity windows:')
for w in wins:
title = w['window_title']
wid = w['window_id']
is_ws = title.startswith('worktree_') or title.startswith('project_') or title.startswith('cmus_')
tag = ' [workspace]' if is_ws else ''
print(f' {wid:6} {title}{tag}')
"