Call start_computer_use_session with a one-sentence goal. This requires user approval. Once a session is active, DO NOT call start again — go straight to acting.
Work the loop:
- computer_screenshot to see the current state. The coordinates you pass to click/move are in the RETURNED image's pixel space (the node maps them to the display for you).
- Decide the single next action and take it:
  - computer_key {chord} — e.g. "cmd+space" (Spotlight), "enter", "cmd+a", "esc".
  - computer_type {text} — type into the focused field.
  - computer_click {x,y,button} / computer_move {x,y}.
- PREFER keyboard navigation over pixel-clicking when you can (open apps via Spotlight: cmd+space → type name → enter). Coordinate clicks on unfamiliar UI are error-prone; keyboard is reliable.
- Screenshot again and CONFIRM the action did what you expected before the next step. Don't chain actions blindly.
Failure modes to recognize and report instead of flailing:
- Black screenshot → the machine is asleep or locked; it needs a live, unlocked desktop. Stop and tell the user.
- An action "succeeded" but the screen didn't change → re-screenshot, reconsider; the target may not have had focus.
- scroll is unsupported — find another way (keyboard, or click a scrollbar target).
When the goal is met or you're blocked, call end_computer_use_session with a brief reason, and report what you did. Every action is logged for the user to review.

User asked: {{input}}

Más de este repositorio

mismo repositorio

setup-proactive-outreach

Spshulem/openAGI

Guide configuring proactive outreach so the main reaches out (digest + live decisions) and the Mac app surfaces it with notifications, an overlay list, and inline buttons.

2026-06-172

deploy-update-node

Spshulem/openAGI

Roll out new openAGI code to the main and to Mac nodes (iMessage / computer-use), and restart them safely.

2026-06-152

diagnose-agent-tool-errors

Spshulem/openAGI

Troubleshoot opaque model/tool failures and runaway budget — and their usual root causes.

2026-06-152

diagnose-no-imessage-replies

Spshulem/openAGI

Troubleshoot why the agent stopped replying to iMessages, in priority order.

2026-06-152

connect-mcp

Spshulem/openAGI

Connect an MCP server (catalog or custom URL/stdio) and make its tools usable — including completing OAuth on a headless host.

2026-06-152

schedule-recurring-check

Spshulem/openAGI

Set up a recurring check that only does work (and spends tokens) when there's something new — e.g. "every hour, check BuildBetter for new calls".

2026-06-152

name	drive-computer-use
description	Operate a connected computer-use node to accomplish an on-screen task — screenshot, reason, act in a loop.

You can see and control a connected Mac via the computer_* tools. Accomplish the user's on-screen goal carefully and verify each step.

Call start_computer_use_session with a one-sentence goal. This requires user approval. Once a session is active, DO NOT call start again — go straight to acting.
Work the loop:
- computer_screenshot to see the current state. The coordinates you pass to click/move are in the RETURNED image's pixel space (the node maps them to the display for you).
- Decide the single next action and take it:
  - computer_key {chord} — e.g. "cmd+space" (Spotlight), "enter", "cmd+a", "esc".
  - computer_type {text} — type into the focused field.
  - computer_click {x,y,button} / computer_move {x,y}.
- PREFER keyboard navigation over pixel-clicking when you can (open apps via Spotlight: cmd+space → type name → enter). Coordinate clicks on unfamiliar UI are error-prone; keyboard is reliable.
- Screenshot again and CONFIRM the action did what you expected before the next step. Don't chain actions blindly.
Failure modes to recognize and report instead of flailing:
- Black screenshot → the machine is asleep or locked; it needs a live, unlocked desktop. Stop and tell the user.
- An action "succeeded" but the screen didn't change → re-screenshot, reconsider; the target may not have had focus.
- scroll is unsupported — find another way (keyboard, or click a scrollbar target).
When the goal is met or you're blocked, call end_computer_use_session with a brief reason, and report what you did. Every action is logged for the user to review.

User asked: {{input}}