somatic

name	somatic
description	Native desktop UI automation via Set-of-Marks screenshots — click, type, and scroll by YOLO-detected element ID without guessing coordinates
allowed-tools	Bash(somatic *) Read

SoMatic

Use SoMatic when you need to operate a native desktop UI with screenshots, mouse, and keyboard.

How images reach you

MCP path: annotated screenshots arrive inline as image content in the tool response — you see the image directly, no extra step needed.

CLI path: somatic screenshot --annotate writes the annotated PNG to disk and returns its path in screenshot.annotated_path. Use the Read tool on that path to see the image. The JSON output is text only — use Read to get the actual image.

Operating Loop

1. At the start of a session, run vision_init (MCP) or somatic vision init (CLI). This loads the YOLO ONNX model into a background daemon. First-ever run may take 1–3 minutes; subsequent runs are near-instant. Require "started": true (or "already_running": true) before continuing.

2. Always begin a task by taking an annotated screenshot and visually inspecting it.

   • MCP: call screenshot_annotated — the image arrives inline.
   • CLI: run somatic screenshot --annotate, then Read the path at screenshot.annotated_path.

3. Scan the image before acting. Check every region — taskbar, dock, desktop icons, system tray, open windows — and identify which numbered mark corresponds to the element you want.

4. Prefer clicks on visible elements over keyboard navigation. If the target is already present in the annotated screenshot (a taskbar icon, a tab, a button, a link), click its mark id. Do not open Start menu / Run / search when the thing you want is already on screen.

5. Use the keyboard for what keyboards are for, not as a shortcut around looking. Keyboard is the right tool for:

   • typing free-form text (type_text "hello")
   • key chords that aren't UI elements (hotkey ctrl s, press enter, hotkey alt tab)
   • launching something that genuinely isn't visible anywhere (then use Win+S, type, screenshot the results, click the best match by mark id — don't blind-press Enter)

6. Inspect the JSON returned by the screenshot tool: marks contains id, bbox, center, and confidence. There are no captions — refer to elements by id and verify visually.

7. Click by mark id. click <id> automatically clicks the center of that mark's bounding box — you never need to calculate or look up any pixel coordinates.

   • click 4 → clicks the center of mark 4's bbox automatically
   • move 7 → moves the cursor to the center of mark 7's bbox automatically
   • scroll -5 --target 2 → scrolls near mark 2 automatically
   • NEVER extract center or bbox from the JSON and pass raw pixel coordinates like click 540,320. That defeats the purpose of mark ids. click <id> does it for you.

8. When YOLO doesn't annotate the exact target — empty text inputs, fields that follow a labelled icon, gaps between buttons — use click_near with a dx/dy offset from the nearest visible mark:

   • click_near 12 --dx 300 --dy 0 (300 px to the right of mark 12's center, automatically)

9. Use raw coordinates only as a last resort when no mark and no nearby anchor exists at all:

   • click 640,420

10. Re-screenshot after every consequential action. Mark IDs are reassigned per screenshot — never apply an id from one screenshot to another screenshot's state.

11. At the end of the session, run vision_stop to free the model's memory.

12. If something goes wrong, run doctor and vision_status.

Decision Rule: Click, Click-Near, or Type?

• Target visible as a mark → click <id>. SoMatic resolves the id to the bbox center for you. Don't calculate coordinates. Don't open a launcher.
• Target NOT visible as a mark but adjacent to one → click_near <id> --dx ... --dy .... SoMatic resolves the anchor id to its center and applies your offset.
• Target's container visible but not the target itself → click into the container first, re-screenshot, then act on the new marks.
• Target genuinely invisible → keyboard shortcut, then screenshot again before doing anything else.

Anti-patterns — never do these:

• ❌ click 540,320 when you could use click 4 — marks are already positioned at the right place
• ❌ Reading center from the marks JSON and passing those pixels to click — click <id> does this automatically
• ❌ Calculating coordinates by eyeballing the annotated image — click <id> is always more accurate

Command Rules

• Treat command output as JSON, not prose.
• If a screenshot tool returns vision_unavailable, call vision_init and retry.
• Don't pre-emptively press Escape or click empty space to "clear state" — trust what the last screenshot shows.

Common Commands

CLI form:

somatic vision init
somatic screenshot --annotate              # returns annotated_path; use Read to view
somatic click <id>
somatic click-near <id> --dx 100 --dy 0
somatic type "text"
somatic hotkey ctrl s
somatic press enter
somatic scroll -4
somatic wait 1
somatic vision stop

MCP form (same names with underscores): vision_init, screenshot_annotated, click, click_near, type_text, hotkey, press, scroll, wait, vision_stop.

Safety

SoMatic controls the real desktop session. Verify visible state with screenshots before typing, clicking destructive controls, submitting forms, deleting files, or changing system settings.