| name | desktop |
| description | Interact with the user's desktop — screenshots, GUI automation, window management, and app launching via computer use tools. |
| metadata | {"author":"fae","version":"1.0"} |
Use Fae's built-in computer use tools to interact with the user's GUI when they ask you to automate desktop tasks.
Available Tools
Fae has these native tools for desktop interaction:
| Tool | What it does |
|---|
screenshot | Capture full screen or a specific app window |
camera | Take a photo using the webcam |
read_screen | Read visible UI elements and their accessibility labels |
click | Click a UI element by label or screen coordinates |
type_text | Type text into the focused field or application |
scroll | Scroll up, down, left, or right |
find_element | Find a UI element by label, role, or coordinates |
When to Use
- User asks to take a screenshot
- User wants to click a button or UI element
- User wants to automate a GUI workflow (e.g. "click the Save button")
- User asks to read what's on screen
- User wants to type text into an application
- User asks to manage windows (list, focus, launch apps)
Safety Rules
- Always describe what you are about to do before executing a desktop action.
- Never automate credential entry (passwords, tokens, keys) without explicit user permission.
- Prefer label-based actions over coordinates when possible — they are more robust.
- Take a screenshot or read the screen first when unsure of the current UI state, so you can plan accurately.
- Maximum 10 action steps per turn. If a task requires more, complete what you can and summarise what remains.
- macOS permissions: Screen Recording and Accessibility permissions are required. If not yet granted, Fae will request them via JIT permission flow.
Platform Notes (macOS)
- Computer use requires Screen Recording permission (for screenshot/read_screen) and Accessibility permission (for click/type_text/scroll) in System Settings > Privacy & Security.
- Fae uses native macOS APIs (Accessibility Inspector) for element targeting — no third-party automation tools needed.
- Do not mention Peekaboo or xdotool to users — the native tools are sufficient.
Do Not Use For
- System-level automation that requires Admin privileges (use
bash for that)
- Multi-step workflows that exceed 10 steps (break into smaller requests)
- Automation of protected system surfaces (System Preferences, login items, etc.)
