desktop-control

name	Desktop Control
description	Control mouse, keyboard, and screen for desktop automation tasks

Desktop Control Skill

This skill provides comprehensive desktop automation capabilities through PyAutoGUI, allowing AI agents to control the mouse, keyboard, take screenshots, and interact with the desktop environment.

How to Use This Skill

Run setup environment script by executing ./setup.sh in this skill folder to install dependencies. As an AI agent, first you should move to this skill folder, then you can invoke desktop automation commands using the uv run desktop-agent CLI.

Command Structure

All commands follow this pattern:

uv run desktop-agent <category> <command> [arguments] [options]

Categories:

• mouse - Mouse control
• keyboard - Keyboard input
• screen - Screenshots and screen analysis
• message - User dialogs
• app - Application control (open, focus, list windows)

Available Commands

🖱️ Mouse Control (mouse)

Control cursor movement and clicks.

# Move cursor to coordinates
uv run desktop-agent mouse move <x> <y> [--duration SECONDS]

# Click at current position or specific coordinates
uv run desktop-agent mouse click [x] [y] [--button left|right|middle] [--clicks N]

# Specialized clicks
uv run desktop-agent mouse double-click [x] [y]
uv run desktop-agent mouse right-click [x] [y]
uv run desktop-agent mouse middle-click [x] [y]

# Drag to coordinates
uv run desktop-agent mouse drag <x> <y> [--duration SECONDS] [--button BUTTON]

# Scroll (positive=up, negative=down)
uv run desktop-agent mouse scroll <clicks> [x] [y]

# Get current mouse position
uv run desktop-agent mouse position

Examples:

# Move to center of 1920x1080 screen
uv run desktop-agent mouse move 960 540 --duration 0.5

# Right-click at specific location
uv run desktop-agent mouse right-click 500 300

# Scroll down 5 clicks
uv run desktop-agent mouse scroll -5

⌨️ Keyboard Control (keyboard)

Type text and execute keyboard shortcuts.

# Type text
uv run desktop-agent keyboard write "<text>" [--interval SECONDS]

# Press keys
uv run desktop-agent keyboard press <key> [--presses N] [--interval SECONDS]

# Execute hotkey combination (comma-separated)
uv run desktop-agent keyboard hotkey "<key1>,<key2>,..."

# Hold/release keys
uv run desktop-agent keyboard keydown <key>
uv run desktop-agent keyboard keyup <key>

Examples:

# Type text with natural delay
uv run desktop-agent keyboard write "Hello World" --interval 0.05

# Copy selected text
uv run desktop-agent keyboard hotkey "ctrl,c"

# Open Task Manager
uv run desktop-agent keyboard hotkey "ctrl,shift,esc"

# Press Enter 3 times
uv run desktop-agent keyboard press enter --presses 3

Common Key Names:

• Modifiers: ctrl, shift, alt, win
• Special: enter, tab, esc, space, backspace, delete
• Function: f1 through f12
• Arrows: up, down, left, right

🖼️ Screen & Screenshots (screen)

Capture screenshots and analyze screen content. Supports targeting specific windows.

# Take full screen screenshot
uv run desktop-agent screen screenshot <filename> [--region "x,y,width,height"]
# Take current active window screenshot
uv run desktop-agent screen screenshot <filename> [--region "x,y,width,height"] [--window <title>] [--active]

# Locate image on screen or within window
uv run desktop-agent screen locate <image_path> [--confidence 0.0-1.0] [--window <title>] [--active]
uv run desktop-agent screen locate-center <image_path> [--confidence 0.0-1.0] [--window <title>] [--active]

# Locate text using OCR fullscreen (requires --with easyocr and --lang)
uv run --with easyocr desktop-agent screen locate-text-coordinates <text> --lang en [--window <title>]
uv run --with easyocr desktop-agent screen read-all-text --lang en [--window <title>]

# Utility commands
uv run desktop-agent screen pixel <x> <y>
uv run desktop-agent screen size
uv run desktop-agent screen on-screen <x> <y>

Examples:

# Full screen screenshot (default, recommended)
uv run desktop-agent screen screenshot fullscreen.png

# Screenshot of a specific application (when targeting a specific window)
uv run desktop-agent screen screenshot chrome.png --window "Google Chrome"

# Screenshot of active window only (when needed)
uv run desktop-agent screen screenshot active.png --active

# Locate image on full screen
uv run desktop-agent screen locate-center button.png

💬 Message Dialogs (message)

Display user interaction dialogs.

# Show alert
uv run desktop-agent message alert "<text>" [--title TITLE] [--button BUTTON]

# Show confirmation dialog
uv run desktop-agent message confirm "<text>" [--title TITLE] [--buttons "OK,Cancel"]

# Prompt for input
uv run desktop-agent message prompt "<text>" [--title TITLE] [--default TEXT]

# Password input
uv run desktop-agent message password "<text>" [--title TITLE] [--mask CHAR]

Examples:

# Simple alert
uv run desktop-agent message alert "Task completed!"

# Get user confirmation
uv run desktop-agent message confirm "Continue with operation?"

# Ask for user input
uv run desktop-agent message prompt "Enter your name:"

📱 Application Control (app)

Control applications across Windows, macOS, and Linux.

# Open an application by name
uv run desktop-agent app open <name> [--arg ARGS...]

# Focus on a window by title/name
uv run desktop-agent app focus <name>

# List all visible windows
uv run desktop-agent app list

Examples:

# Windows: Open Notepad
uv run desktop-agent app open notepad

# Windows: Open Chrome with a URL
uv run desktop-agent app open "chrome" --arg "https://google.com"

# macOS: Open Safari
uv run desktop-agent app open "Safari"

# Focus on a specific window
uv run desktop-agent app focus "Untitled - Notepad"

# List all open windows
uv run desktop-agent app list

Common Automation Workflows

Workflow 1: Open Application and Type

# Open notepad directly (cross-platform)
uv run desktop-agent app open notepad

# Wait for app to open, then focus it
uv run desktop-agent app focus notepad

# Type some text
uv run desktop-agent keyboard write "Hello from Desktop Skill!"

Workflow 2: Screenshot + Analysis

# Get screen size first
uv run desktop-agent screen size

# Take full screenshot
uv run desktop-agent screen screenshot current_screen.png

# Check if specific UI element is visible
uv run desktop-agent screen locate save_button.png

Workflow 3: Form Filling

# Click first field
uv run desktop-agent mouse click 300 200

# Fill field
uv run desktop-agent keyboard write "John Doe"

# Tab to next field
uv run desktop-agent keyboard press tab

# Fill second field
uv run desktop-agent keyboard write "[e-mail]"

# Submit form (Enter)
uv run desktop-agent keyboard press enter

Workflow 4: Copy/Paste Operations

# Select all text
uv run desktop-agent keyboard hotkey "ctrl,a"

# Copy
uv run desktop-agent keyboard hotkey "ctrl,c"

# Click destination
uv run desktop-agent mouse click 500 600

# Paste
uv run desktop-agent keyboard hotkey "ctrl,v"

Safety Considerations

When using this skill, AI agents should:

1. Verify coordinates: Use screen size and on-screen before clicking
2. Add delays: Insert appropriate delays between commands for UI responsiveness
3. Validate images: Ensure image files exist before using locate commands
4. Handle failures: Commands may fail if windows change or elements move
5. User safety: Always confirm destructive actions with user via message confirm

Troubleshooting

PyAutoGUI Fail-Safe

PyAutoGUI has a fail-safe: moving mouse to screen corner aborts operations. This is a safety feature.

Image not found

When using screen locate, ensure:

• Image file exists and path is correct
• Adjust --confidence (try 0.7-0.9)
• Image matches exact screen appearance (resolution, colors)

Getting Help

# Show all available commands
uv run desktop-agent --help

# Show commands for specific category
uv run desktop-agent mouse --help
uv run desktop-agent keyboard --help
uv run desktop-agent screen --help
uv run desktop-agent message --help

# Show help for specific command
uv run desktop-agent mouse move --help

Integration Tips for AI Agents

1. Always check screen size first when working with absolute coordinates
2. Use relative positioning when possible (e.g., get current position, calculate offset)
3. Combine commands for complex workflows
4. Validate before executing (e.g., check if image exists on screen)
5. Provide user feedback using message dialogs for important operations
6. Handle errors gracefully - commands may fail if UI state changes

Performance Notes

• Mouse movements with --duration are animated and take time
• Image location (locate) can be slow on large screens - use regions when possible
• Keyboard commands are generally fast (< 100ms)
• Screenshots depend on screen resolution and region size

Output Format

All commands output structured JSON by default, ideal for programmatic use by AI agents:

uv run desktop-agent mouse position
# Output: {"success": true, "command": "mouse.position", "timestamp": "2026-01-31T10:00:00Z", "duration_ms": 5, "data": {"position": {"x": 960, "y": 540}}}

Response Schema

All JSON responses follow this schema:

{
  "success": true,
  "command": "category.command",
  "timestamp": "2026-01-31T10:00:00Z",
  "duration_ms": 150,
  "data": { ... },
  "error": null
}

Error Response Schema

{
  "success": false,
  "command": "category.command",
  "timestamp": "2026-01-31T10:00:00Z",
  "duration_ms": 50,
  "data": null,
  "error": {
    "code": "image_not_found",
    "message": "Image file 'button.png' not found",
    "details": {},
    "recoverable": true
  }
}

Error Codes

Code	Description
success	Command succeeded
invalid_argument	Invalid command arguments
coordinates_out_of_bounds	Coordinates outside screen
image_not_found	Image file not found or not on screen
window_not_found	Target window not found
ocr_failed	OCR operation failed
application_not_found	Application not found
permission_denied	Permission denied
platform_not_supported	Platform not supported
timeout	Operation timed out
unknown_error	Unknown error

Mouse move:

uv run desktop-agent mouse move 960 540

{"success": true, "command": "mouse.move", "timestamp": "...", "duration_ms": 150, "data": {"x": 960, "y": 540, "duration": 0}, "error": null}

Screen size:

uv run desktop-agent screen size

{"success": true, "command": "screen.size", "timestamp": "...", "duration_ms": 5, "data": {"size": {"width": 1920, "height": 1080}}, "error": null}

Locate image:

uv run desktop-agent screen locate button.png

{"success": true, "command": "screen.locate", "timestamp": "...", "duration_ms": 250, "data": {"image_found": true, "bounding_box": {"left": 100, "top": 200, "width": 50, "height": 30, "center_x": 125, "center_y": 215}}, "error": null}

List windows:

uv run desktop-agent app list

{"success": true, "command": "app.list", "timestamp": "...", "duration_ms": 100, "data": {"windows": ["Untitled - Notepad", "Google Chrome", "Visual Studio Code"]}, "error": null}

Error example:

uv run desktop-agent screen locate missing.png

{"success": false, "command": "screen.locate", "timestamp": "...", "duration_ms": 50, "data": null, "error": {"code": "image_not_found", "message": "Image file 'missing.png' not found", "details": {}, "recoverable": true}}

Effective Usage Guide for AI Agents

This section teaches AI agents how to use this skill effectively with optimal command sequences and best practices.

🎯 Core Strategy: Observe First, Then Act

Always understand the current state before performing actions. This avoids clicking wrong coordinates or typing in the wrong window.

Recommended Initial Sequence:

# 1. Get screen dimensions to understand your workspace
uv run desktop-agent screen size
uv run desktop-agent app list
uv run desktop-agent mouse position

📋 Recommended Command Sequences by Task

Open and Interact with Application

# ✅ CORRECT: Open, wait, verify, then interact
uv run desktop-agent app open notepad              # Step 1: Open app
uv run desktop-agent app list
uv run desktop-agent app focus "Notepad"
uv run desktop-agent keyboard write "Hello World"  # Step 4: Now safe to type

# ❌ WRONG: Type immediately without verification
uv run desktop-agent app open notepad
uv run desktop-agent keyboard write "Hello World"  # May type in wrong window!

Find and Click UI Element (Image-Based)

# ✅ CORRECT: Locate first, click if found
uv run desktop-agent screen locate-center button.png --confidence 0.8
# Check if success=true and coordinates are valid
uv run desktop-agent mouse click 125 215  # Use returned coordinates

# ❌ WRONG: Click without verifying element exists
uv run desktop-agent mouse click 125 215  # Might click wrong area!

Find and Click UI Element (Text-Based with OCR)

# ✅ CORRECT: Read screen text, then locate specific text (fullscreen)
uv run --with easyocr desktop-agent screen read-all-text --lang en
uv run --with easyocr desktop-agent screen locate-text-coordinates "Save" --lang en
# Use returned coordinates to click

# For window-specific OCR (when targeting a specific window):
uv run --with easyocr desktop-agent screen locate-text-coordinates "OK" --lang en --window "Dialog Title"

Fill a Form with Multiple Fields

# ✅ CORRECT: Click each field explicitly before typing
uv run desktop-agent mouse click 300 200           # Click first field
uv run desktop-agent keyboard write "John Doe"
uv run desktop-agent mouse click 300 250           # Click second field (more reliable)
uv run desktop-agent keyboard write "[e-mail]"
uv run desktop-agent mouse click 300 300           # Click third field
uv run desktop-agent keyboard write "555-1234"

# OR use Tab navigation (less reliable if field order changes)
uv run desktop-agent mouse click 300 200
uv run desktop-agent keyboard write "John Doe"
uv run desktop-agent keyboard press tab
uv run desktop-agent keyboard write "[e-mail]"
uv run desktop-agent keyboard press tab
uv run desktop-agent keyboard write "555-1234"
uv run desktop-agent keyboard press enter          # Submit

Take Targeted Screenshots for Analysis

# ✅ CORRECT: Screenshot specific windows for faster processing
uv run desktop-agent app list --json                           # Find exact window title
uv run desktop-agent screen screenshot app.png --window "Google Chrome"

# Full screen (default, recommended)
uv run desktop-agent screen size
uv run desktop-agent screen screenshot fullscreen.png

# For active window only (when needed)
uv run desktop-agent screen screenshot active.png --active

Safe Drag and Drop

# ✅ CORRECT: Move to start, verify position, then drag
uv run desktop-agent mouse move 100 200                 # Move to source
uv run desktop-agent mouse position              # Verify position
uv run desktop-agent mouse drag 500 400 --duration 0.5  # Drag to destination

# For precision, use slower duration
uv run desktop-agent mouse drag 500 400 --duration 1.0

🔄 Error Recovery Patterns

When Window Not Found

# Pattern: List windows, find closest match, retry
uv run desktop-agent app focus "Chrome"             # Fails with window_not_found
uv run desktop-agent app list                # See actual window titles
# Output shows: "Google Chrome - My Page"
uv run desktop-agent app focus "Google Chrome"      # Use correct title

When Image Not Found

# Pattern: Adjust confidence or take new screenshot
uv run desktop-agent screen locate button.png --confidence 0.9
uv run desktop-agent screen locate button.png --confidence 0.7
# If still failing, capture current state for analysis
uv run desktop-agent screen screenshot current.png

When Click Seems to Miss

# Pattern: Verify coordinates are on screen
uv run desktop-agent screen size             # Get screen bounds
uv run desktop-agent screen on-screen 1500 900      # Check if coords are valid
uv run desktop-agent mouse move 1500 900            # Move first to visualize
uv run desktop-agent mouse click                    # Then click at current position

⚡ Performance Optimization

Minimize Screenshots

# ✅ GOOD: Screenshot only the region you need
uv run desktop-agent screen screenshot button_area.png --region "100,200,200,100"

# ✅ GOOD: Screenshot specific window instead of full screen  
uv run desktop-agent screen screenshot chrome.png --window "Google Chrome"

# ❌ SLOW: Full screen capture when you only need a small area
uv run desktop-agent screen screenshot full.png

Batch Keyboard Input

# ✅ FASTER: Write entire text at once
uv run desktop-agent keyboard write "This is a complete sentence with all the text."

# ❌ SLOWER: Multiple write commands
uv run desktop-agent keyboard write "This is "
uv run desktop-agent keyboard write "a complete "
uv run desktop-agent keyboard write "sentence."

Use Hotkeys Over Mouse When Possible

# ✅ FASTER: Use keyboard shortcuts
uv run desktop-agent keyboard hotkey "ctrl,s"       # Save
uv run desktop-agent keyboard hotkey "ctrl,a"       # Select all
uv run desktop-agent keyboard hotkey "ctrl,shift,s" # Save as

# ❌ SLOWER: Navigate menu with mouse
uv run desktop-agent mouse click 50 30              # Click File menu
uv run desktop-agent mouse click 60 80              # Click Save option

🛡️ Defensive Programming Patterns

Always Verify Critical Actions

# Before destructive action, confirm with user
uv run desktop-agent message confirm "This will delete all files. Continue?" --title "Warning"
# Check output: if "Cancel" was clicked, abort operation

Use JSON Mode for Reliable Parsing

# ✅ RELIABLE: Parse structured JSON output
uv run desktop-agent screen locate button.png
# Parse: {"success": true, "data": {"center_x": 125, "center_y": 215}}

# ❌ FRAGILE: Parse text output
uv run desktop-agent screen locate button.png
# Parse: "Found at: Box(left=100, top=200, width=50, height=30)"

Validate Before Multi-Step Operations

# Multi-step file operation with validation
uv run desktop-agent app list
uv run --with easyocr desktop-agent screen locate-text-coordinates "File" --lang en
uv run desktop-agent mouse click <returned_x> <returned_y>
uv run --with easyocr desktop-agent screen locate-text-coordinates "Save As" --lang en
uv run desktop-agent mouse click <returned_x> <returned_y>

🎮 Platform-Specific Considerations

Windows

# Common Windows shortcuts
uv run desktop-agent keyboard hotkey "win,d"        # Show desktop
uv run desktop-agent keyboard hotkey "win,e"        # Open Explorer
uv run desktop-agent keyboard hotkey "alt,tab"      # Switch windows
uv run desktop-agent keyboard hotkey "win,r"        # Run dialog

# Open apps by name
uv run desktop-agent app open notepad
uv run desktop-agent app open calc
uv run desktop-agent app open mspaint

macOS

# Common macOS shortcuts (use 'command' for Cmd key)
uv run desktop-agent keyboard hotkey "command,space"   # Spotlight
uv run desktop-agent keyboard hotkey "command,tab"     # App switcher
uv run desktop-agent keyboard hotkey "command,q"       # Quit app
uv run desktop-agent keyboard hotkey "command,shift,3" # Screenshot

# Open apps
uv run desktop-agent app open "Safari"
uv run desktop-agent app open "TextEdit"

Linux

# Open apps (uses xdg-open or direct command)
uv run desktop-agent app open firefox
uv run desktop-agent app open gedit

# Common shortcuts may vary by DE
uv run desktop-agent keyboard hotkey "alt,f2"       # Run dialog (many DEs)

📊 Decision Tree: Choosing the Right Command

 Want to interact with an app?
├── App not running → `app open <name>`
├── App running but not focused → `app focus <name>` 
└── Need to verify windows → `app list`

Want to find a UI element?
├── Have reference image → `screen locate-center <image>`
├── Know the text label → `screen locate-text-coordinates "<text>" --lang en` (requires --with easyocr)
└── Need to see all text → `screen read-all-text --lang en` (requires --with easyocr)

Want to click something?
├── Know exact coordinates → `mouse click <x> <y>`
├── Need to find first → Use locate commands above, then click returned coords
└── Not sure if on screen → `screen on-screen <x> <y>` first

Want to type something?
├── Regular text → `keyboard write "<text>"`
├── Keyboard shortcut → `keyboard hotkey "<key1>,<key2>"`
├── Single key press → `keyboard press <key>`
└── Multiple of same key → `keyboard press <key> --presses N`

Integration Tips for AI Agents

1. Always check screen size first when working with absolute coordinates
2. Use relative positioning when possible (e.g., get current position, calculate offset)
3. Combine commands for complex workflows
4. Validate before executing (e.g., check if image exists on screen)
5. Provide user feedback using message dialogs for important operations
6. Handle errors gracefully - commands may fail if UI state changes