computer-control

name	computer-control
description	Full computer control via screen capture + vision analysis + mouse/keyboard/window automation. See the screen, understand UI, then click, type, and manage windows.

Computer Control

See → Think → Act → Verify loop for GUI automation on Windows.

Combines screenshot capture, vision model analysis, and Win32 input APIs to give Claude eyes and hands on your desktop.

Architecture

┌──────────┐     ┌──────────┐     ┌──────────────┐     ┌──────────┐
│  See     │ ──▶ │  Think   │ ──▶ │    Act       │ ──▶ │  Verify  │
│ screen.  │     │ vision   │     │ mouse.ps1    │     │ screen.  │
│ ps1      │     │ analysis │     │ keyboard.ps1  │     │ ps1+vis. │
└──────────┘     └──────────┘     │ window.ps1    │     └──────────┘
                                  └──────────────┘

Scripts

All scripts are in scripts/ directory relative to this skill. Call them via PowerShell:

Script	Purpose	Key Actions
screen.ps1	Screenshot capture	capture, monitors
mouse.ps1	Mouse control	getpos, move, click, clickat, dclick, rclick, drag, scroll, mousedown, mouseup
keyboard.ps1	Keyboard input	type, combo, key, shortcut
window.ps1	Window management	list, focus, info, move, resize, minimize, maximize, restore, close, find

All script paths below use $SKILL as shorthand for <path-to-this-skill>/scripts.

Quick Reference

Screen Capture

# Full virtual screen (all monitors) → returns file path
powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet

# Specific monitor
powershell .claude/skills/computer-control/scripts/screen.ps1 -Monitor 0 -Quiet

# Capture a region
powershell .claude/skills/computer-control/scripts/screen.ps1 -Region "100,200,500,400" -Quiet

# List monitors
powershell .claude/skills/computer-control/scripts/screen.ps1 -Action monitors

# Custom output path
powershell .claude/skills/computer-control/scripts/screen.ps1 -Output "C:\temp\shot.png"

Mouse Control

# Get current position
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action getpos

# Move cursor
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action move -X 500 -Y 300

# Click at position
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X 500 -Y 300

# Double-click, right-click
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action dclickat -X 500 -Y 300
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action rclickat -X 500 -Y 300

# Drag from A to B
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action drag -X1 100 -Y1 200 -X2 500 -Y2 600

# Scroll (120 = one notch up, -120 = one notch down)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action scroll -Amount -120

# Hold and release (for complex drags)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action mousedown
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action mouseup

Keyboard Input

# Type text
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "Hello World"

# Key combination
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Ctrl+C"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Alt+Tab"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Win+R"

# Single special key
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Enter"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Escape"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Tab"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "F5"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Down"

# Windows shortcuts
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action shortcut -Keys "Win+D"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action shortcut -Keys "Win+E"

Window Management

# List all visible windows
powershell .claude/skills/computer-control/scripts/window.ps1 -Action list

# Find window by partial title
powershell .claude/skills/computer-control/scripts/window.ps1 -Action find -Title "Chrome"

# Focus a window
powershell .claude/skills/computer-control/scripts/window.ps1 -Action focus -Title "Notepad"

# Get window position/size
powershell .claude/skills/computer-control/scripts/window.ps1 -Action info -Title "Calculator"

# Move window
powershell .claude/skills/computer-control/scripts/window.ps1 -Action move -Title "Chrome" -X 0 -Y 0

# Resize and reposition
powershell .claude/skills/computer-control/scripts/window.ps1 -Action resize -Title "Chrome" -Width 1280 -Height 720 -X 0 -Y 0

# Minimize / Maximize / Restore / Close
powershell .claude/skills/computer-control/scripts/window.ps1 -Action minimize -Title "Settings"
powershell .claude/skills/computer-control/scripts/window.ps1 -Action maximize -Title "Chrome"
powershell .claude/skills/computer-control/scripts/window.ps1 -Action restore -Title "Chrome"
powershell .claude/skills/computer-control/scripts/window.ps1 -Action close -Title "Notepad"

The See → Think → Act → Verify Loop

This is the core workflow. Follow this pattern for every GUI interaction:

Step 1: See (Capture Screen)

powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet
# Output: C:\Users\lanxi\AppData\Local\Temp\claude_screenshot_20260620_143022_456.png

Step 2: Think (Analyze with Vision)

Use the vision skill to understand what's on screen:

python .claude/skills/vision/vision.py "<screenshot_path>" "Describe this screen: what UI elements do you see? Identify buttons, text fields, menus, links, and their approximate positions (estimate x,y coordinates for each interactive element)."

For coordinate estimation, ask specifically:

python .claude/skills/vision/vision.py "<screenshot_path>" "For the [specific UI element I want to interact with], estimate its center pixel coordinates (x,y). The screen resolution is WxH."

Step 3: Act (Execute)

Based on vision analysis, execute the appropriate action:

# Click a button at estimated coordinates
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X <x> -Y <y>

# Type in a text field (click it first, then type)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X <x> -Y <y>
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "..."

Step 4: Verify (Screenshot + Check)

powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet

python .claude/skills/vision/vision.py "<new_screenshot>" "Did the action succeed? Compare: [describe expected outcome]."

If not successful, go back to Step 2 with the new screenshot and adjust.

Patterns

Pattern 1: Click a Button by Description

1. Screenshot full screen
2. Vision: "Find the 'Save' button and estimate its center coordinates"
3. Click at the estimated position
4. Screenshot to verify button was clicked (e.g., dialog appeared, page changed)

Pattern 2: Fill a Form

1. Screenshot
2. Vision: "Find the following fields and estimate coordinates: Name field, Email field, Submit button"
3. Click Name field → type name
4. Click Email field → type email
5. Click Submit button
6. Screenshot → Vision: "Did the form submit successfully?"

Pattern 3: Navigate a Menu

1. Screenshot
2. Vision: "Find the 'File' menu and its position. What menu items are visible?"
3. Click File menu
4. Screenshot → Vision: "What dropdown items appeared? Find 'Export' position."
5. Click Export
6. Screenshot → Verify

Pattern 4: Drag and Drop

1. Screenshot
2. Vision: "Find the element to drag and the drop target. Estimate both positions."
3. Drag from source to target
4. Screenshot → Verify element is in new position

Pattern 5: Scroll and Scan

1. Screenshot top of page
2. Vision: "Is [target element] visible?"
3. If no → scroll down → screenshot → repeat
4. If yes → interact with it

Pattern 6: Wait for UI Change

1. Screenshot
2. Perform action (click button, submit form, etc.)
3. Wait 1-3 seconds
4. Screenshot
5. Vision: "Has [expected change] occurred? Compare with previous state."
6. If not, wait longer or investigate

Coordinate Estimation

When asking vision to estimate coordinates, provide screen context:

python .claude/skills/vision/vision.py "screen.png" \
  "The screen is 1920x1080. 
   Find the 'Start' button (Windows taskbar, bottom-left corner). 
   Estimate the CENTER x,y pixel coordinates of this button. 
   Output format: x,y"

For better accuracy:

• Mention known screen dimensions
• Describe the element clearly by its label, position relative to other elements, or visual appearance
• Ask for the center of the element, not its edge
• If unsure, ask vision to describe the layout first, then pinpoint in a follow-up

Safety Rules

1. Confirm destructive actions — before clicking Delete, Remove, Uninstall, or similar
2. Small movements first — when navigating unfamiliar UI, move step by step
3. Always verify — screenshot after every action to confirm it did what you expected
4. Report coordinates before clicking — tell the user "Clicking at (X,Y) on [element]" before executing
5. Handle errors gracefully — if vision can't find an element, ask the user or try scrolling
6. CLEAN UP SCREENSHOTS — delete screenshot files after the task is complete:

   Remove-Item "C:\Users\lanxi\AppData\Local\Temp\claude_screenshot_*.png" -Force
   

   Or during the session, keep only the most recent 1-2 screenshots and delete older ones.

Troubleshooting

Problem	Solution
Click missed target	Re-screenshot, re-estimate. Try clicking slightly above/below/left/right by 5-10px
Window won't focus	Use window.ps1 -Action focus first, then interact
SendKeys types wrong chars	Use keyboard.ps1 -Action combo with Win32 API instead
Vision can't find element	Scroll or resize window. Provide more context about where element should be
Element behind another window	Focus the target window first, then screenshot
Screen resolution unknown	Run screen.ps1 -Action monitors first

Dependencies

• Windows 10/11 with PowerShell 5.1+
• .NET Framework 4.x (included with Windows)
• Vision skill (~/.claude/skills/vision/) with at least one API key configured
• No external installations required — all scripts use built-in .NET/Win32 APIs

Example: Full Workflow

Here's a complete example — opening Notepad, typing text, and saving:

# 1. Open Run dialog and launch Notepad
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action shortcut -Keys "Win+R"
Start-Sleep -Milliseconds 500
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "notepad{ENTER}"
Start-Sleep -Seconds 1

# 2. Verify Notepad opened
$shot = powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet
# vision: "Is Notepad visible? What's the active window?"

# 3. Type some text
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "Hello from Claude! This is computer control in action."

# 4. Save with Ctrl+S
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Ctrl+S"
Start-Sleep -Milliseconds 500

# 5. Screenshot to see Save dialog
$shot2 = powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet
# vision: "Find the file name field in the Save dialog. Estimate coordinates."

# 6. Type file name and click Save
# (coordinates estimated by vision)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X <x> -Y <y>
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "test_note.txt"
# ... continue with Save button

# 7. Clean up screenshots
Remove-Item "$env:TEMP\claude_screenshot_*.png" -Force