computer-use-agents

name	computer-use-agents
description	Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control. Use when: computer use, desktop automation agent, screen control AI, vision-based agent, GUI automation.
source	vibeship-spawner-skills (Apache 2.0)

Computer Use Agents

Patterns

Perception-Reasoning-Action Loop

The fundamental architecture of computer use agents: observe screen, reason about next action, execute action, repeat. This loop integrates vision models with action execution through an iterative pipeline.

Key components:

1. PERCEPTION: Screenshot captures current screen state
2. REASONING: Vision-language model analyzes and plans
3. ACTION: Execute mouse/keyboard operations
4. FEEDBACK: Observe result, continue or correct

Critical insight: Vision agents are completely still during "thinking" phase (1-5 seconds), creating a detectable pause pattern.

When to use: ['Building any computer use agent from scratch', 'Integrating vision models with desktop control', 'Understanding agent behavior patterns']

from anthropic import Anthropic
from PIL import Image
import base64
import pyautogui
import time

class :
    

     ():
        .client = client
        .model = model
        .max_steps =   
        .action_delay =   

     () -> :
        
        screenshot = pyautogui.screenshot()
        
        screenshot = screenshot.resize((, ), Image.LANCZOS)

         io
        buffer = io.BytesIO()
        screenshot.save(buffer, =)
         base64.b64encode(buffer.getvalue()).decode()

     () -> :
        
        action_type = action.get()

         action_type == :
            x, y = action[], action[]
            button = action.get(, )
            pyautogui.click(x, y, button=button)
             {: , : }

         action_type == :
            text = action[]
            pyautogui.typewrite(text, interval=)
             {: , : }

         action_type == :
            key = action[]
            pyautogui.press(key)
             {: , : }

         action_type == :
            direction = action.get(, )
            amount = action.get(, )
            scroll = -amount  direction ==   amount
            pyautogui.scroll(scroll)
             {: , :