Computer Use Agents workflow skill. Use this skill when the user needs Build AI agents that interact with computers like humans do - and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.
Instrucciones de origen · Vista previa de solo lectura
name
computer-use-agents-v2
description
Computer Use Agents workflow skill. Use this skill when the user needs Build AI agents that interact with computers like humans do - and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.
This public intake copy packages plugins/antigravity-awesome-skills/skills/computer-use-agents from https://github.com/sickn33/antigravity-awesome-skills into the native Omni Skills editorial shape without hiding its origin.
Use it when the operator needs the upstream workflow, support files, and repository context to stay intact while the public validator and private enhancer continue their normal downstream flow.
This intake keeps the copied upstream files intact and uses the external_source block in metadata.json plus ORIGIN.md as the provenance anchor for review.
Computer Use Agents Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control.
Imported source sections that did not map cleanly to the public headings are still preserved below or in the support files. Notable imported sections: Patterns, Sharp Edges, Defense in depth - no single solution works, Add human-like variance to actions, Rotate user agents and fingerprints, Fall back to DOM access for web.
When to Use This Skill
Use this section as the trigger filter. It should make the activation boundary explicit before the operator loads files, runs commands, or opens a pull request.
User mentions or implies: computer use
User mentions or implies: desktop automation agent
User mentions or implies: screen control AI
User mentions or implies: vision-based agent
User mentions or implies: GUI automation
User mentions or implies: Claude computer
Operating Table
Situation
Start here
Why it matters
First-time use
metadata.json
Confirms repository, branch, commit, and imported path through the external_source block before touching the copied workflow
Provenance review
ORIGIN.md
Gives reviewers a plain-language audit trail for the imported source
Workflow execution
SKILL.md
Starts with the smallest copied file that materially changes execution
Supporting context
SKILL.md
Adds the next most relevant copied source file without loading the entire package
Handoff decision
## Related Skills
Helps the operator switch to a stronger native skill when the task drifts
Workflow
This workflow is intentionally editorial and operational at the same time. It keeps the imported source useful to the operator while still satisfying the public intake standards that feed the downstream enhancer flow.
python # Instead of drag-and-drop async def reliable_drag(page, source, target): # Step 1: Click and hold await page.mouse.move(source["x"], source["y"]) await page.mouse.down() await asyncio.sleep(0.2) # Step 2: Move in steps steps = 10 for i in range(steps): x = source["x"] + (target["x"] - source["x"]) i / steps y = source["y"] + (target["y"] - source["y"]) i / steps await page.mouse.move(x, y) await asyncio.sleep(0.05) # Step 3: Release await page.mouse.move(target["x"], target["y"]) await asyncio.sleep(0.1) await page.mouse.up()
Confirm the user goal, the scope of the imported workflow, and whether this skill is still the right router for the task.
Read the overview and provenance files before loading any copied upstream support files.
Load only the references, examples, prompts, or scripts that materially change the outcome for the current request.
Execute the upstream workflow while keeping provenance and source boundaries explicit in the working notes.
Validate the result against the upstream expectations and the evidence you can point to in the copied files.
Escalate or hand off to a related skill when the work moves out of this imported workflow's center of gravity.
Imported Workflow Notes
Imported: Break complex actions into steps
# Instead of drag-and-dropasyncdefreliable_drag(page, source, target):
# Step 1: Click and holdawait page.mouse.move(source["x"], source["y"])
await page.mouse.down()
await asyncio.sleep(0.2)
# Step 2: Move in steps
steps = 10for i inrange(steps):
x = source["x"] + (target["x"] - source["x"]) * i / steps
y = source["y"] + (target["y"] - source["y"]) * i / steps
await page.mouse.move(x, y)
await asyncio.sleep(0.05)
# Step 3: Releaseawait page.mouse.move(target["x"], target["y"])
await asyncio.sleep(0.1)
await page.mouse.up()
Imported: Patterns
Perception-Reasoning-Action Loop
The fundamental architecture of computer use agents: observe screen,
reason about next action, execute action, repeat. This loop integrates
vision models with action execution through an iterative pipeline.
Key components:
PERCEPTION: Screenshot captures current screen state
REASONING: Vision-language model analyzes and plans
ACTION: Execute mouse/keyboard operations
FEEDBACK: Observe result, continue or correct
Critical insight: Vision agents are completely still during "thinking"
phase (1-5 seconds), creating a detectable pause pattern.
When to use: Building any computer use agent from scratch,Integrating vision models with desktop control,Understanding agent behavior patterns
from anthropic import Anthropic
from PIL import Image
import base64
import pyautogui
import time
class ComputerUseAgent:
"""
Perception-Reasoning-Action loop implementation.
Based on Anthropic Computer Use patterns.
"""
def __init__(self, client: Anthropic, model: str = "claude-sonnet-4-20250514"):
self.client = client
self.model = model
self.max_steps = 50 # Prevent runaway loops
self.action_delay = 0.5 # Seconds between actions
def capture_screenshot(self) -> str:
"""Capture screen and return base64 encoded image."""
screenshot = pyautogui.screenshot()
# Resize for token efficiency (1280x800 is good balance)
screenshot = screenshot.resize((1280, 800), Image.LANCZOS)
import io
buffer = io.BytesIO()
screenshot.save(buffer, format="PNG")
return base64.b64encode(buffer.getvalue()).decode()
def execute_action(self, action: dict) -> dict:
"""Execute mouse/keyboard action on the computer."""
action_type = action.get("type")
if action_type == "click":
x, y = action["x"], action["y"]
button = action.get("button", "left")
pyautogui.click(x, y, button=button)
return {"success": True, "action": f"clicked at ({x}, {y})"}
elif action_type == "type":
text = action["text"]
pyautogui.typewrite(text, interval=0.02)
return {"success": True, "action": f"typed {len(text)} chars"}
elif action_type == "key":
key = action["key"]
pyautogui.press(key)
return {"success": True, "action": f"pressed {key}"}
elif action_type == "scroll":
direction = action.get("direction", "down")
amount = action.get("amount", 3)
scroll = -amount if direction == "down" else amount
pyautogui.scroll(scroll)
return {"success": True, "action": f"scrolled {direction}"}
elif action_type == "move":
x, y = action["x"], action["y"]
pyautogui.moveTo(x, y)
return {"success": True, "action": f"moved to ({x}, {y})"}
else:
return {"success": False, "error": f"Unknown action: {action_type}"}
def run(self, task: str) -> dict:
"""
Run perception-reasoning-action loop until task complete.
The loop:
1. Screenshot current state
2. Send to vision model with task context
3. Parse action from response
4. Execute action
5. Repeat until done or max steps
"""
messages = []
step_count = 0
system_prompt = """You are a computer use agent. You can see the screen
and control mouse/keyboard.
Available actions (respond with JSON):
- {"type": "click", "x": 100, "y": 200, "button": "left"}
- {"type": "type", "text": "hello world"}
- {"type": "key", "key": "enter"}
- {"type": "scroll", "direction": "down", "amount": 3}
- {"type": "done", "result": "task completed successfully"}
Always respond with ONLY a JSON action object.
Be precise with coordinates - click exactly where needed.
If you see an error, try to recover.
"""
while step_count < self.max_steps:
step_count += 1
# 1. PERCEPTION: Capture current screen
screenshot_b64 = self.capture_screenshot()
# 2. REASONING: Send to vision model
user_content = [
{"type": "text", "text": f"Task: {task}\n\nStep {step_count}. What action should I take?"},
{"type": "image", "source": {
"type": "base64",
"media_type": "image/png",
"data": screenshot_b64
}}
]
messages.append({"role": "user", "content": user_content})
response = self.client.messages.create(
model=self.model,
max_tokens=1024,
system=system_prompt,
messages=messages
)
assistant_message = response.content[0].text
messages.append({"role": "assistant", "content": assistant_message})
# 3. Parse action from response
import json
try:
action = json.loads(assistant_message)
except json.JSONDecodeError:
# Try to extract JSON from response
import re
match = re.search(r'\{[^}]+\}', assistant_message)
if match:
action = json.loads(match.group())
else:
continue
# Check if done
if action.get("type") == "done":
return {
"success": True,
"result": action.get("result"),
"steps": step_count
}
# 4. ACTION: Execute
result = self.execute_action(action)
# Small delay for UI to update
time.sleep(self.action_delay)
return {
"success": False,
"error": "Max steps reached",
"steps": step_count
}
Usage
agent = ComputerUseAgent(Anthropic())
result = agent.run("Open Chrome and search for 'weather today'")
Anti_patterns
Running without step limits (infinite loops)
No delay between actions (UI can't keep up)
Screenshots at full resolution (token explosion)
Ignoring action failures (no recovery)
Sandboxed Environment Pattern
Computer use agents MUST run in isolated, sandboxed environments.
Never give agents direct access to your main system - the security
risks are too high. Use Docker containers with virtual desktops.
Key isolation requirements:
NETWORK: Restrict to necessary endpoints only
FILESYSTEM: Read-only or scoped to temp directories
CREDENTIALS: No access to host credentials
SYSCALLS: Filter dangerous system calls
RESOURCES: Limit CPU, memory, time
The goal is "blast radius minimization" - if the agent goes wrong,
damage is contained to the sandbox.
When to use: Deploying any computer use agent,Testing agent behavior safely,Running untrusted automation tasks
Dockerfile for sandboxed computer use environment
Based on Anthropic's reference implementation pattern
networks:
agent-network:
driver: bridge
internal: true # No internet by default
volumes:
agent-tmp:
Python wrapper with additional runtime sandboxing
import subprocess
import os
from dataclasses import dataclass
from typing import Optional
@dataclass
class SandboxConfig:
"""Configuration for agent sandbox."""
network_allowed: list[str] = None # Allowed domains
max_runtime_seconds: int = 300
max_memory_mb: int = 2048
allow_downloads: bool = False
allow_clipboard: bool = False
class SandboxedAgent:
"""
Run computer use agent in Docker sandbox.
"""
def __init__(self, config: SandboxConfig):
self.config = config
self.container_id: Optional[str] = None
def start(self):
"""Start sandboxed environment."""
# Build network rules
network_rules = ""
if self.config.network_allowed:
for domain in self.config.network_allowed:
network_rules += f"--add-host={domain}:$(dig +short {domain}) "
else:
network_rules = "--network=none"
cmd = f"""
docker run -d \
--name computer-use-sandbox-$$ \
--security-opt no-new-privileges \
--cap-drop ALL \
--memory {self.config.max_memory_mb}m \
--cpus 2 \
--read-only \
--tmpfs /tmp \
{network_rules} \
computer-use-agent:latest
"""
result = subprocess.run(cmd, shell=True, capture_output=True)
self.container_id = result.stdout.decode().strip()
# Set up kill timer
subprocess.Popen([
"sh", "-c",
f"sleep {self.config.max_runtime_seconds} && docker kill {self.container_id}"
])
return self.container_id
def execute_task(self, task: str) -> dict:
"""Execute task in sandbox."""
if not self.container_id:
self.start()
# Send task to agent via API
import requests
response = requests.post(
f"http://localhost:8080/task",
json={"task": task},
timeout=self.config.max_runtime_seconds
)
return response.json()
def stop(self):
"""Stop and remove sandbox."""
if self.container_id:
subprocess.run(f"docker rm -f {self.container_id}", shell=True)
self.container_id = None
Anti_patterns
Running agents on host system directly
Giving sandbox full network access
Running as root in container
No resource limits (denial of service)
Persistent storage (data can leak between runs)
Anthropic Computer Use Implementation
Official implementation pattern using Claude's computer use capability.
Claude 3.5 Sonnet was the first frontier model to offer computer use.
Claude Opus 4.5 is now the "best model in the world for computer use."
Key capabilities:
screenshot: Capture current screen state
mouse: Click, move, drag operations
keyboard: Type text, press keys
bash: Run shell commands
text_editor: View and edit files
Tool versions:
computer_20251124 (Opus 4.5): Adds zoom action for detailed inspection
computer_20250124 (All other models): Standard capabilities
Critical limitation: "Some UI elements (like dropdowns and scrollbars)
might be tricky for Claude to manipulate" - Anthropic docs
When to use: Building production computer use agents,Need highest quality vision understanding,Full desktop control (not just browser)
from anthropic import Anthropic
from anthropic.types.beta import (
BetaToolComputerUse20241022,
BetaToolBash20241022,
BetaToolTextEditor20241022,
)
import subprocess
import base64
from PIL import Image
import io
class AnthropicComputerUse:
"""
Official Anthropic Computer Use implementation.
Requires:
- Docker container with virtual display
- VNC for viewing agent actions
- Proper tool implementations
"""
def __init__(self):
self.client = Anthropic()
self.model = "claude-sonnet-4-20250514" # Best for computer use
self.screen_size = (1280, 800)
def get_tools(self) -> list:
"""Define computer use tools."""
return [
BetaToolComputerUse20241022(
type="computer_20241022",
name="computer",
display_width_px=self.screen_size[0],
display_height_px=self.screen_size[1],
),
BetaToolBash20241022(
type="bash_20241022",
name="bash",
),
BetaToolTextEditor20241022(
type="text_editor_20241022",
name="str_replace_editor",
),
]
def execute_tool(self, name: str, input: dict) -> dict:
"""Execute a tool and return result."""
if name == "computer":
return self._handle_computer_action(input)
elif name == "bash":
return self._handle_bash(input)
elif name == "str_replace_editor":
return self._handle_editor(input)
else:
return {"error": f"Unknown tool: {name}"}
def _handle_computer_action(self, input: dict) -> dict:
"""Handle computer control actions."""
action = input.get("action")
if action == "screenshot":
# Capture via xdotool/scrot
subprocess.run(["scrot", "/tmp/screenshot.png"])
with open("/tmp/screenshot.png", "rb") as f:
img_data = f.read()
# Resize for efficiency
img = Image.open(io.BytesIO(img_data))
img = img.resize(self.screen_size, Image.LANCZOS)
buffer = io.BytesIO()
img.save(buffer, format="PNG")
return {
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": base64.b64encode(buffer.getvalue()).decode()
}
}
elif action == "mouse_move":
x, y = input.get("coordinate", [0, 0])
subprocess.run(["xdotool", "mousemove", str(x), str(y)])
return {"success": True}
elif action == "left_click":
subprocess.run(["xdotool", "click", "1"])
return {"success": True}
elif action == "right_click":
subprocess.run(["xdotool", "click", "3"])
return {"success": True}
elif action == "double_click":
subprocess.run(["xdotool", "click", "--repeat", "2", "1"])
return {"success": True}
elif action == "type":
text = input.get("text", "")
# Use xdotool type with delay for reliability
subprocess.run(["xdotool", "type", "--delay", "50", text])
return {"success": True}
elif action == "key":
key = input.get("key", "")
# Map common key names
key_map = {
"return": "Return",
"enter": "Return",
"tab": "Tab",
"escape": "Escape",
"backspace": "BackSpace",
}
xdotool_key = key_map.get(key.lower(), key)
subprocess.run(["xdotool", "key", xdotool_key])
return {"success": True}
elif action == "scroll":
direction = input.get("direction", "down")
amount = input.get("amount", 3)
button = "5" if direction == "down" else "4"
for _ in range(amount):
subprocess.run(["xdotool", "click", button])
return {"success": True}
return {"error": f"Unknown action: {action}"}
def _handle_bash(self, input: dict) -> dict:
"""Execute bash command."""
command = input.get("command", "")
# Security: Sanitize and limit commands
dangerous_patterns = ["rm -rf", "mkfs", "dd if=", "> /dev/"]
for pattern in dangerous_patterns:
if pattern in command:
return {"error": "Dangerous command blocked"}
try:
result = subprocess.run(
command,
shell=True,
capture_output=True,
text=True,
timeout=30
)
return {
"stdout": result.stdout[:10000], # Limit output
"stderr": result.stderr[:1000],
"returncode": result.returncode
}
except subprocess.TimeoutExpired:
return {"error": "Command timed out"}
def _handle_editor(self, input: dict) -> dict:
"""Handle text editor operations."""
command = input.get("command")
path = input.get("path")
if command == "view":
try:
with open(path, "r") as f:
content = f.read()
return {"content": content[:50000]} # Limit size
except Exception as e:
return {"error": str(e)}
elif command == "str_replace":
old_str = input.get("old_str")
new_str = input.get("new_str")
try:
with open(path, "r") as f:
content = f.read()
if old_str not in content:
return {"error": "old_str not found in file"}
content = content.replace(old_str, new_str, 1)
with open(path, "w") as f:
f.write(content)
return {"success": True}
except Exception as e:
return {"error": str(e)}
return {"error": f"Unknown editor command: {command}"}
def run_task(self, task: str, max_steps: int = 50) -> dict:
"""Run computer use task with agentic loop."""
messages = [{"role": "user", "content": task}]
tools = self.get_tools()
for step in range(max_steps):
response = self.client.beta.messages.create(
model=self.model,
max_tokens=4096,
tools=tools,
messages=messages,
betas=["computer-use-2024-10-22"]
)
# Check for completion
if response.stop_reason == "end_turn":
return {
"success": True,
"result": response.content[0].text if response.content else "",
"steps": step + 1
}
# Handle tool use
if response.stop_reason == "tool_use":
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type == "tool_use":
result = self.execute_tool(block.name, block.input)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result
})
messages.append({"role": "user", "content": tool_results})
return {"success": False, "error": "Max steps reached"}
Anti_patterns
Not using betas=['computer-use-2024-10-22'] flag
Full resolution screenshots (wasteful)
No command sanitization for bash tool
Unbounded execution time
Browser-Use Pattern (Playwright-based)
For browser-only automation, using structured DOM access is more efficient
than pixel-based computer use. Playwright MCP allows LLMs to control
browsers using accessibility snapshots rather than screenshots.
from anthropic import Anthropic
result = await agent.run_with_llm(
"Go to weather.com and find the weather for New York",
Anthropic()
)
print(result)
await agent.close()
asyncio.run(main())
Anti_patterns
Using screenshots when DOM access works
Not waiting for page loads
Hardcoded selectors that break
No error recovery for stale elements
User Confirmation Pattern
For sensitive actions, agents should pause and ask for human confirmation.
"ChatGPT agent also pauses and asks for confirmation prior to taking
sensitive steps such as completing a purchase."
Sensitivity levels:
LOW: Navigation, reading (auto-approve)
MEDIUM: Form filling, clicking (log, maybe confirm)
When to use: Actions with real-world consequences,Financial transactions,Authentication flows,File modifications
from enum import Enum
from dataclasses import dataclass
from typing import Callable, Optional
import asyncio
class ActionSeverity(Enum):
LOW = "low" # Auto-approve
MEDIUM = "medium" # Log, optional confirm
HIGH = "high" # Always confirm
CRITICAL = "critical" # Confirm + review details
@dataclass
class SensitiveAction:
"""Action that may need user confirmation."""
action_type: str
description: str
severity: ActionSeverity
details: dict
class ConfirmationGate:
"""
Gate sensitive actions through user confirmation.
"""
class ActionLogger:
"""
Comprehensive action logging for computer use agents.
"""
def __init__(self, log_dir: str = "./agent_logs"):
self.log_dir = log_dir
self.screenshot_dir = os.path.join(log_dir, "screenshots")
os.makedirs(self.screenshot_dir, exist_ok=True)
self.current_session: Optional[TaskSession] = None
def start_session(self, task: str) -> str:
"""Start a new task session."""
import uuid
session_id = str(uuid.uuid4())[:8]
self.current_session = TaskSession(
session_id=session_id,
task=task,
start_time=datetime.now()
)
return session_id
def log_action(
self,
action_type: str,
parameters: dict,
success: bool,
error: Optional[str] = None,
screenshot_before: bytes = None,
screenshot_after: bytes = None,
model_reasoning: str = None,
duration_ms: int = None
):
"""Log a single action."""
if not self.current_session:
raise RuntimeError("No active session")
# Save screenshots if provided
screenshot_paths = {}
timestamp_str = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
if screenshot_before:
path = os.path.join(
self.screenshot_dir,
f"{self.current_session.session_id}_{timestamp_str}_before.png"
)
with open(path, "wb") as f:
f.write(screenshot_before)
screenshot_paths["before"] = path
if screenshot_after:
path = os.path.join(
self.screenshot_dir,
f"{self.current_session.session_id}_{timestamp_str}_after.png"
)
with open(path, "wb") as f:
f.write(screenshot_after)
screenshot_paths["after"] = path
# Create log entry
entry = ActionLogEntry(
timestamp=datetime.now(),
action_type=action_type,
parameters=parameters,
success=success,
error=error,
screenshot_before=screenshot_paths.get("before"),
screenshot_after=screenshot_paths.get("after"),
model_reasoning=model_reasoning,
duration_ms=duration_ms
)
self.current_session.actions.append(entry)
# Also append to running log file
self._append_to_log(entry)
def _append_to_log(self, entry: ActionLogEntry):
"""Append entry to JSONL log file."""
log_file = os.path.join(
self.log_dir,
f"session_{self.current_session.session_id}.jsonl"
)
with open(log_file, "a") as f:
f.write(json.dumps(entry.to_dict()) + "\n")
def end_session(self, success: bool, result: str = None):
"""End current session."""
if not self.current_session:
return
self.current_session.end_time = datetime.now()
self.current_session.success = success
self.current_session.final_result = result
# Write session summary
summary_file = os.path.join(
self.log_dir,
f"session_{self.current_session.session_id}_summary.json"
)
summary = {
"session_id": self.current_session.session_id,
"task": self.current_session.task,
"start_time": self.current_session.start_time.isoformat(),
"end_time": self.current_session.end_time.isoformat(),
"duration_seconds": (
self.current_session.end_time -
self.current_session.start_time
).total_seconds(),
"total_actions": len(self.current_session.actions),
"successful_actions": sum(
1 for a in self.current_session.actions if a.success
),
"failed_actions": sum(
1 for a in self.current_session.actions if not a.success
),
"success": success,
"final_result": result
}
with open(summary_file, "w") as f:
json.dump(summary, f, indent=2)
self.current_session = None
def get_session_replay(self, session_id: str) -> list[dict]:
"""Get all actions from a session for replay/debugging."""
log_file = os.path.join(self.log_dir, f"session_{session_id}.jsonl")
actions = []
with open(log_file, "r") as f:
for line in f:
actions.append(json.loads(line))
return actions
Integration with agent
class LoggedComputerUseAgent:
"""Computer use agent with comprehensive logging."""
def __init__(self, base_agent, logger: ActionLogger):
self.agent = base_agent
self.logger = logger
def run_task(self, task: str) -> dict:
"""Run task with full logging."""
session_id = self.logger.start_session(task)
try:
result = self._run_with_logging(task)
self.logger.end_session(
success=result.get("success", False),
result=result.get("result")
)
return result
except Exception as e:
self.logger.end_session(success=False, result=str(e))
raise
def _run_with_logging(self, task: str) -> dict:
"""Internal run with action logging."""
# This would wrap the base agent's run method
# and log each action
pass
Anti_patterns
Not sanitizing sensitive data in logs
Storing screenshots indefinitely (storage costs)
Not rotating log files
Logging synchronously (blocks agent)
Examples
Example 1: Ask for the upstream workflow directly
Use @computer-use-agents-v2 to handle <task>. Start from the copied upstream workflow, load only the files that change the outcome, and keep provenance visible in the answer.
Explanation: This is the safest starting point when the operator needs the imported workflow, but not the entire repository.
Example 2: Ask for a provenance-grounded review
Review @computer-use-agents-v2 against metadata.json and ORIGIN.md, then explain which copied upstream files you would load first and why.
Explanation: Use this before review or troubleshooting when you need a precise, auditable explanation of origin and file selection.
Example 3: Narrow the copied support files before execution
Use @computer-use-agents-v2 for <task>. Load only the copied references, examples, or scripts that change the outcome, and name the files explicitly before proceeding.
Explanation: This keeps the skill aligned with progressive disclosure instead of loading the whole copied package by default.
Example 4: Build a reviewer packet
Review @computer-use-agents-v2 using the copied upstream files plus provenance, then summarize any gaps before merge.
Explanation: This is useful when the PR is waiting for human review and you want a repeatable audit packet.
Best Practices
Treat the generated public skill as a reviewable packaging layer around the upstream repository. The goal is to keep provenance explicit and load only the copied source material that materially improves execution.
Keep the imported skill grounded in the upstream repository; do not invent steps that the source material cannot support.
Prefer the smallest useful set of support files so the workflow stays auditable and fast to review.
Keep provenance, source commit, and imported file paths visible in notes and PR descriptions.
Point directly at the copied upstream files that justify the workflow instead of relying on generic review boilerplate.
Treat generated examples as scaffolding; adapt them to the concrete task before execution.
Route to a stronger native skill when architecture, debugging, design, or security concerns become dominant.
Troubleshooting
Problem: The operator skipped the imported context and answered too generically
Symptoms: The result ignores the upstream workflow in plugins/antigravity-awesome-skills/skills/computer-use-agents, fails to mention provenance, or does not use any copied source files at all.
Solution: Re-open metadata.json, ORIGIN.md, and the most relevant copied upstream files. Check the external_source block first, then restate the provenance before continuing.
Problem: The imported workflow feels incomplete during review
Symptoms: Reviewers can see the generated SKILL.md, but they cannot quickly tell which references, examples, or scripts matter for the current task.
Solution: Point at the exact copied references, examples, scripts, or assets that justify the path you took. If the gap is still real, record it in the PR instead of hiding it.
Problem: The task drifted into a different specialization
Symptoms: The imported skill starts in the right place, but the work turns into debugging, architecture, design, security, or release orchestration that a native skill handles better.
Solution: Use the related skills section to hand off deliberately. Keep the imported provenance visible so the next skill inherits the right context instead of starting blind.
Related Skills
@00-andruia-consultant - Use when the work is better handled by that native specialization after this imported skill establishes context.
@00-andruia-consultant-v2 - Use when the work is better handled by that native specialization after this imported skill establishes context.
@10-andruia-skill-smith - Use when the work is better handled by that native specialization after this imported skill establishes context.
@10-andruia-skill-smith-v2 - Use when the work is better handled by that native specialization after this imported skill establishes context.
Additional Resources
Use this support matrix and the linked files below as the operator packet for this imported skill. They should reflect real copied source material, not generic scaffolding.
Resource family
What it gives the reviewer
Example path
references
copied reference notes, guides, or background material from upstream
references/n/a
examples
worked examples or reusable prompts copied from upstream
examples/n/a
scripts
upstream helper scripts that change execution or validation
scripts/n/a
agents
routing or delegation notes that are genuinely part of the imported package
agents/n/a
assets
supporting assets or schemas copied from the source package
assets/n/a
Imported Reference Notes
Imported: Sharp Edges
Web Content Can Hijack Your Agent
Severity: CRITICAL
Situation: Computer use agent browsing the web
Symptoms:
Agent suddenly performs unexpected actions. Clicks malicious links.
Enters credentials on phishing sites. Downloads files it shouldn't.
Ignores your instructions and follows embedded commands instead.
Why this breaks:
"While all agents that process untrusted content are subject to prompt
injection risks, browser use amplifies this risk in two ways. First,
the attack surface is vast: every webpage, embedded document, advertisement,
and dynamically loaded script represents a potential vector for malicious
instructions. Second, browser agents can take many different actions—
navigating to URLs, filling forms, clicking buttons, downloading files—
that attackers can exploit."
Real attacks have already happened:
"Microsoft Copilot agents were hijacked with emails containing malicious
instructions, which allowed attackers to extract entire CRM databases."
"Google's Workspace services were manipulated—hidden prompts inside
calendar invites and emails tricked Gemini agents into deleting events
and exposing sensitive messages."
Even a 1% attack success rate represents meaningful risk at scale.
Recommended fix:
Imported: Defense in depth - no single solution works
Sandboxing (most effective):
# Docker with strict isolation
docker run \
--security-opt no-new-privileges \
--cap-drop ALL \
--network none \ # No internet!
--read-only \
computer-use-agent
Classifier-based detection:
defscan_for_injection(content: str) -> bool:
"""Detect prompt injection attempts."""
patterns = [
r"ignore.*instructions",
r"disregard.*previous",
r"new.*instructions",
r"you are now",
r"act as if",
r"pretend to be",
]
returnany(re.search(p, content.lower()) for p in patterns)
# Check page content before processing
page_text = await page.text_content("body")
if scan_for_injection(page_text):
return {"error": "Potential injection detected"}
User confirmation for sensitive actions:
SENSITIVE_ACTIONS = {"download", "submit", "login", "purchase"}
if action_type in SENSITIVE_ACTIONS:
ifnotawait get_user_confirmation(action):
return {"error": "User rejected action"}
Scoped credentials:
Never give agent access to all credentials
Use temporary, limited tokens
Revoke after task completion
Vision Agents Click Exact Centers
Severity: MEDIUM
Situation: Agent clicking on UI elements
Symptoms:
Agent's clicks are detectable as non-human. Websites may block or
CAPTCHA the agent. Anti-bot systems flag the interaction.
Why this breaks:
"When a vision model identifies a button, it calculates the center.
Click coordinates land at mathematically precise positions—often exact
element centers or grid-aligned pixel values. Humans don't click centers;
their click distributions follow a Gaussian pattern around targets."
The screenshot loop also creates detectable patterns:
"Predictable pauses. Vision agents are completely still during their
'thinking' phase. The pattern looks like: Action → Complete stillness
(1-5 seconds) → Action → Complete stillness → Action."
Sophisticated anti-bot systems detect:
Perfect center clicks
No mouse movement during "thinking"
Consistent timing between actions
Lack of micro-movements and hesitation
Recommended fix:
Imported: Add human-like variance to actions
import random
import time
defhumanized_click(x: int, y: int) -> tuple[int, int]:
"""Add human-like variance to click coordinates."""# Gaussian distribution around target# Humans typically land within ~10px of target
x_offset = int(random.gauss(0, 5))
y_offset = int(random.gauss(0, 5))
return (x + x_offset, y + y_offset)
defhumanized_delay():
"""Add human-like delay between actions."""# Humans have variable reaction times
base_delay = random.uniform(0.3, 0.8)
# Occasionally longer pauses (reading, thinking)if random.random() < 0.2:
base_delay += random.uniform(0.5, 2.0)
time.sleep(base_delay)
defhumanized_movement(from_pos: tuple, to_pos: tuple):
"""Move mouse in curved path like human."""# Bezier curve or similar# Humans don't move in straight lines
steps = random.randint(10, 20)
for i inrange(steps):
t = i / steps
# Simple curve approximation
x = from_pos[0] + (to_pos[0] - from_pos[0]) * t
y = from_pos[1] + (to_pos[1] - from_pos[1]) * t
# Add wobble
x += random.gauss(0, 2)
y += random.gauss(0, 2)
pyautogui.moveTo(int(x), int(y))
time.sleep(0.01)
Imported: Rotate user agents and fingerprints
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120...",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Safari/...",
# ... more realistic agents
]
await page.set_extra_http_headers({
"User-Agent": random.choice(USER_AGENTS)
})
Dropdowns, Scrollbars, and Drags Are Unreliable
Severity: HIGH
Situation: Agent interacting with complex UI elements
Symptoms:
Agent fails to select dropdown options. Scroll doesn't work as expected.
Drag and drop completely fails. Hover menus disappear before clicking.
Why this breaks:
"Computer Use currently struggles with certain interface interactions,
particularly scrolling, dragging, and zooming operations. Some UI elements
(like dropdowns and scrollbars) might be tricky for Claude to manipulate."
Anthropic documentation
Why these are hard:
Dropdowns: Options appear after click, need second click to select
Scrollbars: Small targets, need precise positioning
Drag: Requires coordinated mouse down, move, mouse up
Hover menus: Disappear when mouse moves away
Canvas elements: No semantic information visible
Vision models see pixels, not DOM structure. They don't "know" that
a dropdown is a dropdown - they have to infer from visual cues.
Recommended fix:
Imported: Fall back to DOM access for web
# If vision fails, try direct DOM manipulationasyncdefrobust_select(page, select_selector, value):
try:
# Try vision approach firstawait vision_agent.select(select_selector, value)
except Exception:
# Fall back to direct DOMawait page.select_option(select_selector, value=value)
Imported: Add verification after action
asyncdefverified_scroll(page, direction):
# Get current scroll position
before = await page.evaluate("window.scrollY")
# Attempt scrollawait page.mouse.wheel(0, 500if direction == "down"else -500)
await asyncio.sleep(0.3)
# Verify it worked
after = await page.evaluate("window.scrollY")
if before == after:
# Try alternative methodawait page.keyboard.press("PageDown"if direction == "down"else"PageUp")
Agents Are 2-5x Slower Than Humans
Severity: MEDIUM
Situation: Automating any computer task
Symptoms:
Task that takes human 1 minute takes agent 3-5 minutes.
Users complain about speed. Timeouts occur.
Why this breaks:
"The technology can be slow compared to human operators, often requiring
multiple screenshots and analysis cycles."
Why so slow:
Screenshot capture: 100-500ms
Vision model inference: 1-5 seconds per screenshot
Action execution: 200-500ms
Wait for UI update: 500-1000ms
Total per action: 2-7 seconds
A task requiring 20 actions takes 40-140 seconds minimum.
Humans do the same actions in 20-30 seconds.
Recommended fix:
Imported: Accept the tradeoff
Computer use is for:
Tasks humans don't want to do (repetitive)
Tasks that can run in background
Tasks where accuracy > speed
Imported: Optimize where possible
# 1. Reduce screenshot resolution
SCREEN_SIZE = (1280, 800) # Not 4K# 2. Batch similar actions# Instead of: type "hello", wait, type " world"await page.type("hello world")
# 3. Parallelize independent tasks# Run multiple sandboxed agents concurrently# 4. Cache repeated computations# If same screenshot, reuse analysis# 5. Use smaller models for simple decisions
simple_model = "claude-haiku-..."# For "is task done?"
complex_model = "claude-sonnet-..."# For complex reasoning
Why this breaks:
Each screenshot is ~1500-3000 tokens. A task with 30 screenshots
uses 45,000-90,000 tokens just for images - before any text.
Claude's context window is finite. When full:
Older context gets dropped
Agent loses memory of earlier steps
Task coherence decreases
"Getting agents to make consistent progress across multiple context
windows remains an open problem. The core challenge is that they must
work in discrete sessions, and each new session begins with no memory
of what came before." - Anthropic engineering blog
Recommended fix:
Imported: Implement context management
classContextManager:
"""Manage context window usage for computer use."""
MAX_SCREENSHOTS = 10# Keep only recent screenshots
MAX_TOKENS = 100000def__init__(self):
self.messages = []
self.screenshot_count = 0defadd_screenshot(self, screenshot_b64: str, description: str):
"""Add screenshot with automatic pruning."""self.screenshot_count += 1# Keep only recent screenshotsifself.screenshot_count > self.MAX_SCREENSHOTS:
self._prune_old_screenshots()
# Store with description for contextself.messages.append({
"role": "user",
"content": [
{"type": "text", "text": description},
{"type": "image", "source": {...}}
]
})
def_prune_old_screenshots(self):
"""Remove old screenshots, keep text summaries."""
new_messages = []
screenshots_kept = 0for msg inreversed(self.messages):
ifself._has_image(msg):
if screenshots_kept < self.MAX_SCREENSHOTS:
new_messages.insert(0, msg)
screenshots_kept += 1else:
# Convert to text summary
summary = self._summarize_screenshot(msg)
new_messages.insert(0, {
"role": msg["role"],
"content": summary
})
else:
new_messages.insert(0, msg)
self.messages = new_messages
def_summarize_screenshot(self, msg) -> str:
"""Summarize screenshot to text."""# Extract any text descriptionfor content in msg.get("content", []):
if content.get("type") == "text":
returnf"[Previous screenshot: {content['text']}]"return"[Previous screenshot - details pruned]"defadd_checkpoint(self):
"""Create a checkpoint summary."""
summary = self._create_progress_summary()
self.messages.append({
"role": "user",
"content": f"CHECKPOINT: {summary}"
})
Imported: Use checkpointing for long tasks
asyncdefrun_with_checkpoints(task: str, checkpoint_every: int = 10):
"""Run task with periodic checkpoints."""
context = ContextManager()
step = 0whilenot task_complete:
step += 1# Take action...if step % checkpoint_every == 0:
# Create checkpoint
context.add_checkpoint()
# Optional: persist to disk
save_checkpoint(context, step)
Imported: Break into subtasks
# Instead of one 50-step task:
subtasks = [
"Navigate to the website and login",
"Find the settings page",
"Update the email address to ...",
"Save and verify the change"
]
for subtask in subtasks:
result = await agent.run(subtask)
ifnot result["success"]:
handle_error(subtask, result)
break
Costs Can Explode Quickly
Severity: HIGH
Situation: Running computer use at scale
Symptoms:
API bill is 10x higher than expected. Single task costs $5+ instead of $0.50.
Monthly costs reach thousands of dollars quickly.
Why this breaks:
Vision tokens are expensive. Each screenshot:
~2000-3000 tokens per image
At $10/million tokens, that's $0.02-0.03 per screenshot
Task with 30 screenshots = $0.60-0.90 just for images
But it compounds:
Screenshots accumulate in context
Model sees ALL previous screenshots each turn
Turn 10 processes 10 screenshots = $0.20-0.30
Turn 20 processes 20 screenshots = $0.40-0.60
Quadratic growth!
Complex task: 50 turns × average 25 images in context = 1250 image tokens
Plus text = could easily hit $5-10 per task.
# 1. Lower resolution
SCREEN_SIZE = (1024, 768) # Smaller = fewer tokens# 2. JPEG instead of PNG (when quality ok)
screenshot.save(buffer, format="JPEG", quality=70)
# 3. Crop to relevant regiondefcrop_relevant(screenshot: Image, focus_area: tuple):
"""Crop to area of interest."""return screenshot.crop(focus_area)
# 4. Don't include screenshot every turnifnot needs_visual_update:
# Text-only turn
messages.append({"role": "user", "content": "Continue..."})
Imported: Use cheaper models strategically
asyncdeftiered_model_selection(task_complexity: str):
"""Use appropriate model for task."""if task_complexity == "simple":
return"claude-haiku-..."# Cheapestelif task_complexity == "medium":
return"claude-sonnet-4-20250514"# Balancedelse:
return"claude-opus-4-5-..."# Best but expensive
Running Agent on Your Actual Computer
Severity: CRITICAL
Situation: Testing or deploying computer use
Symptoms:
Agent deletes important files. Sends emails from your account.
Posts on social media. Accesses sensitive documents.
Why this breaks:
Computer use agents make mistakes. They can:
Misinterpret instructions
Click wrong buttons
Type in wrong fields
Follow prompt injection attacks
Without sandboxing, these mistakes happen on your real system.
There's no undo for "agent sent email to all contacts" or
"agent deleted project folder."
"Autonomous agents that can access external systems and APIs
introduce new security risks. They may be vulnerable to prompt
injection attacks, unauthorized access to sensitive data, or
manipulation by malicious actors."
