| name | open-cua-computer-agents |
| title | OpenCUA - Open Foundations for Computer-Use Agents |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2508.09123 |
| keywords | ["computer-use-agents","agent-scaling","chain-of-thought","dataset-annotation","reflection"] |
| description | Scales computer-use agent capabilities through reflective Chain-of-Thought reasoning in large-scale annotated datasets spanning multiple operating systems and 200+ applications. |
OpenCUA: Open Foundations for Computer-Use Agents
Core Concept
OpenCUA provides comprehensive foundations for computer-use agents through three interconnected components: an annotation infrastructure that captures human-computer interactions, the AgentNet dataset spanning 3 operating systems and 200+ applications, and a transformation pipeline converting demonstrations into state-action pairs with reflective Chain-of-Thought reasoning. This approach enables robust agent scaling with improved reasoning patterns.
Architecture Overview
- Human Demonstration Capture: Seamlessly record human interactions with computers
- Large-Scale AgentNet Dataset: 3 operating systems, 200+ applications/websites
- Reflective CoT Transformation: Convert raw demonstrations to reasoning-annotated pairs
- State-Action Representation: Structured data for agent training
- Multi-OS Support: Windows, macOS, Linux compatibility
Implementation Steps
Step 1: Implement Interaction Capture Infrastructure
Record human demonstrations:
class InteractionCapture:
def __init__(self):
super().__init__()
self.current_session = None
self.interactions = []
def capture_session_start(self, app_name, window_title):
"""
Initialize capture for new application interaction.
Args:
app_name: Name of application
window_title: Window title at start
Returns:
session_id: Unique session identifier
"""
session_id = str(uuid.uuid4())
self.current_session = {
'session_id': session_id,
'app_name': app_name,
'window_title': window_title,
'start_time': time.time(),
'interactions': [],
'screenshots': []
}
return session_id
def capture_action(self, action_type, details):
"""
Capture single user action.
Args:
action_type: Type of action (click, type, scroll, etc)
details: Action-specific details
Returns:
action_record: Captured action with metadata
"""
action_record = {
'timestamp': time.time(),
'type': action_type,
'details': details,
'screenshot_before': self._capture_screenshot(),
'window_state': self._get_window_state()
}
self.current_session[].append(action_record)
.interactions.append(action_record)
action_record
():
action = {
: ,
: x,
: y,
: button
}
.capture_action(, action)
():
action = {
: ,
: text,
: (text)
}
.capture_action(, action)
():
action = {
: ,
: direction,
: amount
}
.capture_action(, action)
():
pyautogui
screenshot = pyautogui.screenshot()
.current_session[].append({
: time.time(),
: screenshot
})
screenshot
():
pyautogui
pyautogui.screenshot()
():
pygetwindow
active_window = pygetwindow.getActiveWindow()
{
: active_window.title active_window ,
: (active_window.left, active_window.top,
active_window.width, active_window.height)
active_window
}
():
.current_session:
.current_session[] = time.time()
.current_session[] = (
.current_session[] - .current_session[]
)
completed = .current_session.copy()
.current_session =
completed
Step 2: Create Reflective CoT Annotation Pipeline
Convert demonstrations to reasoning-annotated training data:
class ReflectiveCoTAnnotator:
def __init__(self, reasoning_model):
super().__init__()
self.reasoning_model = reasoning_model
def generate_cot_for_interaction(self, interaction_sequence, goal):
"""
Generate Chain-of-Thought explanation for action sequence.
Args:
interaction_sequence: List of captured actions
goal: High-level goal being accomplished
Returns:
annotated_sequence: Actions with reasoning
"""
annotated = []
context = f"Goal: {goal}\n\nActions taken:"
for action_idx, action in enumerate(interaction_sequence):
action_description = self._describe_action(action)
reasoning_prompt = f"""{context}
{action_description}
Why was this action taken? What was the agent trying to achieve?"""
with torch.no_grad():
reasoning = self.reasoning_model.generate(
reasoning_prompt,
max_length=150,
temperature=0.7
)
annotated_record = {
'action': action,
'description': action_description,
'reasoning': reasoning,
'screenshot_before': action.get('screenshot_before'),
'window_state': action.get('window_state')
}
annotated.append(annotated_record)
context +=
annotated
():
action_type = action[]
action_type == :
action_type == :
action_type == :
:
(action)
():
annotated_interactions = .generate_cot_for_interaction(
session[],
goal_statement
)
{
: session[],
: session[],
: goal_statement,
: annotated_interactions,
: session.get(),
: ._assess_success(annotated_interactions)
}
():
(annotated_interactions) >
Step 3: Build State-Action Pair Representation
Convert annotated demonstrations to training format:
class StateActionPairBuilder:
def __init__(self, vision_encoder):
super().__init__()
self.vision_encoder = vision_encoder
def build_training_pair(self, annotated_interaction):
"""
Convert annotated interaction to state-action training pair.
Args:
annotated_interaction: Single interaction with reasoning
Returns:
training_pair: Structured state-action pair
"""
screenshot = annotated_interaction['screenshot_before']
screen_embedding = self.vision_encoder.encode(screenshot)
state = {
'visual': screen_embedding,
'window_state': annotated_interaction['window_state'],
'history': annotated_interaction.get('action_history', []),
'goal': annotated_interaction.get('goal')
}
action = annotated_interaction['action']
reasoning = annotated_interaction['reasoning']
return {
'state': state,
'action': action,
'reasoning': reasoning,
'description': annotated_interaction['description']
}
def build_dataset(self, annotated_sessions):
"""
Convert multiple sessions to training dataset.
Args:
annotated_sessions: List of annotated demonstration sessions
Returns:
training_dataset: Ready-to-use training data
"""
training_pairs = []
session annotated_sessions:
interaction session[]:
pair = .build_training_pair(interaction)
training_pairs.append(pair)
{
: training_pairs,
: (training_pairs),
: (s[] s annotated_sessions),
: (annotated_sessions)
}
Step 4: Implement Agent Training on Dataset
Train computer-use agents:
class ComputerUseAgentTrainer:
def __init__(self, model, vision_encoder):
super().__init__()
self.model = model
self.vision_encoder = vision_encoder
def train_agent(self, training_dataset, num_epochs=3):
"""
Train agent on state-action-reasoning data.
Args:
training_dataset: Built from demonstrations
num_epochs: Training epochs
Returns:
trained_agent: Ready for deployment
"""
optimizer = AdamW(self.model.parameters(), lr=2e-5)
for epoch in range(num_epochs):
total_loss = 0
for pair in training_dataset['pairs']:
state = pair['state']
action = pair['action']
reasoning = pair['reasoning']
state_embedding = self._encode_state(state)
predicted_action, predicted_reasoning = self.model(state_embedding)
action_loss = self._action_loss(predicted_action, action)
reasoning_loss = self._reasoning_loss(predicted_reasoning, reasoning)
total_loss_step = 0.7 * action_loss + 0.3 * reasoning_loss
optimizer.zero_grad()
total_loss_step.backward()
torch.nn.utils.clip_grad_norm_(.model.parameters(), )
optimizer.step()
total_loss += total_loss_step.item()
()
.model
():
visual = state[]
context_text = (state[]) + + state.get(, )
context_embedding = .model.tokenizer.encode(context_text)
{
: visual,
: context_embedding
}
():
F.cross_entropy(predicted, ._encode_action(ground_truth))
():
F.cross_entropy(predicted, .model.tokenizer.encode(ground_truth))
():
action_type = action.get(, )
action_mapping = {
: ,
: ,
: ,
:
}
torch.tensor(action_mapping.get(action_type, -))
():
success_count =
total_steps =
task test_tasks:
state = ._prepare_task_state(task)
step_count =
task_success =
step_count < max_steps:
torch.no_grad():
action, reasoning = .model(._encode_state(state))
new_state, task_done, success = ._execute_action(action, state, task)
task_done:
task_success = success
state = new_state
step_count +=
task_success:
success_count +=
total_steps += step_count
{
: success_count / (test_tasks),
: total_steps / (test_tasks),
: success_count
}
():
{
: task[],
: ,
: {}
}
():
state, ,
Practical Guidance
Hyperparameters and Configuration:
- CoT generation temperature: 0.7-0.9
- Agent training learning rate: 2e-5 to 5e-5
- Action/reasoning loss weight ratio: 0.7/0.3
- Maximum steps per task: 50-100
- Training epochs: 2-5
When to Use OpenCUA:
- Building computer-use agents for automation
- Scenarios with diverse applications (need broad coverage)
- Systems where interpretability through reasoning is valuable
- Applications requiring multi-OS support
When NOT to Use:
- Simple single-application automation (specialized tools better)
- Real-time systems with strict latency constraints
- Scenarios with limited demonstration data
- When screen complexity is very high
Implementation Notes:
- Reflective CoT critical for agent robustness and scaling
- Diverse demonstration dataset essential (3 OSes, 200+ apps)
- Vision encoding quality impacts action prediction
- Monitor success rates across different application domains
- Consider curriculum: start with simple tasks, increase complexity
Reference
Paper: OpenCUA: Open Foundations for Computer-Use Agents
ArXiv: 2508.09123
Performance: OpenCUA-72B achieves 45.0% average success rate on OSWorld-Verified, state-of-the-art among open-source models
