open-cua-computer-agents

name	open-cua-computer-agents
title	OpenCUA - Open Foundations for Computer-Use Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.09123
keywords	["computer-use-agents","agent-scaling","chain-of-thought","dataset-annotation","reflection"]
description	Scales computer-use agent capabilities through reflective Chain-of-Thought reasoning in large-scale annotated datasets spanning multiple operating systems and 200+ applications.

OpenCUA: Open Foundations for Computer-Use Agents

Core Concept

OpenCUA provides comprehensive foundations for computer-use agents through three interconnected components: an annotation infrastructure that captures human-computer interactions, the AgentNet dataset spanning 3 operating systems and 200+ applications, and a transformation pipeline converting demonstrations into state-action pairs with reflective Chain-of-Thought reasoning. This approach enables robust agent scaling with improved reasoning patterns.

Architecture Overview

• Human Demonstration Capture: Seamlessly record human interactions with computers
• Large-Scale AgentNet Dataset: 3 operating systems, 200+ applications/websites
• Reflective CoT Transformation: Convert raw demonstrations to reasoning-annotated pairs
• State-Action Representation: Structured data for agent training
• Multi-OS Support: Windows, macOS, Linux compatibility

Implementation Steps

Step 1: Implement Interaction Capture Infrastructure

Record human demonstrations:

# Pseudocode for demonstration capture
class InteractionCapture:
    def __init__(self):
        super().__init__()
        self.current_session = None
        self.interactions = []

    def capture_session_start(self, app_name, window_title):
        """
        Initialize capture for new application interaction.

        Args:
            app_name: Name of application
            window_title: Window title at start

        Returns:
            session_id: Unique session identifier
        """
        session_id = str(uuid.uuid4())

        self.current_session = {
            'session_id': session_id,
            'app_name': app_name,
            'window_title': window_title,
            'start_time': time.time(),
            'interactions': [],
            'screenshots': []
        }

        return session_id

    def capture_action(self, action_type, details):
        """
        Capture single user action.

        Args:
            action_type: Type of action (click, type, scroll, etc)
            details: Action-specific details

        Returns:
            action_record: Captured action with metadata
        """
        action_record = {
            'timestamp': time.time(),
            'type': action_type,
            'details': details,
            'screenshot_before': self._capture_screenshot(),
            'window_state': self._get_window_state()
        }

        self.current_session[].append(action_record)
        .interactions.append(action_record)

         action_record

     ():
        
        action = {
            : ,
            : x,
            : y,
            : button
        }

         .capture_action(, action)

     ():
        
        action = {
            : ,
            : text,
            : (text)
        }

         .capture_action(, action)

     ():
        
        action = {
            : ,
            : direction,  
            : amount
        }

         .capture_action(, action)

     ():
        
         pyautogui

        screenshot = pyautogui.screenshot()
        .current_session[].append({
            : time.time(),
            : screenshot
        })

         screenshot

     ():
        
         pyautogui
         pyautogui.screenshot()

     ():
        
         pygetwindow

        active_window = pygetwindow.getActiveWindow()
         {
            : active_window.title  active_window  ,
            : (active_window.left, active_window.top,
                        active_window.width, active_window.height)
             active_window  
        }

     ():
        
         .current_session:
            .current_session[] = time.time()
            .current_session[] = (
                .current_session[] - .current_session[]
            )

            completed = .current_session.copy()
            .current_session = 

             completed

         

Step 2: Create Reflective CoT Annotation Pipeline

Convert demonstrations to reasoning-annotated training data:

# Pseudocode for reflective CoT generation
class ReflectiveCoTAnnotator:
    def __init__(self, reasoning_model):
        super().__init__()
        self.reasoning_model = reasoning_model

    def generate_cot_for_interaction(self, interaction_sequence, goal):
        """
        Generate Chain-of-Thought explanation for action sequence.

        Args:
            interaction_sequence: List of captured actions
            goal: High-level goal being accomplished

        Returns:
            annotated_sequence: Actions with reasoning
        """
        annotated = []

        context = f"Goal: {goal}\n\nActions taken:"

        for action_idx, action in enumerate(interaction_sequence):
            # Generate reasoning for this action
            action_description = self._describe_action(action)

            reasoning_prompt = f"""{context}
{action_description}

Why was this action taken? What was the agent trying to achieve?"""

            with torch.no_grad():
                reasoning = self.reasoning_model.generate(
                    reasoning_prompt,
                    max_length=150,
                    temperature=0.7
                )

            annotated_record = {
                'action': action,
                'description': action_description,
                'reasoning': reasoning,
                'screenshot_before': action.get('screenshot_before'),
                'window_state': action.get('window_state')
            }

            annotated.append(annotated_record)
            context += 

         annotated

     ():
        
        action_type = action[]

         action_type == :
             
         action_type == :
             
         action_type == :
             
        :
             (action)

     ():
        
        annotated_interactions = .generate_cot_for_interaction(
            session[],
            goal_statement
        )

         {
            : session[],
            : session[],
            : goal_statement,
            : annotated_interactions,
            : session.get(),
            : ._assess_success(annotated_interactions)
        }

     ():
        
        
         (annotated_interactions) > 

Step 3: Build State-Action Pair Representation

Convert annotated demonstrations to training format:

# Pseudocode for state-action pair creation
class StateActionPairBuilder:
    def __init__(self, vision_encoder):
        super().__init__()
        self.vision_encoder = vision_encoder

    def build_training_pair(self, annotated_interaction):
        """
        Convert annotated interaction to state-action training pair.

        Args:
            annotated_interaction: Single interaction with reasoning

        Returns:
            training_pair: Structured state-action pair
        """
        # State: screenshot + context + goal
        screenshot = annotated_interaction['screenshot_before']
        screen_embedding = self.vision_encoder.encode(screenshot)

        state = {
            'visual': screen_embedding,
            'window_state': annotated_interaction['window_state'],
            'history': annotated_interaction.get('action_history', []),
            'goal': annotated_interaction.get('goal')
        }

        # Action: what the human did
        action = annotated_interaction['action']

        # Reasoning: why they did it
        reasoning = annotated_interaction['reasoning']

        return {
            'state': state,
            'action': action,
            'reasoning': reasoning,
            'description': annotated_interaction['description']
        }

    def build_dataset(self, annotated_sessions):
        """
        Convert multiple sessions to training dataset.

        Args:
            annotated_sessions: List of annotated demonstration sessions

        Returns:
            training_dataset: Ready-to-use training data
        """
        training_pairs = []

         session  annotated_sessions:
             interaction  session[]:
                pair = .build_training_pair(interaction)
                training_pairs.append(pair)

         {
            : training_pairs,
            : (training_pairs),
            : (s[]  s  annotated_sessions),
            : (annotated_sessions)
        }

Step 4: Implement Agent Training on Dataset

Train computer-use agents:

# Pseudocode for agent training
class ComputerUseAgentTrainer:
    def __init__(self, model, vision_encoder):
        super().__init__()
        self.model = model
        self.vision_encoder = vision_encoder

    def train_agent(self, training_dataset, num_epochs=3):
        """
        Train agent on state-action-reasoning data.

        Args:
            training_dataset: Built from demonstrations
            num_epochs: Training epochs

        Returns:
            trained_agent: Ready for deployment
        """
        optimizer = AdamW(self.model.parameters(), lr=2e-5)

        for epoch in range(num_epochs):
            total_loss = 0

            for pair in training_dataset['pairs']:
                state = pair['state']
                action = pair['action']
                reasoning = pair['reasoning']

                # Encode state
                state_embedding = self._encode_state(state)

                # Forward pass: predict action and reasoning
                predicted_action, predicted_reasoning = self.model(state_embedding)

                # Compute loss
                action_loss = self._action_loss(predicted_action, action)
                reasoning_loss = self._reasoning_loss(predicted_reasoning, reasoning)

                total_loss_step = 0.7 * action_loss + 0.3 * reasoning_loss

                # Backward
                optimizer.zero_grad()
                total_loss_step.backward()
                torch.nn.utils.clip_grad_norm_(.model.parameters(), )
                optimizer.step()

                total_loss += total_loss_step.item()

            ()

         .model

     ():
        
        
        visual = state[]
        context_text = (state[]) +  + state.get(, )

        
        context_embedding = .model.tokenizer.encode(context_text)

         {
            : visual,
            : context_embedding
        }

     ():
        
        
         F.cross_entropy(predicted, ._encode_action(ground_truth))

     ():
        
        
         F.cross_entropy(predicted, .model.tokenizer.encode(ground_truth))

     ():
        
        action_type = action.get(, )
        action_mapping = {
            : ,
            : ,
            : ,
            : 
        }
         torch.tensor(action_mapping.get(action_type, -))

     ():
        
        success_count = 
        total_steps = 

         task  test_tasks:
            
            state = ._prepare_task_state(task)

            
            step_count = 
            task_success = 

             step_count < max_steps:
                
                 torch.no_grad():
                    action, reasoning = .model(._encode_state(state))

                
                new_state, task_done, success = ._execute_action(action, state, task)

                 task_done:
                    task_success = success
                    

                state = new_state
                step_count += 

             task_success:
                success_count += 

            total_steps += step_count

         {
            : success_count / (test_tasks),
            : total_steps / (test_tasks),
            : success_count
        }

     ():
        
         {
            : task[],
            : ,
            : {}
        }

     ():
        
        
         state, , 

Practical Guidance

Hyperparameters and Configuration:

• CoT generation temperature: 0.7-0.9
• Agent training learning rate: 2e-5 to 5e-5
• Action/reasoning loss weight ratio: 0.7/0.3
• Maximum steps per task: 50-100
• Training epochs: 2-5

When to Use OpenCUA:

• Building computer-use agents for automation
• Scenarios with diverse applications (need broad coverage)
• Systems where interpretability through reasoning is valuable
• Applications requiring multi-OS support

When NOT to Use:

• Simple single-application automation (specialized tools better)
• Real-time systems with strict latency constraints
• Scenarios with limited demonstration data
• When screen complexity is very high

Implementation Notes:

• Reflective CoT critical for agent robustness and scaling
• Diverse demonstration dataset essential (3 OSes, 200+ apps)
• Vision encoding quality impacts action prediction
• Monitor success rates across different application domains
• Consider curriculum: start with simple tasks, increase complexity

Reference

Paper: OpenCUA: Open Foundations for Computer-Use Agents ArXiv: 2508.09123 Performance: OpenCUA-72B achieves 45.0% average success rate on OSWorld-Verified, state-of-the-art among open-source models