computer-use-hybrid-actions

name	computer-use-hybrid-actions
title	UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.17790
keywords	["computer use","hybrid actions","GUI automation","tool calling","agent framework"]
description	Enable computer-use agents to flexibly choose between GUI primitives (click, type) and high-level tool calls, reducing cascading errors by 22% and improving execution speed by 11%.

Technique: Hybrid Actions for Computer-Use Agents

Traditional computer-use agents rely exclusively on low-level GUI primitives (click, type, scroll), which creates fragile execution chains vulnerable to UI detection errors. One wrong click location cascades into multiple failures. UltraCUA solves this by enabling agents to dynamically choose between GUI primitives and direct API tool calls, bypassing brittle UI interactions when more reliable tool access is available.

The key insight is that agents should have options: when GUI interaction is unreliable or inefficient, fall back to tool calls; when direct tool access is unavailable, use GUI. This flexibility reduces error cascades while maintaining generality.

Core Concept

Hybrid actions operate on three principles:

• Flexible Action Space: Agents can choose to invoke primitive GUI actions OR high-level tool APIs
• Error Recovery: Failed GUI actions → fallback to tool calls instead of cascading errors
• Efficiency: Some tasks complete faster via tool calls (API call < 5 UI steps)
• Integration: Unified decision-making: "should I click this button or call the API directly?"

The result is 22% relative gains on OSWorld and 11% faster execution by intelligently routing to the right action type.

Architecture Overview

• GUI Environment Simulator: Provide screenshots and UI element coordinates
• Tool Registry: Catalog of available APIs (from documentation/GitHub)
• Action Router: LLM decides action type (GUI primitive vs tool call)
• GUI Executor: Handle click, type, scroll actions
• Tool Executor: Invoke APIs with parameter binding
• State Monitor: Track execution history, detect failures
• Feedback Loop: Learn which action types work best for which subtasks

Implementation Steps

The core decision point is the action router: when should the agent invoke a tool vs interact with the GUI? This example shows the hybrid action framework.

from typing import Union, List, Dict, Literal
from dataclasses import dataclass

@dataclass
class GUIPrimitive:
    """Low-level GUI action."""
    action_type: Literal["click", "type", "scroll"]
    coordinates: tuple = None  # (x, y) for click
    text: str = None           # for type
    direction: str = None      # "up" or "down" for scroll
    times: int = 1             # repeat count

@dataclass
class ToolCall:
    """High-level tool API invocation."""
    tool_name: str
    function_name: str
    parameters: Dict[str, any]
    description: str

HybridAction = Union[GUIPrimitive, ToolCall]


class HybridActionRouter:
    """
    Decide whether to use GUI primitive or tool call for each action.
    """

    def __init__(self, model, tool_registry: Dict[str, Dict]):
        self.model = model
        .tools = tool_registry  
        .execution_history = []

     () -> :
        
        prompt = 
         prompt

     () -> HybridAction:
        
        prompt = .build_action_prompt(screenshot, state, goal, available_tools)

        response = .model.generate(prompt)
        decision = parse_json_response(response)

         decision[] == :
            action = GUIPrimitive(
                action_type=decision[][],
                coordinates=decision[].get(),
                text=decision[].get(),
                direction=decision[].get()
            )
        :
            action = ToolCall(
                tool_name=decision[][],
                function_name=decision[][],
                parameters=decision[][],
                description=decision[]
            )

        .execution_history.append({
            : goal,
            : action,
            : decision[]
        })

         action

     () -> :
        
        catalog = []
         tool_name  available_tools:
             tool_name  .tools:
                funcs = .tools[tool_name].get(, {})
                 func_name, func_spec  funcs.items():
                    catalog.append(
                        
                    )
         .join(catalog)


 :
    

     ():
        .gui = gui_executor
        .tools = tool_executor
        .router = router

     () -> :
        
         (action, GUIPrimitive):
            
            :
                result = .gui.execute(action)
                 result[]:
                     result
             Exception  e:
                ()

                
                 (.router, ):
                    history = .router.execution_history[-]
                    
                    

         (action, ToolCall):
            
            :
                result = .tools.invoke(
                    tool_name=action.tool_name,
                    function_name=action.function_name,
                    parameters=action.parameters
                )
                 {
                    : ,
                    : result,
                    : 
                }
             Exception  e:
                ()
                 {
                    : ,
                    : (e),
                    : 
                }

         {: }


 ():
    
    current_state = get_initial_state()
    remaining_steps = max_steps

     step  (max_steps):
        
        screenshot = get_screenshot(current_state)

        
        action = router.decide_action(
            screenshot=screenshot,
            state=current_state,
            goal=initial_task,
            available_tools=[, , ]
        )

        
        result = executor.execute_action(action)

          result[]:
            ()
            remaining_steps -= 
             remaining_steps <= :
                
        :
            ()
            current_state = update_state(current_state, result)

     current_state

The key insight is teaching agents to compare action types: is this task easier via GUI or tool? For example, "open a file" might be easier via click→navigate dialog or directly via file_system.read_file(). Let the agent decide.

Practical Guidance

Task Type	GUI Primitives	Tool Calls	Hybrid Win
Fill form field	3-5 steps	1 API call	+40%
Search and click	2-3 steps	1 API call	+30%
Navigate pages	5-8 steps	Direct access	+50%

When to Use:

• Complex computer-use tasks where GUI can fail
• Mix of UI-dependent and API-available functionality
• Error recovery is important (cascading UI failures)
• You have tool/API documentation available

When NOT to Use:

• GUI-only interfaces (no tools/APIs available)
• Real-time interactive tasks requiring UI feedback
• Tasks where tool invocation requires manual setup

Common Pitfalls:

• Router indecisive → falls back too often
• Tool documentation incomplete → function calls fail
• GUI executor unreliable → defeats hybrid advantage
• Not tracking which action type succeeds (lose learning signal)
• Overly complex tool registry → router confused by too many options

Reference

UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action