cua-skill-computer-use

name	cua-skill-computer-use
title	CUA-Skill: Develop Skills for Computer Using Agent
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.21123
keywords	["Computer-Use Agents","Skill Abstraction","GUI Interaction","Parameterized Graphs","Skill Composition"]
description	Build desktop agents via reusable, parameterized skills encoding human computer-use knowledge. Skills combine execution graphs (handling UI variations) with composition graphs (chaining strategies). 57.5% success on WindowsAgentArena.

CUA-Skill: Reusable Computer-Use Skills

Problem

Desktop agents typically model interaction as flat sequences of low-level actions, forcing agents to rediscover common workflows from scratch. There is no reusable abstraction layer for human computer-use knowledge.

Multi-step desktop tasks require recovering common patterns (opening applications, finding UI elements, editing documents) repeatedly.

Core Concept

CUA-Skill encodes computer-use knowledge as reusable, parameterized skills. Each skill captures a coherent interaction pattern with built-in handling for UI variations and contingencies. Skills compose hierarchically into complete task workflows.

A skill comprises: target application, natural language intent, parameters, and execution graph (handling variations). Composition graphs capture how skills chain into higher-level strategies.

Architecture Overview

• Skill Definition: Application, intent, parameters, execution constraints
• Execution Graph: Parameterized paths handling common UI variations
• Contingency Handling: Alternative branches for common failures
• Composition Graph: Directed graph of skill dependencies and ordering
• Skill Retrieval: Dynamic selection based on current UI state
• Memory Integration: State tracking across multi-step workflows

Implementation

Step 1: Define Parameterized Skill Structure

Create skill representation with execution graphs.

from dataclasses import dataclass
from typing import List, Dict, Any

@dataclass
class Skill:
    """Reusable computer-use skill with parameterized execution."""
    skill_id: str
    application: str
    intent: str  # Natural language description
    parameters: Dict[str, str]  # Input parameters
    execution_graph: [, ]  

     ():
        .skill_id = skill_id
        .application = application
        .intent = intent
        .parameters = parameters
        .execution_graph = ._build_execution_graph()

     ():
        
         .application == :
             .intent == :
                 {
                    : ,
                    : {
                        : ,
                        : {: .parameters[]},
                        : ,
                        : 
                    },
                    : {
                        : ,
                        : 
                    },
                    : {
                        : ,
                        : {: .parameters[]},
                        : 
                    },
                    : {
                        : ,
                        : {: .parameters[]},
                        : 
                    },
                    : {
                        : ,
                        : ,
                        : 
                    },
                    : {
                        : ,
                        : 
                    },
                    : 
                }

     ():
        
        current_node = .execution_graph[]

         current_node   :
            node = .execution_graph[current_node]

            
            action = node[]
            params = node.get(, {}).copy()

            
             key, value  params.items():
                 (value, )  value.startswith():
                    param_name = value.strip()
                    params[key] = .parameters.get(param_name, value)

            result = ui_agent.execute_action(action, params, current_ui_state)

            
             result.success:
                current_node = node.get(, node.get())
            :
                current_node = node.get(, node.get())

         result