gui-360-desktop-agent-dataset

name	gui-360-desktop-agent-dataset
title	GUI-360°: Comprehensive Dataset for Computer-Using Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.04307
keywords	["Desktop Agents","GUI Grounding","Action Prediction","Multimodal Agents","Autonomous Agents"]
description	Enable training and evaluation of desktop computer-using agents through 1.2M action steps across diverse Windows applications, covering GUI grounding, screen parsing, and action prediction with hybrid GUI+API action space reflecting modern agent designs.

Title: Build Comprehensive Training Data for Desktop Autonomous Agents

Desktop agents that autonomously complete computer tasks face unique challenges: grounding spatial coordinates, parsing complex UIs, predicting appropriate next actions. GUI-360° provides 1.2M executed action steps across thousands of real trajectories in Word, Excel, and PowerPoint, with full screenshots, accessibility metadata, task goals, reasoning traces, and both successful and failed action sequences. The dataset reveals that state-of-the-art models struggle with grounding and action prediction despite strong general capabilities.

The benchmark spans three canonical tasks: GUI grounding, screen parsing, and action prediction.

Core Concept

Comprehensive Desktop Agent Training and Evaluation:

• Large-Scale Data: 1.2M action steps, 13,750 trajectories across three Office applications
• Rich Annotations: Screenshots + accessibility metadata + task descriptions + reasoning + failure cases
• Hybrid Action Space: GUI actions (click, type, drag) + app-specific APIs (table insertion, cell editing)
• Three Evaluation Tasks: Grounding (screen→coordinate), parsing (screen→UI elements), prediction (state→action)
• Real-World Complexity: Diverse task types, variable UI layouts, complex tool interactions

Architecture Overview

• Data Collection: Instrumented Windows environment recording human demonstrations
• Annotation Pipeline: Automatic extraction of actions, coordinates, UI elements via accessibility APIs
• Task Specification: Goal descriptions with reasoning traces from human performers
• Corpus Composition: 41% Word, 31.6% Excel, 27.4% PowerPoint
• Split Design: Train/test maintaining consistent application distributions

Implementation Steps

1. Design Data Collection Pipeline

Capture human demonstrations with full metadata.

class DesktopAgentDataCollector:
    def __init__(self, recording_dir="./gui_recordings"):
        self.recording_dir = recording_dir
        self.current_trajectory = 

     ():
        
        .current_trajectory = {
            : task_description,
            : [],
            : [],
            : {
                : ,
                : datetime.now(),
                : []
            }
        }

     ():
        
        
        screenshot = .screenshot_full_screen()

        
        a11y_tree = .get_accessibility_tree()

        
        step = {
            : {
                : action_type,  
                : coordinates,
                : text,
                : keystroke
            },
            : screenshot,
            : a11y_tree,
            : datetime.now()
        }

        .current_trajectory[].append(step)
         (.current_trajectory[])

     ():
        
        .current_trajectory[][] = task_completed
        .current_trajectory[][] = final_notes
        .current_trajectory[][] = datetime.now()

        
        trajectory_id = .save_trajectory(.current_trajectory)
         trajectory_id

     ():
        
         pyautogui
        screenshot = pyautogui.screenshot()
         np.array(screenshot)

     ():
        
         pywinauto
        app = pywinauto.GetFocusedWindow()

        
        a11y_tree = .build_tree(app)
         a11y_tree

     ():
        
         current_depth > max_depth:
             

        tree = {
            : element.element_info.name,
            : element.element_info.control_type,
            : element.rectangle().to_dict(),
            : element.is_enabled(),
            : element.is_visible(),
            : []
        }

        
        :
             child  element.children():
                child_tree = .build_tree(child, max_depth, current_depth + )
                 child_tree:
                    tree[].append(child_tree)
        :
            

         tree