computer-using-world-model

name	computer-using-world-model
title	Computer-Using World Model
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.17365
keywords	["world modeling","UI simulation","agent planning","desktop automation","think-then-act"]
description	Enable AI agents to safely explore action outcomes before execution by predicting UI state changes in desktop applications. Two-stage approach: first predict textual description of what changes, then synthesize visual representation of resulting screen. Allows agents to compare multiple candidate actions without risky trial-and-error, trained on Microsoft Office interactions (Word, Excel, PowerPoint).

Computer-Using World Model: Safe Desktop Automation through Outcome Prediction

Web and desktop automation agents face a critical challenge: many actions are irreversible and costly. Deleting a file, overwriting a spreadsheet cell, or closing an unsaved document cannot be undone, yet agents often learn through trial-and-error exploration. Unlike robotics where physical environments are forgiving, software environments demand careful planning before execution.

The standard approach—act first, observe consequences—is unsafe in software contexts where action consequences are immediate and permanent. A safer strategy is to enable agents to simulate action outcomes before committing to execution, supporting "think-then-act" decision-making that compares multiple candidate paths without risky exploration.

Core Concept

The Computer-Using World Model (CUWM) predicts how desktop applications will change in response to user actions without actually executing them. The system operates in two stages:

1. Textual Change Prediction: Generate a natural-language description of what state changes will occur (e.g., "the cell A1 value will become 'Q4 Revenue'")
2. Visual Synthesis: Render a synthetic screenshot showing the predicted resulting state

This separation allows the model to focus on decision-relevant information (what changes) separately from appearance details (how it looks), improving generalization and prediction accuracy.

Architecture Overview

• State Encoder: Encode current UI screenshot into a latent representation capturing semantic state (open documents, form field values, selection state)
• Action Embedding: Encode the candidate action (click coordinates, text input, keyboard shortcut) into semantic representation
• Change Predictor: Decode change description from encoded state + action using causal language modeling
• Visual Renderer: Synthesize predicted screenshot using base screenshot + change description, applying element-level modifications
• Value Aggregator: For multi-action planning, score predicted outcome against goal representation to guide action selection

Implementation

Implement a two-stage predictor combining text generation and visual modification:

def predict_ui_state_change(screenshot, action, model):
    """
    Predict what will change when action is taken.
    screenshot: PIL Image of current state
    action: dict with type, target, value (e.g., {'type': 'click', 'target': (x, y)})
    Returns: (change_description, predicted_screenshot)
    """
    
    state_latent = model.encode_screenshot(screenshot)
    action_latent = model.embed_action(action)

    
    change_description = model.predict_change(
        state_latent, action_latent, temperature=
    )

    
    predicted_screenshot = model.render_screenshot(
        screenshot, change_description, action
    )

     change_description, predicted_screenshot