| name | computer-using-world-model |
| title | Computer-Using World Model |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2602.17365 |
| keywords | ["world modeling","UI simulation","agent planning","desktop automation","think-then-act"] |
| description | Enable AI agents to safely explore action outcomes before execution by predicting UI state changes in desktop applications. Two-stage approach: first predict textual description of what changes, then synthesize visual representation of resulting screen. Allows agents to compare multiple candidate actions without risky trial-and-error, trained on Microsoft Office interactions (Word, Excel, PowerPoint). |
Computer-Using World Model: Safe Desktop Automation through Outcome Prediction
Web and desktop automation agents face a critical challenge: many actions are irreversible and costly. Deleting a file, overwriting a spreadsheet cell, or closing an unsaved document cannot be undone, yet agents often learn through trial-and-error exploration. Unlike robotics where physical environments are forgiving, software environments demand careful planning before execution.
The standard approach—act first, observe consequences—is unsafe in software contexts where action consequences are immediate and permanent. A safer strategy is to enable agents to simulate action outcomes before committing to execution, supporting "think-then-act" decision-making that compares multiple candidate paths without risky exploration.
Core Concept
The Computer-Using World Model (CUWM) predicts how desktop applications will change in response to user actions without actually executing them. The system operates in two stages:
- Textual Change Prediction: Generate a natural-language description of what state changes will occur (e.g., "the cell A1 value will become 'Q4 Revenue'")
- Visual Synthesis: Render a synthetic screenshot showing the predicted resulting state
This separation allows the model to focus on decision-relevant information (what changes) separately from appearance details (how it looks), improving generalization and prediction accuracy.
Architecture Overview
- State Encoder: Encode current UI screenshot into a latent representation capturing semantic state (open documents, form field values, selection state)
- Action Embedding: Encode the candidate action (click coordinates, text input, keyboard shortcut) into semantic representation
- Change Predictor: Decode change description from encoded state + action using causal language modeling
- Visual Renderer: Synthesize predicted screenshot using base screenshot + change description, applying element-level modifications
- Value Aggregator: For multi-action planning, score predicted outcome against goal representation to guide action selection
Implementation
Implement a two-stage predictor combining text generation and visual modification:
def predict_ui_state_change(screenshot, action, model):
"""
Predict what will change when action is taken.
screenshot: PIL Image of current state
action: dict with type, target, value (e.g., {'type': 'click', 'target': (x, y)})
Returns: (change_description, predicted_screenshot)
"""
state_latent = model.encode_screenshot(screenshot)
action_latent = model.embed_action(action)
change_description = model.predict_change(
state_latent, action_latent, temperature=
)
predicted_screenshot = model.render_screenshot(
screenshot, change_description, action
)
change_description, predicted_screenshot
