Predict GUI state evolution by generating HTML code rather than pixel images. Combines visual fidelity of pixel-based approaches with structural precision of code-based methods through deterministic rendering. Enables agents to evaluate action consequences and select best decisions before execution.
Instrucciones de origen · Vista previa de solo lectura
name
code2world-gui-synthesis
title
Code2World: A GUI World Model via Renderable Code Generation
version
0.0.2
engine
skillxiv-v0.0.2-claude-opus-4.6
license
MIT
url
https://arxiv.org/abs/2602.09856
keywords
["GUI World Model","HTML Generation","Structured Prediction","Vision-Language","Deterministic Rendering"]
description
Predict GUI state evolution by generating HTML code rather than pixel images. Combines visual fidelity of pixel-based approaches with structural precision of code-based methods through deterministic rendering. Enables agents to evaluate action consequences and select best decisions before execution.
Code2World: HTML-Based GUI World Modeling
Predicting GUI state changes requires balancing visual realism against structural controllability. Pixel-based models achieve realism but lack precision; text-only code lacks visual grounding. Code2World bridges this gap: VLMs generate HTML code for the next GUI state, then deterministically render it to visual images. This unifies the strengths of both approaches while enabling agents to anticipate action consequences with both visual and structural clarity.
Core Concept
Standard pixel prediction: image + action → predicted next image. Realistic but spatially imprecise; hard for agents to verify if buttons appear in correct locations.
Code2World: image + action → predicted HTML code → render to image. Generates structured representation (HTML) that captures layout, text, and interaction patterns, then renders deterministically for visual output. Agents can parse HTML for precise interaction targets or render for visual context.
Architecture Overview
Code Generation Stage: VLM generates HTML code representing next GUI state
Deterministic Rendering: Render HTML to image pixel-perfectly
"""
Generate HTML for next GUI state.
Args:
current_screenshot: PIL Image of current GUI
action_description: str describing action to perform
Returns:
html_code: str with HTML markup for next state
"""
# Prompt the VLM
f"""Given this GUI screenshot and the action '{action_description}',
generate the HTML code for the next GUI state. Include all visible elements, their positions,
text content, and styling. Return valid HTML only.
Previous state screenshot: [provided]
Output HTML code for next state:
```html
"""
# Generate HTML
self
500
# Extract HTML from response
self
return
def
_extract_html
self, text
"""Extract HTML block from model output."""
import
# Find content between ```html and ```
match
r'```html\n(.*?)\n```'
if
match
return
match
1
return
class
HTMLRenderer
"""Deterministically render HTML to images."""
def
__init__
self, viewport_width=1280, viewport_height=720
self
self
self
self
def
_init_selenium
self
"""Initialize headless browser for rendering."""
from
import
"--headless"
"--no-sandbox"
f"--window-size={self.width},{self.height}"
"--disable-blink-features=AutomationControlled"
return
def
render_html_to_image
self, html_code
"""Render HTML to PNG image."""
# Create data URL
f"data:text/html,{html_code}"
# Navigate to URL
self
# Take screenshot
self
open
return
def
close
self
self
class
Code2WorldWorldModel
"""Full world model with HTML generation + rendering."""
"""
Evaluate multiple candidate actions and select best.
Args:
current_screenshot: PIL Image
candidate_actions: List of action descriptions
num_samples: Number of rollouts per action
Returns:
best_action: Str with highest expected reward
predictions: Dict mapping actions to predicted screenshots
"""
for
in
# Sample multiple predictions for uncertainty
for
in
range
self
# Score prediction (e.g., via reward model)
self
0
# Store best
sum
len
# Select best action
max
return
def
_score_prediction
self, predicted_image, action
"""Score quality of prediction (action following + visual fidelity)."""
# Example scoring function
# In practice, use trained reward model
# Check for common visual artifacts
0.0
# ... artifact detection logic ...
# Score action consistency
0.5
# Placeholder
# ... action evaluation logic ...
return
Integrate into training with RL:
defcode2world_training_step(code_gen, renderer, batch, optimizer):
"""Training step combining SFT + RL."""
current_screenshots, actions, target_screenshots = batch
# Generate HTML predictions
predicted_htmls = []
for i, (screenshot, action) inenumerate(zip(current_screenshots, actions)):
html = code_gen.predict_next_html(screenshot, action)
predicted_htmls.append(html)
# Render predictions
predicted_screenshots = []
for html in predicted_htmls:
img = renderer.render_html_to_image(html)
predicted_screenshots.append(img)
# Compute rewards# 1. Visual similarity (LPIPS or SSIM)
visual_loss = F.mse_loss(
torch.tensor(predicted_screenshots),
torch.tensor(target_screenshots)
)
# 2. Action consistency (does predicted state reflect action?)
action_consistency = evaluate_action_consistency(predicted_screenshots, actions)
# Combined loss
loss = 0.7 * visual_loss + 0.3 * (1.0 - action_consistency)
# Update
optimizer.zero_grad()
loss.backward()
optimizer.step()
return loss.item()
Practical Guidance
Component
Recommendation
Notes
HTML specification
HTML5 subset
Stick to standard features; complex CSS breaks rendering.
Viewport size
1280×720
Standard; adjust per target interface.
Rendering engine
Headless Chrome/Firefox
Consistent rendering; use same engine always.
Training data
1K+ screenshots
Need diverse GUI states; pair with real trajectories.
RL weight
30-50%
Balance supervised signal with action consistency.
When to Use
GUI agent needs to evaluate action consequences before executing
Precise spatial reasoning matters (button locations, text regions)
Generating diverse action rollouts for planning
Interactive environments where agent makes sequential decisions
When rendering performance is critical (rendering adds latency)
Common Pitfalls
HTML generation too verbose/invalid; use prompt engineering for concise valid code
Rendering inconsistencies between training and deployment browsers; standardize
Not handling dynamic content (JavaScript, animations); code2world assumes static output
Over-fitting to training screenshots; augment data and regularize
Reference
See https://arxiv.org/abs/2602.09856 for full architecture, including action evaluation, GUI agent baselines, and validation on real-world web navigation tasks.
