code2world-gui-synthesis

name	code2world-gui-synthesis
title	Code2World: A GUI World Model via Renderable Code Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.09856
keywords	["GUI World Model","HTML Generation","Structured Prediction","Vision-Language","Deterministic Rendering"]
description	Predict GUI state evolution by generating HTML code rather than pixel images. Combines visual fidelity of pixel-based approaches with structural precision of code-based methods through deterministic rendering. Enables agents to evaluate action consequences and select best decisions before execution.

Code2World: HTML-Based GUI World Modeling

Predicting GUI state changes requires balancing visual realism against structural controllability. Pixel-based models achieve realism but lack precision; text-only code lacks visual grounding. Code2World bridges this gap: VLMs generate HTML code for the next GUI state, then deterministically render it to visual images. This unifies the strengths of both approaches while enabling agents to anticipate action consequences with both visual and structural clarity.

Core Concept

Standard pixel prediction: image + action → predicted next image. Realistic but spatially imprecise; hard for agents to verify if buttons appear in correct locations.

Code2World: image + action → predicted HTML code → render to image. Generates structured representation (HTML) that captures layout, text, and interaction patterns, then renders deterministically for visual output. Agents can parse HTML for precise interaction targets or render for visual context.

Architecture Overview

• Code Generation Stage: VLM generates HTML code representing next GUI state
• Deterministic Rendering: Render HTML to image pixel-perfectly
• Dual Representation: Structured code (for parsing) + rendered image (for visual grounding)
• Training Strategy: Supervised fine-tuning (HTML generation) + RL (optimize for action consistency)
• Action Evaluation: Agents can sample multiple actions, render outcomes, and select best before execution

Implementation

Implement HTML generation and rendering:

from selenium import webdriver
from PIL import Image
import io
import torch
import torch.nn as nn

class Code2WorldModel(nn.Module):
    """VLM fine-tuned for HTML generation."""

    def __init__(self, base_vlm_model):
        super().__init__()
        self.vlm = base_vlm_model

    def predict_next_html(self, current_screenshot, action_description, temperature=):
        
        
        prompt = 

        
        html_output = .vlm.generate(
            current_screenshot,
            prompt,
            max_tokens=,
            temperature=temperature
        )

        
        html_code = ._extract_html(html_output)
         html_code

     ():
        
         re

        
         = re.search(, text, re.DOTALL)
         :
             .group()
         text

 :
    

     ():
        .width = viewport_width
        .height = viewport_height
        .driver = ._init_selenium()

     ():
        
         selenium.webdriver.chrome.options  Options

        chrome_options = Options()
        chrome_options.add_argument()
        chrome_options.add_argument()
        chrome_options.add_argument()
        chrome_options.add_argument()

        driver = webdriver.Chrome(options=chrome_options)
         driver

     ():
        
        
        data_url = 

        
        .driver.get(data_url)

        
        screenshot = .driver.get_screenshot_as_png()
        image = Image.(io.BytesIO(screenshot))

         image

     ():
        .driver.quit()

 :
    

     ():
        .code_gen = Code2WorldModel(vlm_model)
        .renderer = HTMLRenderer(viewport_size[], viewport_size[])

     ():
        
        
        html = .code_gen.predict_next_html(current_screenshot, action)

        
        next_screenshot = .renderer.render_html_to_image(html)

         next_screenshot, html

     ():
        
        predictions = {}
        scores = {}

         action  candidate_actions:
            rollout_images = []
            rollout_scores = []

            
             _  (num_samples):
                next_img, html = .predict_next_state(current_screenshot, action)
                rollout_images.append(next_img)

                
                score = ._score_prediction(next_img, action)
                rollout_scores.append(score)

            predictions[action] = rollout_images[]  
            scores[action] = (rollout_scores) / (rollout_scores)

        
        best_action = (scores, key=scores.get)

         best_action, predictions, scores

     ():
        
        
        

        
        artifact_penalty = 
        

        
        action_consistency =   
        

         action_consistency - artifact_penalty