neuralos-gui-simulation

name	neuralOS-gui-simulation
title	NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.08800
keywords	["GUI Simulation","Diffusion Models","RNN State Tracking","Interactive Systems"]
description	Simulate GUI behavior by predicting screen frames in response to user inputs. NeuralOS combines hierarchical RNNs for state tracking with diffusion-based rendering, capturing mouse interactions and application state transitions. Trains on synthetic demonstrations plus random exploration; achieves 50-61% human indistinguishability on basic operations while maintaining 18 fps inference on single H100.

NeuralOS: Simulate Operating System Interactions with Neural Models

Operating system GUIs involve complex state tracking and visual rendering—applications open, windows resize, text appears. Traditional automation approaches use rigid rules; NeuralOS learns to predict GUI evolution from user inputs (mouse position, clicks, keyboard) using a recurrent neural network for state tracking and diffusion models for frame generation. The system captures state transitions implicitly, enabling realistic simulation of multi-step interactions without explicit state machines.

The key insight is that GUI behavior is spatiotemporally continuous: mouse movements lead to smooth cursor motion, clicks trigger state changes with visual consequences, keyboard input fills text fields. By combining RNNs (efficient state tracking without quadratic complexity) with diffusion rendering (high-fidelity frame generation), you can simulate OS interactions at near-realistic quality.

Core Concept

NeuralOS operates as a three-component pipeline:

1. RNN State Tracker: Maintains hidden system state (what windows are open, text field contents) based on accumulated user inputs, avoiding transformer quadratic complexity during inference
2. Latent Frame Encoder: Compresses video frames into latent codes to reduce diffusion model compute
3. Diffusion Renderer: Generates new frames conditioned on RNN state, user input, and spatial cursor encoding

The system learns from two data sources: agent-generated demonstrations (synthetic interactions) and random exploration (avoiding spurious agent patterns).

Architecture Overview

• RNN State Module: 2-level hierarchical LSTM tracking long-term state and short-term dynamics (avoids transformer quadratic cost)
• Latent Encoder-Decoder: Converts 1280×720 RGB frames to 160×90 latent codes for efficient diffusion
• Diffusion UNet Renderer: Denoising network conditioned on RNN state via cross-attention
• Spatial Cursor Encoding: Gaussian spatial embeddings precisely localizing cursor (hundreds-of-pixels error without it)
• Scheduled Sampling Trainer: Gradually shifts from training-data frames to model-generated frames during training
• Context Manager: Extends from 32 to 64 frame context for capturing longer dependencies

Implementation

The following demonstrates the hierarchical RNN state tracker and diffusion renderer:

import torch
import torch.nn as nn
 torch.nn.functional  F
 typing  , 

 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .context_len = context_len

        
        .frame_lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim // ,
            num_layers=,
            batch_first=
        )

        
        .context_lstm = nn.LSTM(
            input_size=hidden_dim // ,
            hidden_size=hidden_dim,
            num_layers=,
            batch_first=
        )

        
        .input_proj = nn.Linear(input_dim, input_dim)

     ():
        
        
        projected = .input_proj(input_sequence)

        
        frame_outputs, frame_hidden = .frame_lstm(projected, frame_hidden)

        
        context_outputs, context_hidden = .context_lstm(frame_outputs, context_hidden)

         context_outputs, frame_hidden, context_hidden

 (nn.Module):
    
     ():
        ().__init__()
        .height = height
        .width = width
        .embedding_dim = embedding_dim

        
        .sigma = nn.Parameter(torch.ones() * )
        .learnable_embedding = nn.Embedding(, embedding_dim)  

     ():
        
        batch_size = cursor_x.shape[]

        
        y_coords = torch.linspace(, , .height, device=cursor_x.device)
        x_coords = torch.linspace(, , .width, device=cursor_x.device)
        yy, xx = torch.meshgrid(y_coords, x_coords, indexing=)

        
        yy = yy.unsqueeze().expand(batch_size, -, -)  
        xx = xx.unsqueeze().expand(batch_size, -, -)

        
        dist = torch.sqrt(
            (xx - cursor_x.view(batch_size, , )) **  +
            (yy - cursor_y.view(batch_size, , )) ** 
        )

        
        gaussian = torch.exp(-dist **  / ( * .sigma ** ))  

        
        spatial_emb = gaussian.unsqueeze(-).expand(-, -, -, .embedding_dim)
         spatial_emb

 (nn.Module):
    
     ():
        ().__init__()
        .latent_dim = latent_dim
        .num_diffusion_steps = num_diffusion_steps

        
        .state_projection = nn.Linear(state_dim, hidden_channels)
        .cross_attention = nn.MultiheadAttention(
            hidden_channels, num_heads=, batch_first=
        )

        
        .down_blocks = nn.ModuleList([
            nn.Conv2d(latent_dim + hidden_channels, hidden_channels, kernel_size=, padding=),
            nn.Conv2d(hidden_channels, hidden_channels * , kernel_size=, padding=, stride=),
        ])

        .up_blocks = nn.ModuleList([
            nn.ConvTranspose2d(hidden_channels * , hidden_channels, kernel_size=, stride=, padding=),
            nn.Conv2d(hidden_channels, latent_dim, kernel_size=, padding=),
        ])

        .time_embedding = nn.Embedding(num_diffusion_steps, hidden_channels)

     ():
        
        batch_size, _, h, w = x_t.shape

        
        state_proj = .state_projection(state)  
        state_proj = state_proj.unsqueeze()  

        
        time_emb = .time_embedding(torch.tensor(timestep, device=x_t.device))
        time_emb = time_emb.unsqueeze().expand(batch_size, -)

        
        cursor_flat = cursor_encoding.view(batch_size, h, w, -).permute(, , , )

        
        x_with_cursor = torch.cat([x_t, cursor_flat[:, :x_t.shape[]]], dim=)

        
        x = x_with_cursor
         block  .down_blocks:
            x = block(x)
            x = F.relu(x)

        
        x_flat = x.view(batch_size, -, x.shape[]).permute(, , )
        attended, _ = .cross_attention(
            query=x_flat,
            key=state_proj,
            value=state_proj
        )
        x = attended.permute(, , ).view(x.shape)

        
         block  .up_blocks:
            x = block(x)
            x = F.relu(x)

         x

 (nn.Module):
    
     ():
        ().__init__()
        .rnn_tracker = HierarchicalRNNStateTracker(
            input_dim=,  
            hidden_dim=hidden_dim
        )
        .cursor_encoder = SpatialCursorEncoding(embedding_dim=hidden_dim)
        .frame_renderer = DiffusionFrameRenderer(
            latent_dim=latent_dim, state_dim=hidden_dim
        )

     ():
        
        
        state_seq, _, _ = .rnn_tracker(input_sequence)

        predictions = []
         t  (input_sequence.shape[]):
            
            cursor_emb = .cursor_encoder(
                cursor_positions[:, t, ],
                cursor_positions[:, t, ]
            )

            
            pred = .frame_renderer(
                latent_frames[:, t],
                timesteps[:, t].item(),
                state_seq[:, t],
                cursor_emb
            )
            predictions.append(pred)

         torch.stack(predictions, dim=)

 () -> :
    
    optimizer.zero_grad()

    
    input_events = batch_inputs[]  
    latent_frames = batch_inputs[]  
    cursor_pos = batch_inputs[]  
    timesteps = batch_inputs[]  
    real_frames = batch_inputs[]  

    
    predictions = model(input_events, latent_frames, cursor_pos, timesteps)

    
    loss = diffusion_loss_fn(predictions, real_frames)

    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), )
    optimizer.step()

     loss.item()