monet-latent-visual-reasoning

name	monet-latent-visual-reasoning
title	Monet: Reasoning in Latent Visual Space Beyond Images and Language
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.21395
keywords	["Latent Visual Reasoning","Chain of Thought","VLM Enhancement","Vision-Language Models"]
description	Enable multimodal LLMs to generate and reason with latent visual embeddings as intermediate thoughts: implement supervised fine-tuning to produce continuous visual representations, then optimize via VLPO to treat embeddings as learnable actions in reinforcement learning.

Monet: Latent Visual Reasoning for Multimodal Models

Vision-language models often struggle with complex spatial reasoning because they must choose between text-only reasoning (losing visual detail) or generating external auxiliary images (expensive and error-prone). This skill demonstrates how to enable MLLMs to internally generate and reason with latent visual embeddings—intermediate visual thoughts that act as a third modality alongside text and images.

The core innovation is treating continuous latent embeddings as learnable "actions" that can be generated and optimized through reinforcement learning, enabling flexible reasoning that interleaves text and visual thought.

Core Concept

Monet enables MLLMs to generate continuous latent visual embeddings through:

1. Supervised Fine-Tuning (SFT): Three-stage training progressively teaching the model to generate and reason with latent embeddings

2. Vision-Language Policy Optimization (VLPO): Novel RL algorithm computing policy gradients directly for latent embeddings by treating them as continuous actions

3. Flexible Interleaving: Models learn to automatically decide when to generate latent reasoning tokens vs. continuing text generation

Architecture Overview

• Latent Embedding Generation: Special tokens that decode to continuous visual embeddings
• Attention Control Mechanism: Learnable flow control maintaining visual information through generation
• SFT Pipeline: Stage 1 (observation alignment), Stage 2 (latent generation), Stage 3 (reasoning chains)
• VLPO Optimizer: Gradient computation for continuous embeddings via probability estimation
• Flexible Token Interleaving: Text and latent tokens can be mixed in reasoning chains

Implementation Steps

The system trains through SFT stages then optimizes with reinforcement learning.

1. Prepare Hidden State Alignment

Align hidden representations of key observation tokens to initialize latent reasoning.

def compute_hidden_state_alignment(model, observation_tokens, hidden_dim=4096, num_layers=32):
    """
    Align hidden states of observation tokens for efficient latent encoding.
    Reduces computational cost of full image embedding alignment.
    Args:
        model: MLLM with vision encoder
        observation_tokens: Special tokens representing image observations
        hidden_dim: Model hidden dimension
        num_layers: Number of transformer layers
    Returns:
        alignment_matrix: Maps observation hidden states to latent space
    """
    
    observation_hiddens = []

     obs_token  observation_tokens:
         torch.no_grad():
            output = model.encode_observation(obs_token, output_hidden_states=)

            
            key_layer_idx = (num_layers * )
            hidden = output.hidden_states[key_layer_idx]

            observation_hiddens.append(hidden.mean(dim=))  

    observation_hiddens = torch.stack(observation_hiddens)

    
    alignment_matrix = torch.nn.Linear(hidden_dim, hidden_dim)

    
    optimizer = torch.optim.Adam([alignment_matrix.parameters()], lr=)

     _  ():
        projected = alignment_matrix(observation_hiddens)
        reconstruction_loss = torch.nn.functional.mse_loss(projected, observation_hiddens)
        reconstruction_loss.backward()
        optimizer.step()
        optimizer.zero_grad()

     alignment_matrix