latent-sketchpad-visual-reasoning

name	latent-sketchpad-visual-reasoning
title	Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.24514
keywords	["Visual Reasoning","Multimodal","Internal Scratchpad","Sketch Generation"]
description	Enables multimodal reasoning by interleaving visual sketches with text. MLLMs generate latent visual representations during reasoning, with sketch decoder converting them to human-interpretable images. Improves reasoning performance while maintaining interpretability through visual thinking aids.

Latent Sketchpad: Internal Visual Reasoning for Multimodal Models

Text-only reasoning in multimodal models wastes visual capabilities. Latent Sketchpad enables models to sketch during reasoning—generating internal visual representations that improve problem-solving while remaining interpretable.

The approach extends textual thinking to include visual-spatial reasoning, treating sketches as thought aids for complex tasks.

Core Concept

Key insight: models can reason better when able to externalize visual thinking, enabling:

• Interleaved text-visual reasoning
• Internal sketch generation for spatial problems (mazes, geometry)
• Sketch decoder makes thinking interpretable
• Improved performance without architectural changes

Architecture Overview

• Multimodal backbone unchanged
• Context-aware vision head: generates latent sketches autoregressively
• Sketch decoder: converts latents to interpretable images
• Token-level integration: sketches appear naturally in reasoning

Implementation Steps

Implement sketch generation as additional output head that produces visual latents during decoding:

class LatentSketchpad(nn.Module):
    def __init__(self, model_dim=768, sketch_dim=256, h=32, w=32):
        super().__init__()
        self.model_dim = model_dim
        self.sketch_dim = sketch_dim
        self.h, self.w = h, w

        # Context-aware vision head
        self.sketch_generator = nn.Sequential(
            nn.Linear(model_dim, model_dim),
            nn.ReLU(),
            nn.Linear(model_dim, sketch_dim * h * w)
        )

        # Sketch decoder: latent -> interpretable image
        self.sketch_decoder = nn.Sequential(
            nn.Linear(sketch_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(, )  
        )

     ():
        
        sketch_logits = .sketch_generator(hidden_state)
        sketch_latent = sketch_logits.reshape(
            -, .h, .w, .sketch_dim
        )
         sketch_latent

     ():
        
        batch_size, h, w, d = sketch_latent.shape
        sketch_flat = sketch_latent.reshape(-, d)
        pixels = .sketch_decoder(sketch_flat)
         pixels.reshape(batch_size, h, w, )