semantic-visual-reconstruction

name	semantic-visual-reconstruction
title	Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.09040
keywords	["vision-language","semantic reconstruction","visual tokenization","multimodal supervision"]
description	Add explicit visual supervision to VLMs by training models to autoregressively reconstruct semantic image tokens, achieving 2-3% average gains and 10-point improvements on hallucination robustness.

Autoregressive Semantic Visual Reconstruction

Core Concept

Traditional Vision-Language Models (VLMs) apply supervision only to text outputs while leaving rich visual input unsupervised. ASVR (Autoregressive Semantic Visual Reconstruction) addresses this asymmetry by training models to predict both semantic visual tokens and text tokens within a unified framework. This establishes a "perceptual foundation for image understanding" that improves robustness and reduces hallucinations.

Architecture Overview

• Visual tokenizer: VQ-SigLIP converts images to discrete semantic tokens capturing high-level features
• Joint training objective: Unified loss on both visual token and text token prediction
• Two-stage training: Pre-training aligns visual representations, instruction tuning refines understanding
• Semantic > Appearance: High-level semantic information matters more than pixel-level reconstruction

Implementation

Step 1: Build Semantic Visual Tokenizer

Create tokenizer that captures high-level semantic features:

class SemanticVisualTokenizer:
    def __init__(self, model_name: str = "vq-siglip"):
        self.model_name = model_name
        # Load pretrained semantic tokenizer
        self.tokenizer = self._load_semantic_tokenizer(model_name)
        self.vocab_size = self.tokenizer.config.vocab_size

    def _load_semantic_tokenizer(self, model_name: str):
        """Load pretrained semantic visual tokenizer."""
        # VQ-SigLIP: VQ-VAE variant using SigLIP embeddings
        from transformers import AutoModel
        return AutoModel.from_pretrained(
            f"semantic-tokenizers/{model_name}",
            trust_remote_code=
        )

     () -> torch.Tensor:
        
        
        image = (image - image.mean()) / image.std()

        
         torch.no_grad():
            tokens = .tokenizer.encode(image)

         tokens  

     () -> torch.Tensor:
        
         torch.no_grad():
            image = .tokenizer.decode(tokens)

         image