vqrae-representation-quantization

name	vqrae-representation-quantization
title	VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.23386
keywords	["quantization autoencoders","multimodal learning","discrete tokens","vector quantization","unified tokenizer"]
description	Unify multimodal understanding, generation, and reconstruction using high-dimensional codebooks for semantic information. VQRAE achieves 100% codebook utilization at 1536 dimensions—ideal when you need a single tokenizer for vision-language tasks.

Overview

VQRAE combines continuous semantic representations and discrete tokens in a single framework through high-dimensional vector quantization autoencoders. Two-stage training first learns semantic quantization, then jointly optimizes with self-distillation for all tasks.

When to Use

• Unified multimodal tokenization (understanding + generation + reconstruction)
• Visual semantic understanding and generation
• Need for single tokenizer across multiple tasks
• High-dimensional codebooks for semantic information
• Autoregressive model training on multimodal data

When NOT to Use

• Task-specific tokenizers already optimal
• Scenarios where separate understanding/generation models work
• Limited codebook dimension resources

Core Technique

High-dimensional vector quantization for semantic tokens:

# VQRAE: Unified multimodal tokenization
class RepresentationQuantizationAutoencoder:
    def __init__(self, codebook_dim=1536):
        self.encoder = nn.Sequential(
            # Visual feature extraction with ViT
            VisionTransformer(pretrained=True),
            nn.Linear(768, 512)
        )

        self.decoder = nn.Sequential(
            nn.Linear(512, 768),
            nn.ReLU(),
            nn.Linear(768, image_size * image_size * 3)
        )

        # High-dimensional codebook
        self.codebook_dim = codebook_dim
        self.codebook = nn.Embedding(
            codebook_dim,
            embedding_dim=512
        )

    def encode_with_quantization(self, image):
        """Encode to discrete tokens."""
        # Continuous features
        features = .encoder(image)

        
        token_indices = torch.argmin(
            torch.cdist(features.unsqueeze(), .codebook.weight),
            dim=
        ).squeeze()

         token_indices

     ():
        
        
         batch  image_dataset:
            
            features = .encoder(batch)

            
            token_indices = .quantize(features)
            quantized_features = .codebook(token_indices)

            
            reconstructed = .decoder(quantized_features)

            
            recon_loss = torch.nn.functional.mse_loss(
                reconstructed,
                batch
            )

            recon_loss.backward()
            .optimizer.step()

        
         batch  image_dataset:
            
            teacher_features = .teacher_encoder(batch)

            
            student_tokens = .encode_with_quantization(batch)
            student_features = .codebook(student_tokens)

            
            distill_loss = torch.nn.functional.mse_loss(
                student_features,
                teacher_features.detach()
            )

            
            reconstructed = .decoder(student_features)
            recon_loss = torch.nn.functional.mse_loss(
                reconstructed,
                batch
            )

            total_loss = distill_loss + recon_loss
            total_loss.backward()
            .optimizer.step()

     ():
        
        
        flat_features = features.reshape(-, features.shape[-])

        
        distances = torch.cdist(
            flat_features.unsqueeze(),
            .codebook.weight.unsqueeze()
        )

        indices = torch.argmin(distances, dim=).squeeze()

         indices.reshape(features.shape[:-])

     ():
        
        tokens = .encode_with_quantization(image_batch)
        unique_tokens = (torch.unique(tokens))

        utilization = unique_tokens / .codebook_dim

         utilization