vfm-visual-tokenizer

name	vfm-visual-tokenizer
title	Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.08441
keywords	["Image Tokenization","Vision Foundation Models","Autoregressive Generation","Efficient Codecs"]
description	Use frozen vision foundation models like DINOv2 and CLIP as image tokenizers for autoregressive generation. Region-adaptive quantization identifies semantically coherent areas and reduces redundancy. Achieves 256-token encoding (vs. 576), 3× AR model speedup, state-of-the-art 1.36 gFID on ImageNet while eliminating classifier-free guidance.

VFMTok: Foundation Model-Based Visual Tokenization for Efficient AR Image Generation

Image generation typically requires tokenizing images into compact codes for autoregressive modeling. Learned tokenizers (VAE variants) need training and may lose semantic information. VFMTok leverages frozen vision foundation models—already trained on massive datasets to understand images semantically—as tokenizers. By adding region-adaptive quantization (identifying semantically coherent clusters rather than fixed grids), the approach reduces tokens by 55% (256 vs. 576), accelerates AR model training by 3×, and eliminates classifier-free guidance while maintaining state-of-the-art quality.

The key insight is that foundation models already extract meaningful features; you only need to make them quantize-friendly by identifying semantic regions and learning lightweight codebooks, not building entire new encoders.

Core Concept

VFMTok operates through a frozen pipeline:

1. Frozen VFM Encoder: Extract multi-level features from DINOv2 or CLIP (layers 6, 12, 18, 24)
2. Deformable Attention Sampler: Use learnable anchor queries to identify semantically coherent regions (adaptive quantization)
3. Lightweight Codebook: Learn small discrete vocabulary (16K codes) mapping regions to indices
4. Semantic Reconstruction: Dual objectives—pixel-level fidelity and VFM feature preservation

The frozen foundation model provides semantic understanding; the learnable components focus purely on efficient discretization.

Architecture Overview

• Frozen VFM Backbone: DINOv2-L or CLIP-L (no gradient flow)
• Multi-level Feature Extraction: Concatenate features from 4 layers (dims: 256→512→768→1024)
• Deformable Attention Module: Learnable anchor queries with deformable sampling to find semantic regions
• Codebook Vector Quantizer (VQ): Maps regional features to discrete codes (size: 16384, dim: 12)
• Lightweight Decoder: Learnable ViT-style decoder reconstructing pixels from codes
• Dual Loss: Pixel-level L2 + VFM feature reconstruction via cosine distance
• Positional Embeddings: 2D spatial encoding for anchor positions

Implementation

The following demonstrates region-adaptive quantization and the tokenization pipeline:

import torch
import torch.nn as nn
import torch.nn.functional  F
 typing  

 (nn.Module):
    
     ():
        ().__init__()
        .feature_dim = feature_dim
        .num_anchors = num_anchors
        .num_heads = num_heads

        
        .anchor_queries = nn.Parameter(torch.randn(, num_anchors, feature_dim))

        
        .offset_regression = nn.Linear(feature_dim, )  

        
        .cross_attention = nn.MultiheadAttention(
            feature_dim, num_heads, batch_first=
        )

        
        .pos_embedding = nn.Embedding(num_anchors, feature_dim)

     () -> [torch.Tensor, torch.Tensor]:
        
        batch_size = features.shape[]

        
        positions = torch.arange(.num_anchors, device=features.device)
        pos_emb = .pos_embedding(positions).unsqueeze().expand(batch_size, -, -)
        anchors_with_pos = .anchor_queries.expand(batch_size, -, -) + pos_emb

        
        offsets = torch.tanh(.offset_regression(anchors_with_pos)) *   
        anchor_positions = offsets  

        
        
        sampled_features = ._deformable_sample(features, anchor_positions)

        
        attended, _ = .cross_attention(
            query=anchors_with_pos,
            key=sampled_features,
            value=sampled_features
        )

         attended, anchor_positions

     () -> torch.Tensor:
        
        
        

        batch_size, seq_len, feature_dim = features.shape
        num_anchors = positions.shape[]
        h = w = (seq_len ** )  

        
        pixel_x = (positions[:, :, ] + ) * (w - )
        pixel_y = (positions[:, :, ] + ) * (h - )

        
        grid = torch.stack([pixel_x / (w - ) *  - ,
                           pixel_y / (h - ) *  - ], dim=-)

        
        features_spatial = features.view(batch_size, h, w, feature_dim).permute(, , , )

        sampled = F.grid_sample(
            features_spatial.(),
            grid.unsqueeze().(),  
            align_corners=,
            mode=
        )

         sampled.permute(, , , ).squeeze()  

 (nn.Module):
    
     ():
        ().__init__()
        .codebook_size = codebook_size
        .feature_dim = feature_dim
        .beta = beta

        
        .embedding = nn.Embedding(codebook_size, feature_dim)
        .embedding.weight.data.uniform_(- / codebook_size,  / codebook_size)

     () -> [torch.Tensor, torch.Tensor, torch.Tensor]:
        
        
        
        distances = torch.cdist(x, .embedding.weight)  

        
        indices = distances.argmin(dim=-)  
        quantized = .embedding(indices)  

        
        loss = F.mse_loss(x.detach(), quantized) + .beta * F.mse_loss(x, quantized.detach())

         quantized, loss, indices

 (nn.Module):
    
     ():
        ().__init__()
        .num_anchor_regions = num_anchor_regions

        
        .vfm =   
        .vfm_feature_dim =   

        
         param  .vfm.parameters():
            param.requires_grad = 

        
        .deformable_sampler = DeformableAttentionSampler(
            feature_dim=.vfm_feature_dim,
            num_anchors=num_anchor_regions,
            num_heads=
        )

        .quantizer = VectorQuantizer(
            codebook_size=codebook_size,
            feature_dim=  
        )

        
        .decoder = nn.Sequential(
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, )  
        )

     () -> [torch.Tensor, torch.Tensor, torch.Tensor]:
        
        
         torch.no_grad():
            
            feat_layer6 = .vfm.get_layer_output(image, layer_idx=)   
            feat_layer12 = .vfm.get_layer_output(image, layer_idx=)  
            feat_layer18 = .vfm.get_layer_output(image, layer_idx=)  
            feat_layer24 = .vfm.get_layer_output(image, layer_idx=)  

            
            features = torch.cat([
                F.interpolate(feat_layer6, size=feat_layer24.shape[:], mode=),
                F.interpolate(feat_layer12, size=feat_layer24.shape[:], mode=),
                F.interpolate(feat_layer18, size=feat_layer24.shape[:], mode=),
                feat_layer24
            ], dim=-)  

        
        regional_features, anchor_positions = .deformable_sampler(features)
        

        
        projected = F.linear(regional_features, torch.randn(, ))  

        
        quantized, quant_loss, codes = .quantizer(projected)
        

        
        reconstructed = .decoder(quantized)  

        
        
        grid_h = grid_w = (.num_anchor_regions ** )
        reconstructed_image = reconstructed.view(-, grid_h, grid_w, ).permute(, , , )
        reconstructed_image = F.interpolate(reconstructed_image, size=(, ), mode=)

        
        recon_loss = F.mse_loss(reconstructed_image, image)
        total_loss = quant_loss + recon_loss

         codes, total_loss, reconstructed_image

 ():
    

     epoch  (num_epochs):
        total_loss = 

         batch_idx, images  (train_loader):
            optimizer.zero_grad()

            
            codes, loss, reconstructed = model(images)

            
            
            

            total_loss_iter = loss

            total_loss_iter.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), )
            optimizer.step()

            total_loss += total_loss_iter.item()

        avg_loss = total_loss / (train_loader)
        ()

         (epoch + ) %  == :
            
            sample_codes, _, sample_recon = model(((train_loader)))
            avg_tokens = (sample_codes >= ).(dim=).().mean().item()
            ()