adaptive-token-reduction-image-representation

name	adaptive-token-reduction-image-representation
title	When Less is Enough: Adaptive Token Reduction for Efficient Image Representation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2503.16660
keywords	["Token Reduction","Vision Transformers","Feature Selection","Image Compression","Efficiency"]
description	Adaptively prune visual tokens from vision encoders by reconstructing discarded features from retained ones, reducing computational cost by 50% while maintaining task performance on OCR and image understanding tasks.

Core Concept

This skill implements adaptive token reduction for vision encoders. The key insight is that many visual tokens are redundant—their information can be reconstructed from a smaller subset of more informative tokens. Rather than random pruning, this approach learns which tokens are essential by training a selector network to identify valuable tokens and a reconstructor to verify that discarded tokens can be faithfully recovered.

Architecture Overview

The system has three main components:

• Feature Selector (S): Three Transformer layers with a Gumbel-Softmax head that generates binary masks, choosing which tokens to keep or discard
• Feature Reconstructor (R): Three Transformer layers that reconstruct discarded tokens from retained ones plus a shared learnable masked embedding
• Optimization Objective: Balances reconstruction fidelity against pruning efficiency using modified regularization

The training uses an autoencoder-like framework where the selector learns to identify redundant tokens, and the reconstructor validates that removed tokens are recoverable.

Implementation

The feature selection mechanism uses Gumbel-Softmax for differentiable discrete choices. The following code shows the core selector and reconstructor modules:

import torch
import torch.nn as nn
from torch.nn.functional import gumbel_softmax

class TokenSelector(nn.Module):
    """Selects which tokens to retain using Gumbel-Softmax."""
    def __init__(self, hidden_dim, num_layers=3, num_heads=8):
        super().__init__()
        self.transformer_layers = nn.ModuleList([
            nn.TransformerEncoderLayer(
                d_model=hidden_dim,
                nhead=num_heads,
                dim_feedforward=hidden_dim * 4,
                batch_first=True
            ) for _ in range(num_layers)
        ])
        self.selection_head = nn.Linear(hidden_dim, )

     ():
        
        x = features
         layer  .transformer_layers:
            x = layer(x)

        logits = .selection_head(x).squeeze(-)

         training:
            
            masks = gumbel_softmax(logits.unsqueeze(-), tau=temperature, hard=)
            masks = masks.squeeze(-)
        :
            masks = (logits > ).()

         masks, logits