Skip to main content Inicio Creadores adu2021 skillxiv dynamic-visual-token-pruning-vlm
dynamic-visual-token-pruning-vlm Dynamic token pruning framework for VLMs that adapts compression to scene complexity through single-pass selection. Removes 92.6% of visual tokens while maintaining performance and enabling superior fine-tuning.
Ir a la instalación Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/ADu2021/skillXiv --skill dynamic-visual-token-pruning-vlmEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Más de este repositorio meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
name dynamic-visual-token-pruning-vlm title GlimpsePrune Dynamic Visual Token Pruning for Vision-Language Models version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2508.01548 keywords ["vision-language-models","token-pruning","efficiency","visual-compression","inference"] description Dynamic token pruning framework for VLMs that adapts compression to scene complexity through single-pass selection. Removes 92.6% of visual tokens while maintaining performance and enabling superior fine-tuning.
GlimpsePrune: Dynamic Visual Token Pruning for Vision-Language Models
GlimpsePrune addresses computational inefficiency in Vision-Language Models by dynamically pruning irrelevant visual tokens based on input complexity. Rather than fixed compression ratios, the framework intelligently adapts token retention to scene characteristics, achieving massive efficiency gains while improving downstream task performance.
Core Concept
The fundamental insight is that not all image regions contribute equally to answering queries. Complex scenes need more visual tokens while simple scenes can work with fewer. GlimpsePrune:
Analyzes scene complexity to determine necessary compression ratioScores token importance dynamically based on content and query relevancePrunes tokens adaptively in a single forward passPreserves critical information while removing redundancyEnables better fine-tuning through cleaner token representations
Architecture Overview
The framework consists of:
Visual Encoder : Processes images into tokensComplexity Analyzer : Estimates scene complexity from visual featuresToken Scorer : Assigns importance scores to each tokenAdaptive Pruning Module : Selects tokens to retain based on complexityLanguage Model : Processes pruned tokens for downstream tasksFine-Tuning Adapter : Leverages pruned tokens for improved learning
Implementation Steps
Step 1: Build visual token scorer
Create a module that assigns importance scores to visual tokens:
import torch
import torch.nn as nn
from typing import Tuple
import numpy as np
class VisualTokenScorer (nn.Module):
"""Scores visual tokens for importance-based pruning"""
def __init__ (self, hidden_size: int = ):
768
int
2
super
self
for
in
range
1
1
def
forward
self, token_embeddings: torch.Tensor
"""
Score each visual token for importance.
Args:
token_embeddings: Shape (batch, num_tokens, hidden_size)
Returns:
Importance scores of shape (batch, num_tokens)
"""
self
return
class
QueryAwareScorer
"""Scores tokens based on relevance to input query"""
def
__init__
self, hidden_size: int = 768
super
self
self
def
forward
self, token_embeddings: torch.Tensor,
query_embedding: torch.Tensor
"""
Score tokens based on query relevance.
Args:
token_embeddings: (batch, num_tokens, hidden_size)
query_embedding: (batch, hidden_size)
Returns:
Query-aware importance scores (batch, num_tokens)
"""
self
self
2
2
return
This creates learnable importance scoring mechanisms.
Step 2: Analyze scene complexity
Estimate visual scene complexity to determine pruning ratio:
class ComplexityAnalyzer :
"""Analyzes visual scene complexity to guide pruning"""
def __init__ (self, method: str = 'entropy' ):
self .method = method
def compute_complexity (self, token_embeddings: torch.Tensor ) -> torch.Tensor:
"""
Estimate scene complexity from token embeddings.
Args:
token_embeddings: (batch, num_tokens, hidden_size)
Returns:
Complexity scores for each image (batch,), range [0, 1]
"""
if self .method == 'entropy' :
return self ._entropy_complexity(token_embeddings)
elif self .method == 'variance' :
return self ._variance_complexity(token_embeddings)
elif self .method == 'histogram' :
return self ._histogram_complexity(token_embeddings)
else :
return torch.ones(token_embeddings.shape[0 ])
def _entropy_complexity (self, embeddings: torch.Tensor ) -> torch.Tensor:
"""
Measure complexity via entropy of token distribution.
Higher entropy = more diverse tokens = higher complexity
"""
batch_size = embeddings.shape[0 ]
complexities = []
for i in range (batch_size):
tokens = embeddings[i]
normalized = torch.nn.functional.normalize(tokens, dim=1 )
sim_matrix = torch.mm(normalized, normalized.t())
probs = torch.softmax(sim_matrix.mean(dim=0 ), dim=0 )
entropy = -(probs * torch.log(probs + 1e-8 )).sum ()
max_entropy = np.log(tokens.shape[0 ])
normalized_entropy = (entropy / max_entropy).clamp(0 , 1 )
complexities.append(normalized_entropy)
return torch.stack(complexities)
def _variance_complexity (self, embeddings: torch.Tensor ) -> torch.Tensor:
"""Estimate complexity from feature variance"""
batch_size = embeddings.shape[0 ]
complexities = []
for i in range (batch_size):
tokens = embeddings[i]
variance = tokens.var(dim=0 ).mean()
normalized_variance = torch.tanh(variance)
complexities.append(normalized_variance)
return torch.stack(complexities)
def _histogram_complexity (self, embeddings: torch.Tensor ) -> torch.Tensor:
"""Estimate from spatial distribution of tokens"""
batch_size = embeddings.shape[0 ]
complexities = []
for i in range (batch_size):
tokens = embeddings[i]
mean = tokens.mean(dim=0 )
distances = torch.norm(tokens - mean, dim=1 )
spread = distances.std()
normalized = torch.tanh(spread)
complexities.append(normalized)
return torch.stack(complexities)
def determine_target_ratio (self, complexity: torch.Tensor,
min_ratio: float = 0.1 ,
max_ratio: float = 0.9 ) -> torch.Tensor:
"""
Map complexity to target retention ratio.
Low complexity → lower ratio (more pruning)
High complexity → higher ratio (fewer prune)
"""
ratio = min_ratio + (1 - complexity) * (max_ratio - min_ratio)
return ratio
This enables adaptive pruning based on image properties.
Step 3: Implement adaptive token selection
Select which tokens to keep based on scores and complexity:
class AdaptiveTokenSelector (nn.Module):
"""Selects tokens adaptively based on importance and complexity"""
def __init__ (self, hidden_size: int = 768 ):
super ().__init__()
self .token_scorer = VisualTokenScorer(hidden_size)
self .complexity_analyzer = ComplexityAnalyzer()
def forward (self, token_embeddings: torch.Tensor,
attention_mask: torch.Tensor = None ) -> Tuple [torch.Tensor, torch.Tensor]:
"""
Adaptively select tokens to retain.
Args:
token_embeddings: (batch, num_tokens, hidden_size)
attention_mask: (batch, num_tokens), optional
Returns:
(selected_tokens, selection_mask)
"""
batch_size, num_tokens, hidden_size = token_embeddings.shape
importance_scores = self .token_scorer(token_embeddings)
complexity = self .complexity_analyzer.compute_complexity(token_embeddings)
target_ratio = self .complexity_analyzer.determine_target_ratio(complexity)
selected_tokens_list = []
selection_masks = []
for b in range (batch_size):
num_keep = max (1 , int (num_tokens * target_ratio[b].item()))
scores = importance_scores[b]
topk_scores, topk_indices = torch.topk(
scores[1 :],
k=min (num_keep - 1 , num_tokens - 1 )
)
all_indices = torch.cat([
torch.tensor([0 ], device=scores.device),
topk_indices + 1
])
all_indices = all_indices.sort()[0 ]
mask = torch.zeros(num_tokens, dtype=torch.bool , device=scores.device)
mask[all_indices] = True
selected_token = token_embeddings[b][mask]
selected_tokens_list.append(selected_token)
selection_masks.append(mask)
max_selected = max (t.shape[0 ] for t in selected_tokens_list)
padded_tokens = torch.zeros(
batch_size, max_selected, hidden_size,
device=token_embeddings.device,
dtype=token_embeddings.dtype
)
for b, tokens in enumerate (selected_tokens_list):
padded_tokens[b, :tokens.shape[0 ]] = tokens
return padded_tokens, torch.stack(selection_masks)
This adaptively selects tokens in a single pass.
Step 4: Implement GlimpsePrune forward pass
Integrate all components into unified pruning module:
class GlimpsePrune (nn.Module):
"""Complete dynamic visual token pruning module"""
def __init__ (self, hidden_size: int = 768 , enable_query_awareness: bool = True ):
super ().__init__()
self .hidden_size = hidden_size
self .token_scorer = VisualTokenScorer(hidden_size)
if enable_query_awareness:
self .query_scorer = QueryAwareScorer(hidden_size)
else :
self .query_scorer = None
self .complexity_analyzer = ComplexityAnalyzer()
self .selector = AdaptiveTokenSelector(hidden_size)
def forward (self, image_tokens: torch.Tensor,
query_embedding: torch.Tensor = None ,
min_retention_ratio: float = 0.1 ,
max_retention_ratio: float = 0.9 ) -> Tuple [torch.Tensor, Dict ]:
"""
Perform adaptive visual token pruning.
Args:
image_tokens: (batch, num_tokens, hidden_size)
query_embedding: (batch, hidden_size), optional
min_retention_ratio: Minimum tokens to keep
max_retention_ratio: Maximum tokens to keep
Returns:
(pruned_tokens, pruning_stats)
"""
batch_size, num_tokens, _ = image_tokens.shape
importance = self .token_scorer(image_tokens)
if query_embedding is not None and self .query_scorer is not None :
query_importance = self .query_scorer(image_tokens, query_embedding)
importance = 0.6 * importance + 0.4 * query_importance
complexity = self .complexity_analyzer.compute_complexity(image_tokens)
target_ratio = self .complexity_analyzer.determine_target_ratio(
complexity,
min_ratio=min_retention_ratio,
max_ratio=max_retention_ratio
)
pruned_tokens = []
num_kept_list = []
for b in range (batch_size):
num_keep = max (1 , int (num_tokens * target_ratio[b].item()))
num_kept_list.append(num_keep)
topk_scores, topk_indices = torch.topk(
importance[b],
k=num_keep,
sorted =True
)
selected = image_tokens[b][topk_indices]
pruned_tokens.append(selected)
max_kept = max (num_kept_list)
padded = torch.zeros(
batch_size, max_kept, self .hidden_size,
device=image_tokens.device,
dtype=image_tokens.dtype
)
for b, tokens in enumerate (pruned_tokens):
padded[b, :tokens.shape[0 ]] = tokens
stats = {
'original_tokens' : num_tokens,
'retained_tokens' : max_kept,
'compression_ratio' : max_kept / num_tokens,
'pruning_ratio' : 1 - (max_kept / num_tokens),
'complexity' : complexity.mean().item(),
'target_ratio' : target_ratio.mean().item(),
'importance_scores' : importance
}
return padded, stats
This is the main pruning module combining all components.
Step 5: Enable superior fine-tuning
Leverage pruned tokens for improved learning:
class PruneAwareFineTuner :
"""Fine-tuning that benefits from cleaner pruned tokens"""
def __init__ (self, model, pruner: GlimpsePrune ):
self .model = model
self .pruner = pruner
def finetune_step (self, images: torch.Tensor,
questions: torch.Tensor,
answers: torch.Tensor ) -> float :
"""
Fine-tuning step using dynamically pruned tokens.
Args:
images: Batch of images
questions: Questions about images
answers: Ground truth answers
Returns:
Loss value
"""
image_tokens = self .model.vision_encoder(images)
query_embedding = self .model.text_encoder(questions)
pruned_tokens, stats = self .pruner(
image_tokens,
query_embedding=query_embedding,
min_retention_ratio=0.1 ,
max_retention_ratio=0.9
)
logits = self .model(pruned_tokens, questions)
loss_fn = torch.nn.CrossEntropyLoss()
loss = loss_fn(logits, answers)
loss.backward()
return loss.item()
def full_finetune (self, dataloader, num_epochs: int = 3 ,
learning_rate: float = 1e-5 ):
"""Full fine-tuning with pruned tokens"""
optimizer = torch.optim.AdamW(
self .model.parameters(),
lr=learning_rate
)
for epoch in range (num_epochs):
total_loss = 0.0
for batch in dataloader:
images = batch['images' ]
questions = batch['questions' ]
answers = batch['answers' ]
loss = self .finetune_step(images, questions, answers)
optimizer.step()
optimizer.zero_grad()
total_loss += loss
avg_loss = total_loss / len (dataloader)
print (f"Epoch {epoch} : {avg_loss:.4 f} " )
This enables fine-tuning that benefits from token pruning.
Practical Guidance When to use GlimpsePrune:
Deploying VLMs with computational constraints
Batch inference where efficiency matters
Fine-tuning VLMs where token quality improves learning
Variable image complexity scenarios
When model quality preservation is critical
When NOT to use GlimpsePrune:
Systems with abundant computational resources
Specialized visual tasks requiring all details (e.g., small object detection)
Real-time systems where pruning overhead matters
Already-optimized models without efficiency bottleneck
min_retention_ratio: 0.05-0.15 typical for aggressive pruningmax_retention_ratio: 0.5-0.9 typical for preserving complex scenesComplexity method: entropy generally best, variance fastest
Query awareness weight: 0.3-0.5 balances content vs query
Token reduction: 85-92% typical
Inference speedup: ~2-3x throughput improvement
Quality preservation: >95% of baseline on most tasks
Fine-tuning improvement: 5-15% performance gain observed
Recommended configurations:
Simple images: 10% retention ratio
Complex scenes: 40-50% retention
Document images: 30% retention
Portrait/simple objects: 15% retention
Reference A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models. arXiv:2508.01548
