era-entropy-token-pruning-mllm

name	era-entropy-token-pruning-mllm
version	1.0.0
description	ERA (Entropy-guided Visual Token Pruning with Rectified Attention) for efficient Multimodal Large Language Models
tags	["multimodal-llm","token-pruning","attention-mechanisms","efficiency","vision-language-models"]
categories	["ai_collection","vision"]
source	arXiv 2606.31982
date_collected	2026-07-02T00:00:00.000Z

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention

Overview

ERA is a training-free visual token pruning framework for Multimodal Large Language Models (MLLMs) that addresses the Attention Logit Collapse phenomenon caused by existing token reduction methods. It combines entropy-guided pruning with attention rectification to preserve visual evidence under aggressive compression.

Problem Statement

MLLMs incur prohibitive inference costs due to long visual token sequences. Training-free token reduction methods exist but have a critical flaw:

• Attention Logit Collapse: Existing methods distort attention distributions, causing logits to collapse and losing discriminative power
• This distortion degrades performance even when the "right" tokens are preserved

Core Methodology

1. Dual-View Entropy Pruning (DEP)

Identifies representative anchor tokens by jointly modeling:

• Visual diversity: Ensures coverage of different visual concepts
• Head-wise saliency: Accounts for attention patterns across different heads

2. Bias-Aware Token Recycling (BTR)

Recycles pruned tokens into their corresponding anchors:

• Estimates cluster-level logit bias
• Preserves information from removed tokens
• Maintains representational capacity despite reduction

3. Logit-Preserving Attention Rectification (LAR)

Injects estimated bias into attention logits:

• Rectifies the collapse induced by token reduction
• Preserves attention distribution semantics
• Establishes logit-preserving pruning as a principled framework

Key Innovations

Attention Logit Collapse Diagnosis

First work to formally characterize why existing token pruning methods fail:

• Token reduction distorts attention logits
• Distortion compounds across layers
• Leads to degraded reasoning despite preserving "important" tokens

Principled Framework

ERA establishes logit-preserving visual token pruning as a unified approach:

• Theoretical foundation: Information-theoretic justification
• Algorithmic design: Three-component pipeline
• Practical deployment: Training-free, works across MLLM architectures

Results

ERA achieves:

• Robust performance across single-image, multi-image, and video settings
• Works with a wide range of MLLMs without fine-tuning
• Practical acceleration while maintaining accuracy
• Establishes new state-of-the-art for training-free token reduction

Implementation Patterns

Three-Component Pipeline

def era_forward(visual_tokens, llm_backbone):
    # 1. Dual-View Entropy Pruning
    anchor_tokens, pruned_tokens = dep(visual_tokens)
    
    # 2. Bias-Aware Token Recycling
    recycled_anchors, logit_bias = btr(anchor_tokens, pruned_tokens)
    
    # 3. Logit-Preserving Attention Rectification
    rectified_attention = lar(recycled_anchors, logit_bias)
    
    # Forward through LLM
    output = llm_backbone(rectified_attention)
    return output

Entropy Computation

def compute_dual_view_entropy(tokens):
    # Visual diversity: feature variance across tokens
    visual_diversity = compute_feature_variance(tokens)
    
    # Head-wise saliency: attention entropy per head
    head_saliency = compute_attention_entropy(tokens)
    
    # Joint score
    importance = visual_diversity * head_saliency
    return importance

Logit Bias Estimation

def estimate_logit_bias(anchor_tokens, pruned_tokens):
    # Cluster pruned tokens by similarity to anchors
    clusters = cluster_by_similarity(pruned_tokens, anchor_tokens)
    
    # Estimate bias as mean contribution of cluster
    bias_per_cluster = [mean(cluster.contributions) for cluster in clusters]
    
    return bias_per_cluster

When to Use

Apply ERA when:

• Deploying MLLMs in latency-sensitive applications
• Need training-free token reduction (no fine-tuning budget)
• Working with long visual sequences (video, multi-image)
• Existing pruning methods show accuracy degradation

Skip ERA when:

• Can afford to fine-tune with reduced tokens (learned pruning may be better)
• Visual sequences are already short (<64 tokens)
• Not using attention-based MLLM architecture

Activation Keywords

token pruning, MLLM, visual tokens, attention collapse, entropy, multimodal, efficiency, training-free, vision-language

Related Patterns

• [[uncertainty-token-pruning-spiking]] - Token pruning for spiking transformers
• [[streaming-attention-space-optimization]] - Streaming attention for efficiency
• [[clsa-cross-layer-sparse-attention]] - Cross-layer sparse attention

References

• Paper: ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
• arXiv: 2606.31982
• Date: 2026-06-30
• Categories: cs.CV
• Code: https://github.com/924973292/ERA