pop-prefill-only-pruning

name	pop-prefill-only-pruning
title	POP: Prefill-Only Pruning for Efficient Large Model Inference
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.03295
keywords	["Layer Pruning","Inference Optimization","Prefill Stage","Model Efficiency","No Retraining"]
description	Remove deep layers during context encoding (prefill) while keeping them for token generation (decode). Identifies layer importance asymmetry via virtual gates; achieves 1.37x prefill speedup without retraining on any pre-trained model.

POP: Stage-Aware Layer Pruning for Inference

Standard inference removes entire layers, but different layers matter differently across inference stages. During prefill (context encoding), deep layers are largely redundant; during decode (token generation), they're critical. POP exploits this asymmetry by removing layers only during prefill, using virtual gates to identify which layers to prune without requiring model retraining.

The key insight is that layer importance changes dramatically between prefill and decode. This suggests a pragmatic pruning strategy that's stage-aware rather than uniform.

Core Concept

POP operates on the observation that:

1. Prefill stage: Context encoding where tokens attend over full sequence; deep layers contribute minimally
2. Decode stage: Single-token generation where all layers are needed for quality

By identifying prunable layers via a virtual gate mechanism, POP removes them during prefill only, maintaining full model capacity for decode.

Architecture Overview

• Virtual Gate Estimator: Lightweight mechanism to estimate layer importance without retraining
• Prunable Layer Identification: Determines which deep layers can be removed during prefill
• Stage-Aware Execution: Disable prunable layers only in prefill; enable in decode
• KV Projection Handling: Ensure cache dimensions remain consistent
• Boundary Safeguards: Final token always uses full model for stability

Implementation

Step 1: Design Virtual Gate Mechanism

Create an importance estimator using second-order information.

# Virtual gate for layer importance
import torch
import torch.nn.functional as F

class VirtualGate:
    def __init__(self, model: nn.Module, eps: float = 1e-8):
        """
        Estimate layer importance via Fisher information.

        Args:
            model: Language model to analyze
            eps: Numerical stability
        """
        self.model = model
        self.eps = eps

     () -> :
        
        layer = .model.layers[layer_idx]

        
        hidden_states.requires_grad = 
        output = layer(hidden_states)
        loss = loss_fn(output)
        loss.backward()

        gradient = hidden_states.grad

        
        fisher_diag = torch.mean(gradient ** , dim=[, ])

        
        importance = torch.(
            torch.(gradient.mean(dim=[, ])) / (fisher_diag + .eps)
        )

         (, importance.item())

     () -> [[, ]]:
        
        layer_scores = []

         layer_idx  ((.model.layers)):
            scores = []

             hidden_states  val_dataset:
                score = .estimate_layer_importance(
                    hidden_states,
                    layer_idx,
                    loss_fn
                )
                scores.append(score)

            avg_score = (scores) / (scores)
            layer_scores.append((layer_idx, avg_score))

        
        sorted_layers = (layer_scores, key= x: x[])
         sorted_layers