rocket-model-compression

name	rocket-model-compression
title	ROCKET: Rapid Optimization via Calibration-guided Knapsack for Model Compression
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.11008
keywords	["Model Compression","Pruning","Quantization","Knapsack Optimization","Layer-wise Allocation"]
description	Compress language models by formulating heterogeneous layer-wise allocation as a constrained knapsack problem. Measure per-layer compression error, solve via dynamic programming to minimize total error within target model size, and avoid pathological solutions where some layers are severely damaged.

ROCKET: Rapid Optimization via Calibration-guided Knapsack for Model Compression

Problem Context

Uniform compression (same quantization/sparsity across all layers) is suboptimal—attention layers are more sensitive than MLPs. Manual layer-wise allocation requires expertise. ROCKET automates this by profiling each layer's reconstruction error under different compression settings, then solving a constrained knapsack problem to distribute the global compression budget optimally.

Core Concept

ROCKET operates in two phases: (1) profile layers with multiple compression configurations (rank, sparsity pairs), measuring error per configuration, (2) solve knapsack via dynamic programming to minimize total reconstruction error while staying within target model size. The approach handles per-layer error caps to prevent pathological solutions.

Implementation

Step 1: Layer-wise compression profiling

import torch
from typing import Dict, List, Tuple

class LayerCompressionProfiler:
    """Profile layers under different compression settings."""

    def __init__(self, model):
        self.model = model

    def profile_layer_compression(
        self,
        layer_name: str,
        layer: torch.nn.Module,
        calibration_data: torch.Tensor,
        compression_configs: List[Dict]
    ) -> List[Dict]:
        """
        Test multiple compression configs on layer.

        Args:
            compression_configs: List of {rank, sparsity} pairs

        Returns:
            results: [{'config': {rank, sparsity}, 'error': float, 'params': int}, ...]
        """
        results = []
        original_output = layer(calibration_data)

        for config in compression_configs:
            rank = config.get('rank', layer.out_features)
            sparsity = config.get('sparsity', )

            
            compressed_layer = ._compress_layer(layer, rank, sparsity)

            
             torch.no_grad():
                compressed_output = compressed_layer(calibration_data)

            error = torch.norm(original_output - compressed_output).item()

            
            param_count = (p.numel()  p  compressed_layer.parameters())

            results.append({
                : config,
                : error,
                : param_count,
                : layer.out_features / (, rank)
            })

         results

     () -> torch.nn.Module:
        
        
         (layer, ):
            W = layer.weight.data
            U, S, V = torch.linalg.svd(W, full_matrices=)
            S[rank:] =   
            W_compressed = U @ torch.diag(S) @ V

            
            threshold = torch.quantile(torch.(W_compressed), sparsity)
            W_compressed[torch.(W_compressed) < threshold] = 

            layer.weight.data = W_compressed

         layer

     () -> [, []]:
        
        layer_profiles = {}

         name, module  .model.named_modules():
             (module, (torch.nn.Linear, torch.nn.Conv2d)):
                profile = .profile_layer_compression(
                    name, module, calibration_data, compression_configs
                )
                layer_profiles[name] = profile

         layer_profiles