sq-format-sparse-quantized

name	sq-format-sparse-quantized
title	SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.05409
keywords	["quantization","sparsification","post-training quantization","hardware efficiency","LLM inference"]
description	Unify sparse and quantized representations in a single hardware-friendly format for efficient LLM inference. Exploit complementary acceleration properties—high precision for sparse operations, low precision for dense—when W4A8 bottlenecks GPU throughput.

Overview

SQ-format combines quantization and sparsification in a unified framework compatible with both new hardware and existing GPUs. The innovation leverages complementary acceleration properties where sparse matrices accelerate effectively at high precision while low-precision multiplication also benefits from acceleration.

When to Use

• Post-training quantization of large language models
• LLM inference with W4A8 or similar bottlenecking GPU performance
• Scenarios where activation patterns contain outlier inequality
• Requiring static compression with minimal accuracy loss
• Deploying LLMs on resource-constrained hardware
• Applications needing Pareto improvements in accuracy-efficiency tradeoff

When NOT to Use

• Models where sparsity is minimal (dense computations dominate)
• Cases where activation distributions are uniform
• Scenarios already achieving good results with W8A8
• Applications requiring full-precision model inference
• Hardware without sparse operation support

Core Technique

Unified sparse-quantized format exploiting dual acceleration pathways:

# SQ-format implementation overview
class SparseQuantizedFormat:
    def __init__(self, quantization_bits=4, sparse_threshold=0.5):
        self.q_bits = quantization_bits
        self.sparse_threshold = sparse_threshold

    def compress_weights(self, weights):
        """
        Compress weights using combined sparsification and quantization.
        Weights are both sparsified (set small values to zero) and quantized.
        """
        # Identify sparse patterns (large magnitude threshold)
        mask = torch.abs(weights) > self.sparse_threshold
        sparse_weights = weights * mask

        # Quantize sparse weights at high precision (8-bit)
        # Dense regions at low precision (4-bit)
        quantized = torch.zeros_like(weights)

        
        sparse_region = sparse_weights[mask]
        quantized[mask] = .quantize_8bit(sparse_region)

        
        dense_region = weights[~mask]
        quantized[~mask] = .quantize_4bit(dense_region)

         quantized, mask

     ():
        
        
        percentile_90 = torch.quantile(activations, )
        percentile_10 = torch.quantile(activations, )
        inequality_ratio = percentile_90 / (percentile_10 + )

         inequality_ratio > :
            
            threshold = percentile_10 +  * (percentile_90 - percentile_10)
            mask = torch.(activations) > threshold
            compressed = activations * mask
             .quantize_8bit(compressed), mask
        :
            
             .quantize_4bit(activations), 

     ():
        
        
        scale = (values.() - values.()) / 
        zero_point = values.()
        quantized = torch.clamp(
            (values - zero_point) / scale,
            , 
        ).()
         quantized * scale + zero_point

     ():
        
        
        scale = (values.() - values.()) / 
        zero_point = values.()
        quantized = torch.clamp(
            (values - zero_point) / scale,
            , 
        ).()
         quantized * scale + zero_point

     ():
        
        weight_q, weight_mask = .compress_weights(weight)
        act_q, act_mask = .compress_activations(input_tensor)

        
        output = torch.nn.functional.linear(act_q, weight_q, bias)
         output