minicpm-sala

name	minicpm-sala
title	MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.11761
keywords	["Attention Mechanisms","Long Context","Hybrid Architecture","Training Efficiency","Token Scaling"]
description	Combine sparse attention (25% of layers) and linear attention (75% of layers) via strategic layer placement to handle 1M-token contexts with 75% training cost reduction. Hybrid positional encoding preserves long-range information while maintaining position awareness.

MiniCPM-SALA: Hybrid Attention for Efficient Long-Context

Problem Context

Standard Transformer attention faces two prohibitive bottlenecks: quadratic computational complexity O(N²) and linear KV-cache memory growth. For million-token contexts, these constraints make full-attention models impractical. Sparse attention alone sacrifices global information. Linear attention lacks fidelity for fine-grained dependencies. Training new models from scratch incurs enormous computational cost.

Core Concept

MiniCPM-SALA achieves efficient long-context modeling through strategic mixing of complementary attention mechanisms in a 1:3 ratio. Rather than using uniform attention throughout, 25% of layers employ sparse attention (InfLLM-V2) for precise long-range modeling while 75% use linear attention (Lightning Attention) for O(N) efficiency.

The key insight: sparse and linear attention have complementary strengths. A learned layer selection mechanism determines optimal sparse placement, enabling continual training to convert pre-trained models instead of training from scratch (75% cost reduction).

Architecture Overview

• Hybrid Mixing Ratio: 25% sparse attention (InfLLM-V2), 75% linear attention (Lightning Attention)
• Layer Selection Algorithm: Determines sparse placement rather than uniform interleaving
• Hybrid Positional Encoding (HyPE): RoPE on linear layers only, removed from sparse layers
• Architectural Stability: First and last layers remain softmax attention
• Five-Stage Training Pipeline: HALO conversion → stable training → short-decay → long-decay → SFT
• QK-Normalization and Output Gates: Stabilize gradients and prevent attention sinks

Implementation

The hybrid attention layer selection mechanism:

def select_layer_types(num_layers, sparse_ratio=0.25):
    """
    Determine which layers get sparse vs linear attention.
    Uses strategic placement rather than uniform distribution.
    """
    num_sparse = int(num_layers * sparse_ratio)

    # Strategic placement: concentrate sparse layers where most useful
    # (typically middle layers for long-range dependencies)
    sparse_positions = []
    for i in range(num_sparse):
        pos = ((i + ) * num_layers / (num_sparse + ))
        sparse_positions.append(pos)

    layer_types = []
     i  (num_layers):
         i  sparse_positions:
            layer_types.append()
        :
            layer_types.append()

     layer_types