psa-pyramid-sparse-attention

name	psa-pyramid-sparse-attention
title	PSA: Pyramid Sparse Attention for Video Understanding and Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.04025
keywords	["sparse-attention","video-understanding","video-generation","hierarchical-pooling","efficient-transformers"]
description	Replaces binary keep/drop masks with multi-level pooled key-value representations, allowing queries to access larger receptive fields under same compute budget through hierarchical aggregation without discarding information.

Summary

Pyramid Sparse Attention (PSA) improves block sparse attention by replacing binary keep/drop decisions with multi-level pooled KV representations. Instead of discarding unimportant blocks entirely, PSA assigns them to progressively coarser pooling levels, maintaining information fidelity while enabling larger receptive fields under computational constraints.

Core Technique

Hierarchical Pooling: Instead of binary keep/drop:

keep_blocks: full attention  [level 0]
skip_blocks: 2× pooled       [level 1]
skip_blocks: 4× pooled       [level 2]
skip_blocks: 8× pooled       [level 3]

Progressive Coarsening: Create multiple downsampled versions of each KV cache:

• Level 0: Original KV (keep blocks)
• Level 1: 2× pooled KV (medium importance)
• Level 2: 4× pooled KV (low importance)
• Level 3: 8× pooled KV (minimal importance)

Query-Aware Routing: Route queries to appropriate pooling levels based on importance scores.

Implementation

Pooled KV cache creation:

def create_pyramid_cache(kv_cache, num_levels=4):
    pyramid = {}
    pyramid[0] = kv_cache  # Full resolution

    for level in range(1, num_levels):
        pool_factor = 2 ** level
        # Average pooling over tokens
        pooled_k = average_pool(kv_cache[0], pool_factor)
        pooled_v = average_pool(kv_cache[1], pool_factor)
        pyramid[level] = (pooled_k, pooled_v)

    return pyramid

Block importance scoring:

def score_blocks(query, kv_cache, block_size=64):
    num_blocks = len(kv_cache) // block_size
    scores = []

    for block_idx in range(num_blocks):
        block_kv = kv_cache[block_idx*block_size:(block_idx+)*block_size]
        
        score = query @ block_kv.mean().T
        scores.append(score)

     scores