dope-denoising-rotary-embeddings

name	dope-denoising-rotary-embeddings
title	DoPE: Denoising Rotary Position Embedding
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.09146
keywords	["Position Embeddings","RoPE","Long-context","Length Extrapolation","Attention Stability"]
description	Improve long-context length extrapolation by denoising instabilities in Rotary Position Embeddings (RoPE) through spectral analysis and selective head rewriting—training-free post-hoc intervention for longer context windows.

Stabilize Long-Context Reasoning by Denoising Rotary Position Embeddings

RoPE (Rotary Position Embedding) enables models to extrapolate beyond training context, but its low-frequency components concentrate energy in narrow angular cones, creating over-aligned attention patterns that destabilize performance on long sequences. DoPE (Denoising Rotary Position Embedding) identifies and corrects these problematic attention heads using spectral analysis, improving extrapolation without fine-tuning.

The core insight is that certain heads amplify positional noise—they suffer from attention sinks where energy concentrates on specific tokens. By detecting these heads via entropy and reparameterizing their attention maps with isotropic noise, DoPE stabilizes extrapolation and improves needle-in-haystack and in-context learning tasks.

Core Concept

RoPE uses low-frequency sinusoidal encodings to maintain rotational equivariance across position shifts. However, these low frequencies create pathological activation patterns: they concentrate spectral energy, producing massive attention scores that collapse into sinks (single tokens grabbing all attention) rather than distributing appropriately.

DoPE solves this by:

1. Spectral Detection: Identify heads where RoPE induces concentrated energy using truncated matrix entropy on query/key representations
2. Head Selection: Sort by entropy and choose heads to denoise (typically 1-32 per layer)
3. Reparameterization: Replace problematic RoPE with isotropic Gaussian noise in selected heads, breaking the alignment pathology

This achieves long-context extrapolation gains while keeping the model fully frozen—denoising happens at inference without model retraining.

Architecture Overview

• Spectral Analysis Layer: Compute truncated matrix entropy ℋ = 1/r Σᵢ₌₁ʳ λᵢ log λᵢ on RoPE-transformed representations to quantify concentration
• Entropy Thresholding: Select heads with entropy below a calibrated threshold (indicating spectral amplification)
• Three Denoising Strategies:
  • DoPE-by-parts: Suppress low-frequency RoPE bands
  • DoPE-by-all: Completely mask RoPE
  • DoPE-by-Gaussian: Replace with isotropic Gaussian noise
• Inference-time Application: Apply denoising only at inference; no training needed

Implementation Steps

Step 1: Entropy Computation. Compute singular values of query/key representations and calculate truncated entropy to identify problematic heads.

import numpy  np

 ():
    
    U, S, Vt = np.linalg.svd(representations, full_matrices=)
    S_sorted = np.sort(S)[::-]  
    S_trunc = S_sorted[:(r, (S_sorted))]
    S_norm = S_trunc / np.(S_trunc)  
    entropy = -np.(S_norm * np.log(S_norm + ))
     entropy