菜单
Improve long-context length extrapolation by denoising instabilities in Rotary Position Embeddings (RoPE) through spectral analysis and selective head rewriting—training-free post-hoc intervention for longer context windows.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill dope-denoising-rotary-embeddings命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
基于 SOC 职业分类
正在显示 SKILL.md
| name | dope-denoising-rotary-embeddings |
| title | DoPE: Denoising Rotary Position Embedding |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2511.09146 |
| keywords | ["Position Embeddings","RoPE","Long-context","Length Extrapolation","Attention Stability"] |
| description | Improve long-context length extrapolation by denoising instabilities in Rotary Position Embeddings (RoPE) through spectral analysis and selective head rewriting—training-free post-hoc intervention for longer context windows. |
RoPE (Rotary Position Embedding) enables models to extrapolate beyond training context, but its low-frequency components concentrate energy in narrow angular cones, creating over-aligned attention patterns that destabilize performance on long sequences. DoPE (Denoising Rotary Position Embedding) identifies and corrects these problematic attention heads using spectral analysis, improving extrapolation without fine-tuning.
The core insight is that certain heads amplify positional noise—they suffer from attention sinks where energy concentrates on specific tokens. By detecting these heads via entropy and reparameterizing their attention maps with isotropic noise, DoPE stabilizes extrapolation and improves needle-in-haystack and in-context learning tasks.
RoPE uses low-frequency sinusoidal encodings to maintain rotational equivariance across position shifts. However, these low frequencies create pathological activation patterns: they concentrate spectral energy, producing massive attention scores that collapse into sinks (single tokens grabbing all attention) rather than distributing appropriately.
DoPE solves this by:
This achieves long-context extrapolation gains while keeping the model fully frozen—denoising happens at inference without model retraining.
Step 1: Entropy Computation. Compute singular values of query/key representations and calculate truncated entropy to identify problematic heads.
import numpy np
():
U, S, Vt = np.linalg.svd(representations, full_matrices=)
S_sorted = np.sort(S)[::-]
S_trunc = S_sorted[:(r, (S_sorted))]
S_norm = S_trunc / np.(S_trunc)
entropy = -np.(S_norm * np.log(S_norm + ))
entropy
Step 2: Head Selection. After profiling on matched-length data, identify heads with low entropy (concentration) and mark them for denoising.
def identify_problematic_heads(model, calibration_data, entropy_threshold=0.5):
"""
Identify heads to denoise by computing entropy across attention heads.
Returns list of (layer_idx, head_idx) tuples for heads with entropy < threshold.
"""
problematic_heads = []
for layer_idx, layer in enumerate(model.layers):
entropies = []
for head_idx in range(layer.num_heads):
# Extract head representation during forward pass
head_reps = get_head_representations(layer, calibration_data, head_idx)
ent = compute_truncated_entropy(head_reps)
entropies.append(ent)
# Select bottom-k heads (lowest entropy = highest concentration)
for head_idx, ent in enumerate(entropies):
if ent < entropy_threshold:
problematic_heads.append((layer_idx, head_idx))
return problematic_heads
Step 3: Gaussian Reparameterization. At inference, replace RoPE in problematic heads with isotropic Gaussian noise.
def apply_dope_denoising(model, problematic_heads, strategy='gaussian'):
"""
Apply denoising strategies to identified heads at inference time.
strategy: 'gaussian' (replace with noise), 'mask-parts' (suppress low freq), 'mask-all' (remove RoPE)
"""
for layer_idx, head_idx in problematic_heads:
layer = model.layers[layer_idx]
if strategy == 'gaussian':
# Replace positional encoding with isotropic Gaussian noise
layer.attention.heads[head_idx].use_rope = False
layer.attention.heads[head_idx].use_gaussian_noise = True
elif strategy == 'mask-parts':
# Suppress low-frequency RoPE bands
layer.attention.heads[head_idx].suppress_low_freq_rope = True
elif strategy == 'mask-all':
# Completely disable RoPE
layer.attention.heads[head_idx].use_rope = False
return model
When to Use: Long-context reasoning tasks (document QA, long-form summarization, many-shot ICL) where models need to reference distant context or integrate information across 10K+ token windows.
Hyperparameters:
Pitfalls:
When NOT to Use: Short-context or fixed-length tasks where extrapolation is not a concern; denoising adds inference overhead without benefit.
Integration: Apply at inference after model loading; pairs well with ALiBi or other position-independent attention variants for complementary improvements.
Reference: https://arxiv.org/abs/2511.09146