| name | attention-sink-structural |
| description | Mechanistic explanation of attention sink phenomenon. Variance discrepancy amplified by FFN super neurons. Head-wise RMSNorm fixes it. Based on arXiv 2605.06611. |
| category | transformer-attention |
Attention Sink: Structural Origin and Control
Overview
Attention sinks (initial tokens monopolizing attention scores) emerge from a causal chain: value aggregation → variance discrepancy → FFN super neuron amplification → dimension disparity → sink formation.
Causal Chain
- Value aggregation in self-attention induces systematic variance discrepancy
- FFN super neurons (channel-sparse down-projections) amplify this discrepancy
- Dimension disparity of first-token representation emerges
- Attention sinks form as structural anchors
Key Interventions
- Isolate aggregation: Attention mask modifications replicate sinks at arbitrary positions
- Amplify variance: Targeted variance amplification creates sinks on demand
- Head-wise RMSNorm: Normalizes attention head outputs independently, restores statistical parity, accelerates convergence
Key Findings
- Sinks are structural, not accidental
- Channel-sparse FFN down-projections are the amplification mechanism
- Can be created at arbitrary positions through controlled interventions
- Head-wise RMSNorm significantly accelerates pre-training
Implementation Steps
- Monitor per-token variance in value aggregation outputs
- Identify FFN super neurons (channel-sparse high-activation units)
- Apply head-wise RMSNorm to normalize attention head outputs
- Verify statistical parity across token positions
Applicable Use Cases
- Pre-training new LLM architectures
- Debugging attention sink-related instability
- Improving convergence in long-context models
Triggers / Keywords
attention sink, variance discrepancy, super neurons, head-wise RMSNorm, transformer debugging, pre-training stability
