Implement CASTLE, a causal attention mechanism that dynamically updates key representations as context expands. Reduces validation loss by 0.006-0.037 across model scales while maintaining O(L²d) training complexity and O(td) decoding speed. Deploy for improved language model perplexity without inference overhead.
Implement CASTLE, a causal attention mechanism that dynamically updates key representations as context expands. Reduces validation loss by 0.006-0.037 across model scales while maintaining O(L²d) training complexity and O(td) decoding speed. Deploy for improved language model perplexity without inference overhead.
Outcome: Improve Language Model Perplexity with Dynamic Causal Attention
CASTLE (Causal Attention with Lookahead Keys) improves autoregressive language model perplexity by enabling tokens to incorporate information from future context during training, while maintaining causal integrity and efficient inference. Typical validation loss improvements range from 0.0059 to 0.0369 across model scales (160M to 1.3B parameters).
Problem Context
Standard causal attention enforces strict causality: each token only attends to tokens at earlier positions. This constraint is necessary for valid autoregressive generation but fundamentally limits information flow during training. A token's representation remains static regardless of what appears later in the sequence, even though lookahead information is available during batch training.
Traditional approaches to incorporate future context either:
Violate causality by allowing attending to future positions (invalid for generation)
Use bidirectional encoders (incompatible with autoregressive decoding)
CASTLE resolves this through hybrid key design: partition attention into causal keys (static, causally valid) and lookahead keys (dynamically updated to incorporate future tokens). This preserves autoregressive guarantees while improving training efficiency.
Core Concept
CASTLE modifies the attention computation to split keys into two categories operating in parallel:
Causal Keys (Static): Standard key vectors from positions 0 to i, allowing position i to attend backward. These remain fixed throughout generation and training.
Lookahead Keys (Dynamic): Evolved key representations that aggregate information from positions i+1 through current generation step t. During training with full context, these synthesize future information. During inference, lookahead keys update incrementally as each token generates.
The attention output combines both pathways using a gated mechanism:
This design maintains causality because lookahead keys only incorporate information up to the current generation step—no future information escapes during decoding.
Architecture Overview
Dual-Path Attention Flow
Causal path: Queries attend to historical causal keys via standard scaled dot-product attention
Lookahead path: Queries attend to evolved lookahead keys updated by sigmoid-gated aggregation
Gating mechanism: Lookahead scores modulate causal attention via SiLU activation, preserving gradient flow
Key Evolution During Training
Lookahead keys initialized at sequence position i remain zero for positions earlier than i
As training progresses through position i to t, lookahead keys accumulate attention-weighted values from positions i+1 to t
Mask matrices enforce causality: position i only attends to positions j where i < j in lookahead computation
UQ-KV Cache for Inference
Unified query cache (U) holds lookahead queries for recursive updates
Causal keys (K_C) and values (V_C) stored conventionally
Updated lookahead keys (U_t) cached as rank-1 updates for O(td) per-token complexity
Cache composition: [U_t, Q_U, K_C, V_C] replaces standard KV cache with minimal overhead
Efficiency Mechanism
Naive lookahead materialization: O(L³d) complexity (infeasible for long sequences)
Mathematical equivalence: Reformulate as masked low-rank operations
Parallel training: Avoid step-by-step lookahead key updates; compute efficiently in vectorized form
Training complexity reduced to O(L²d), matching standard attention
Implementation
Step 1: Define Lookahead Key Update Function
Lookahead keys evolve through sigmoid-gated aggregation. At each sequence position, new tokens contribute their values to preceding positions' lookahead representations.
Python
import torch
import torch.nn.functional as F
defcompute_lookahead_keys(
queries_u: torch.Tensor,
values_u: torch.Tensor,
mask_matrix: torch.Tensor,
d: int) -> torch.Tensor:
"""
Compute lookahead key updates via attention-weighted aggregation.
Args:
queries_u: (batch, seq_len, d) lookahead queries
values_u: (batch, seq_len, d) lookahead values (typically same as input)
mask_matrix: (seq_len, seq_len) causal mask with 1 where i < j
d: embedding dimension for scaling
Returns:
updated_keys: (batch, seq_len, d) evolved lookahead keys
"""
batch_size, seq_len, dim = queries_u.shape
# Compute attention scores: Q @ K^T / sqrt(d)
scores = torch.matmul(queries_u, queries_u.transpose(-2, -1)) / (d ** 0.5)
# Apply causal mask (enforce i < j)
scores = scores.masked_fill(mask_matrix == 0, float('-inf'))
# Attention weights with sigmoid gating
attention_weights = F.sigmoid(scores)
# Aggregate values: apply attention to values
updated_keys = torch.matmul(attention_weights, values_u)
return updated_keys
Step 2: Implement Causal Attention with Lookahead Integration
Combine causal and lookahead pathways into unified attention computation.
Debugging complexity: dual-path attention harder to reason about than standard mechanisms
Known Pitfalls
Lookahead mask causality errors: Ensure lookahead mask strictly enforces i < j, never i <= j. Permitting self-attention in lookahead violates autoregressive guarantees.
Gate saturation: If SiLU(lookahead_scores) approaches 1.0 for all positions, lookahead dominates and causal structure erodes. Monitor gate statistics during training.
Initialization mismatch: Lookahead queries initialized identically to regular queries may produce identical scores initially. Use slightly different initialization (e.g., scaled by 0.9) for diversity.
Cache invalidation during generation: UQ-KV updates assume strictly sequential token generation. Batched decoding with variable-length sequences requires masking cache positions.
Downstream task mismatch: Perplexity improvements don't always transfer to downstream tasks if they have distribution shift. Validate on target benchmarks.
Attention head coordination: In multi-head attention, different heads may learn to specialize in causal vs. lookahead paths. Ensure regularization doesn't suppress this beneficial diversity.
Technical Details and Validation
The paper demonstrates CASTLE's effectiveness across four model scales trained on 50 billion tokens of FineWeb-Edu data:
