sleep-like-consolidation-llm

# Sleep-Like Consolidation for LLMs ## Overview Methodology from arXiv:2605.26099 (May 2026). Transformer-based LLMs suffer from poor context length scaling due to attention mechanism's O(n²) complexity. This paper introduces a **sleep-like consolidation mechanism** that periodically converts recent context into persistent fast weights before clearing the KV cache. ## Core Mechanism ### 1. Wake Phase - Model processes input tokens normally with attention + SSM blocks - Context accumulates in KV cache as usual - No additional computation overhead during inference ### 2. Sleep Phase (Triggered When) - KV cache reaches a predefined threshold (e.g., 80% capacity) - Or periodically after N tokens processed - Model performs **N offline recurrent passes** over accumulated context - Updates **fast weights** in state-space model (SSM) blocks via learned local rule - KV cache is cleared after consolidation ### 3. Inference After Sleep - Fast weights carry consolidated knowledge from past context - Wake-time prediction latency is preserved - Model can handle effectively unlimited context by cycling sleep/wake ## Key Findings - **Increasing sleep duration N improves performance** - Largest gains on examples requiring **deeper reasoning** - Regular transformers and SSM-attention hybrids **fail** on multi-hop graph retrieval and math reasoning tasks that this method solves - Works on controlled synthetic tasks (cellular automata, multi-hop graph retrieval) and realistic math reasoning ## Implementation Patterns ### Pattern 1: Fast Weight Update Rule ``` For each sleep pass k = 1..N: For each position i in context: h_i^{k+1} = h_i^k + η · ∇_fast_weights L(context, h_i^k) Fast weights W_fast ← update(W_fast, {h_i^N}) ``` ### Pattern 2: KV Cache Management ```python if len(kv_cache) > threshold: # Enter sleep phase for _ in range(num_sleep_passes): fast_weights = recurrent_pass(context, fast_weights, local_rule) kv_cache.clear() # Resume inference with consolidated fast weights ``` ### Pattern 3: Hybrid Attention-SSM Architecture - Use attention for immediate context (wake phase) - Use SSM with fast weights for long-term memory (consolidated) - Shift computation to offline sleep to preserve inference latency ## When to Apply - **Trigger words**: context length, KV cache, long-horizon tasks, memory compression, sleep consolidation, fast weights - **Use cases**: Long-context LLM serving, agentic workflows, multi-step reasoning tasks - **Benefits**: O(1) memory growth with context, preserved inference latency, improved deep reasoning ## Pitfalls - Sleep phase requires careful tuning of N (number of passes) — too few loses information, too many wastes compute - Fast weight update rule must be learned during pre-training, not added post-hoc - Works best with SSM-attention hybrid architectures, not pure transformers - Consolidation quality depends on the learned local rule — must be trained on diverse tasks ## Related Concepts - Hebbian learning (biological inspiration for fast weight updates) - Compressive transformers (alternative context compression) - State-space models (SSMs like Mamba, RWKV) - Working memory consolidation in neuroscience