| name | language-models-need-sleep |
| description | Sleep paradigm for LLMs that enables continual learning through memory consolidation and dreaming phases. Use when: (1) implementing continual learning for LLMs; (2) designing memory consolidation mechanisms; (3) creating autonomous self-improvement systems; (4) addressing catastrophic forgetting in sequential tasks; (5) developing RL-based curriculum generation for synthetic data. Trigger words: sleep paradigm, memory consolidation, dreaming process, knowledge seeding, LLM sleep. |
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Overview
This methodology introduces a "Sleep" paradigm for Large Language Models (LLMs) that enables continual learning by distilling short-term fragile memories into stable long-term knowledge through replay, and recursively improving themselves with a "Dreaming" process.
Core Components
1. Memory Consolidation (Knowledge Seeding)
- Upward distillation process: Memories of a smaller-self are distilled into a larger network to provide more capacity while preserving knowledge
- Generalized Distillation: Combines on-policy distillation with Reinforcement Learning (RL)-based imitation learning
2. Dreaming (Self-Improvement Phase)
- RL-based curriculum generation: Model uses RL to generate synthetic data to rehearse new knowledge and refine existing capabilities without human supervision
- Autonomous self-improvement: No external supervision required during the dreaming phase
Applications
- Long-horizon tasks
- Continual learning scenarios
- Knowledge incorporation
- Few-shot generalization tasks
Implementation Guidelines
- Two-stage sleep process: Implement both memory consolidation and dreaming phases sequentially
- Knowledge seeding: Use generalized distillation combining on-policy distillation with RL-based imitation learning
- Dreaming curriculum: Generate synthetic data through RL that challenges the model's current capabilities
- Evaluation metrics: Test on long-horizon, continual learning, knowledge incorporation, and few-shot generalization tasks
Key Insights
- Existing LLMs lack the ability to continually learn and effectively transfer temporal in-context knowledge to long-term parameters
- The sleep paradigm addresses this gap by providing mechanisms for both consolidation and self-improvement
- Experiments demonstrate the importance of both stages working together for optimal performance
References
