Sleep framework: A new way for LLMs to retain knowledge in continual learning
Education Innovation
First seen by Oriel on .
AI summary
Google’s sleep framework alternates active interaction with offline memory consolidation and dreaming, enabling LLMs to retain new knowledge, handle contexts up to 10M tokens, and improve reasoning without catastrophic forgetting.
- During sleep, pre-allocated MoE memory blocks unlock and low-rank experts absorb short-term knowledge through GKD, on-policy distillation, RL-based imitation learning, and subsequent pruning.
- Dreaming randomly activates non-standard experts, generates synthetic samples, filters them using gradient-based selection, and fine-tunes on highly rated examples to improve downstream performance.
- The framework showed near-zero performance loss when sequentially learning unseen languages and near-perfect scaling to 10M tokens on BABILong.
- It scored 48.9% on single-task and 46.2% on CPT factual-knowledge evaluations, outperforming the base model, SFT, and SEAL.
- Sleep-enhanced models reached 79.2%, 69.0%, and 46.1% accuracy on AIME 24, AIME 25, and HMMT 25, respectively.
#continual learning #llm #mixture of experts #knowledge distillation
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 271, latest observation ; +267 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 56, latest observation ; +56 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Google Research Blog | https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/ | https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/ Read discussion |
| Hacker News | 50272479 | https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/ Read discussion |