Oriel — AI & technology worth noticing, every day.

Sleep framework: A new way for LLMs to retain knowledge in continual learning

Web page · https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning

Education Innovation

First seen by Oriel on .

AI summary

Google’s sleep framework alternates active interaction with offline memory consolidation and dreaming, enabling LLMs to retain new knowledge, handle contexts up to 10M tokens, and improve reasoning without catastrophic forgetting.

  • During sleep, pre-allocated MoE memory blocks unlock and low-rank experts absorb short-term knowledge through GKD, on-policy distillation, RL-based imitation learning, and subsequent pruning.
  • Dreaming randomly activates non-standard experts, generates synthetic samples, filters them using gradient-based selection, and fine-tunes on highly rated examples to improve downstream performance.
  • The framework showed near-zero performance loss when sequentially learning unseen languages and near-perfect scaling to 10M tokens on BABILong.
  • It scored 48.9% on single-task and 46.2% on CPT factual-knowledge evaluations, outperforming the base model, SFT, and SEAL.
  • Sleep-enhanced models reached 79.2%, 69.0%, and 46.1% accuracy on AIME 24, AIME 25, and HMMT 25, respectively.

#continual learning #llm #mixture of experts #knowledge distillation

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 271, latest observation ; +267 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 56, latest observation ; +56 over observed period since

Aliases

SourceExternal IDURL
Google Research Bloghttps://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/ Read discussion
Hacker News50272479https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/ Read discussion