Attention to Mamba: A Recipe for Cross-Architecture Distillation
Web page · https://machinelearning.apple.com/research/attention-mamba-distillation
First seen by Oriel on .
AI summary
Apple researchers present a two-stage recipe for distilling pretrained Attention-based Transformers into pure Mamba models while retaining most of the teacher’s performance.
- The first stage distills a Transformer into linearized Attention using an adaptation of the kernel trick.
- The second stage distills the linearized model into an adapted Mamba architecture containing no Attention blocks.
- The distilled Mamba reaches 14.01 perplexity, close to Pythia-1B’s 13.86, but shows gaps on downstream tasks requiring in-context recall.
- Ablations cover 1B-parameter models trained on 10B tokens, sequence mixer architectures, model and token scaling, and token allocation between stages.
#mamba #transformers #knowledge distillation #state space models
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 74, latest observation ; +69 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 5, latest observation ; +5 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50453171 | https://machinelearning.apple.com/research/attention-mamba-distillation Read discussion |