Oriel — AI & technology worth noticing, every day.

Attention to Mamba: A Recipe for Cross-Architecture Distillation

Web page · https://machinelearning.apple.com/research/attention-mamba-distillation

First seen by Oriel on .

AI summary

Apple researchers present a two-stage recipe for distilling pretrained Attention-based Transformers into pure Mamba models while retaining most of the teacher’s performance.

  • The first stage distills a Transformer into linearized Attention using an adaptation of the kernel trick.
  • The second stage distills the linearized model into an adapted Mamba architecture containing no Attention blocks.
  • The distilled Mamba reaches 14.01 perplexity, close to Pythia-1B’s 13.86, but shows gaps on downstream tasks requiring in-context recall.
  • Ablations cover 1B-parameter models trained on 10B tokens, sequence mixer architectures, model and token scaling, and token allocation between stages.

#mamba #transformers #knowledge distillation #state space models

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 74, latest observation ; +69 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 5, latest observation ; +5 over observed period since

Aliases

SourceExternal IDURL
Hacker News50453171https://machinelearning.apple.com/research/attention-mamba-distillation Read discussion