Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch
Learn how i14 trained and served Laplace 3B, a linear attention model, on AWS Trainium2 using native PyTorch. Its recurrent state stays fixed at 29.5 MB per sequence regardless of context length, avoiding the growing key-value cache of transformer models and keeping serving memory predictable.
First seen by Oriel on .
AI summary
i14 and AWS adapted Laplace 3B’s linear-attention architecture for AWS Trainium2, reporting about 120,000 training tokens per second and 11,544 inference tokens per second on one trn2.48xlarge.
- Laplace 3B replaces softmax attention with Gated Linear Attention, keeping a 29.5 MB recurrent state per sequence regardless of context length and changing training attention from O(T2) to O(T).
- The Trainium2 implementation uses native PyTorch operators, 16 chunks of 128 tokens, and statically unrolls the sequential backward pass to meet compiler and on-chip memory limits.
- MuonFSDP2 uses two all_to_all collectives to reduce per-rank communication from 10.87 GB to 0.45 GB; measured throughput increased from 73,378 to 119,934 tokens per second, with a bit-identical loss curve.
- Training on 64 logical NeuronCores reached roughly 120,000 tokens per second through step 84,210, with loss decreasing from about 11.4 to 2.3729.
- Using an unmodified vLLM engine, inference reached 11,544 output tokens per second across 1,280 concurrent sequences; outputs matched the i14 reference token for token on the reported tests.
#linear attention #aws trainium #pytorch #llm inference
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| AWS Machine Learning Blog | 733fa955e4db373fd2ec88938f4d9a6aa8ad0a49 | https://aws.amazon.com/blogs/machine-learning/beyond-the-kv-cache-training-and-serving-i14s-laplace-3b-linear-attention-model-on-aws-trainium2-with-native-pytorch/ Read discussion |