Oriel — AI & technology worth noticing, every day.

Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch

Web page · https://aws.amazon.com/blogs/machine-learning/beyond-the-kv-cache-training-and-serving-i14s-laplace-3b-linear-attention-model-on-aws-trainium2-with-native-pytorch

Learn how i14 trained and served Laplace 3B, a linear attention model, on AWS Trainium2 using native PyTorch. Its recurrent state stays fixed at 29.5 MB per sequence regardless of context length, avoiding the growing key-value cache of transformer models and keeping serving memory predictable.

First seen by Oriel on .

AI summary

i14 and AWS adapted Laplace 3B’s linear-attention architecture for AWS Trainium2, reporting about 120,000 training tokens per second and 11,544 inference tokens per second on one trn2.48xlarge.

  • Laplace 3B replaces softmax attention with Gated Linear Attention, keeping a 29.5 MB recurrent state per sequence regardless of context length and changing training attention from O(T2) to O(T).
  • The Trainium2 implementation uses native PyTorch operators, 16 chunks of 128 tokens, and statically unrolls the sequential backward pass to meet compiler and on-chip memory limits.
  • MuonFSDP2 uses two all_to_all collectives to reduce per-rank communication from 10.87 GB to 0.45 GB; measured throughput increased from 73,378 to 119,934 tokens per second, with a bit-identical loss curve.
  • Training on 64 logical NeuronCores reached roughly 120,000 tokens per second through step 84,210, with loss decreasing from about 11.4 to 2.3729.
  • Using an unmodified vLLM engine, inference reached 11,544 output tokens per second across 1,280 concurrent sequences; outputs matched the i14 reference token for token on the reported tests.

#linear attention #aws trainium #pytorch #llm inference

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
AWS Machine Learning Blog733fa955e4db373fd2ec88938f4d9a6aa8ad0a49https://aws.amazon.com/blogs/machine-learning/beyond-the-kv-cache-training-and-serving-i14s-laplace-3b-linear-attention-model-on-aws-trainium2-with-native-pytorch/ Read discussion