Attention Is Not What You Need
Paper · https://arxiv.org/abs/2512.19428
First seen by Oriel on .
AI summary
The paper proposes Causal Grassmann, an attention-free sequence architecture that propagates information through low-rank subspaces, with language-model results near size-matched Transformers and competitive SNLI accuracy.
- Causal Grassmann reduces token states, encodes local token pairs as Grassmann-manifold subspaces using Plucker coordinates, then fuses them through gated mixing.
- On Wikitext-2, models with 13 to 18 million parameters achieve validation perplexities about 10 to 15 percent worse than size-matched Transformers.
- On SNLI, a Grassmann-Plucker head on DistilBERT reaches 0.8550 validation and 0.8538 test accuracy, slightly above the Transformer head.
- The authors report linear sequence-length scaling for fixed rank and frame manifold-based computation as a more structured route to geometric interpretations of reasoning.
#sequence modeling #grassmann manifolds #attention-free neural networks #natural language inference
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 190, latest observation ; +180 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 71, latest observation ; +69 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50237224 | https://arxiv.org/abs/2512.19428 Read discussion |