Oriel — AI & technology worth noticing, every day.

Attention Is Not What You Need

Paper · https://arxiv.org/abs/2512.19428

First seen by Oriel on .

AI summary

The paper proposes Causal Grassmann, an attention-free sequence architecture that propagates information through low-rank subspaces, with language-model results near size-matched Transformers and competitive SNLI accuracy.

  • Causal Grassmann reduces token states, encodes local token pairs as Grassmann-manifold subspaces using Plucker coordinates, then fuses them through gated mixing.
  • On Wikitext-2, models with 13 to 18 million parameters achieve validation perplexities about 10 to 15 percent worse than size-matched Transformers.
  • On SNLI, a Grassmann-Plucker head on DistilBERT reaches 0.8550 validation and 0.8538 test accuracy, slightly above the Transformer head.
  • The authors report linear sequence-length scaling for fixed rank and frame manifold-based computation as a more structured route to geometric interpretations of reasoning.

#sequence modeling #grassmann manifolds #attention-free neural networks #natural language inference

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 190, latest observation ; +180 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 71, latest observation ; +69 over observed period since

Aliases

SourceExternal IDURL
Hacker News50237224https://arxiv.org/abs/2512.19428 Read discussion