Oriel — AI & technology worth noticing, every day.

Understanding Positional Embeddings in Modern LLMs

Web page · https://magazine.sebastianraschka.com/p/understanding-positional-embeddings

First seen by Oriel on .

AI summary

A technical tutorial explains why LLMs need positional information and compares learned absolute and fixed sinusoidal embeddings as groundwork for RoPE and modern architecture choices.

  • Causal masks provide implicit ordering by limiting attention to earlier tokens, but they do not directly distinguish distances such as one position versus ten.
  • GPT-2 adds learned absolute position vectors to token embeddings before the first transformer block, while MiniMax-M3 applies RoPE within attention in every block.
  • GPT-2 has 1,024 learned position vectors; extending GPT-2 Medium to around 100k positions would increase positional weights from 0.30% to over 22% of parameters.
  • The original transformer uses fixed sine and cosine functions whose different frequencies encode positions without adding trainable parameters.
  • Sinusoidal encodings can be evaluated beyond trained positions, though performance is not guaranteed, and equal positional offsets produce the same rotation for each feature pair.

#positional embeddings #rope #transformers #llm architecture

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 241, latest observation ; +215 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 27, latest observation ; +27 over observed period since

Aliases

SourceExternal IDURL
Hacker News50506376https://magazine.sebastianraschka.com/p/understanding-positional-embeddings Read discussion