RC RANDOM CHAOS

Positional Embeddings: Classical to RoPE

· via Hacker News

Original source

Understanding Positional Embeddings in Modern LLMs

Hacker News →

Language models must track token position to maintain meaning—“the red fox jumps over the gray fox” and “the gray fox jumps over the red fox” have different meanings despite identical token sequences. Without explicit position information, the same word appearing twice receives identical embeddings, making sequence order indistinguishable.

Models handle positional information differently. GPT-2 learns a table of position vectors during training, adding them to token embeddings before the first transformer block. Newer models like MiniMax-M3 instead apply positional information within transformer blocks using a method called RoPE. While causal attention masks provide implicit ordering by restricting tokens to attend only to earlier positions, explicit positional methods remain essential because they encode the actual distance between positions—a separation of one differs fundamentally from a separation of ten, information the masking alone cannot convey.

This technical survey covers multiple positional encoding approaches used in 2026 open-weight models, examining both classical methods and contemporary techniques alongside their mathematical foundations and PyTorch implementations.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.