Raschka explains RoPE and positional embeddings in GPT-2 to MiniMax-M3

Raschka notes that after years of reader requests for a tutorial on relative positional embeddings, he revisited an old draft to produce this walkthrough, which also serves as a foundation for future articles.

The lineage runs from GPT-2's learned absolute position table, which is capped at 1,024 positions and becomes uneconomical to extend to a 100k context, to the fixed sine-cosine encodings of the original 2017 transformer, and then to RoPE from the 2021 RoFormer paper. Raschka highlights that the original transformer's sinusoidal design was motivated by a property where a fixed offset corresponds to a linear rotation, which he suggests may make it easier for attention to learn relative relationships.

MiniMax-M3 applies RoPE inside attention with no separate position vector added to the token embedding, and a partial RoPE variant is to be covered later. The reference to a gallery of 2026 open-weight architectures, grouped by positional encoding, indicates the piece is intended as a map of current practice as well as a tutorial.