Positional Encodings (incl. RoPE)
How does a Transformer know word order, and why RoPE?
The situation
"The biryani was cold but the naan was great" and "the naan was cold but the biryani was great" use the same words, with opposite meanings for each dish. Plain self-attention can't tell them apart. The interviewer asks:
How does a Transformer know word order, and why RoPE?
Running example: "the biryani was cold but the naan was great" vs "the naan was cold but the biryani was great": same words, opposite meaning for each dish. Without position, plain self-attention can't tell them apart.
Where this lesson sits: positions enter right after the token embeddings (RoPE applies them inside every attention layer).
Full walkthrough: The Transformer, End to End.
Attention is order-blind
Scores q·k depend only on the two tokens' contents. Permute the input and the outputs are permuted the same way: self-attention is permutation-equivariant. For a bag of words that's fine; for language, "dog bites man" ≠ "man bites dog". RNNs got order for free by reading in sequence; Transformers have to be told.
The idea
Think of giving each word a clock hand, turned according to its position. Any two words a fixed distance apart have the same angle between their hands, wherever they sit. That's RoPE: attention sees how far apart words are, not just where they are.
Play
Try to break it. Move the query and key positions together, keeping the gap between them the same. Then change the gap.
RoPE: rotate queries and keys by position
Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding", 2021 arXiv:2104.09864: RoPE encodes absolute position with a rotation matrix and makes the attention score depend on relative position. Used in LLaMA (Touvron et al. 2023), PaLM, Mistral, Qwen and most open LLMs.
A fixed 8-dimensional query and key (4 rotating pairs, θi = 10000−2i/8). The circle shows pair 1 of the rotated query (teal) and key (orange); the score uses all 4 pairs.
Shift both positions by 40 and the score is identical: only m − n matters. Change the gap instead (move n) and the score changes. That's why RoPE models generalise across positions within the trained range: "the adjective two words before the noun" looks the same at the start or the end of a document.
Now say why: why does the score depend on the gap, and not on the absolute positions?
Maths
In one breath
Attention compares every token with every other but has no idea of order: shuffle the words and each token gets the same weights. So position must be injected. The original Transformer added fixed sinusoidal vectors to the embeddings; BERT and GPT-2 learned one vector per position. Most modern LLMs use RoPE: rotate each pair of query and key dimensions by an angle proportional to the token's position. Then the attention score depends only on the relative distance between two tokens, which is what language mostly cares about. Extending a model to longer contexts usually means adjusting these position schemes.What you'll be able to do after this: explain why position is needed, compare absolute, learned, relative, RoPE and ALiBi, and show why RoPE gives relative positions.Sinusoidal positional encoding
Vaswani et al. (2017), §3.5: chosen because PE(pos + k) is a linear function (a rotation) of PE(pos), which may let the model attend by relative position; learned embeddings performed nearly identically in their tests.
Top: the encoding for positions 0–63 (rows) × 64 dimensions (columns), orange = +1, blue = −1. Bottom: cosine similarity between PE(p) and PE(p + k) for k = 0…100. d = 64.
Neighbouring positions get similar vectors and similarity falls with distance. Move p from 20 to 150: the similarity values for +1, +10 and +50 stay the same (0.97, 0.66 and 0.49 for d = 64), because sinusoidal similarity depends only on the offset, not on where you are in the sentence.
In practice
ALiBi and context extension
- ALiBi (Press et al., ICLR 2022): no position vectors at all; subtract a penalty proportional to distance from each attention score. Trained on short sequences, it extrapolated to longer ones at test time.
- Position interpolation (Chen et al. 2023): to run a RoPE model on 4× longer inputs, squeeze the positions (divide them by 4) so the angles stay in the trained range, then fine-tune briefly; later methods (NTK-aware scaling, YaRN) rescale the frequencies instead.
- Long-context models combine these with attention optimisations; position handling is usually the first thing to change when extending context.
The answer that sounds right
Learned absolute positions don't extrapolate
BERT and GPT-2 learn one vector per position (512 and 1,024 positions). Simple and effective, but a model has never seen position 1,025 and has no vector for it, and absolute position is often less useful than "how far apart are these two words?". Relative schemes (Shaw et al. 2018; T5's relative biases) add a learned term to each attention score that depends on the distance i − j.
Say it
How does a Transformer know word order, and why RoPE?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
Why is RoPE applied to Q and K but not V?
Answer it out loud, then check.
Position should affect who attends to whom (the scores), not the content that is passed along.
Follow-up 2 of 3
Added vs multiplied position information?
Answer it out loud, then check.
Sinusoidal and learned encodings are added once to the input; RoPE multiplies (rotates) inside every attention layer, so relative position is present at every layer.
Follow-up 3 of 3
Why multiple frequencies?
Answer it out loud, then check.
Fast-rotating pairs distinguish nearby positions; slow ones cover long ranges without repeating, like the hands of a clock.
The 30-second answer
Self-attention is permutation-equivariant, so without position information a Transformer can't tell word order. The original paper added fixed sinusoidal encodings at different frequencies to the embeddings; BERT and GPT-2 learned absolute position embeddings, which can't go beyond the trained length. Most modern LLMs use RoPE: rotate each 2-D pair of query and key dimensions by an angle proportional to position, so the dot product depends only on the relative offset. ALiBi instead adds a distance penalty to the scores. To extend context, people interpolate or rescale RoPE frequencies and fine-tune.
Go further
What not to say
- "Transformers know order from the input sequence." Without encodings they don't.
- "RoPE lets any model handle unlimited length." It still degrades beyond the trained range unless rescaled.
Key terms and sources
Why attention needs position information, sinusoidal encodings, learned absolute positions, relative schemes, rotary position embeddings (RoPE) used in most modern LLMs, ALiBi, and stretching the context window · built from Vaswani et al. (2017), Devlin et al. (2019), Shaw et al. (2018), Su et al. (RoFormer, 2021), Press et al. (ALiBi, ICLR 2022), Chen et al. (2023) and Touvron et al. (2023) · checked 29 Oct 2026
Sources
- A. Vaswani et al., "Attention Is All You Need", NeurIPS 2017, arXiv:1706.03762.
- P. Shaw, J. Uszkoreit & A. Vaswani, "Self-Attention with Relative Position Representations", NAACL 2018, arXiv:1803.02155.
- J. Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding", arXiv:2104.09864, 2021.
- O. Press, N. Smith & M. Lewis, "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ALiBi), ICLR 2022, arXiv:2108.12409.
- S. Chen, S. Wong, L. Chen & Y. Tian, "Extending Context Window of Large Language Models via Positional Interpolation", arXiv:2306.15595, 2023.
- H. Touvron et al., "LLaMA: Open and Efficient Foundation Language Models", arXiv:2302.13971, 2023.