Attention Is All You Need
What happens if you remove recurrence and keep only attention?
The paper
The problem
Recurrent models read a sentence one word at a time, which makes them slow to train and weak over long distances. Each step has to wait for the one before it, so training is hard to run in parallel.
The idea
Think of a meeting. In a recurrent model, a message passes from person to person down a line, and it fades on the way. With attention, everyone in the room can hear everyone else directly, and each person decides who to listen to.
Method
- An encoder and a decoder, each a stack of 6 identical layers.
- Multi-head attention with 8 heads and a model size of 512.
- A feed-forward layer applied to each position on its own.
- Sine and cosine position signals, since attention alone ignores word order.
What they found
- On English-to-German translation (WMT 2014), it reached a BLEU score of 28.4, more than 2 points above the best existing results at the time, including ensembles.
- It trained in 12 hours on eight GPUs, far less time than the best earlier models reported.
- It also worked on English parsing, which showed the idea wasn't limited to translation.
Figures from the paper's abstract. Read the abstract.
Where it falls short
- Attention compares every position with every other, so its cost grows with the square of the sequence length.
- Word order has to be added separately, because attention alone ignores it.
- The results cover translation and parsing, so the paper itself doesn't show how far the idea would go.
What it changed
Training could now run in parallel across a sequence, and distant words were one step apart. This architecture became the base of modern LLMs.
What to remember. Attention can replace recurrence entirely, and that's what made today's large models practical to train.
Verdict
Read the architecture section and the attention figures. Skim the training details unless you plan to reproduce it.