QUIZ
Attention and Transformers
10 questions on Attention and Transformers, from 6 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering, ML Engineering.
Question 1 of 10
In self-attention, what do the queries, keys and values do?
- A. A query is matched to every key to get weights that mix the values
- B. Queries are the labels the model is trying to predict
- C. They're three identical copies of the token embedding
- D. Keys hold the final output, and the values are only ever used for masking
View answer
✓ A. A query is matched to every key to get weights that mix the values
Each token asks (query) which tokens are relevant (keys) and takes a weighted mix of their content (values).
Lesson: Self-Attention (Q, K, V)
Question 2 of 10
Why are attention scores divided by √d before the softmax?
- A. To save memory when the head dimension is very large
- B. To make every attention score positive before it goes into the softmax
- C. To normalise each value vector to unit length first
- D. Dot products grow with d; scaling stops the softmax saturating
View answer
✓ D. Dot products grow with d; scaling stops the softmax saturating
Without scaling, large scores push the softmax towards one-hot, and gradients become tiny.
Lesson: Self-Attention (Q, K, V)
Question 3 of 10
How does the cost of full self-attention grow with sequence length n?
- A. Quadratically, O(n²)
- B. It doesn't depend on n
- C. Linearly, O(n)
- D. Logarithmically, O(log n)
View answer
✓ A. Quadratically, O(n²)
Every token scores every other token, which makes an n × n matrix.
Lesson: Multi-Head Attention
Question 4 of 10
Why use several attention heads instead of one big one?
- A. It's much cheaper to compute than one big head of the same total size
- B. Several heads remove the need for positional information
- C. Extra heads replace the need for a feed-forward layer
- D. Each head can learn a different relationship, at about the same cost
View answer
✓ D. Each head can learn a different relationship, at about the same cost
Splitting d into h heads keeps compute similar while letting heads specialise.
Lesson: Multi-Head Attention
Question 5 of 10
Grouped-query attention (GQA) shares keys and values across groups of query heads. What does that mainly save?
- A. The amount of training data needed
- B. The number of layers in the model
- C. KV-cache memory at long context
- D. The size of the vocabulary
View answer
✓ C. KV-cache memory at long context
Fewer K/V heads means a smaller cache per token, which matters most when serving long contexts.
Lesson: Multi-Head Attention
Question 6 of 10
Without any positional information, what is self-attention's problem with word order?
- A. It's permutation-invariant: shuffled words give the same outputs
- B. It reads the sentence right to left by default
- C. It can't process words longer than one token, so it splits them into letters
- D. It can only attend to the first word in the sentence
View answer
✓ A. It's permutation-invariant: shuffled words give the same outputs
Attention treats its inputs as a set; positional encodings or RoPE bring order in.
Question 7 of 10
How does RoPE (rotary positional embedding) encode position?
- A. By adding a learned position vector to each token
- B. By sorting the tokens into order before attention
- C. By rotating queries and keys so scores depend on relative distance
- D. By running a separate small model that predicts each token's position
View answer
✓ C. By rotating queries and keys so scores depend on relative distance
Rotations make the score between two tokens depend on how far apart they are, not on where they sit.
Question 8 of 10
In a standard Transformer block with a 4× feed-forward expansion, where do most of the parameters sit?
- A. In the feed-forward (MLP) layers, about two-thirds
- B. In the positional encodings, one vector per position
- C. In the layer norms, which have a gain per channel
- D. In the attention softmax, which learns its own weights
View answer
✓ A. In the feed-forward (MLP) layers, about two-thirds
Attention projections are about 4d², and the FFN is about 8d² (d → 4d → d), so the FFN holds roughly two-thirds.
Lesson: The Transformer Block
Question 9 of 10
Which pairing is correct?
- A. BERT, GPT and T5 are all encoder-decoder models
- B. BERT: encoder-decoder; GPT: encoder-only; T5: decoder-only
- C. BERT: causal decoder; GPT: bidirectional encoder; T5: a decoder-only model
- D. BERT: bidirectional encoder; GPT: causal decoder; T5: encoder-decoder
View answer
✓ D. BERT: bidirectional encoder; GPT: causal decoder; T5: encoder-decoder
BERT sees both sides to understand text; GPT predicts the next token; T5 encodes an input and decodes an output.
Question 10 of 10
What does the causal mask in a decoder do?
- A. It stops each token attending to later tokens, matching generation
- B. It randomly drops some tokens during training, a lot like dropout does
- C. It hides padding tokens and nothing else
- D. It removes the first token so the model can't copy it
View answer
✓ A. It stops each token attending to later tokens, matching generation
Without it, the model would see the answer while learning to predict it.
Lesson: The Transformer, End to End