AIML Rounds

QUIZ

Attention and Transformers

10 questions on Attention and Transformers, from 6 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering, ML Engineering.

  1. Question 1 of 10

    In self-attention, what do the queries, keys and values do?

    1. A. A query is matched to every key to get weights that mix the values
    2. B. Queries are the labels the model is trying to predict
    3. C. They're three identical copies of the token embedding
    4. D. Keys hold the final output, and the values are only ever used for masking
    View answer

    ✓ A. A query is matched to every key to get weights that mix the values

    Each token asks (query) which tokens are relevant (keys) and takes a weighted mix of their content (values).

    Lesson: Self-Attention (Q, K, V)

  2. Question 2 of 10

    Why are attention scores divided by √d before the softmax?

    1. A. To save memory when the head dimension is very large
    2. B. To make every attention score positive before it goes into the softmax
    3. C. To normalise each value vector to unit length first
    4. D. Dot products grow with d; scaling stops the softmax saturating
    View answer

    ✓ D. Dot products grow with d; scaling stops the softmax saturating

    Without scaling, large scores push the softmax towards one-hot, and gradients become tiny.

    Lesson: Self-Attention (Q, K, V)

  3. Question 3 of 10

    How does the cost of full self-attention grow with sequence length n?

    1. A. Quadratically, O(n²)
    2. B. It doesn't depend on n
    3. C. Linearly, O(n)
    4. D. Logarithmically, O(log n)
    View answer

    ✓ A. Quadratically, O(n²)

    Every token scores every other token, which makes an n × n matrix.

    Lesson: Multi-Head Attention

  4. Question 4 of 10

    Why use several attention heads instead of one big one?

    1. A. It's much cheaper to compute than one big head of the same total size
    2. B. Several heads remove the need for positional information
    3. C. Extra heads replace the need for a feed-forward layer
    4. D. Each head can learn a different relationship, at about the same cost
    View answer

    ✓ D. Each head can learn a different relationship, at about the same cost

    Splitting d into h heads keeps compute similar while letting heads specialise.

    Lesson: Multi-Head Attention

  5. Question 5 of 10

    Grouped-query attention (GQA) shares keys and values across groups of query heads. What does that mainly save?

    1. A. The amount of training data needed
    2. B. The number of layers in the model
    3. C. KV-cache memory at long context
    4. D. The size of the vocabulary
    View answer

    ✓ C. KV-cache memory at long context

    Fewer K/V heads means a smaller cache per token, which matters most when serving long contexts.

    Lesson: Multi-Head Attention

  6. Question 6 of 10

    Without any positional information, what is self-attention's problem with word order?

    1. A. It's permutation-invariant: shuffled words give the same outputs
    2. B. It reads the sentence right to left by default
    3. C. It can't process words longer than one token, so it splits them into letters
    4. D. It can only attend to the first word in the sentence
    View answer

    ✓ A. It's permutation-invariant: shuffled words give the same outputs

    Attention treats its inputs as a set; positional encodings or RoPE bring order in.

    Lesson: Positional Encodings (incl. RoPE)

  7. Question 7 of 10

    How does RoPE (rotary positional embedding) encode position?

    1. A. By adding a learned position vector to each token
    2. B. By sorting the tokens into order before attention
    3. C. By rotating queries and keys so scores depend on relative distance
    4. D. By running a separate small model that predicts each token's position
    View answer

    ✓ C. By rotating queries and keys so scores depend on relative distance

    Rotations make the score between two tokens depend on how far apart they are, not on where they sit.

    Lesson: Positional Encodings (incl. RoPE)

  8. Question 8 of 10

    In a standard Transformer block with a 4× feed-forward expansion, where do most of the parameters sit?

    1. A. In the feed-forward (MLP) layers, about two-thirds
    2. B. In the positional encodings, one vector per position
    3. C. In the layer norms, which have a gain per channel
    4. D. In the attention softmax, which learns its own weights
    View answer

    ✓ A. In the feed-forward (MLP) layers, about two-thirds

    Attention projections are about 4d², and the FFN is about 8d² (d → 4d → d), so the FFN holds roughly two-thirds.

    Lesson: The Transformer Block

  9. Question 9 of 10

    Which pairing is correct?

    1. A. BERT, GPT and T5 are all encoder-decoder models
    2. B. BERT: encoder-decoder; GPT: encoder-only; T5: decoder-only
    3. C. BERT: causal decoder; GPT: bidirectional encoder; T5: a decoder-only model
    4. D. BERT: bidirectional encoder; GPT: causal decoder; T5: encoder-decoder
    View answer

    ✓ D. BERT: bidirectional encoder; GPT: causal decoder; T5: encoder-decoder

    BERT sees both sides to understand text; GPT predicts the next token; T5 encodes an input and decodes an output.

    Lesson: Encoder vs Decoder vs Encoder-Decoder

  10. Question 10 of 10

    What does the causal mask in a decoder do?

    1. A. It stops each token attending to later tokens, matching generation
    2. B. It randomly drops some tokens during training, a lot like dropout does
    3. C. It hides padding tokens and nothing else
    4. D. It removes the first token so the model can't copy it
    View answer

    ✓ A. It stops each token attending to later tokens, matching generation

    Without it, the model would see the answer while learning to predict it.

    Lesson: The Transformer, End to End