QUIZ
LLM architecture
8 questions on LLM architecture, from 8 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering, ML Engineering.
Question 1 of 8
Which change is typical of modern LLMs compared with the original 2017 Transformer?
- A. No attention, only feed-forward layers
- B. Decoder-only, with RMSNorm, RoPE and SwiGLU
- C. Encoder-decoder models for chat
- D. Decoder-only, with LayerNorm, learned positions, ReLU
View answer
✓ B. Decoder-only, with RMSNorm, RoPE and SwiGLU
Most current open models share this recipe, plus grouped-query attention for cheaper inference.
Lesson: Anatomy of a Modern LLM
Question 2 of 8
How do multi-query (MQA) and grouped-query attention (GQA) let more conversations fit on one GPU?
- A. They shorten the vocabulary to save memory
- B. They share key/value heads, shrinking the KV cache
- C. They remove attention from some layers
- D. They quantise the attention weights down to 4 bits each
View answer
✓ B. They share key/value heads, shrinking the KV cache
MQA uses one KV head, GQA a few groups; MLA compresses keys and values into a smaller latent.
Question 3 of 8
Why is decoding limited by memory bandwidth rather than compute?
- A. GPUs have little compute compared with their memory
- B. Little maths per byte: all weights read for one token
- C. Prefill is slower and holds up every decode step that follows
- D. Decoding runs on the CPU, not on the GPU at all
View answer
✓ B. Little maths per byte: all weights read for one token
Low arithmetic intensity means the GPU waits on memory. Batching many requests raises it.
Lesson: GPUs, TPUs and Memory Bandwidth
Question 4 of 8
How can FlashAttention be faster while computing exact attention?
- A. It uses lower precision, and that alone is the whole speed-up
- B. It works in tiles on-chip, never writing the full matrix
- C. It skips tokens that get low attention scores
- D. It approximates the softmax with a cheaper function
View answer
✓ B. It works in tiles on-chip, never writing the full matrix
Memory traffic, not maths, was the bottleneck; fewer reads and writes beat the extra recomputation.
Lesson: FlashAttention
Question 5 of 8
A model trained on 4k tokens is used at 16k and breaks down. What does RoPE scaling (e.g. YaRN) do?
- A. It retrains the whole model from scratch at 16k
- B. It adds new layers that are trained only on long documents
- C. It rescales rotary frequencies to fit long positions
- D. It truncates the input to the first 4k tokens
View answer
✓ C. It rescales rotary frequencies to fit long positions
Positions beyond training are unfamiliar; rescaling them, plus brief fine-tuning, extends usable context.
Lesson: Long Context: RoPE Scaling, Sliding Windows and Attention Sinks
Question 6 of 8
What do linear or sparse attention alternatives usually give up for million-token contexts?
- A. Some precision recalling exact details from far back
- B. All positional information about the tokens in the sequence
- C. The ability to generate text one token at a time
- D. Nothing: they match full attention at lower cost
View answer
✓ A. Some precision recalling exact details from far back
Compressing or skipping parts of the context saves compute but can lose information exact attention would keep.
Lesson: Attention Alternatives: Sparse, Linear and Compressed
Question 7 of 8
How does a Mixture-of-Experts model with 47B parameters run at roughly 13B cost per token?
- A. It uses only half of its vocabulary at a time
- B. A router sends each token to a few experts only
- C. It's quantised to a quarter of its size
- D. It skips half of its layers for every token it processes
View answer
✓ B. A router sends each token to a few experts only
Compute follows the active parameters, but memory still has to hold all experts.
Lesson: Mixture of Experts
Question 8 of 8
How does a vision-language model 'see' an image?
- A. It can't; it only describes images it was trained on
- B. A vision encoder turns patches into tokens the LLM reads
- C. It runs OCR and then reads only the text found in the image
- D. It reads the image's file name and metadata
View answer
✓ B. A vision encoder turns patches into tokens the LLM reads
Each image becomes many tokens, often hundreds or more, which costs context and compute.
Lesson: Vision-Language Models