AIML Rounds

QUIZ

LLM architecture

8 questions on LLM architecture, from 8 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering, ML Engineering.

  1. Question 1 of 8

    Which change is typical of modern LLMs compared with the original 2017 Transformer?

    1. A. No attention, only feed-forward layers
    2. B. Decoder-only, with RMSNorm, RoPE and SwiGLU
    3. C. Encoder-decoder models for chat
    4. D. Decoder-only, with LayerNorm, learned positions, ReLU
    View answer

    ✓ B. Decoder-only, with RMSNorm, RoPE and SwiGLU

    Most current open models share this recipe, plus grouped-query attention for cheaper inference.

    Lesson: Anatomy of a Modern LLM

  2. Question 2 of 8

    How do multi-query (MQA) and grouped-query attention (GQA) let more conversations fit on one GPU?

    1. A. They shorten the vocabulary to save memory
    2. B. They share key/value heads, shrinking the KV cache
    3. C. They remove attention from some layers
    4. D. They quantise the attention weights down to 4 bits each
    View answer

    ✓ B. They share key/value heads, shrinking the KV cache

    MQA uses one KV head, GQA a few groups; MLA compresses keys and values into a smaller latent.

    Lesson: Attention Variants: MQA, GQA and MLA

  3. Question 3 of 8

    Why is decoding limited by memory bandwidth rather than compute?

    1. A. GPUs have little compute compared with their memory
    2. B. Little maths per byte: all weights read for one token
    3. C. Prefill is slower and holds up every decode step that follows
    4. D. Decoding runs on the CPU, not on the GPU at all
    View answer

    ✓ B. Little maths per byte: all weights read for one token

    Low arithmetic intensity means the GPU waits on memory. Batching many requests raises it.

    Lesson: GPUs, TPUs and Memory Bandwidth

  4. Question 4 of 8

    How can FlashAttention be faster while computing exact attention?

    1. A. It uses lower precision, and that alone is the whole speed-up
    2. B. It works in tiles on-chip, never writing the full matrix
    3. C. It skips tokens that get low attention scores
    4. D. It approximates the softmax with a cheaper function
    View answer

    ✓ B. It works in tiles on-chip, never writing the full matrix

    Memory traffic, not maths, was the bottleneck; fewer reads and writes beat the extra recomputation.

    Lesson: FlashAttention

  5. Question 5 of 8

    A model trained on 4k tokens is used at 16k and breaks down. What does RoPE scaling (e.g. YaRN) do?

    1. A. It retrains the whole model from scratch at 16k
    2. B. It adds new layers that are trained only on long documents
    3. C. It rescales rotary frequencies to fit long positions
    4. D. It truncates the input to the first 4k tokens
    View answer

    ✓ C. It rescales rotary frequencies to fit long positions

    Positions beyond training are unfamiliar; rescaling them, plus brief fine-tuning, extends usable context.

    Lesson: Long Context: RoPE Scaling, Sliding Windows and Attention Sinks

  6. Question 6 of 8

    What do linear or sparse attention alternatives usually give up for million-token contexts?

    1. A. Some precision recalling exact details from far back
    2. B. All positional information about the tokens in the sequence
    3. C. The ability to generate text one token at a time
    4. D. Nothing: they match full attention at lower cost
    View answer

    ✓ A. Some precision recalling exact details from far back

    Compressing or skipping parts of the context saves compute but can lose information exact attention would keep.

    Lesson: Attention Alternatives: Sparse, Linear and Compressed

  7. Question 7 of 8

    How does a Mixture-of-Experts model with 47B parameters run at roughly 13B cost per token?

    1. A. It uses only half of its vocabulary at a time
    2. B. A router sends each token to a few experts only
    3. C. It's quantised to a quarter of its size
    4. D. It skips half of its layers for every token it processes
    View answer

    ✓ B. A router sends each token to a few experts only

    Compute follows the active parameters, but memory still has to hold all experts.

    Lesson: Mixture of Experts

  8. Question 8 of 8

    How does a vision-language model 'see' an image?

    1. A. It can't; it only describes images it was trained on
    2. B. A vision encoder turns patches into tokens the LLM reads
    3. C. It runs OCR and then reads only the text found in the image
    4. D. It reads the image's file name and metadata
    View answer

    ✓ B. A vision encoder turns patches into tokens the LLM reads

    Each image becomes many tokens, often hundreds or more, which costs context and compute.

    Lesson: Vision-Language Models