AIML Rounds

QUIZ

Inference and serving

12 questions on Inference and serving, from 14 lessons. Pick an answer in your head, then open it.

On the roadmap for: AI Engineering, ML Engineering.

  1. Question 1 of 12

    Why does an LLM read your prompt quickly but write its answer slowly?

    1. A. Output tokens are longer than input tokens on average
    2. B. Prefill runs the prompt in parallel; decode makes one token at a time
    3. C. The GPU slows down as it heats up during long generations of many tokens
    4. D. The prompt is cached on disk, so it loads almost instantly
    View answer

    ✓ B. Prefill runs the prompt in parallel; decode makes one token at a time

    Each decode step reloads the weights and KV cache for a single token, so memory bandwidth, not FLOPs, sets the speed.

    Lesson: Prefill vs Decode

  2. Question 2 of 12

    Which of these does NOT change the KV-cache size per token?

    1. A. Bytes per value (e.g. FP16 vs FP8)
    2. B. Vocabulary size
    3. C. Number of layers
    4. D. Number of KV heads
    View answer

    ✓ B. Vocabulary size

    Per token the cache is 2 × layers × KV heads × head dimension × bytes. The vocabulary only affects the output layer.

    Lesson: KV Cache Management and PagedAttention

  3. Question 3 of 12

    What problem does PagedAttention solve?

    1. A. Long training times for large models
    2. B. Slow tokenisation of very long prompts
    3. C. Lower model accuracy at long context lengths, as attention gets diluted
    4. D. KV memory wasted by reserving one big block per request
    View answer

    ✓ D. KV memory wasted by reserving one big block per request

    Allocating KV in small blocks on demand, like OS paging, lets far more requests share the GPU.

    Lesson: KV Cache Management and PagedAttention

  4. Question 4 of 12

    Why does 4-bit weight quantization often speed up decoding?

    1. A. It adds extra layers that run in parallel on the GPU
    2. B. It removes the need for a KV cache during generation
    3. C. Decode is memory-bound, and smaller weights load faster
    4. D. It shrinks the vocabulary, so the softmax is much cheaper
    View answer

    ✓ C. Decode is memory-bound, and smaller weights load faster

    Fewer bytes per weight means less data to read for every generated token, usually with small accuracy loss.

    Lesson: LLM Quantization

  5. Question 5 of 12

    What does continuous batching change compared with static batching?

    1. A. It batches only the prompts together, then decodes each request on its own
    2. B. It runs requests one at a time to keep latency low
    3. C. Requests join and leave every step, so done ones free the GPU
    4. D. It waits for a full batch of requests before it starts
    View answer

    ✓ C. Requests join and leave every step, so done ones free the GPU

    Scheduling per iteration keeps the GPU busy despite very different output lengths.

    Lesson: Continuous Batching and Scheduling

  6. Question 6 of 12

    A team adds the current timestamp at the start of a long system prompt, and their prompt-cache hit rate drops to almost zero. Why?

    1. A. Models refuse to cache prompts that start with a date
    2. B. Caching needs an exact prefix; a changing first line breaks it
    3. C. Prompt caches expire every second by default
    4. D. Timestamps make the prompt too long to fit inside the provider's cache
    View answer

    ✓ B. Caching needs an exact prefix; a changing first line breaks it

    Put stable content first and anything that changes, like dates or user data, at the end.

    Lesson: Prefix and Prompt Caching

  7. Question 7 of 12

    How can speculative decoding speed up generation without changing the output distribution?

    1. A. A small draft proposes tokens; the big model checks them at once
    2. B. It lowers the temperature, so fewer tokens are needed to reach the answer
    3. C. It switches to a smaller vocabulary for common words
    4. D. It skips every other token and fills the gaps in later
    View answer

    ✓ A. A small draft proposes tokens; the big model checks them at once

    The acceptance rule guarantees samples follow the target model's distribution; you just get several tokens per big-model pass.

    Lesson: Speculative Decoding

  8. Question 8 of 12

    What does TTFT measure, and which phase mostly sets it?

    1. A. Tokens per user per day; mostly tokenisation
    2. B. Time to finish training; mostly pretraining
    3. C. Time to first token; mostly prefill, plus any queueing
    4. D. Total tokens per second across all users; mostly decode time
    View answer

    ✓ C. Time to first token; mostly prefill, plus any queueing

    TTFT covers waiting plus processing the prompt. Time per output token (TPOT) reflects decode.

    Lesson: Serving Metrics and SLOs

  9. Question 9 of 12

    Why does p99 latency shoot up as a server approaches full capacity?

    1. A. GPUs cool down and then slow down when they are under very heavy load
    2. B. Requests get shorter, which adds overhead per token
    3. C. The model gets bigger as more users connect
    4. D. Queueing: near 100% utilisation, waiting time grows very fast
    View answer

    ✓ D. Queueing: near 100% utilisation, waiting time grows very fast

    Near saturation, small bursts create long queues. Plan capacity for latency targets, not peak throughput.

    Lesson: Serving Metrics and SLOs

  10. Question 10 of 12

    A Mixture-of-Experts model activates only a small share of its parameters per token. Why is it still expensive to serve?

    1. A. Only one user at a time can be routed to each expert
    2. B. All experts stay in memory, with lots of traffic between GPUs
    3. C. Its outputs are much longer than a dense model's outputs for the same prompt
    4. D. It needs special hardware that no cloud provider offers
    View answer

    ✓ B. All experts stay in memory, with lots of traffic between GPUs

    Compute per token is low, but memory holds the full model, often spread across many GPUs.

    Lesson: Serving Mixture-of-Experts Models

  11. Question 11 of 12

    Reasoning models write thousands of hidden thinking tokens. What does that do to serving?

    1. A. Makes it prefill-heavy, because prompts get much longer
    2. B. Removes the KV cache, since thinking tokens are discarded
    3. C. Has no effect, because hidden tokens aren't billed or stored
    4. D. Makes it decode-heavy, with bigger KV caches and cost
    View answer

    ✓ D. Makes it decode-heavy, with bigger KV caches and cost

    Those tokens are generated one by one like any output, so latency, memory and cost all rise.

    Lesson: Serving Agents and Reasoning Models

  12. Question 12 of 12

    When is self-hosting an open model usually cheaper per million tokens than an API?

    1. A. Always, because there are no API fees
    2. B. Never, because GPUs cost more than any API
    3. C. When you can keep the GPUs busy with steady traffic
    4. D. When traffic is tiny and arrives in short, unpredictable bursts
    View answer

    ✓ C. When you can keep the GPUs busy with steady traffic

    You pay for GPU hours whether or not they're used, so idle capacity makes each token expensive.

    Lesson: Sizing and Cost: Cost per Million Tokens