QUIZ
Inference and serving
12 questions on Inference and serving, from 14 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering, ML Engineering.
Question 1 of 12
Why does an LLM read your prompt quickly but write its answer slowly?
- A. Output tokens are longer than input tokens on average
- B. Prefill runs the prompt in parallel; decode makes one token at a time
- C. The GPU slows down as it heats up during long generations of many tokens
- D. The prompt is cached on disk, so it loads almost instantly
View answer
✓ B. Prefill runs the prompt in parallel; decode makes one token at a time
Each decode step reloads the weights and KV cache for a single token, so memory bandwidth, not FLOPs, sets the speed.
Lesson: Prefill vs Decode
Question 2 of 12
Which of these does NOT change the KV-cache size per token?
- A. Bytes per value (e.g. FP16 vs FP8)
- B. Vocabulary size
- C. Number of layers
- D. Number of KV heads
View answer
✓ B. Vocabulary size
Per token the cache is 2 × layers × KV heads × head dimension × bytes. The vocabulary only affects the output layer.
Question 3 of 12
What problem does PagedAttention solve?
- A. Long training times for large models
- B. Slow tokenisation of very long prompts
- C. Lower model accuracy at long context lengths, as attention gets diluted
- D. KV memory wasted by reserving one big block per request
View answer
✓ D. KV memory wasted by reserving one big block per request
Allocating KV in small blocks on demand, like OS paging, lets far more requests share the GPU.
Question 4 of 12
Why does 4-bit weight quantization often speed up decoding?
- A. It adds extra layers that run in parallel on the GPU
- B. It removes the need for a KV cache during generation
- C. Decode is memory-bound, and smaller weights load faster
- D. It shrinks the vocabulary, so the softmax is much cheaper
View answer
✓ C. Decode is memory-bound, and smaller weights load faster
Fewer bytes per weight means less data to read for every generated token, usually with small accuracy loss.
Lesson: LLM Quantization
Question 5 of 12
What does continuous batching change compared with static batching?
- A. It batches only the prompts together, then decodes each request on its own
- B. It runs requests one at a time to keep latency low
- C. Requests join and leave every step, so done ones free the GPU
- D. It waits for a full batch of requests before it starts
View answer
✓ C. Requests join and leave every step, so done ones free the GPU
Scheduling per iteration keeps the GPU busy despite very different output lengths.
Question 6 of 12
A team adds the current timestamp at the start of a long system prompt, and their prompt-cache hit rate drops to almost zero. Why?
- A. Models refuse to cache prompts that start with a date
- B. Caching needs an exact prefix; a changing first line breaks it
- C. Prompt caches expire every second by default
- D. Timestamps make the prompt too long to fit inside the provider's cache
View answer
✓ B. Caching needs an exact prefix; a changing first line breaks it
Put stable content first and anything that changes, like dates or user data, at the end.
Lesson: Prefix and Prompt Caching
Question 7 of 12
How can speculative decoding speed up generation without changing the output distribution?
- A. A small draft proposes tokens; the big model checks them at once
- B. It lowers the temperature, so fewer tokens are needed to reach the answer
- C. It switches to a smaller vocabulary for common words
- D. It skips every other token and fills the gaps in later
View answer
✓ A. A small draft proposes tokens; the big model checks them at once
The acceptance rule guarantees samples follow the target model's distribution; you just get several tokens per big-model pass.
Lesson: Speculative Decoding
Question 8 of 12
What does TTFT measure, and which phase mostly sets it?
- A. Tokens per user per day; mostly tokenisation
- B. Time to finish training; mostly pretraining
- C. Time to first token; mostly prefill, plus any queueing
- D. Total tokens per second across all users; mostly decode time
View answer
✓ C. Time to first token; mostly prefill, plus any queueing
TTFT covers waiting plus processing the prompt. Time per output token (TPOT) reflects decode.
Lesson: Serving Metrics and SLOs
Question 9 of 12
Why does p99 latency shoot up as a server approaches full capacity?
- A. GPUs cool down and then slow down when they are under very heavy load
- B. Requests get shorter, which adds overhead per token
- C. The model gets bigger as more users connect
- D. Queueing: near 100% utilisation, waiting time grows very fast
View answer
✓ D. Queueing: near 100% utilisation, waiting time grows very fast
Near saturation, small bursts create long queues. Plan capacity for latency targets, not peak throughput.
Lesson: Serving Metrics and SLOs
Question 10 of 12
A Mixture-of-Experts model activates only a small share of its parameters per token. Why is it still expensive to serve?
- A. Only one user at a time can be routed to each expert
- B. All experts stay in memory, with lots of traffic between GPUs
- C. Its outputs are much longer than a dense model's outputs for the same prompt
- D. It needs special hardware that no cloud provider offers
View answer
✓ B. All experts stay in memory, with lots of traffic between GPUs
Compute per token is low, but memory holds the full model, often spread across many GPUs.
Question 11 of 12
Reasoning models write thousands of hidden thinking tokens. What does that do to serving?
- A. Makes it prefill-heavy, because prompts get much longer
- B. Removes the KV cache, since thinking tokens are discarded
- C. Has no effect, because hidden tokens aren't billed or stored
- D. Makes it decode-heavy, with bigger KV caches and cost
View answer
✓ D. Makes it decode-heavy, with bigger KV caches and cost
Those tokens are generated one by one like any output, so latency, memory and cost all rise.
Question 12 of 12
When is self-hosting an open model usually cheaper per million tokens than an API?
- A. Always, because there are no API fees
- B. Never, because GPUs cost more than any API
- C. When you can keep the GPUs busy with steady traffic
- D. When traffic is tiny and arrives in short, unpredictable bursts
View answer
✓ C. When you can keep the GPUs busy with steady traffic
You pay for GPU hours whether or not they're used, so idle capacity makes each token expensive.