QUIZ
Generation and LLM evals
6 questions on Generation and LLM evals, from 6 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering.
Question 1 of 6
What does raising the sampling temperature do?
- A. It flattens the distribution; rare tokens appear more
- B. It removes all randomness from sampling
- C. It changes the model's weights for that request
- D. It makes outputs shorter and more focused on the likeliest words
View answer
✓ A. It flattens the distribution; rare tokens appear more
Logits are divided by T before the softmax: higher T means more diverse and riskier text.
Question 2 of 6
How does min-p sampling choose which tokens to keep?
- A. Only the single most likely token
- B. Tokens above a fraction of the top token's probability
- C. Tokens until their probabilities add up to p, in order of likelihood
- D. The k tokens with the highest probability
View answer
✓ B. Tokens above a fraction of the top token's probability
The cut-off adapts to how confident the model is, unlike a fixed top-k or top-p.
Question 3 of 6
Why does chain-of-thought prompting help on multi-step maths problems?
- A. It lowers the temperature automatically
- B. It makes the model larger for that question
- C. It lets the model search the web for the answer while it writes
- D. Writing steps gives it room to compute before answering
View answer
✓ D. Writing steps gives it room to compute before answering
Each generated step becomes context for the next, so hard problems are broken into easier ones.
Question 4 of 6
Why do LLMs hallucinate with confidence?
- A. They read the internet live while answering and repeat its errors
- B. They are lazy and choose the easiest answer
- C. Temperature is always set too high
- D. Training rewards a plausible guess over 'I don't know'
View answer
✓ D. Training rewards a plausible guess over 'I don't know'
Ground answers in sources, allow abstaining, and measure it, rather than relying on the model to know its limits.
Question 5 of 6
A new model scores suspiciously high on a public benchmark. What should you suspect?
- A. Contamination: test questions were in its training data
- B. It was evaluated on faster GPUs than the other models
- C. Nothing: it's simply a much better model overall
- D. The benchmark is too hard for the other models tested
View answer
✓ A. Contamination: test questions were in its training data
Use held-out or freshly written test sets and check small score gaps for statistical noise.
Lesson: Benchmarks and Contamination
Question 6 of 6
Roughly how many FLOPs does it take to train a model with N parameters on D tokens?
- A. About N² · D operations
- B. About 6 · N · D
- C. About D / N operations
- D. About N + D operations
View answer
✓ B. About 6 · N · D
About 2ND for the forward pass and 4ND for the backward pass is the standard back-of-the-envelope estimate.