QUIZ
Pretraining
7 questions on Pretraining, from 5 lessons. Pick an answer in your head, then open it.
On the roadmap for: AI Engineering.
Question 1 of 7
How does one sentence of 8 tokens become 8 training examples?
- A. Every position predicts its next token, all in one pass
- B. The sentence is copied 8 times into the dataset
- C. Each word is treated as a separate document with its own label
- D. It doesn't: one sentence is one example
View answer
✓ A. Every position predicts its next token, all in one pass
Next-token prediction makes every position a target, which is why pretraining is so data-efficient per sequence.
Lesson: The Next-Token Objective
Question 2 of 7
Why is deduplication important for pretraining data?
- A. Models can't read the same piece of text twice
- B. Only to save disk space on the training cluster
- C. Duplicates cause memorisation, waste compute, leak tests
- D. Duplicates help, because they reinforce the high-quality text
View answer
✓ C. Duplicates cause memorisation, waste compute, leak tests
Removing near-duplicates improves quality per token and reduces regurgitation.
Lesson: Pretraining Data: Deduplication, Filtering and Mixtures
Question 3 of 7
By the Chinchilla scaling result, roughly how many training tokens per parameter is compute-optimal?
- A. It doesn't matter
- B. About 1,000
- C. About 1
- D. About 20
View answer
✓ D. About 20
Earlier models were undertrained. Chinchilla found model size and tokens should grow together, around 20 tokens per parameter.
Question 4 of 7
Why was a small model like Llama 3 8B trained on about 15 trillion tokens, far beyond 'compute-optimal'?
- A. To test how many GPUs the cluster could use
- B. Over-training a small model makes it better and cheaper to serve
- C. By mistake: the run was left going too long
- D. Small models need much more data than large ones just to work at all
View answer
✓ B. Over-training a small model makes it better and cheaper to serve
Compute-optimal minimises training cost; when inference dominates total cost, a smaller, longer-trained model wins.
Question 5 of 7
Why does training a 70B model with Adam in mixed precision need around a terabyte of memory?
- A. ~16 bytes per parameter: weights, grads, FP32 copy, Adam
- B. Activations alone take a terabyte at this model size
- C. The vocabulary's embedding table is extremely large at this size
- D. The training dataset itself must sit in GPU memory
View answer
✓ A. ~16 bytes per parameter: weights, grads, FP32 copy, Adam
70B × 16 bytes ≈ 1.1 TB before activations, so the model is sharded across many GPUs with FSDP or ZeRO.
Lesson: Training at Scale: Mixed Precision, Parallelism, ZeRO/FSDP
Question 6 of 7
What do ZeRO and FSDP do?
- A. They compress the training data so that it takes up less GPU memory
- B. They lower the learning rate as training goes on
- C. They shard parameters, gradients and optimiser state
- D. They skip the backward pass for frozen layers
View answer
✓ C. They shard parameters, gradients and optimiser state
Each GPU holds only a slice and gathers the rest when needed, cutting memory per GPU.
Lesson: Training at Scale: Mixed Precision, Parallelism, ZeRO/FSDP
Question 7 of 7
The loss spikes suddenly at step 40,000. Which is a standard response?
- A. Ignore it; loss spikes always recover on their own
- B. Roll back, skip bad batches or lower LR, add z-loss
- C. Delete the optimiser state and keep training on
- D. Double the learning rate to push through the spike fast
View answer
✓ B. Roll back, skip bad batches or lower LR, add z-loss
Spikes often come from exploding logits or attention scores; stabilisers keep them bounded.
Lesson: Training Instabilities: Loss Spikes, z-loss and QK-norm