QUIZ
Training deep nets
10 questions on Training deep nets, from 11 lessons. Pick an answer in your head, then open it.
On the roadmap for: ML Engineering.
Question 1 of 10
Why does He initialisation scale weights by about √(2 / fan-in) for ReLU networks?
- A. To make every initial weight positive and equal
- B. To speed up the GPU's matrix multiplications
- C. To keep activation variance steady through ReLU layers
- D. To avoid needing dropout later on in training
View answer
✓ C. To keep activation variance steady through ReLU layers
With the wrong scale, signals shrink to zero or blow up across 20 layers before training even starts.
Lesson: Weight Initialisation
Question 2 of 10
How do residual (skip) connections help very deep networks train?
- A. They give gradients a direct identity path backwards
- B. They add extra parameters, so the network can fit more of the data
- C. They remove non-linearities from deep layers
- D. They lower the learning rate for early layers
View answer
✓ A. They give gradients a direct identity path backwards
Each block learns a change to its input, and the identity path stops gradients from vanishing.
Lesson: Vanishing & Exploding Gradients
Question 3 of 10
Why do Transformers use LayerNorm (or RMSNorm) instead of BatchNorm?
- A. BatchNorm can't run efficiently on modern GPUs at all
- B. LayerNorm has no learnable parameters to train
- C. BatchNorm is newer and less tested than LayerNorm
- D. Batch statistics are unreliable for sequences, small batches
View answer
✓ D. Batch statistics are unreliable for sequences, small batches
Per-token normalisation behaves the same in training and inference, whatever the batch.
Question 4 of 10
Why do LLM training runs warm up the learning rate at the start?
- A. Early Adam statistics are unreliable, so big steps blow up
- B. Warm-up isn't actually used in LLM training runs
- C. To save electricity at the very start of the run
- D. To make the loss rise first, so that it falls faster later in the run
View answer
✓ A. Early Adam statistics are unreliable, so big steps blow up
Ramping the learning rate up over the first steps, then decaying it, gives stable and strong training.
Lesson: Learning-Rate Schedules
Question 5 of 10
You increase the batch size 8×. What should usually happen to the learning rate?
- A. Raise it roughly in proportion (linear scaling), with warm-up
- B. Set it to zero for the first thousand steps only
- C. Keep it the same, since batch size doesn't affect it
- D. Divide it by 8, so each update stays the same size
View answer
✓ A. Raise it roughly in proportion (linear scaling), with warm-up
Larger batches give less noisy gradients, so bigger steps are safe, up to a limit.
Lesson: Batch Size and Learning Rate
Question 6 of 10
How does dropout help a network generalise?
- A. It speeds up inference by skipping some units at test time
- B. Dropping units stops co-adaptation, like an ensemble
- C. It removes the need for so much training data
- D. It makes the network bigger at test time
View answer
✓ B. Dropping units stops co-adaptation, like an ensemble
Activations are rescaled during training, so nothing changes at inference.
Lesson: Dropout
Question 7 of 10
Why is Adam with L2 regularisation not the same as AdamW?
- A. AdamW doesn't apply any weight decay
- B. They are exactly the same thing, just given different names by different libraries
- C. L2 regularisation only works with SGD
- D. In Adam, L2 goes through adaptive scaling; AdamW decays directly
View answer
✓ D. In Adam, L2 goes through adaptive scaling; AdamW decays directly
Decoupling weight decay from the gradient update makes regularisation behave as intended.
Lesson: Weight Decay vs L2, and AdamW
Question 8 of 10
What does mixup do?
- A. It trains on blends of two examples and their labels
- B. It shuffles the labels between classes
- C. It averages the weights of several separately trained models
- D. It mixes the training and test sets together
View answer
✓ A. It trains on blends of two examples and their labels
Like other augmentation, it creates new training signal without collecting data.
Lesson: Data Augmentation and Mixup
Question 9 of 10
What does label smoothing change?
- A. It removes the labels from noisy examples
- B. It adds extra 'unsure' classes for uncertain examples
- C. Targets become slightly soft, like 0.9 instead of 1
- D. It averages predictions across several epochs
View answer
✓ C. Targets become slightly soft, like 0.9 instead of 1
Softer targets often improve calibration and generalisation.
Question 10 of 10
Your model can't overfit a batch of just 8 examples. What does that tell you?
- A. You need much more data before it can fit
- B. The model is too big to fit so few examples without heavy regularisation
- C. There's likely a bug: labels, loss, frozen weights or LR
- D. Training is working as expected for small batches
View answer
✓ C. There's likely a bug: labels, loss, frozen weights or LR
Any working model should memorise 8 examples; failing to is a quick bug detector.