AIML Rounds

QUIZ

Training deep nets

10 questions on Training deep nets, from 11 lessons. Pick an answer in your head, then open it.

On the roadmap for: ML Engineering.

  1. Question 1 of 10

    Why does He initialisation scale weights by about √(2 / fan-in) for ReLU networks?

    1. A. To make every initial weight positive and equal
    2. B. To speed up the GPU's matrix multiplications
    3. C. To keep activation variance steady through ReLU layers
    4. D. To avoid needing dropout later on in training
    View answer

    ✓ C. To keep activation variance steady through ReLU layers

    With the wrong scale, signals shrink to zero or blow up across 20 layers before training even starts.

    Lesson: Weight Initialisation

  2. Question 2 of 10

    How do residual (skip) connections help very deep networks train?

    1. A. They give gradients a direct identity path backwards
    2. B. They add extra parameters, so the network can fit more of the data
    3. C. They remove non-linearities from deep layers
    4. D. They lower the learning rate for early layers
    View answer

    ✓ A. They give gradients a direct identity path backwards

    Each block learns a change to its input, and the identity path stops gradients from vanishing.

    Lesson: Vanishing & Exploding Gradients

  3. Question 3 of 10

    Why do Transformers use LayerNorm (or RMSNorm) instead of BatchNorm?

    1. A. BatchNorm can't run efficiently on modern GPUs at all
    2. B. LayerNorm has no learnable parameters to train
    3. C. BatchNorm is newer and less tested than LayerNorm
    4. D. Batch statistics are unreliable for sequences, small batches
    View answer

    ✓ D. Batch statistics are unreliable for sequences, small batches

    Per-token normalisation behaves the same in training and inference, whatever the batch.

    Lesson: BatchNorm vs LayerNorm vs RMSNorm

  4. Question 4 of 10

    Why do LLM training runs warm up the learning rate at the start?

    1. A. Early Adam statistics are unreliable, so big steps blow up
    2. B. Warm-up isn't actually used in LLM training runs
    3. C. To save electricity at the very start of the run
    4. D. To make the loss rise first, so that it falls faster later in the run
    View answer

    ✓ A. Early Adam statistics are unreliable, so big steps blow up

    Ramping the learning rate up over the first steps, then decaying it, gives stable and strong training.

    Lesson: Learning-Rate Schedules

  5. Question 5 of 10

    You increase the batch size 8×. What should usually happen to the learning rate?

    1. A. Raise it roughly in proportion (linear scaling), with warm-up
    2. B. Set it to zero for the first thousand steps only
    3. C. Keep it the same, since batch size doesn't affect it
    4. D. Divide it by 8, so each update stays the same size
    View answer

    ✓ A. Raise it roughly in proportion (linear scaling), with warm-up

    Larger batches give less noisy gradients, so bigger steps are safe, up to a limit.

    Lesson: Batch Size and Learning Rate

  6. Question 6 of 10

    How does dropout help a network generalise?

    1. A. It speeds up inference by skipping some units at test time
    2. B. Dropping units stops co-adaptation, like an ensemble
    3. C. It removes the need for so much training data
    4. D. It makes the network bigger at test time
    View answer

    ✓ B. Dropping units stops co-adaptation, like an ensemble

    Activations are rescaled during training, so nothing changes at inference.

    Lesson: Dropout

  7. Question 7 of 10

    Why is Adam with L2 regularisation not the same as AdamW?

    1. A. AdamW doesn't apply any weight decay
    2. B. They are exactly the same thing, just given different names by different libraries
    3. C. L2 regularisation only works with SGD
    4. D. In Adam, L2 goes through adaptive scaling; AdamW decays directly
    View answer

    ✓ D. In Adam, L2 goes through adaptive scaling; AdamW decays directly

    Decoupling weight decay from the gradient update makes regularisation behave as intended.

    Lesson: Weight Decay vs L2, and AdamW

  8. Question 8 of 10

    What does mixup do?

    1. A. It trains on blends of two examples and their labels
    2. B. It shuffles the labels between classes
    3. C. It averages the weights of several separately trained models
    4. D. It mixes the training and test sets together
    View answer

    ✓ A. It trains on blends of two examples and their labels

    Like other augmentation, it creates new training signal without collecting data.

    Lesson: Data Augmentation and Mixup

  9. Question 9 of 10

    What does label smoothing change?

    1. A. It removes the labels from noisy examples
    2. B. It adds extra 'unsure' classes for uncertain examples
    3. C. Targets become slightly soft, like 0.9 instead of 1
    4. D. It averages predictions across several epochs
    View answer

    ✓ C. Targets become slightly soft, like 0.9 instead of 1

    Softer targets often improve calibration and generalisation.

    Lesson: Early Stopping and Label Smoothing

  10. Question 10 of 10

    Your model can't overfit a batch of just 8 examples. What does that tell you?

    1. A. You need much more data before it can fit
    2. B. The model is too big to fit so few examples without heavy regularisation
    3. C. There's likely a bug: labels, loss, frozen weights or LR
    4. D. Training is working as expected for small batches
    View answer

    ✓ C. There's likely a bug: labels, loss, frozen weights or LR

    Any working model should memorise 8 examples; failing to is a quick bug detector.

    Lesson: Debugging Training: Loss Won't Go Down