ML ENGINEERING · ROUND 2 OF 3
Deep learning and training
The interviewer checks you understand how networks train, and how to fix them when they don't.
Question 1 of 8
Explain backpropagation. What does one backward pass cost, and what does it need to store?
Answer out loud first, then open the model answer.
View model answer
Backpropagation applies the chain rule from the loss back through the network. Each layer receives the gradient of the loss with respect to its output, computes gradients for its own parameters, and passes the gradient for its input to the layer before.
One backward pass costs roughly twice a forward pass and gives every gradient at once. It needs the activations saved during the forward pass, which is why training uses much more memory than inference; gradient checkpointing trades compute to save that memory.
What the interviewer listens for
- Chain rule, layer by layer, backwards
- All gradients in one pass, ~2× forward cost
- Stored activations → memory
- Checkpointing trade-off
Lessons: Backpropagation Through a Network, Derivatives, Gradients & the Chain Rule
Question 2 of 8
Why do gradients vanish or explode in deep networks, and how do modern networks avoid it?
Answer out loud first, then open the model answer.
View model answer
Backprop multiplies many per-layer factors. If they're mostly below 1 the gradient shrinks towards zero in early layers; above 1, it blows up. Saturating activations like sigmoid make vanishing worse.
Fixes: ReLU-family activations, careful initialisation such as He init, normalisation layers, residual connections that give gradients a direct path, and gradient clipping for explosions.
What the interviewer listens for
- Product of many factors
- Sigmoid saturation
- Init, ReLU, normalisation, residuals
- Clipping for exploding gradients
Lessons: Vanishing & Exploding Gradients, Weight Initialisation, ResNets and Skip Connections
Question 3 of 8
BatchNorm, LayerNorm or RMSNorm: what's the difference, and why do Transformers use LayerNorm or RMSNorm?
Answer out loud first, then open the model answer.
View model answer
BatchNorm normalises each feature using statistics across the batch, which works well for CNNs with large batches but behaves differently in training and inference, and struggles with small batches or variable-length sequences.
LayerNorm normalises across the features of each example or token, independent of the batch. RMSNorm is a cheaper variant that only rescales by the root mean square. Transformers use them because each token is normalised the same way regardless of batch size or sequence length.
What the interviewer listens for
- Batch statistics vs per-example statistics
- BatchNorm's train/inference and small-batch issues
- RMSNorm = cheaper LayerNorm
- Why Transformers prefer per-token norms
Lessons: BatchNorm vs LayerNorm vs RMSNorm
Question 4 of 8
Compare SGD with momentum, Adam and AdamW.
Answer out loud first, then open the model answer.
View model answer
SGD with momentum takes steps along a running average of gradients, which smooths noise and speeds up consistent directions. Adam also keeps a running average of squared gradients and scales each parameter's step by it, so it adapts per parameter and needs less tuning.
AdamW fixes how Adam handles weight decay: with plain L2 inside Adam, the decay gets scaled by the adaptive term, so weights with big gradients are barely regularised. AdamW applies decay directly to the weights, which is why it's the default for Transformers.
What the interviewer listens for
- Momentum = running average of gradients
- Adam = per-parameter adaptive steps
- AdamW decouples weight decay
- When each is used
Lessons: Gradient Descent, Momentum & Adam, Weight Decay vs L2, and AdamW
Question 5 of 8
Why do large models use learning-rate warm-up and decay? And what happens to the learning rate if you increase the batch size 8×?
Answer out loud first, then open the model answer.
View model answer
At the start, the weights are random and Adam's statistics are unreliable, so large steps can destabilise training; warm-up ramps the rate up over the first steps. Later, decaying it, for example with a cosine schedule, lets the model settle into a good minimum.
With a larger batch the gradient is less noisy, so you can take bigger steps: for SGD, the linear scaling rule says scale the learning rate roughly with the batch size, with warm-up. It breaks down at very large batches, so I'd tune it.
What the interviewer listens for
- Warm-up for early instability
- Decay to settle at the end
- Linear scaling rule with warm-up
- Limits at very large batches
Lessons: Learning-Rate Schedules, Batch Size and Learning Rate
Question 6 of 8
Your training loss won't go down. Walk me through how you debug it.
Answer out loud first, then open the model answer.
View model answer
I'd start small: can the model overfit a batch of 8 examples? If not, there's a bug. I'd check the inputs and labels by eye, the loss function and its inputs (logits vs probabilities), that gradients are non-zero and the weights actually update, and the learning rate, trying a range.
If I see NaNs, I'd look for a learning rate that's too high, division by zero or log of zero, and use gradient clipping or mixed-precision loss scaling. Once a tiny run works, I'd scale back up step by step.
What the interviewer listens for
- Overfit a tiny batch first
- Check data, labels and loss inputs
- Check gradients and updates; LR sweep
- NaN causes and fixes
Lessons: Debugging Training: Loss Won't Go Down, Debugging Training: It Trains, But Something's Off
Question 7 of 8
Your network overfits. What would you try?
Answer out loud first, then open the model answer.
View model answer
More and more varied data first: data augmentation, or mixup to blend examples. Then regularisation: weight decay, dropout, and early stopping on validation loss. Label smoothing can also reduce over-confidence.
If the dataset is small, I'd start from a pretrained model and fine-tune it rather than train from scratch, freezing early layers at first.
What the interviewer listens for
- Augmentation / mixup
- Weight decay, dropout, early stopping
- Label smoothing
- Transfer learning for small data
Lessons: Dropout, Data Augmentation and Mixup, Early Stopping and Label Smoothing, Transfer Learning and Fine-Tuning
Question 8 of 8
Why did residual connections make very deep networks trainable?
Answer out loud first, then open the model answer.
View model answer
Before ResNet, deeper plain networks had higher training error than shallower ones, so it wasn't overfitting; they were hard to optimise. A residual block outputs its input plus a learned change, so the identity path carries signals forward and gradients backward without fading.
Each block only needs to learn a small correction, and if a block isn't useful it can learn close to zero. That's why residuals are in every modern deep architecture, including Transformers.
What the interviewer listens for
- Degradation problem: worse training error, not overfitting
- Identity path for signals and gradients
- Blocks learn corrections
- Used everywhere now
Lessons: ResNets and Skip Connections