Debugging Training: Loss Won't Go Down

Answer: 2.30 = ln 10: the model predicts all 10 classes equally and isn't learning at all

1. Cross-entropy for a uniform guess over K classes is ln K; ln 10 ≈ 2.303. So the starting value is perfect, but nothing has moved.
2. A loss that doesn't move at all usually means the weights aren't being updated: optimizer.step() not called, the optimiser built on the wrong parameters, gradients zeroed after backward instead of before, or a learning rate of 0.
3. Check: print the gradient norm and one weight before and after a step. If the weight doesn't change, it's the training loop, not the model.

Follow-up: What if the loss at step 0 were 9.5 instead? The initial logits are far too large (overconfident at random). Initialise the last layer smaller (or zero) so the first loss is near ln K.

Answer: Neither: this is a bug in the model, loss or data pipeline

1. Any reasonable network can memorise 8 examples. If it can't, more data or regularisation won't help.
2. Usual culprits: labels misaligned with inputs (shuffled separately), applying softmax before a loss that expects logits (CrossEntropyLoss already includes it), wrong label format (one-hot vs index), inputs not normalised, or a frozen layer.
3. Turn off dropout and augmentation, print a few (input, label) pairs right before the forward pass, and check the loss on a hand-computed example.

Follow-up: Once it can overfit one batch, what next? Train on the full training set with no regularisation and confirm training loss falls; only then add regularisation to close the train–validation gap.

Answer: Usually exploding updates or an invalid operation: too-high learning rate, log(0) or division by zero, fp16 overflow, or bad data

1. Log the gradient norm every step. A sharp rise just before the NaN points to exploding gradients: lower the learning rate, add warmup and gradient clipping (e.g. max norm 1.0).
2. If gradients are calm, look for invalid maths: a hand-written log(softmax(x)) (use a fused loss on logits), dividing by a std of 0, sqrt of a negative.
3. With mixed precision, fp16 can overflow: use loss scaling or bf16.
4. Check the batch that triggered it for NaN/inf inputs; torch.autograd.set_detect_anomaly(True) reports the operation that produced the NaN.

Follow-up: Why does a NaN in one batch ruin the whole model? One NaN gradient makes the weights NaN, and every later output becomes NaN. Skip or clip before the update, and checkpoint so you can resume.

Answer: It predicts “not fraud” for everything; 0.20 is just the entropy of the 5% base rate

1. Predicting the base rate p = 0.05 for every example gives loss −[0.05 ln 0.05 + 0.95 ln 0.95] ≈ 0.199: exactly where it's stuck.
2. Accuracy hides it: 95% is what “always no” scores. Look at recall, precision and PR-AUC instead (class imbalance).
3. Fixes: class weights or focal loss, re-sampling, and check the features actually carry signal. Initialising the output bias to ln(0.05/0.95) ≈ −2.94 starts the model at the base rate so it doesn't waste the first steps learning it.

Follow-up: After reweighting, recall is high but precision is terrible. Is that a bug? No: it's the threshold. Choose the operating threshold on validation data for the cost of a missed fraud vs a false alarm.

Answer: The learning rate is too high for the later phase: lower it or add a decay schedule

1. Fast early progress then noisy bouncing is the classic sign that steps overshoot the minimum.
2. Try 10× smaller, or keep the peak but add cosine or step decay so late steps are small (learning-rate schedules).
3. For a principled start, run an LR range test: increase the learning rate exponentially over a short run and pick a value somewhat below where the loss blows up.

Follow-up: Loss decreases smoothly but extremely slowly. Same cure? The opposite: the learning rate is probably too small (or gradients vanish). Check per-layer gradient norms before raising it.

Answer: BatchNorm's running statistics don't match the batch statistics used in training

1. In train mode BatchNorm normalises with each batch's own mean and variance; in eval mode it uses running averages collected during training.
2. If batches are unrepresentative (data sorted by class or source, tiny batch sizes) or the running averages are stale, the eval-mode normalisation differs and outputs shift.
3. Fix: shuffle the data, use larger batches, recompute BatchNorm statistics with a pass over the training data, or switch to GroupNorm/LayerNorm for small batches (normalisation layers).

Follow-up: What other layer behaves differently between train and eval? Dropout: but it usually makes eval loss lower, not higher.