Entropy, Cross-Entropy & KL Divergence
Why every classifier is trained on cross-entropy, and what KL adds.
The situation
Two models have the same accuracy, but one says 99% on its top guess and the other says 51%. You need a score that tells them apart. The interviewer asks:
How are entropy, cross-entropy and KL divergence related?
Running example: the food-delivery app predicts which of 4 cuisines a customer will order next: North Indian, Chinese, Pizza or Biryani. The customer's true habits are a distribution p (e.g. 50%, 25%, 15%, 10%); the model predicts a distribution q. How good is q, in bits?
Measuring how wrong a probability is
Accuracy only checks the top guess. A model that says "North Indian 51%" and one that says "North Indian 99%" are equally accurate when the customer orders North Indian, but very different when they don't. We need a score for whole probability distributions that rewards confident correct predictions and punishes confident wrong ones. Information theory gives it, and it's the loss behind almost every classifier and language model.
The idea
Think of surprise as a cost. Entropy is the surprise you can't avoid, even with perfect knowledge. Cross-entropy is the surprise you get when you bet using your model. KL divergence is the difference: the extra price of being wrong.
Play
Try to break it. Move the model's prediction from close, to uniform, to confident and wrong. Watch the KL and compare its two directions.
Move the model's distribution, watch the bits
Teal bars: p. Orange bars: q. All values in bits (log base 2).
The "close" prediction costs only about 0.03 extra bits. "Uniform" costs about 0.26 bits: it knows nothing, but never bets heavily against reality. "Confident and wrong" costs 2.03 extra bits (cross-entropy 3.77 bits), more than the entropy itself, because it put 5% on the customer's favourite cuisine. Notice KL(p ‖ q) ≠ KL(q ‖ p): for the confident-wrong case they differ (2.03 vs 2.06 bits), and dragging Biryani in p to 0 makes KL(q ‖ p) infinite while KL(p ‖ q) stays small (0.15 bits for "close").
Now say why: why does uniform cost less than confident and wrong, and why do the two KL directions differ?
Maths
In one breath
The surprise of an outcome with probability p is −log p: rare events are surprising. Entropy H(p) is the average surprise, a measure of uncertainty (bits with log₂). Cross-entropy H(p, q) is the average surprise when reality follows p but you predicted q; it's always ≥ H(p), and with a one-hot true label it is exactly the log loss used to train classifiers and language models. The gap is the KL divergence, KL(p ‖ q) = H(p, q) − H(p) ≥ 0: the extra cost of using the wrong distribution. KL is asymmetric, which matters: it's the penalty in knowledge distillation, the regulariser in VAEs, and the leash that keeps an RLHF-tuned model close to its starting point.What you'll be able to do after this: compute entropy, cross-entropy and KL for small distributions, explain why cross-entropy is the classification loss, and explain KL's asymmetry and its uses.Surprise and entropy
Shannon, "A mathematical theory of communication", 1948; Cover & Thomas ch. 2. Entropy is also the minimum average number of bits per symbol for a lossless code. For p = (0.5, 0.25, 0.15, 0.1), H ≈ 1.74 bits (a uniform choice of 4 would be 2 bits).
Cross-entropy and KL divergence
Goodfellow et al. §3.13; Cover & Thomas §2.3. Minimising cross-entropy over the training labels is maximum likelihood (MLE & MAP).
Forward vs reverse KL
- KL(p ‖ q) ("forward", what cross-entropy training minimises) is huge wherever p has mass and q doesn't, so q is pushed to cover everything p does: mass-covering, sometimes spreading too thin.
- KL(q ‖ p) ("reverse", used in variational inference) is huge wherever q has mass and p doesn't, so q prefers to sit inside one mode of p and ignore others: mode-seeking.
- KL isn't a distance (not symmetric, no triangle inequality). The Jensen–Shannon divergence is a symmetric alternative.
In practice
KL and cross-entropy across ML
- Classification and language modelling: the loss is cross-entropy with the one-hot label; for an LLM, the next-token cross-entropy; perplexity = 2cross-entropy in bits.
- Knowledge distillation: a small student matches a large teacher's softened output distribution by minimising cross-entropy / KL to the teacher's "soft targets", which carry more information than one-hot labels (Hinton, Vinyals & Dean, 2015) arXiv:1503.02531.
- RLHF: InstructGPT optimises a reward model's score with a per-token KL penalty from the supervised fine-tuned model, to stop the policy drifting into reward hacking (Ouyang et al., 2022) arXiv:2203.02155.
- Variational autoencoders: the loss includes KL(q(z | x) ‖ p(z)), keeping the latent code close to the prior.
- Decision trees: information gain is the drop in entropy after a split (Decision Trees).
- Monitoring: KL or population stability index between training and live feature distributions detects drift.
The answer that sounds right
"Use squared error between probability vectors"
Squared error (the Brier score) is a valid proper score, but as a training loss with softmax it gives weak gradients when the model is confidently wrong, exactly when learning should be fastest. Cross-entropy's −log q(correct) grows without bound as q(correct) → 0, and with softmax its gradient with respect to the logits is simply q − p: large when wrong, zero when right (see the chain rule lesson).
Say it
How are entropy, cross-entropy and KL divergence related?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
Why is cross-entropy preferred to MSE for classification?
Answer it out loud, then check.
It's the negative log-likelihood of a categorical model, heavily penalises confident mistakes, and with softmax gives the simple gradient q − p, avoiding the weak gradients MSE gives through a saturating output.
Follow-up 2 of 3
What's perplexity?
Answer it out loud, then check.
2^H (or e^H in nats) of the per-token cross-entropy: an LLM with perplexity 10 is as uncertain as choosing uniformly among 10 tokens.
Follow-up 3 of 3
Why a KL penalty in RLHF?
Answer it out loud, then check.
Maximising a learned reward alone lets the policy exploit the reward model; the KL term to the reference model keeps outputs in the region where the reward model is reliable and preserves language quality.
The 30-second answer
Entropy is the expected surprise, minus the sum of p log p, measuring uncertainty. Cross-entropy is the expected surprise when outcomes follow p but you predict q; with one-hot labels it's the log loss, and minimising it is maximum likelihood. Cross-entropy equals entropy plus the KL divergence, so KL is the extra cost of using q instead of p; it's non-negative, zero only when they match, and asymmetric: forward KL is mass-covering, reverse KL is mode-seeking. KL appears in knowledge distillation, the VAE objective, and the KL penalty that keeps RLHF policies close to the reference model.
Go further
What not to say
- "KL divergence is a distance." It's asymmetric.
- "Cross-entropy and KL give different optimal models." With fixed p they differ by the constant H(p), so they have the same minimiser.
Key terms and sources
Information as surprise, entropy, cross-entropy and why it is the classification loss, KL divergence and its asymmetry, and where KL appears in distillation, VAEs and RLHF · built from Shannon (1948), Cover & Thomas, Elements of Information Theory, Goodfellow et al., Deep Learning (§3.13), Hinton et al. (2015) and Ouyang et al. (2022) · checked 28 Oct 2026
Key terms
| Term | Meaning |
|---|---|
| Entropy | Average surprise; uncertainty of a distribution. |
| Cross-entropy | Average surprise using model q when truth is p; log loss. |
| KL divergence | Cross-entropy minus entropy; extra cost of q; asymmetric. |
| Perplexity | 2 to the cross-entropy (bits): effective number of equally likely choices. |
| Mutual information | How much knowing X reduces uncertainty about Y. |
Sources
- C. Shannon, "A mathematical theory of communication", Bell System Technical Journal 27, 1948.
- T. Cover & J. Thomas, Elements of Information Theory, 2nd ed., Wiley 2006, ch. 2.
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, MIT Press 2016, §3.13, deeplearningbook.org.
- G. Hinton, O. Vinyals & J. Dean, "Distilling the knowledge in a neural network", arXiv:1503.02531, 2015.
- L. Ouyang et al., "Training language models to follow instructions with human feedback", NeurIPS 2022, arXiv:2203.02155.