QUIZ
Information theory
8 questions on Information theory, from 5 lessons. Pick an answer in your head, then open it.
On the roadmap for: Data Science.
Question 1 of 8
What does the entropy of a distribution measure?
- A. The largest single probability anywhere in the distribution
- B. The average surprise, or bits needed per outcome
- C. The variance of the distribution
- D. The mean of the distribution
View answer
✓ B. The average surprise, or bits needed per outcome
Entropy is −Σ p log p. A fair coin has 1 bit; a coin that always lands heads has 0.
Question 2 of 8
Why do classifiers train with cross-entropy?
- A. Minimising it maximises the true labels' likelihood
- B. It's faster to compute than plain accuracy on large batches
- C. It ignores the wrong classes completely
- D. It's always zero for a well-trained model
View answer
✓ A. Minimising it maximises the true labels' likelihood
Cross-entropy = entropy of the labels + KL from the labels to the model, so minimising it pushes the model's distribution towards the true one.
Question 3 of 8
Which statement about KL divergence is true?
- A. It's symmetric: KL(P‖Q) equals KL(Q‖P)
- B. It's a true distance metric in the usual sense
- C. It's zero only when they match, and isn't symmetric
- D. It can be negative when Q is a much wider distribution than P
View answer
✓ C. It's zero only when they match, and isn't symmetric
KL(P‖Q) and KL(Q‖P) generally differ, which is why which way you compute it matters.
Question 4 of 8
A language model's perplexity on test text is 20. What does that mean?
- A. It used 20 layers to make each prediction
- B. It's as unsure as picking among 20 equal tokens
- C. It saw 20 tokens of context for each guess
- D. It got 20% of the next tokens exactly right
View answer
✓ B. It's as unsure as picking among 20 equal tokens
Perplexity is the exponential of the average negative log-likelihood per token; lower is better.
Question 5 of 8
What does entropy have to do with compression?
- A. Only images can be compressed using entropy
- B. Nothing: they are unrelated ideas
- C. Entropy is the lower limit on bits per symbol
- D. Compression increases the entropy of data
View answer
✓ C. Entropy is the lower limit on bits per symbol
A good model of the data predicts it well, and good prediction means fewer bits to encode it.
Question 6 of 8
When can mutual information find a relationship that correlation misses?
- A. Never: correlation catches every relationship
- B. For non-linear links, like y = x² around zero
- C. When both variables are constant
- D. Only for linear relationships
View answer
✓ B. For non-linear links, like y = x² around zero
Mutual information measures any dependence; correlation only measures linear association.
Lesson: Mutual Information
Question 7 of 8
Two distributions don't overlap at all. Which comparison still gives a useful, finite value that reflects how far apart they are?
- A. Wasserstein distance
- B. KL divergence
- C. Cosine similarity
- D. Jensen–Shannon divergence
View answer
✓ A. Wasserstein distance
KL becomes infinite and Jensen–Shannon saturates without overlap; Wasserstein measures how far mass must move.
Lesson: Comparing Distributions: KL, Jensen-Shannon and Wasserstein
Question 8 of 8
A decision tree split reduces label entropy from 1.0 bit to 0.6 bits on average. What is the information gain?
- A. 0.4 bits
- B. 1.6 bits
- C. 0.6 bits
- D. It can't be computed
View answer
✓ A. 0.4 bits
Information gain is the drop in entropy: 1.0 − 0.6 = 0.4 bits.
Lesson: Information Theory Practice