AIML Rounds

QUIZ

Information theory

8 questions on Information theory, from 5 lessons. Pick an answer in your head, then open it.

On the roadmap for: Data Science.

  1. Question 1 of 8

    What does the entropy of a distribution measure?

    1. A. The largest single probability anywhere in the distribution
    2. B. The average surprise, or bits needed per outcome
    3. C. The variance of the distribution
    4. D. The mean of the distribution
    View answer

    ✓ B. The average surprise, or bits needed per outcome

    Entropy is −Σ p log p. A fair coin has 1 bit; a coin that always lands heads has 0.

    Lesson: Entropy, Cross-Entropy & KL Divergence

  2. Question 2 of 8

    Why do classifiers train with cross-entropy?

    1. A. Minimising it maximises the true labels' likelihood
    2. B. It's faster to compute than plain accuracy on large batches
    3. C. It ignores the wrong classes completely
    4. D. It's always zero for a well-trained model
    View answer

    ✓ A. Minimising it maximises the true labels' likelihood

    Cross-entropy = entropy of the labels + KL from the labels to the model, so minimising it pushes the model's distribution towards the true one.

    Lesson: Entropy, Cross-Entropy & KL Divergence

  3. Question 3 of 8

    Which statement about KL divergence is true?

    1. A. It's symmetric: KL(P‖Q) equals KL(Q‖P)
    2. B. It's a true distance metric in the usual sense
    3. C. It's zero only when they match, and isn't symmetric
    4. D. It can be negative when Q is a much wider distribution than P
    View answer

    ✓ C. It's zero only when they match, and isn't symmetric

    KL(P‖Q) and KL(Q‖P) generally differ, which is why which way you compute it matters.

    Lesson: Entropy, Cross-Entropy & KL Divergence

  4. Question 4 of 8

    A language model's perplexity on test text is 20. What does that mean?

    1. A. It used 20 layers to make each prediction
    2. B. It's as unsure as picking among 20 equal tokens
    3. C. It saw 20 tokens of context for each guess
    4. D. It got 20% of the next tokens exactly right
    View answer

    ✓ B. It's as unsure as picking among 20 equal tokens

    Perplexity is the exponential of the average negative log-likelihood per token; lower is better.

    Lesson: Information, Compression and Perplexity

  5. Question 5 of 8

    What does entropy have to do with compression?

    1. A. Only images can be compressed using entropy
    2. B. Nothing: they are unrelated ideas
    3. C. Entropy is the lower limit on bits per symbol
    4. D. Compression increases the entropy of data
    View answer

    ✓ C. Entropy is the lower limit on bits per symbol

    A good model of the data predicts it well, and good prediction means fewer bits to encode it.

    Lesson: Information, Compression and Perplexity

  6. Question 6 of 8

    When can mutual information find a relationship that correlation misses?

    1. A. Never: correlation catches every relationship
    2. B. For non-linear links, like y = x² around zero
    3. C. When both variables are constant
    4. D. Only for linear relationships
    View answer

    ✓ B. For non-linear links, like y = x² around zero

    Mutual information measures any dependence; correlation only measures linear association.

    Lesson: Mutual Information

  7. Question 7 of 8

    Two distributions don't overlap at all. Which comparison still gives a useful, finite value that reflects how far apart they are?

    1. A. Wasserstein distance
    2. B. KL divergence
    3. C. Cosine similarity
    4. D. Jensen–Shannon divergence
    View answer

    ✓ A. Wasserstein distance

    KL becomes infinite and Jensen–Shannon saturates without overlap; Wasserstein measures how far mass must move.

    Lesson: Comparing Distributions: KL, Jensen-Shannon and Wasserstein

  8. Question 8 of 8

    A decision tree split reduces label entropy from 1.0 bit to 0.6 bits on average. What is the information gain?

    1. A. 0.4 bits
    2. B. 1.6 bits
    3. C. 0.6 bits
    4. D. It can't be computed
    View answer

    ✓ A. 0.4 bits

    Information gain is the drop in entropy: 1.0 − 0.6 = 0.4 bits.

    Lesson: Information Theory Practice