Concept20 minBefore this:Common Probability Distributions

Expectation, Variance & Covariance

Why averaging correlated models helps less than you expect.

The situation

Your team averages 100 ETA models, expecting the error to fall tenfold, but it barely moves. The interviewer asks:

Why does averaging correlated models help less than averaging independent ones?

Running example: the food-delivery app averages several models' ETA predictions. Each model's error has a standard deviation of 4 minutes (variance 16). How much does averaging n models help, and what if their errors are correlated because they were trained on the same data?

Summarising randomness, and combining it

Losses are expectations (average error over the data distribution); ensembles average predictions; A/B tests compare means; feature correlations decide whether PCA or regularisation is needed. The handful of rules for means, variances and covariances is the toolkit behind all of these.

The idea

Think of asking friends to guess a delivery time. If they guess independently, their errors cancel out. If they all read the same forecast, they share the same mistake, and asking more friends barely helps.

Illustrative. The floor depends on how correlated the errors are.

Play

Try to break it. Set the correlation to 0.3 and raise the number of models as far as it goes. Then set the correlation to 0.

Average some models

How much does an ensemble help?

Number of models n Correlation between their errors ρ

std dev of the averaged ETA error (one model: 4.0 min)

–

floor as n → ∞ (√(ρσ²))

–

if the models were independent

–

With ρ = 0.3, averaging 20 models cuts the error spread from 4.0 to about 2.3 minutes, and no number of models gets it below 2.2. Independent models (ρ = 0) would reach 0.9. That's exactly the design goal of a random forest: bagging alone leaves trees highly correlated, so random forests also sample features at each split to lower ρ, trading a little bias per tree for a lower variance floor.

Now say why: why can't more models get you below the floor, and what would you change to lower it?

Maths

In one breath

The expectation E[X] is the long-run average, and it's linear: E[aX + bY] = aE[X] + bE[Y] even when X and Y depend on each other, which cracks many interview puzzles. Variance measures spread, Var(X) = E[X²] − E[X]². Covariance measures whether two quantities move together; correlation is covariance rescaled to −1…1. The variance of a sum includes covariance terms, so averaging n independent noisy estimates divides variance by n, but averaging correlated ones hits a floor of ρσ². That single formula explains why random forests decorrelate their trees.What you'll be able to do after this: use linearity of expectation, compute variance and covariance, explain correlation vs covariance, and derive the variance of an average of correlated models.

variance of the averagefloor ρσ²independent case σ²/nnumbers

Expectation and variance

Blitzstein & Hwang ch. 4; MML §6.4. Variance rules: Var(aX + b) = a²Var(X); adding a constant doesn't change spread.

Covariance and correlation

Zero correlation means no linear relationship: Y = X² with X symmetric around 0 has zero correlation but complete dependence. Independence implies zero covariance, not the reverse. my take

Averaging correlated estimates

The ensemble variance formula

Hastie, Tibshirani & Friedman, ESL 2nd ed., Eq. 15.1 (§15.2, random forests).

In practice

Where it shows up

Expectation and variance across ML

Losses are expectations: training minimises the average loss over data (empirical risk), an estimate of the expected loss on new data.
Bias–variance: expected error = bias² + variance + noise (Bias–Variance).
Standard errors: the mean of n independent samples has variance σ²/n, the basis of confidence intervals and A/B-test sample sizes (Confidence Intervals).
Covariance matrices: PCA, Gaussian mixtures and Mahalanobis distance all use them.
Mini-batch gradients: the gradient's variance falls like 1/batch size, which is why larger batches are less noisy.

The answer that sounds right

The trap

"Variances add, so averaging n models divides the error by n"

Var(X + Y) = Var(X) + Var(Y) only when X and Y are uncorrelated. Models trained on the same data make similar mistakes; their errors are positively correlated, and the covariance terms don't average away. With 4-minute errors and correlation 0.6, averaging 100 models leaves a standard deviation of 3.1 minutes, not 0.4.

Say it

Why does averaging correlated models help less than averaging independent ones?

Say your answer out loud

Aim for 30 seconds. It feels different from reading it, and that's the point.

Tick what you covered

Expectation is the probability-weighted average, and it's linear even for dependent variables.
Covariance measures joint movement, and correlation is its scaled version.
The variance of a sum adds the variances plus twice the covariance.
Independent errors shrink like 1/n, while correlated ones hit a floor.
That's why random forests decorrelate their trees.

The interviewer pushes further

Follow-up 1 of 3

Does zero correlation mean independence?

Answer it out loud, then check.

No. It means no linear relationship. Y = X² with symmetric X is uncorrelated with X but fully dependent. (For jointly Gaussian variables they are equivalent.)

Follow-up 2 of 3

Use linearity of expectation: expected number of fixed points in a random permutation of n?

Answer it out loud, then check.

Each position is fixed with probability 1/n; summing n indicators gives 1, without worrying about dependence.

Follow-up 3 of 3

Why does bagging reduce variance but not bias?

Answer it out loud, then check.

Averaging keeps the mean (bias) of the base model and shrinks the variance, down to the correlation floor ρσ².

The 30-second answer

Expectation is the probability-weighted average and it's linear even for dependent variables. Variance is E[X²] minus E[X]², and covariance measures joint movement; correlation is covariance scaled to between −1 and 1. Variance of a sum adds the variances plus twice the covariance, so the average of n estimates with variance σ² and pairwise correlation ρ has variance ρσ² + (1 − ρ)σ²/n: independent errors shrink like 1/n, correlated ones hit a floor. That's why random forests decorrelate trees, why standard errors fall like 1/√n, and why bigger mini-batches give less noisy gradients.

Go further

What not to say

"Variance of a sum is the sum of variances." Only when uncorrelated.
"Correlation shows causation." See Correlation vs Causation.

Key terms and sources

Expected value and its linearity, variance and standard deviation, covariance and correlation, the variance of sums and averages, and why averaging correlated models has a floor (bagging and random forests) · built from Blitzstein & Hwang, Introduction to Probability (ch. 4, 7), Hastie et al., ESL (§15.2) and Deisenroth et al., MML (§6.4) · checked 28 Oct 2026

Key terms

Term	Meaning
Expectation	Probability-weighted average.
Linearity of expectation	E of a sum = sum of Es, always.
Variance / std dev	Average squared spread / its square root.
Covariance / correlation	Joint movement / its unit-free version in [−1, 1].
Standard error	Std dev of an estimate, e.g. σ/√n for a mean.

Sources

J. Blitzstein & J. Hwang, Introduction to Probability, 2nd ed., 2019, ch. 4 and 7, stat110.
T. Hastie, R. Tibshirani & J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer 2009, §15.2 (Eq. 15.1).
M. P. Deisenroth, A. A. Faisal & C. S. Ong, Mathematics for Machine Learning, CUP 2020, §6.4.