Confidence Intervals & the Bootstrap
What does 95% confidence actually promise?
The situation
You report 87% accuracy on 500 test rows, and a stakeholder asks how sure you are. The interviewer asks:
What does a 95% confidence interval actually mean, and when would you use the bootstrap?
Running example: delivery times in minutes, simulated from a right-skewed distribution (log-normal: median 30 minutes, true mean 34.0 minutes, a long tail of slow deliveries). Operations samples n deliveries and reports the average delivery time with a 95% interval.
A number without an error bar is a guess
"Average delivery time: 33.1 minutes" from 30 deliveries could easily be 36 on the next 30. "Model accuracy 87.2%" on 500 test rows has an uncertainty of about ±3 points. Interviewers and stakeholders ask "how sure are you?" and a confidence interval is the standard answer, provided you know what it means and when it's off.
The idea
Think of throwing a ring at a hidden peg. Once it lands, it either circles the peg or it doesn't. The 95% describes how good your throw is: about 95 of every 100 throws land around the peg.
Play
Try to break it. Start at n = 10 and read the coverage. Then move up to n = 100.
Draw 100 samples, build 100 intervals
Each horizontal line is one sample's 95% t-interval; the orange vertical line is the true mean (34.0 min). Red lines miss it.
With skewed data the t-interval under-covers for small n: about 92.5% at n = 10 and 93.7% at n = 30 instead of 95%, approaching 95% by n = 50–100 as the central limit theorem kicks in. Intervals also narrow like 1/√n. For heavy tails or tiny samples, report this caveat, transform the data (log), or use the bootstrap.
Now say why: why do intervals on skewed data miss more often when the sample is small?
Maths
In one breath
Any number computed from a sample (a mean, a median, an accuracy) would come out different on another sample. Its spread across samples is the sampling distribution; its standard deviation is the standard error, σ/√n for a mean. The central limit theorem says means are roughly Gaussian for large n, so mean ± 1.96 × SE gives a 95% confidence interval: a procedure that captures the true value in about 95% of repeated samples (not "95% chance the truth is in this one interval"). With small or skewed samples coverage drops below 95%. The bootstrap resamples your own data thousands of times to get intervals for any statistic, no formula needed.What you'll be able to do after this: compute a CI for a mean, state its correct interpretation, explain when it's unreliable, and build a bootstrap CI for a median or a model metric.Standard error and the CLT
Wasserman, All of Statistics §6.3.2, §7; the central limit theorem (Blitzstein & Hwang §10.3) makes x̄ approximately Gaussian for large n whatever the shape of the data.
Resample your own data to see the sampling distribution
Efron (1979) noticed that if your sample is representative, drawing samples from your sample (n draws with replacement) mimics drawing new samples from the population. Compute the statistic on each of, say, 2,000 resamples; the spread of those values estimates its sampling distribution. The simplest 95% interval takes the 2.5th and 97.5th percentiles (the percentile bootstrap); refinements such as BCa correct for bias and skew.
One sample of 50 deliveries, 2,000 bootstrap resamples. There's no simple formula for the standard error of a median or a 90th percentile; the bootstrap doesn't need one.
In practice
Error bars for models and metrics
- Test-set metrics: accuracy p on n test rows has SE ≈ √(p(1 − p)/n): 87% on 500 rows is ±2.9 points (95%). Small test sets can't separate models 1–2 points apart.
- Bootstrap a metric: resample test rows to get intervals for AUC, F1 or MAE; resample both models' predictions together to compare them fairly (paired).
- Bagging: the bootstrap is literally how random forests make each tree's training set (Random Forest).
- A/B tests: intervals for the difference between variants (next lesson).
The answer that sounds right
"There's a 95% chance the true mean is in my interval"
In the frequentist framework, the true mean is a fixed number; this particular interval either contains it or doesn't. The 95% describes the procedure: if you repeated the sampling many times, about 95% of the intervals built this way would contain the truth. The lab in the Intuition tab shows 100 such intervals at once. (A Bayesian credible interval does support the "95% probability" reading, given a prior.)
Caveats
- Both methods assume independent observations; repeated orders from the same customer need clustered or block resampling.
- The bootstrap can't invent information: with n = 10 the interval is unreliable too; extreme statistics (max, min) bootstrap badly.
- "Statistically significant" isn't "important": a tiny difference can have a tight interval with lots of data.
Say it
What does a 95% confidence interval actually mean, and when would you use the bootstrap?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
How many test examples do I need to measure accuracy to ±1 point?
Answer it out loud, then check.
95% half-width 1.96√(p(1 − p)/n) ≤ 0.01; at p ≈ 0.85, n ≈ 1.96² × 0.1275 / 0.0001 ≈ 4,900.
Follow-up 2 of 3
When does the bootstrap fail?
Answer it out loud, then check.
Tiny samples, dependent data (use block bootstrap), statistics at the extremes (max), and heavy-tailed distributions without enough data.
Follow-up 3 of 3
Confidence interval vs credible interval?
Answer it out loud, then check.
Frequentist: a procedure with 95% coverage over repetitions. Bayesian: given the prior and data, the parameter lies in the interval with 95% posterior probability.
The 30-second answer
A statistic like a sample mean varies from sample to sample; its standard deviation is the standard error, s over root n for a mean. By the central limit theorem the mean is roughly Gaussian, so mean plus or minus about two standard errors is a 95% confidence interval, meaning the procedure captures the true value in 95% of repeated samples, not that this interval has a 95% probability. With small or skewed samples, coverage falls short. The bootstrap resamples the observed data with replacement thousands of times to estimate the sampling distribution of any statistic, such as a median, AUC or F1, and the percentiles give an interval.
Go further
What not to say
- "There's a 95% chance the true value is in this interval" (for a frequentist CI).
- "Non-overlapping intervals are needed for a significant difference." Overlapping CIs can still differ significantly; test the difference directly.
Key terms and sources
Sampling distributions, standard error, the central limit theorem, what a 95% confidence interval does and doesn't mean, coverage with skewed data, and bootstrap intervals for any statistic · built from Wasserman, All of Statistics (ch. 6–8), Efron (1979), Efron & Tibshirani, An Introduction to the Bootstrap, and Blitzstein & Hwang (ch. 10) · checked 28 Oct 2026
Key terms
| Term | Meaning |
|---|---|
| Sampling distribution | How a statistic varies across repeated samples. |
| Standard error | Std dev of a statistic; s/√n for a mean. |
| Central limit theorem | Means of many independent draws are approximately Gaussian. |
| Coverage | How often the interval procedure contains the truth. |
| Bootstrap | Resampling the data with replacement to estimate sampling variability. |
Sources
- L. Wasserman, All of Statistics, Springer 2004, ch. 6–8.
- B. Efron, "Bootstrap methods: another look at the jackknife", Annals of Statistics 7(1), 1979; B. Efron & R. Tibshirani, An Introduction to the Bootstrap, Chapman & Hall 1993.
- J. Blitzstein & J. Hwang, Introduction to Probability, 2nd ed., 2019, §10.3 (central limit theorem), stat110.
Delivery times are simulated (log-normal) with fixed seeds; intervals and bootstraps are computed live in your browser.