Probability Metrics
The model ranks perfectly, so why are the bids wrong?
The situation
Your ad model bids price = P(click) × value. It ranks ads perfectly, but it says 4% when the true rate is 2%, so every bid is double what it should be. The interviewer asks:
How do you check whether a model's probabilities can be trusted?
Scenarios: ad click-through prediction for bidding, insurance and credit risk pricing, medical risk scores shown to patients, weather forecasting (where the Brier score was born), and deep-network confidence. Predictions come from a simple model (illustrative) whose calibration you can break and fix live; the published results are from Guo et al. (2017).
"The model ranks perfectly, so why are we losing money on bids?"
An ad platform bids price = P(click) × value of a click. If the model says 4% when the true rate is 2%, every bid is double what it should be. The model can still rank ads perfectly, so its AUC looks great. Ranking metrics can't see this error. You need metrics that check whether the numbers themselves are right.
The idea
Think of a weather forecaster. When they say 70% chance of rain, it should rain on about 70% of those days. That's calibration. A forecaster can order days perfectly by rain risk and still get the chances wrong.
Play
Try to break it. Move the temperature away from 1 in both directions, and watch AUC, log-loss and the diagram. Then apply temperature scaling.
Reliability diagram: break the calibration, watch which metrics notice
10% positives; predictions computed over 20,000 cases. Left: reliability diagram (bars = observed frequency per confidence bin; diagonal = perfect; red = gap). Right: how many predictions fall in each bin. Overconfidence (T < 1) pushes predictions toward 0 and 1, so bars sit below the diagonal on the right. AUC never changes with T, because temperature doesn't change the order.
Now say why: why can AUC stay put while the probabilities become badly wrong?
Maths
In one breath
When decisions use the probability itself (pricing, bidding, risk scores, the cost thresholds from lesson 3), you need probabilities that are right, not just well ranked. Calibration means "of all cases predicted 70%, about 70% happen". Log-loss and the Brier score are proper scoring rules: they're best only when predicted probabilities equal the true ones, so they reward both sharpness and calibration. A reliability diagram and the expected calibration error (ECE) isolate calibration. Key insight: you can wreck calibration without changing AUC at all. Modern deep networks tend to be overconfident, and a one-parameter fix, temperature scaling, usually repairs it.What you'll be able to do after this: read a reliability diagram, explain why AUC can't detect miscalibration, choose between log-loss, Brier and ECE, and calibrate a model properly.Calibration and sharpness
Calibration
Predicted probabilities match observed frequencies: P(y = 1 | p̂ = p) = p for every p. Guo et al.: "given 100 predictions, each with confidence of 0.8, we expect that 80 should be correctly classified" Guo et al. 2017, p. 2.
Sharpness (resolution)
Predictions far from the base rate. Always predicting the base rate (say 10%) is perfectly calibrated and useless. Good probabilistic models are both calibrated and sharp.
Two proper scoring rules and one calibration error
Murphy's decomposition: reliability − resolution + uncertainty
Murphy (1973), via the Brier score article. scikit-learn's warning follows from it: "A lower Brier loss, for instance, does not necessarily mean a better calibrated model" docs; a sharper model can win on resolution while being worse calibrated. The Brier skill score 1 − BS / BSbase-rate rescales it so that predicting the base rate scores 0.
Platt scaling, isotonic regression, temperature scaling
| Method | What it fits | Use when |
|---|---|---|
| Platt scaling | A logistic regression on the model's score (2 parameters) | Binary classifiers with sigmoid-shaped distortion (SVMs, boosted trees); small validation sets |
| Temperature scaling | One parameter on the logits | Neural networks, multi-class; the default first try |
| Isotonic regression | Any monotonic step function | Large validation sets and odd-shaped distortions; can overfit small sets |
| Histogram binning | One probability per score bin | Simple, robust; can change class predictions |
In scikit-learn: CalibratedClassifierCV(method="sigmoid" | "isotonic"), fitted on data the model wasn't trained on. Logistic regression is often well calibrated already: it "is more likely to return well calibrated predictions by itself" thanks to its canonical link scikit-learn calibration docs.
In practice
Bigger, more accurate, and worse calibrated
Guo et al. found that "modern neural networks, unlike those from a decade ago, are poorly calibrated": a 110-layer ResNet on CIFAR-100 is more accurate than a 5-layer LeNet, but its average confidence is far above its accuracy. Depth, width, batch normalisation and less weight decay all made calibration worse; their explanation is that networks keep lowering training log-loss by growing more confident after they already classify the training data correctly Guo et al. 2017, §3.
ECE on CIFAR-100 (ResNet-110), before and after post-hoc calibration (↓ better)
15 bins. Matrix scaling has too many parameters for 100 classes and overfits the validation set Table 1, p. 6. Across vision and NLP datasets, uncalibrated ECE was "typically between 4 to 10%", and temperature scaling was best on the vision tasks p. 7.
Which metric, in which situation
| Scenario | Report | Why |
|---|---|---|
| Ad click prediction for bidding | Log-loss + calibration (predicted/actual CTR ratio by segment) | Bids multiply by p; a 2× bias doubles spend |
| Credit default, insurance claims pricing | Brier / log-loss + reliability by score band; AUC/Gini for ranking | Prices and capital depend on the probability level, not just the order |
| Weather forecasts ("70% chance of rain") | Brier score and its decomposition | Where it came from: Brier, Monthly Weather Review, 1950 |
| Medical risk shown to patients or doctors | Calibration plot + Brier | A "12% risk" must mean 12%; patients act on the number |
| Deep-network classifier confidence (abstain / route to a human) | ECE + log-loss; temperature-scale first | Overconfidence makes "only answer when sure" unreliable |
| Fraud alert ranking with a capacity limit | PR-AUC; calibration matters less | Only the order of the top alerts is used (unless costs use p) |
The answer that sounds right
"AUC is 0.92, the probabilities must be fine"
AUC only looks at the order of the scores. Squash or stretch every probability with any monotonic function (make them all twice as confident) and the order, and the AUC, are unchanged, while every probability is now wrong. The Play lab, in the Intuition tab, proves it: move the temperature and AUC stays fixed at its value while log-loss, Brier and ECE all move.
What to keep
- ECE depends on binning (number of bins, equal-width vs equal-count) and is noisy with few samples per bin; it also ignores sharpness. Report it with log-loss or Brier, never alone.
- Calibrate on held-out data, never the training set, and re-check after data drift.
- Resampling and class weights break calibration (lesson 3); recalibrate afterwards.
- Calibration is about groups, not individuals. "70%" being right on average says nothing about any one case, and can hide miscalibration within subgroups (check fairness-relevant segments). my take
Say it
How do you check whether a model's probabilities can be trusted?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 4
Log-loss vs Brier: which?
Answer it out loud, then check.
Both are proper. Log-loss is unbounded and harshly punishes confident mistakes, which suits systems where a 0.1% prediction for a frequent event is catastrophic, and it's what models are trained on. Brier is bounded, gentler, decomposes into reliability, resolution and uncertainty, and is common in forecasting and medicine.
Follow-up 2 of 4
Can a model have high AUC and poor calibration?
Answer it out loud, then check.
Yes. Any monotonic distortion of the scores keeps AUC identical while making probabilities wrong. Conversely, predicting the base rate for everyone is perfectly calibrated with AUC 0.5.
Follow-up 3 of 4
How does temperature scaling work, and why doesn't it change accuracy?
Answer it out loud, then check.
Divide the logits by a scalar T fitted to minimise validation log-loss. It rescales confidence but keeps the logits' order, so the argmax (and accuracy) is unchanged.
Follow-up 4 of 4
What's wrong with ECE?
Answer it out loud, then check.
It depends on the binning scheme, is biased with few samples, ignores sharpness, and only looks at the top-class confidence. Pair it with log-loss or Brier and a reliability diagram.
The 30-second answer
When decisions use the probability itself, like bidding, pricing or risk thresholds, I check calibration as well as ranking. AUC can't detect miscalibration because it only depends on order. Log-loss and Brier are proper scoring rules that reward both calibration and sharpness; a reliability diagram and ECE isolate calibration. Modern neural networks tend to be overconfident, and temperature scaling, one parameter fitted on validation data, usually fixes that without changing accuracy. For other models I'd use Platt scaling or isotonic regression on held-out data.
Go further
What not to say
- "Softmax outputs are probabilities." They sum to 1, but they're often miscalibrated.
- "Lower Brier means better calibrated." It also rewards sharpness.
- "Calibrate on the training set." Use held-out data.
Key terms and sources
Log-loss, Brier score, reliability diagrams, ECE and calibration methods · built from Guo et al. (2017), Brier (1950), Murphy (1973) and scikit-learn's docs · checked 28 Oct 2026
Glossary
| Term | Meaning |
|---|---|
| Calibration | Predicted probabilities match observed frequencies. |
| Proper scoring rule | Expected score is optimised by reporting the true probability (log-loss, Brier). |
| Log-loss / NLL / cross-entropy | −mean log probability of what happened. |
| Brier score | Mean squared error of probabilities. |
| Reliability diagram | Observed frequency vs predicted confidence, per bin. |
| ECE / MCE | Weighted average / maximum gap between confidence and accuracy over bins. |
| Temperature scaling | Divide logits by a fitted T. |
| Platt / isotonic | Sigmoid / monotonic step-function recalibration. |
What this page is built from
- C. Guo, G. Pleiss, Y. Sun & K. Weinberger, On Calibration of Modern Neural Networks, ICML 2017 (ECE, reliability diagrams, temperature scaling, Table 1).
- G. W. Brier, "Verification of forecasts expressed in terms of probability", Monthly Weather Review (1950); A. H. Murphy, "A new vector partition of the probability score" (1973); summary in Brier score.
- scikit-learn, Metrics and scoring (log-loss, Brier, strictly proper) and Probability calibration.
- A. Niculescu-Mizil & R. Caruana, "Predicting good probabilities with supervised learning", ICML 2005 (Platt vs isotonic; background).
The live lab uses an illustrative two-bell-curve score model with 10% positives; the published numbers are Guo et al.'s.