Concept30 minBefore this:The Bias–Variance Trade-off

Cross-Validation & Data Leakage

How did a model score 86% on pure noise, and how do you make sure yours isn't?

The situation

Your cross-validation says 0.92 AUC, and production says 0.70. Nothing in the code threw an error. The interviewer asks:

What is data leakage, and how does cross-validation help you avoid it?

Running example: the food-delivery app's data team. Three simulated situations, each with a known true answer so the leak is visible: (1) predicting whether a customer will churn from 10,000 behavioural signals that are in fact pure noise (true accuracy 50%); (2) trying many model variants and keeping the one with the best validation score; (3) predicting a rider's "safe driver" badge from repeated trip records, several per rider.

The score you report is a prediction too

"Validation accuracy 86%" is a claim about data the model hasn't seen. If any information about that data leaked into training, the claim is wrong, and nothing in the code will tell you. scikit-learn: testing on the training data "is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data" cross-validation guide. Every technique on this page exists to keep the scoring data genuinely unseen.

The idea

Think of a practice exam where you've already seen the answer key. You'll score brilliantly, and the score tells you nothing. Leakage is any way the answers reach your model before it's tested, and it often looks like good hygiene.

5-fold cross-validation: each block gets one turn being held out.

Play

Try to break it. Select features on all the rows, then cross-validate, and read the accuracy. The labels here are random, so the truth is 50%. Then select inside each fold.

Leak 1: fitting steps on all rows

Feature selection outside CV finds signal in noise

5-fold CV accuracy

–

true accuracy (labels are random)

50%

200 customers, 10,000 random features, random churn labels. Pick the 25 features whose class means differ most, then classify with nearest centroid. Generated in your browser (takes a second).

Selecting on all rows: CV accuracy 86.5% on data with no signal at all. Selecting inside each training fold: 51%, the truth. With 10,000 random features, some will separate the two classes in these 200 rows by pure chance; selecting them with the validation rows included bakes that chance into the "test". scikit-learn's own version of this experiment: 0.76 accuracy when selecting on all data vs 0.5 when splitting first common pitfalls. The same leak, smaller, happens with scaling, imputation, target encoding and SMOTE fitted before the split.

Now say why: how did pure noise score so well, when no line of code looks wrong?

Maths

In one breath

An evaluation is only honest if the data used to score the model played no part in building it. k-fold cross-validation rotates which slice is held out, so every row is tested once and the estimate is less noisy than a single split. The split must copy how the model will meet new data in production: stratify rare classes, keep each group (patient, customer, device) on one side, and never let a model see the future. Leakage is any information at training time that won't exist at prediction time: preprocessing or feature selection fitted on all rows, tuning on the test set, duplicated or related rows across the split, or features that are recorded after the outcome. Leaks don't crash anything; they just give you a great score that vanishes in production.What you'll be able to do after this: choose the right splitter for a dataset, explain nested CV, spot the five common leaks, and reproduce three of them with numbers.

training datavalidation / test dataleaked, optimistic scorehonest scorenumbers

Train, validation, test, and k-fold

Swipe sideways to see the whole diagram. Fold scores are illustrative.

Start with all the labelled data.
Set aside a test set first and don't touch it until the very end. It's your one honest final exam.
Split the rest into k = 5 folds. Train on four (teal), validate on the fifth (orange), and rotate so every fold is the validation set once.
Average the five validation scores. The spread across folds tells you how much the score depends on the particular split.
Use the CV score to choose models and settings; report the test score once, for the final choice.

scikit-learn also notes leave-one-out "often results in high variance as an estimator for the test error" and costs n fits cross-validation guide.

One principle

Cross-validate the whole pipeline, split like production

Two rules cover almost everything:

Everything that learns from data goes inside the fold. Scaling, imputation, encoding, feature selection, resampling, target encoding, PCA and the model itself are one pipeline, refitted on each training fold and only applied to its validation fold. In scikit-learn that means a Pipeline passed to cross_val_score or GridSearchCV.
The validation fold must look like the future. If production sees new customers, validation must contain customers the model never saw; if production sees next month, validation must be later than training.

Kaufman, Rosset, Perlich & Stitelman define leakage as "the introduction of information about the data mining target that should not be legitimately available to mine from", and recommend a strict "learn-predict separation" ACM TKDD 2012.

Nested CV · and the sneakiest leak

Tune inside, score outside

If you choose hyperparameters with the same CV loop that produces the score you report, the score is optimistic (the same leak as tuning on test, in a milder form). Nested CV wraps the whole tuning procedure (e.g. a GridSearchCV) inside an outer loop: each outer fold is scored by a model whose settings were chosen using only the other folds. Varma & Simon (2006) showed that tuning and estimating in one CV loop gives a biased error estimate, and that the nested procedure "gives an estimate of the error that is very close to that obtained on the independent testing set" BMC Bioinformatics 7. It costs k_outer × k_inner × grid fits; with plenty of data, a single untouched test set does the same job.

Target leakage: features that encode the answer

A feature recorded after or because of the outcome: "refund issued" when predicting fraud, "account closed date" when predicting churn, "treatment given" when predicting diagnosis. CV can't catch it (it's present in every fold); only asking "would I have this value at prediction time?" does. Leaks like these are a recurring theme in data-mining competitions; Kaufman et al. survey several, including KDD Cup cases TKDD 2012. A warning sign: a single feature with suspiciously high importance, or validation accuracy far above what domain experts manage. my take

In practice

Choose the splitter that matches production

Data	Splitter	Why
i.i.d. rows, balanced	`KFold(shuffle=True)`	Plain rotation
Rare classes	`StratifiedKFold`	Each fold keeps "approximately the same percentage of samples of each target class"
Several rows per patient / customer / device	`GroupKFold`, `StratifiedGroupKFold`	"ensures that the same group is not represented in both testing and training sets"
Time-ordered data	`TimeSeriesSplit` (or a date cut-off)	Train on the past, test on the future; shuffled CV on time series gives "poor estimates of generalization error"
Tuning + honest estimate	Nested CV	Inner loop tunes, outer loop scores (see the Maths tab)

Quotes from scikit-learn's cross-validation guide docs, which adds: "While i.i.d. data is a common assumption in machine learning theory, it rarely holds in practice."

The whole recipe

Split first, by the production rule (time cut-off, groups, or stratified random), and lock the test set away.
Put every learned step in a Pipeline; never call fit on validation or test rows.
Cross-validate the pipeline (5 or 10 folds, with the matching splitter) to compare models and tune.
Audit features for timing: would each one exist at prediction time?
Evaluate once on the test set. If it's much worse than CV, look for leakage before blaming overfitting.

Scenarios

"CV said 0.92 AUC, production says 0.70"

Check, in order: preprocessing fitted on all rows, groups split across folds, time order ignored, features not available at prediction time, then genuine distribution shift.

"Which CV for churn with monthly snapshots per customer?"

Time-based split (train on earlier months, validate on later) and keep each customer on one side, or validate on customers who signed up later.

The answer that sounds right

The trap

"Prepare the data first, then cross-validate"

The most common leak looks like good hygiene: scale, impute, select features or balance classes on the full dataset, then run CV on the result. But every one of those steps learned something from the rows that will later be the validation folds. scikit-learn: "Data leakage occurs when information that would not be available at prediction time is used when building the model … The general rule is to never call fit on the test data" common pitfalls. How big can it get? Chapter 6 shows a model scoring 86% on pure noise.

Leak 2: choosing by the score you report

Try enough models and one will look good

Model variants tried

best validation accuracy (50 rows)

–

that model on 10,000 fresh rows

–

Every variant is a coin-flip model (true accuracy 50%). Each dot is one variant's validation accuracy on the same 50 rows.

With 50 validation rows, a single coin-flip model already scored 58% here by luck; the best of 10 scored 62%, of 100 68%, and of 1,000 70%. On 10,000 fresh rows, the "winner" scores 50%: it was never better, just luckier on that sample. Hyperparameter searches, feature experiments and "let me try one more thing" all do this. scikit-learn: tuning against the test set means "knowledge about the test set can 'leak' into the model and evaluation metrics no longer report on generalization performance" cross-validation guide. The cure: choose on validation/CV, and report a test set used once. With small data and heavy tuning, use nested CV.

Leak 3: related rows on both sides

The same rider in train and test

1-nearest-neighbour CV accuracy

–

true skill on new riders

50%

40 riders × 8 trips. Each rider's trips share a personal "signature" (phone, route, riding style) but the badge is random: trip features carry no real signal about it. Strip: each square is a rider; split = trips of that rider on both sides.

Random K-fold: 92.5%. The model just recognises the rider from their other trips in the training folds. GroupKFold: 47%, the honest coin flip. scikit-learn's example is the same: a model "flexible enough to learn from highly person specific features … could fail to generalize to new subjects. GroupKFold makes it possible to detect this kind of overfitting" docs. Same story for medical images (several scans per patient), sessions per user, near-duplicate documents, and augmented copies of one image.

Time leaks

If production predicts next week, validation must be later than training. Shuffled CV on time-ordered data tests on rows "artificially similar (close in time) to training samples" docs. Also watch features computed with future data: a 30-day rolling average that includes days after the prediction, "total orders this month" used on day 3, or a customer's lifetime value as of today.

Caveats

CV estimates the performance of the procedure (pipeline + tuning), not of one fitted model.
Fold scores aren't independent (training sets overlap), so the fold spread understates the true uncertainty; don't over-read small differences between models.
No splitter fixes distribution shift you didn't anticipate (a new city, a new app version). Validate on the most production-like slice you have.
Nested CV is expensive; critics (e.g. Wainer & Cawley 2021) argue it's often overkill when comparing a few model types. Use it when tuning is heavy and data is small. my take

Say it

What is data leakage, and how does cross-validation help you avoid it?

Say your answer out loud

Aim for 30 seconds. It feels different from reading it, and that's the point.

Tick what you covered

Cross-validation rotates held-out folds and averages the scores, with a test set used once at the end.
The split must mirror production: stratified, grouped or time-based.
Leakage is information at training time that wouldn't exist at prediction time.
Common forms are preprocessing on all rows, tuning on the test set and related rows on both sides.
You put every learned step in a pipeline, refitted inside each fold.
You use nested CV when tuning.