Python & Data Science
Machine Learning Under review

Reference: Cross-Validation

Cross-validation (CV) is the practice of splitting your data into training and evaluation partitions multiple times so that every observation is used for evaluation exactly once (or a controlled number of times). The averaged score across folds is a less biased estimate of out-of-sample performance than a single train/test split, and the spread of fold scores tells you how sensitive the model is to which data it saw. This reference catalogues the major variants, when to reach for each, and the leakage traps that turn CV from a safeguard into a mirage.

Roster of cross-validation strategies

StrategyOne-line definitionRange / constraintsWhen to useCorpus article
k-foldSplit data into k contiguous folds; train on k−1, test on the held-out one; rotate.k typically 5–10; every sample scored once.Default for i.i.d. tabular data with a reasonable sample size.What is cross-validation?)
Stratified k-foldk-fold where each fold preserves the class proportion of the target (or of a stratification key).Classification only by default; works for regression via StratifiedKSplit-style binning.Imbalanced classification; any case where rare classes must appear in each fold.What is cross-validation?)
Leave-one-out (LOO)k-fold with k = N; each fold holds out exactly one observation.Cost ∝ N model fits; deterministic for a given N.Tiny N (<100) where dropping a whole fold wastes data; low-variance but high-bias estimate.What is cross-validation?)
Leave-one-group-out (LOGO)Like LOO but the “one” is a group identifier; entire groups are held out.Requires a group label; fold sizes = group sizes.Repeated measurements per subject, per customer, per site — anything with correlated blocks.Why your model looks great in training but fails in production
GroupKFoldsklearn’s implementation of LOGO; non-overlapping group splits.Same as LOGO; folds balanced by group count, not sample count.Same as LOGO; pick by API preference.Why your model looks great in training but fails in production
Nested CVOuter loop estimates generalization; inner loop tunes hyperparameters.Cost ∝ k_outer × k_inner × tune_trials.Anytime you tune hyperparameters and report a single score — single-loop CV overfitting is real.Nested cross-validation)
Walk-forward / TimeSeriesSplitTrain on the past, test on the next block; expand the training window forward in time.No shuffling; later folds always larger.Time series, survival, log data — anything with autocorrelation or drift.Time-series cross-validation)
Train / val / test splitSingle static partition; train and validate on train+val, test exactly once.One holdout; no rotation.The “production truth” — the final unbiased check after all CV is done.Calibration curves)

The decision tree: which CV when

This is the value-add over a glossary — the choice depends on three axes you should be able to answer before writing any code:

  • How big is N?
    • N < 50 → Leave-one-out (or repeated k-fold with k = N//2); you can’t afford to hold out 20%.
    • 50 ≤ N ≤ ~10k → 5- or 10-fold is the default.
    • N > 10k → 5-fold or even holdout; marginal gains from larger k are smaller than the cost.
  • Is the target imbalanced (or are there rare strata)?
    • Yes → Stratified k-fold (or stratified group k-fold if groups are also present).
    • No → Plain k-fold is fine.
  • Are observations correlated (groups, subjects, sites, time)?
    • Repeated subjects → GroupKFold (or StratifiedGroupKFold if classes are also imbalanced).
    • Temporal ordering matters → TimeSeriesSplit / walk-forward, never shuffle.
    • Both (e.g. multiple sites observed over time) → block-respecting walk-forward with group awareness; there is no clean sklearn primitive, build it manually.
  • Are you tuning hyperparameters?
    • Yes, and you’ll report one number → Nested CV (outer for the estimate, inner for tuning).
    • No, fixed model → single-loop k-fold is enough.
  • What do you actually report?
    • The CV mean ± std is your expected generalization.
    • The final test-set score, run once, is the number you put in the paper / the slide / the deployment doc. CV does not replace it.

Quick flowchart in prose:

Imbalanced? → stratified. Grouped? → group-aware. Temporal? → walk-forward. Tuning? → wrap the whole thing in nested. Tiny N? → LOO. Otherwise → 5-fold.

The leakage trap: split before preprocessing

The single most common CV mistake is to fit imputers, scalers, encoders, feature selectors, or even the whole pipeline on the full dataset and then run CV. The model has already seen statistics (means, medians, vocabularies, target correlations) computed from the held-out fold. The CV score is now optimistically biased, sometimes dramatically so.

Counter-example. Suppose a dataset has 1000 rows, 5% of which are missing in a numeric column income. You impute with the global median (computed on all 1000 rows), then run 5-fold CV. In fold 1, the test set’s income column has been pre-filled with the median computed including the test rows themselves — and more subtly, the distribution the model sees in training is identical to what it sees in test because both came through the same imputer. The model gets an unrealistically clean view; missingness becomes invisible; the test fold is no longer a fair proxy for new data.

The fix is mechanical: the transformer and the model go inside one Pipeline, and you pass the pipeline to cross_val_score. sklearn fits the pipeline on each training fold only, then applies it to the test fold. No leakage.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, KFold

X, y = make_classification(n_samples=1000, n_informative=5, random_state=0)
rng = np.random.default_rng(0)
mask = rng.random(X.shape) < 0.05
X[mask] = np.nan  # inject ~5% missingness

# WRONG: fit imputer on all data, then CV
X_imp = SimpleImputer().fit_transform(X)
X_scaled = StandardScaler().fit_transform(X_imp)
wrong = cross_val_score(LogisticRegression(max_iter=1000),
                        X_scaled, y, cv=KFold(5, shuffle=True, random_state=0))
print("Leaky CV accuracy:", wrong.mean().round(3), "±", wrong.std().round(3))
# Leaky CV accuracy: 0.884 ± 0.018

# RIGHT: pipeline fitted inside each fold
pipe = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
    LogisticRegression(max_iter=1000),
)
clean = cross_val_score(pipe, X, y, cv=KFold(5, shuffle=True, random_state=0))
print("Clean CV accuracy:", clean.mean().round(3), "±", clean.std().round(3))
# Clean CV accuracy: 0.889 ± 0.02

On this particular toy split the two numbers are close (0.884 vs 0.889) — the fold-to-fold noise on 1000 synthetic rows is large enough to swallow the leakage effect, and the leaky version isn’t even reliably the higher one. That’s not an argument against the fix: it’s a reminder that a single small comparison proves little either way, and the mechanical rule (pipeline inside cross_val_score, always) is correct regardless of what any one toy run shows. In real pipelines with target encoding, feature selection, or PCA — where the leaked statistic is doing more work — the same mechanism routinely produces gaps of 5–15 points of AUC.

The time-series variant: why shuffle breaks autocorrelation

Standard k-fold shuffles the rows so that fold membership is independent of row order. For an i.i.d. dataset that’s correct. For a time series it is a silent disaster: the model trains on September and December to predict August. August’s outcome is autocorrelated with September’s, so the test fold is effectively a lag-1 copy of a training observation. The score reflects “how well can you predict today from a slightly shifted copy of today,” not “how well can you predict the future from the past.”

The honest alternative is walk-forward: train on [0, t), evaluate on [t, t+h), then expand the training window to [0, t+h) and evaluate on [t+h, t+2h). sklearn’s TimeSeriesSplit does this with a growing window; a “rolling” variant slides a fixed-size window instead, which matters when the series drifts and old data is more noise than signal.

Concrete failure. A model trained on Jan–Jun to predict July scores 0.93 AUC under shuffled k-fold. Re-evaluated properly — train Jan–Jun, predict July, then train Jan–Jul, predict August — the AUC collapses to 0.71. The shuffle was hiding the fact that the model had no real lead-lag signal; it was just memorizing near-duplicate rows.

import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge
from sklearn.model_selection import TimeSeriesSplit, KFold, cross_val_score

# 36 months of a noisy AR(1) series
n = 36
dates = pd.date_range("2022-01-01", periods=n, freq="MS")
y = pd.Series(0.0, index=dates)
y.iloc[0] = 1.0
for i in range(1, n):
    y.iloc[i] = 0.6 * y.iloc[i-1] + np.random.default_rng(i).normal(0, 1.0)
X = pd.DataFrame({"lag1": y.shift(1).fillna(0),
                  "month": dates.month})

# Wrong: shuffled k-fold
kf = KFold(n_splits=5, shuffle=True, random_state=0)
leaky = cross_val_score(Ridge(alpha=1.0), X, y, cv=kf, scoring="neg_mean_squared_error")
print("Shuffled CV MSE:", (-leaky.mean()).round(3))
# Shuffled CV MSE: 0.908  -- looks better than it should

# Right: walk-forward, no shuffle
tscv = TimeSeriesSplit(n_splits=5)
honest = cross_val_score(Ridge(alpha=1.0), X, y, cv=tscv, scoring="neg_mean_squared_error")
print("Walk-forward CV MSE:", (-honest.mean()).round(3))
# Walk-forward CV MSE: 1.163  -- the real out-of-sample number
The expected test error of a model $\hat f$ trained on dataset $D$ is $$ \mathbb{E}_{(x,y)\sim P}\bigl[\ell(y, \hat f(x; D))\bigr], $$ where $P$ is the joint distribution of features and target. Cross-validation estimates this by averaging over $k$ splits $D = T_i \cup V_i$: $$ \widehat{\mathrm{Err}}_{\mathrm{CV}} = \frac{1}{k}\sum_{i=1}^{k} \frac{1}{|V_i|}\sum_{j \in V_i} \ell\bigl(y_j, \hat f(x_j; T_i)\bigr). $$
Plain EnglishStatistical symbolPython equivalent
Test error of model trained on DDE[ℓ(y,f^(x;D))]\mathbb{E}[\ell(y, \hat f(x;D))]model.score(X_test, y_test)
Fold-level test loss$\frac{1}{V_i
CV point estimateErr^CV\widehat{\mathrm{Err}}_{\mathrm{CV}}cross_val_score(...).mean()
CV uncertaintyVari[Err^i]/k\sqrt{\mathrm{Var}_i[\widehat{\mathrm{Err}}_i]/k}cross_val_score(...).std() / sqrt(k)

For nested CV the outer loop estimates E[ℓ]\mathbb{E}[\ell] while the inner loop picks λ^⋆(Ti)=arg⁡min⁡λErr^(Ti;λ)\hat\lambda^\star(T_i) = \arg\min_\lambda \widehat{\mathrm{Err}}(T_i; \lambda). The reported number is the outer mean; the inner mean is a tuning artifact and should not be reported as performance.

Worked comparison: multiple CV strategies on one dataset

The cleanest way to feel the differences between strategies is to run them on the same data with the same model and look at the score spread. Below we use the breast-cancer dataset (small, binary, mild imbalance) and a single LogisticRegression. Notice how each strategy produces a different mean and a different variance.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (KFold, StratifiedKFold, LeaveOneOut,
                                     GroupKFold, RepeatedStratifiedKFold,
                                     cross_val_score)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
# Synthetic groups: pretend each tumor site produces ~10 samples
groups = np.arange(len(y)) // 10

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

def report(name, scores):
    print(f"{name:28s}  mean={scores.mean():.3f}  std={scores.std():.3f}  n={len(scores)}")

report("5-fold (plain, shuffled)",  cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0)))
report("5-fold (stratified)",       cross_val_score(model, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=0)))
report("10-fold (stratified)",      cross_val_score(model, X, y, cv=StratifiedKFold(10, shuffle=True, random_state=0)))
report("Repeated 5x2 stratified",   cross_val_score(model, X, y, cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)))
report("Leave-one-out",             cross_val_score(model, X, y, cv=LeaveOneOut()))
report("GroupKFold (site=group)",   cross_val_score(model, X, y, cv=GroupKFold(5), groups=groups))

# 5-fold (plain, shuffled)        mean=0.972  std=0.017  n=5
# 5-fold (stratified)             mean=0.979  std=0.014  n=5
# 10-fold (stratified)            mean=0.977  std=0.019  n=10
# Repeated 5x2 stratified        mean=0.977  std=0.015  n=50
# Leave-one-out                   mean=0.979  std=0.144  n=569
# GroupKFold (site=group)         mean=0.976  std=0.017  n=5

What this is telling us, in order:

  1. Plain vs stratified — close means (0.972 vs 0.979), but stratified’s std is a bit lower (0.014 vs 0.017) because no fold is missing the minority class.
  2. Larger k and repeated CV — not automatically a tighter std on a dataset this size. 10-fold’s std (0.019) is actually higher than 5-fold stratified’s (0.014) here, because each fold now holds only ~57 rows and single-fold noise dominates. What repeated CV buys isn’t a smaller point std — it’s 50 scores instead of 5 or 10, which makes downstream paired tests (bootstrap, confidence intervals) more honest even when the point estimate of variance doesn’t move.
  3. LOO — mean (0.979) ties the best k-fold variant here, but std explodes to 0.144. That’s not evidence LOO is worse: each fold’s score is a single 0/1 outcome (Bernoulli variance), so the number isn’t directly comparable to the multi-row folds above it — it means each fold is noisier and the summary is harder to interpret.
  4. GroupKFold — mean (0.976) sits just under stratified 5-fold’s (0.979); std (0.017) matches the unstratified split rather than shrinking. The “sites” here are synthetic (just contiguous row blocks), so the gap is modest — but the mechanism is real: holding out a whole group removes any correlated signal the model could lean on, and on genuinely clustered data (real patients, sensors, sites) expect a much larger gap than this toy grouping shows.
Line-by-line:
  • X, y = load_breast_cancer(return_X_y=True) — loads 569 samples, 30 numeric features, binary target (~63% malignant-benign split, mild but not extreme imbalance).
  • groups = np.arange(len(y)) // 10 — manufactures 57 synthetic “sites” of 10 samples each (the last site has 9, since 569 isn’t a multiple of 10). In a real medical dataset this would be the hospital ID; here we just want the structural effect. Watch the shape: groups needs exactly one label per row — a shorter or mispadded array here makes cross_val_score raise ValueError: Found input variables with inconsistent numbers of samples.
  • model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)) — pipeline prevents scaler leakage. The high max_iter avoids convergence warnings on some folds.
  • report(...) — utility prints mean, std, and number of scores. The n matters: 5-fold gives you 5 numbers, LOO gives you N, and RepeatedStratifiedKFold(n_repeats=10) gives you 50 — which changes how you’d compute a confidence interval.
  • GroupKFold(5) and groups=... — note the kwarg: GroupKFold needs the groups argument at cross_val_score time, not at splitter construction. A common bug is to forget the kwarg and get a “groups cannot be None” error.

Why the means differ:

  • Plain 5-fold mean (0.972) is close to stratified 5-fold’s (0.979). The marginal imbalance isn’t enough to swing the mean much on its own — it mostly shows up in the (lower) std for the stratified version.
  • GroupKFold mean (0.976) sits just under stratified 5-fold’s (0.979) — a modest gap on this synthetic grouping. Holding out a whole site means the model has never seen that site’s row-to-row correlation. On a real medical dataset with an actual hospital ID instead of row-order buckets, expect the gap to be much wider — and the group-held-out number is the one that matches your deployment, not the row-shuffled one.
  • LOO mean (0.979) ties stratified 5-fold rather than under-performing it. LOO’s training sets are nearly the full data, so there’s no meaningful bias from a smaller training set here. What actually differs is the std (0.144 vs 0.014–0.019 for the others) — a change in what’s being measured (single-row Bernoulli variance per fold), not a worse model.

Nested CV: the only honest way to tune and report

Single-loop CV tunes hyperparameters on the same folds used to score the model. With enough trials (grid points, random searches, Optuna iterations) the search finds hyperparameters that exploit the specific fold boundaries — a small but real optimistic bias. The cure is two loops: the outer loop produces k honest test scores, each computed on a model whose hyperparameters were selected using only the corresponding inner training data.

from sklearn.model_selection import KFold, GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

inner = KFold(n_splits=3, shuffle=True, random_state=0)
outer = KFold(n_splits=5, shuffle=True, random_state=1)

pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": [0.01, 0.1, 1, 10], "svc__gamma": ["scale", "auto"]}

tuner = GridSearchCV(pipe, grid, cv=inner, scoring="roc_auc", n_jobs=-1)
nested_scores = cross_val_score(tuner, X, y, cv=outer, scoring="roc_auc")
print("Nested CV AUC:", nested_scores.mean().round(3), "±", nested_scores.std().round(3))
# Nested CV AUC: 0.996 ± 0.002

# Naive single-loop version for comparison
naive = cross_val_score(tuner, X, y, cv=inner, scoring="roc_auc")
print("Single-loop AUC:", naive.mean().round(3), "±", naive.std().round(3))
# Single-loop AUC: 0.996 ± 0.003  -- ties nested to 3 decimals here

On this dataset the two numbers are indistinguishable at 3 decimals (0.996 vs 0.996) — breast-cancer with an SVC over a 4×2 grid is separable enough that there’s little room for the inner loop to overfit its own folds. That doesn’t make nested CV pointless: on noisier data, or with a wider grid, the same single-loop-vs-nested gap opens up to 1–2 points of AUC, which is the difference between “we beat the baseline” and “we don’t.” The single-loop number is what you get if you write the naive code; the nested number is what you should report, independent of how large the gap happens to be on any one dataset.

Walk-forward on a real-ish series

import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error

# 60 months of seasonal + trend + noise
t = np.arange(60)
y = 0.1*t + 5*np.sin(2*np.pi*t/12) + np.random.default_rng(0).normal(0, 1, 60)
df = pd.DataFrame({"y": y, "t": t, "month": (t % 12) + 1})

model = GradientBoostingRegressor(random_state=0)
scores, preds = [], []
horizon = 3
for start in range(24, len(df) - horizon, horizon):
    train = df.iloc[:start]
    test  = df.iloc[start:start+horizon]
    model.fit(train[["t", "month"]], train["y"])
    p = model.predict(test[["t", "month"]])
    scores.append(mean_squared_error(test["y"], p))
print("Walk-forward MSE:", np.mean(scores).round(3))
# Walk-forward MSE: 1.951

The first 24 months are the burn-in; from there the training window only grows. This is the rolling-origin scheme from Bergmeir & Hyndman (2014). Variants include a rolling window (drop old data as you go) which is better for drifting series, and a purged window (drop the h observations immediately before the test block to avoid direct leakage from correlated neighbors).

Edge cases and common mistakes

Leakage traps: shuffling, preprocessing, and re-tuning

Mistake 1: shuffling a time series. Already covered above; the score will look great and the model will be useless. Symptom: training AUC ≈ CV AUC ≫ test AUC.

Mistake 2: fitting the scaler / imputer / encoder outside the pipeline. Use make_pipeline or Pipeline. Anything computed on all rows — including PCA loadings, target encodings, vocabulary sets, kmeans cluster centers used as features — is a leakage vector.

Mistake 3: tuning on the test fold. If you call GridSearchCV and then cross_val_score on the same data with the same folds, the second call is a re-evaluation of already-tuned hyperparameters on the same splits. It’s not nested CV — it’s just running the same CV twice. Wrap the whole GridSearchCV inside the outer cross_val_score.

Reporting mistakes: which number to trust

Mistake 4: reporting the inner-loop score from nested CV. The inner mean is a tuning artifact; it’s the best achievable on the inner folds, biased upward. Only the outer scores are honest.

Mistake 5: choosing k by looking at the CV score. “5-fold gave 0.91, 10-fold gave 0.92, let me use 10.” The difference is sampling noise, not a real signal. Pick k on bias-variance grounds, not on the score it produces.

Mistake 6: ignoring the std. A mean of 0.90 ± 0.02 and a mean of 0.90 ± 0.10 are very different models. Report both, or report a confidence interval. With only 5 folds the std itself has high uncertainty; RepeatedStratifiedKFold gives you a denser sampling of the variance.

Splitter API gotchas

Mistake 7: GroupKFold without passing groups=. The splitter raises at fit time. Pass groups= to cross_val_score, not to the splitter constructor.

Mistake 8: stratifying on the wrong variable. StratifiedKFold stratifies on y. If your rare class is in a feature (e.g. minority group of a sensitive attribute), you need a custom splitter or StratifiedGroupKFold with combined keys.

Mistake 9: LOO on classification with a hard decision function. LOO test sets have N=1, so accuracy is 0 or 1 per fold. The std looks alarming but is just Bernoulli noise; the mean is still interpretable but paired comparisons become noisy. Prefer LOO for regression or for neg_log_loss-style probabilistic scores, where the per-fold score is continuous.

Final-number and dispatch mistakes

Mistake 10: using CV as the final number. CV is an estimate of generalization. The deployed number is the test set, run once, on data the model has never touched — and ideally collected after the model was frozen. CV alone does not protect you from distribution shift between training and production; that’s what the held-out test set and ongoing monitoring are for.

Mistake 11: time-series CV with overlapping test windows. If you don’t purge the gap between train and test in walk-forward, lag-1 features leak. For a horizon h, drop the h observations immediately before the test block from training.

Mistake 12: using cv=5 (int) in cross_val_score for classification. sklearn auto-dispatches to StratifiedKFold for classifiers and KFold for regressors — but only if you pass an int. If you pass cv=KFold(5) you get plain (un-stratified) folds even on classification, which can produce a fold with zero positive examples.

Cross-references

Further reading

  • Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B, 36(2), 111–147. The original LOO/CV formulation.
  • Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. IJCAI. The classic experimental comparison of k-fold, leave-one-out, and bootstrap; argues for 10-fold stratified.
  • Bergmeir, C., & Hyndman, R. J. (2014). Note on the invalidity of cross-validation for evaluating autoregressive time series prediction. Computational Statistics & Data Analysis. Why ordinary CV is biased for time series and what to use instead.
  • Kaggle: Porto Seguro’s Safe Driver Prediction — a canonical imbalanced-tabular competition where the public leaderboard divergence between plain and stratified CV is the entire game. Compare your local stratified CV to a plain 5-fold and watch the ranking shift.
  • scikit-learn user guide, Cross-validation: evaluating estimator performance and Tuning the hyper-parameters of an estimator — the canonical API reference for KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit, and the nested-CV idiom.

Related articles

Looking for something else?

Search every article by title, summary or topic.