Cross Validation Strategies K Fold Stratified Grou

You’ve been there. It’s 11 p.m. on a Thursday. You’ve cleaned the data, engineered the features, and trained a model that makes you proud. You run cross-validation, and the accuracy comes back at 95%. You smile, push the model to production, and go to sleep.

The next morning, your phone is buzzing. The model that was supposed to predict customer churn is barely hitting 60% accuracy on real users. Your boss wants to know what happened. You stare at the screen, confused. The cross-validation score was great. Why did the model lie to you?

Here’s the answer, and it’s one of the hardest lessons in data science: your model wasn’t learning to predict churn. It was learning to recognize individual users who appeared in both the training and the test folds. Standard cross-validation assumes that every row in your dataset is independent—like flipping a coin over and over. But your data had ten rows for each customer. When you randomly split those rows, the same customer ended up in both training and test. The model memorized the customer, not the pattern. In validation, it looked like a genius. In production, where every customer was truly new, it was guessing.

This article is your field guide to never making that mistake again. Every specialized cross-validation strategy exists because a specific real-world dataset violates the independence assumption in a specific way. By the time you finish reading, you won’t blindly reach for KFold anymore. You’ll diagnose your data’s dependency structure—Do I have groups? Is this a time series? Are my classes rare?—and pick the right splitter on purpose.

Think of it this way: standard cross-validation is a fantastic tool, but only when your data behaves like a perfectly shuffled deck of cards. The moment your data has structure—patients, stores, dates, users—you need a splitter that respects that structure. Let’s build that intuition from the ground up.

The Baseline: Plain K-Fold and the i.i.d. Assumption

Before we can understand why specialized splitters exist, we need to understand what standard K-Fold actually does and what it assumes about your data. The assumption has a fancy name—i.i.d.—but the idea is simple.

Imagine you’re shuffling a deck of cards and dealing five hands. Each hand is a random mini-version of the full deck. Some hands might have a few more hearts, some a few more spades, but on average, each hand looks roughly like the whole deck. That’s what K-Fold cross-validation does to your data. It shuffles the rows, splits them into k equal-sized chunks (the “folds”), and then trains on k-1 folds while testing on the held-out fold. It repeats this k times, each time holding out a different fold, and averages the scores.

What does i.i.d. actually mean in plain English? “Independent” means one row’s value tells you nothing about another row’s value. If I tell you the purchase amount in row 47, you can’t guess anything about the purchase amount in row 48. “Identically distributed” means all rows come from the same underlying process. The data wasn’t collected differently halfway through, and the patterns haven’t shifted over time.

Here’s a concrete counterexample. Imagine you have ten blood-pressure measurements from the same patient. Those rows are not independent. If I tell you the patient’s first measurement was high, you’d be smart to bet their second measurement is also high. Knowing one row gives you information about another. If you use plain K-Fold on this data, measurements from the same patient will scatter across training and test folds, and your model will learn to recognize the patient, not the medical condition.

Let’s see what happens when we run plain K-Fold on some simple data. We’ll create 100 rows, split them into 5 folds, and look at which rows go where.

import numpy as np
from sklearn.model_selection import KFold

# Create a tiny dataset: 100 rows, just an ID for each row
rng = np.random.default_rng(42)
X = rng.random((100, 2))  # 100 rows, 2 features
y = rng.integers(0, 2, 100)  # binary target, roughly balanced

# Set up K-Fold: 5 splits, shuffle ON, fixed random state for reproducibility
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# Loop through one split to see what's happening
for fold_num, (train_idx, test_idx) in enumerate(kf.split(X)):
    print(f"--- Fold {fold_num + 1} ---")
    print(f"Training rows: {len(train_idx)} (indices {train_idx[:5]}...{train_idx[-5:]})")
    print(f"Test rows:     {len(test_idx)} (indices {test_idx[:5]}...{test_idx[-5:]})")
    # Quick dummy "accuracy" — just to show the pattern
    dummy_score = np.mean(y[test_idx])  # proportion of class 1 in test fold
    print(f"Proportion of class 1 in test fold: {dummy_score:.2f}")
    print()
    break  # Just show the first fold for clarity

When you run this, you’ll see something like: training rows 0–79, test rows 80–99. Because we set shuffle=True, these are random subsets of the full dataset. The test fold contains 20 rows, and the proportion of class 1 in that fold will be somewhere around 0.5—roughly matching the full dataset. That’s the i.i.d. assumption working as intended.

Why does the shuffle matter so much? Let me tell you a quick horror story. Imagine your data is sorted: all the class-A rows first, then all the class-B rows. If you run K-Fold without shuffling, some folds will contain only class A, and some only class B. Your model trains on one class and tests on the other—and reports 0% accuracy. You’ll think your model is broken when really your splitter just handed it an impossible task. Always shuffle unless you have a specific reason not to.

How many folds should you use? Five and ten are the most common defaults, and there’s a practical reason for that. With too few folds—say, two—each training set is only half your data, and your performance estimate bounces around a lot (high variance). With too many folds—say, leaving out just one row at a time—each training set is almost the full dataset, so the estimates are stable, but they’re all nearly identical, and you might not notice overfitting (potentially biased). Five or ten folds sit in the sweet spot: enough splits to get a stable average, but each test fold is large enough to give a meaningful score.

Now here’s the interesting part. Plain K-Fold works beautifully when your data is i.i.d. But the moment your data has structure—rare classes, grouped rows, or time ordering—the i.i.d. assumption breaks, and K-Fold starts lying to you. Let’s look at the first violation: class imbalance.

When Classes Are Rare: Stratified K-Fold

Imagine you’re auditing 1,000 credit-card transactions, and only 10 of them are fraudulent. That’s a 1% fraud rate. You want to build a model to catch those rare fraud cases. You reach for plain K-Fold with 5 splits.

What happens? Each fold gets 200 transactions. On average, each fold should get 2 fraud cases—but “on average” doesn’t mean “every fold.” Random chance means one fold might get zero fraud cases. Your model trains on zero fraud, tests on zero fraud, and reports 100% accuracy. You celebrate. But the model has learned exactly nothing about detecting fraud—it just learned to say “not fraud” every time, and on that particular test fold, that lazy strategy scored perfectly.

This is the “identically distributed” part of i.i.d. breaking at the fold level. The class distribution in a random fold can deviate wildly from the full dataset’s distribution when classes are rare. The fix is StratifiedKFold.

Stratified K-Fold works like dealing cards but making sure every hand gets the same proportion of face cards. It shuffles the data, splits it into k folds, but ensures that each fold has approximately the same percentage of each class as the full dataset. If your full dataset is 1% fraud, every fold will be roughly 1% fraud.

Let’s check that against the data. We’ll create an imbalanced dataset—95% class A, 5% class B—and run both K-Fold and StratifiedKFold side by side.

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold

# Create an imbalanced dataset: 1000 rows, 95% class 0, 5% class 1
rng = np.random.default_rng(42)
n_samples = 1000
X = rng.random((n_samples, 2))
y = np.concatenate([np.zeros(950), np.ones(50)]).astype(int)
# Shuffle y so the classes aren't all bunched up at the end
rng.shuffle(y)

print("Full dataset class distribution:")
print(f"  Class 0: {np.mean(y == 0):.2%}")
print(f"  Class 1: {np.mean(y == 1):.2%}")
print()

# Plain K-Fold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
print("=== Plain K-Fold (class distribution per fold) ===")
for fold_num, (train_idx, test_idx) in enumerate(kf.split(X)):
    test_y = y[test_idx]
    pct_class1 = np.mean(test_y == 1)
    print(f"Fold {fold_num + 1}: Class 1 = {pct_class1:.1%}  ({np.sum(test_y == 1)} out of {len(test_y)} rows)")

print()

# Stratified K-Fold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
print("=== Stratified K-Fold (class distribution per fold) ===")
for fold_num, (train_idx, test_idx) in enumerate(skf.split(X, y)):
    test_y = y[test_idx]
    pct_class1 = np.mean(test_y == 1)
    print(f"Fold {fold_num + 1}: Class 1 = {pct_class1:.1%}  ({np.sum(test_y == 1)} out of {len(test_y)} rows)")

Look at the output. With plain K-Fold, you might see Fold 3 with only 2% class 1, or Fold 5 with 8% class 1—the proportion bounces around. If you’re unlucky, one fold gets 0% class 1, and that fold is useless for evaluating a fraud detector. With StratifiedKFold, every fold has almost exactly 5% class 1. The model is tested on a realistic mix of classes every single time.

This is the hardest part to internalize about stratification: it only works when you have a clear classification target. What if you’re doing regression and your target variable is skewed—say, predicting house prices where most homes are $200k–$400k but a few are $$2 million? There’s no built-in StratifiedKFold for regression. You’d need to bin your target into categories (like “low,” “medium,” “high” price) and use those bins as pseudo-classes for stratification. That’s a whole separate topic, but the principle is the same: make sure every fold reflects the distribution you care about.

When Rows Are Connected: Group K-Fold

Now we arrive at the violation that burned our data scientist in the opening story: non-independent rows. This is, in my experience, the most common cross-validation mistake in industry, and it’s also the hardest to detect because the leakage is invisible until you test on truly new groups.

Let’s make it concrete. You’re building a model to detect disease from medical images. You have 1,000 images from 100 patients—10 images per patient. Each image is a row in your dataset. If you use plain K-Fold, images from Patient #7 might scatter across three different folds. Some of Patient #7’s images land in training, some in test. The model learns to recognize Patient #7’s specific anatomy—the shape of their bones, the quirks of their scan—not the disease you’re trying to detect. In validation, it scores 98%. Then you test it on a new hospital’s patients, and it drops to 60%.

What’s actually going on here? The “independent” part of i.i.d. is broken. Rows from the same patient are correlated. If I tell you the label for one of Patient #7’s images, you can make a pretty good guess about the label for Patient #7’s other images. The model exploits that correlation instead of learning the true signal.

The fix is GroupKFold. You provide a groups array—patient IDs, user IDs, store IDs, device IDs—and the splitter ensures that all rows with the same group ID are either entirely in training or entirely in test. They are never split across both. It’s like saying, “Patient #7’s images all go in Fold 3, and nowhere else.”

Let’s see this in action. We’ll create a toy dataset with 3 patients, 5 measurements each, and a binary outcome.

import numpy as np
from sklearn.model_selection import KFold, GroupKFold

# Create grouped data: 3 patients, 5 measurements each = 15 rows
rng = np.random.default_rng(42)
n_patients = 3
n_measurements = 5
X = rng.random((n_patients * n_measurements, 2))
y = rng.integers(0, 2, n_patients * n_measurements)
groups = np.repeat(np.arange(n_patients), n_measurements)  # [0,0,0,0,0, 1,1,1,1,1, 2,2,2,2,2]

print("Group IDs for each row:", groups)
print()

# Plain K-Fold — notice it doesn't even look at the groups array
kf = KFold(n_splits=3, shuffle=True, random_state=42)
print("=== Plain K-Fold (which groups appear in each fold?) ===")
for fold_num, (train_idx, test_idx) in enumerate(kf.split(X)):
    train_groups = set(groups[train_idx])
    test_groups = set(groups[test_idx])
    overlap = train_groups.intersection(test_groups)
    print(f"Fold {fold_num + 1}:")
    print(f"  Training groups: {train_groups}")
    print(f"  Test groups:     {test_groups}")
    print(f"  Groups in BOTH:  {overlap} <-- LEAKAGE!")
    print()

# GroupKFold — respects group boundaries
gkf = GroupKFold(n_splits=3)
print("=== GroupKFold (which groups appear in each fold?) ===")
for fold_num, (train_idx, test_idx) in enumerate(gkf.split(X, y, groups=groups)):
    train_groups = set(groups[train_idx])
    test_groups = set(groups[test_idx])
    overlap = train_groups.intersection(test_groups)
    print(f"Fold {fold_num + 1}:")
    print(f"  Training groups: {train_groups}")
    print(f"  Test groups:     {test_groups}")
    print(f"  Groups in BOTH:  {overlap}")
    print()

Look at the Plain K-Fold output. You’ll see something like “Groups in BOTH: {0, 1}”—Patient 0 and Patient 1 appear in both training and test within the same fold. The model can cheat by memorizing those patients. Now look at GroupKFold: the “Groups in BOTH” set is always empty. Each patient is entirely in training or entirely in test. The model must generalize to a patient it has never seen before, which is exactly what it’ll face in production.

But wait—what if you have both groups and class imbalance? Imagine you’re classifying disease from those medical images, and the disease is rare. You need to keep patients together (GroupKFold) but also ensure each fold has roughly the same proportion of diseased vs. healthy images (StratifiedKFold). This is where StratifiedGroupKFold comes in.

It’s a combined strategy that balances both constraints. The implementation in scikit-learn uses a greedy algorithm—it tries to assign whole groups to folds while keeping the class distribution as balanced as possible. It’s not guaranteed to be perfect (the optimization problem is genuinely hard), but it’s far better than ignoring one of the constraints. Here’s what it looks like:

import numpy as np
from sklearn.model_selection import StratifiedGroupKFold

# Create grouped data with class imbalance
rng = np.random.default_rng(42)
n_patients = 6
n_measurements = 5
X = rng.random((n_patients * n_measurements, 2))
# Make class 1 rare: only 2 out of 6 patients have the disease
y = np.array([0,0,0,0,0,  0,0,0,0,0,  1,1,1,1,1,  1,1,1,1,1,  0,0,0,0,0,  0,0,0,0,0])
groups = np.repeat(np.arange(n_patients), n_measurements)

sgkf = StratifiedGroupKFold(n_splits=3, shuffle=True, random_state=42)
print("=== StratifiedGroupKFold ===")
for fold_num, (train_idx, test_idx) in enumerate(sgkf.split(X, y, groups=groups)):
    test_y = y[test_idx]
    test_groups = set(groups[test_idx])
    train_groups = set(groups[train_idx])
    pct_class1 = np.mean(test_y == 1)
    print(f"Fold {fold_num + 1}:")
    print(f"  Test groups: {test_groups}")
    print(f"  Overlap with train: {test_groups.intersection(train_groups)}")
    print(f"  Class 1 in test: {pct_class1:.1%}")
    print()

Every fold keeps patients intact (no group appears in both train and test), and the class-1 proportion stays close to the overall rate. That’s the gold standard for grouped classification problems.

When Yesterday Predicts Today: Time-Series Split

Now let’s talk about the third major i.i.d. violation: time. If you’re predicting the future—stock prices, sales, weather, website traffic—shuffling your data is catastrophic.

Imagine you’re predicting tomorrow’s stock price. You have a year of daily prices. If you shuffle the rows and put some of December’s prices in the training set and some in the test set, your model learns to peek into the future. It sees December 15th’s price while “predicting” December 10th’s price. In validation, it looks like a prophet. In production, where tomorrow’s price is genuinely unknown, it’s guessing.

Both parts of i.i.d. break here. “Independent” breaks because time-series data is autocorrelated—today’s value depends on yesterday’s. “Identically distributed” can break because the data-generating process can shift over time (concept drift). Customer behavior in December 2020 is not the same as customer behavior in December 2019.

The fix is TimeSeriesSplit. It uses an expanding-window strategy: Fold 1 trains on the first 20% of the timeline and tests on the next chunk. Fold 2 trains on the first 40% and tests on the next chunk. The training set always precedes the test set in time. No shuffling, no peeking.

Let’s build a simple time-indexed dataset and see how the splits work.

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

# Create 100 days of fake daily sales data
dates = pd.date_range(start='2024-01-01', periods=100, freq='D')
sales = np.random.default_rng(42).random(100) * 100

print("First 5 rows of our time series:")
for i in range(5):
    print(f"  {dates[i].date()} : ${sales[i]:.2f}")
print()

# TimeSeriesSplit with 5 splits, no gap
tscv = TimeSeriesSplit(n_splits=5)
print("=== TimeSeriesSplit (n_splits=5, gap=0) ===")
for fold_num, (train_idx, test_idx) in enumerate(tscv.split(sales)):
    train_dates = dates[train_idx]
    test_dates = dates[test_idx]
    print(f"Fold {fold_num + 1}:")
    print(f"  Train: {train_dates[0].date()} to {train_dates[-1].date()} ({len(train_idx)} days)")
    print(f"  Test:  {test_dates[0].date()} to {test_dates[-1].date()} ({len(test_idx)} days)")
    print()

Notice the pattern: Fold 1 trains on days 1–16 and tests on days 17–33. Fold 2 trains on days 1–33 and tests on days 34–50. The training window expands to include all past data, and the test window always comes after. This respects the arrow of time.

Now here’s the secret weapon for realistic forecasting: the gap parameter. In production, you rarely predict the very next time step. You might predict sales 7 days ahead, or 30 days ahead. Without a gap, autocorrelation can leak information across the boundary—day 20 and day 21 are almost identical, so testing on day 21 after training on day 20 is easier than the real production task. A gap=7 means “skip 7 days between the last training observation and the first test observation,” simulating a 7-day forecast horizon.

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

# Same data as before
dates = pd.date_range(start='2024-01-01', periods=100, freq='D')
sales = np.random.default_rng(42).random(100) * 100

# TimeSeriesSplit WITH a gap
tscv_gap = TimeSeriesSplit(n_splits=5, gap=7)
print("=== TimeSeriesSplit (n_splits=5, gap=7) ===")
for fold_num, (train_idx, test_idx) in enumerate(tscv_gap.split(sales)):
    train_dates = dates[train_idx]
    test_dates = dates[test_idx]
    print(f"Fold {fold_num + 1}:")
    print(f"  Train: {train_dates[0].date()} to {train_dates[-1].date()} ({len(train_idx)} days)")
    print(f"  Gap:   7 days")
    print(f"  Test:  {test_dates[0].date()} to {test_dates[-1].date()} ({len(test_idx)} days)")
    print()

With gap=7, the model must predict a full week ahead. That’s much closer to the real production task, and your cross-validation score will be a much more honest estimate of production performance.

What if your old data becomes irrelevant? For example, customer behavior from three years ago might not help predict today. In that case, you might prefer a rolling window—a fixed-size training set that slides forward, dropping old data as new data arrives. Scikit-learn supports this through the max_train_size parameter:

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

# Same data as before
dates = pd.date_range(start='2024-01-01', periods=100, freq='D')
sales = np.random.default_rng(42).random(100) * 100

# Rolling window: train on at most 30 days
tscv_rolling = TimeSeriesSplit(n_splits=5, max_train_size=30)
print("=== TimeSeriesSplit with rolling window (max_train_size=30) ===")
for fold_num, (train_idx, test_idx) in enumerate(tscv_rolling.split(sales)):
    train_dates = dates[train_idx]
    test_dates = dates[test_idx]
    print(f"Fold {fold_num + 1}:")
    print(f"  Train: {train_dates[0].date()} to {train_dates[-1].date()} ({len(train_idx)} days)")
    print(f"  Test:  {test_dates[0].date()} to {test_dates[-1].date()} ({len(test_idx)} days)")
    print()

Notice how the training window stays at 30 days even as time advances—old data gets dropped. This is especially useful when you suspect concept drift.

This isn’t just scikit-learn’s opinion, by the way. Researchers like Bergmeir and Benítez (2012) have argued that blocked cross-validation—exactly what TimeSeriesSplit does—should be the standard procedure for time-series evaluation, not a single train/test split. The research consensus is clear: if your data has a time axis, respect it.

The Hidden Danger: Cross-Validation Is Not Leakage-Proof

So you’ve picked the right splitter. You’re using GroupKFold for your patient data, or TimeSeriesSplit for your forecasts. You’re feeling confident. But there’s a hidden danger that can still inflate your scores, and it has nothing to do with which splitter you chose.

Let me tell you a story from the scikit-learn documentation. A researcher standardizes their features—subtracts the mean, divides by the standard deviation—before running cross-validation. They run the CV, get an AUC of 0.76, and think they’ve found a signal. But the data is completely random. The true AUC should be 0.5—coin-flip territory. What happened?

When they standardized the features before splitting, they used the mean and standard deviation of the entire dataset, including the test folds. That mean and standard deviation leaked information about the test data into the training process. The model didn’t learn any real pattern—it learned the global scaling parameters, which contained a whisper of information about the held-out data.

Here’s the rule in plain English: any transformation that learns parameters from data—scaling, imputation, feature selection, target encoding—must be fit only on the training fold and then applied to the test fold. If you fit on the whole dataset first, your CV score is a lie.

The fix is sklearn.pipeline.Pipeline inside cross_val_score. The pipeline ensures that StandardScaler.fit() is called only on the training portion of each fold. Let’s see the difference:

import numpy as np
from sklearn.model_selection import cross_val_score, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

# Create completely random data — there is NO signal here
rng = np.random.default_rng(42)
X_random = rng.random((500, 10))
y_random = rng.integers(0, 2, 500)

kf = KFold(n_splits=5, shuffle=True, random_state=42)
model = LogisticRegression()

# LEAKY version: scale everything first, then cross-validate
scaler_leaky = StandardScaler()
X_scaled_leaky = scaler_leaky.fit_transform(X_random)  # Fit on ALL data — including test folds!
scores_leaky = cross_val_score(model, X_scaled_leaky, y_random, cv=kf, scoring='roc_auc')
print(f"Leaky CV AUC scores:     {scores_leaky}")
print(f"Leaky CV mean AUC:       {np.mean(scores_leaky):.3f}  <-- Should be ~0.5 but is inflated!")
print()

# CORRECT version: pipeline inside cross_val_score
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', LogisticRegression())
])
scores_correct = cross_val_score(pipeline, X_random, y_random, cv=kf, scoring='roc_auc')
print(f"Correct CV AUC scores:   {scores_correct}")
print(f"Correct CV mean AUC:     {np.mean(scores_correct):.3f}  <-- Honestly ~0.5, as expected")

Look at the leaky mean AUC. On completely random data, it might be 0.52 or 0.55 or even higher—it bounces around because of the leakage, and you might convince yourself there’s a pattern where none exists. The correct pipeline version reports something very close to 0.5, which is the truth: there’s no signal here.

This is the hardest part of cross-validation to internalize: choosing the right splitter prevents structural leakage. Preventing preprocessing leakage requires pipeline discipline. Both are necessary. Neither is sufficient alone.

When You’re Also Tuning Hyperparameters: Nested Cross-Validation

There’s one more layer. What if you’re using cross-validation not just to evaluate your model, but also to pick the model—tuning hyperparameters like the depth of a decision tree or the regularization strength of a logistic regression?

Imagine you’re a coach picking the best sprinter for the Olympics. You hold five time trials—that’s your inner cross-validation loop. You pick the runner with the best average time. Then you report that average time as their expected Olympic performance. But wait—you just picked the runner who got lucky in those specific trials. Maybe they had a tailwind, or the track was slightly downhill. Their true performance is probably worse than that average. You’ve been fooled by what statisticians call “optimistic selection bias.”

That’s exactly what happens when you use the same cross-validation scores for both tuning hyperparameters and reporting final performance. The more hyperparameter combinations you try, the worse the bias—you’re essentially multiple-testing your way to an inflated score. Cawley and Talbot (2010) showed this definitively: model selection using the same CV score you report as the final estimate introduces a systematic optimism.

The fix is nested cross-validation. The outer loop splits the data into K outer folds. For each outer fold, the inner loop takes the outer training set and runs a full grid search (with its own CV) to find the best hyperparameters. Then that best model is evaluated on the outer test fold—data it has never seen, even during tuning. The final reported score is the average across outer folds.

Let’s see the structure:

import numpy as np
from sklearn.model_selection import cross_val_score, KFold, GridSearchCV
from sklearn.svm import SVC

# Create some data with actual signal this time
rng = np.random.default_rng(42)
X = rng.random((200, 5))
y = (X[:, 0] + X[:, 1] > 1).astype(int)  # Simple decision boundary

# Define inner and outer CV
inner_cv = KFold(n_splits=3, shuffle=True, random_state=42)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=42)

# The model we want to tune
base_model = SVC()
param_grid = {'C': [0.1, 1, 10], 'gamma': [0.01, 0.1, 1]}

# Nested CV: GridSearchCV wrapped inside cross_val_score
# This will train 3 (inner folds) * 9 (param combos) * 5 (outer folds) = 135 models!
grid_search = GridSearchCV(base_model, param_grid, cv=inner_cv, scoring='accuracy')
nested_scores = cross_val_score(grid_search, X, y, cv=outer_cv, scoring='accuracy')

print("Nested CV outer-fold scores:", nested_scores)
print(f"Nested CV mean accuracy: {np.mean(nested_scores):.3f}")
print()
print("Each of these 5 scores comes from a model whose hyperparameters")
print("were tuned on a completely separate subset of data.")
print("The average is an honest estimate of how your tuning pipeline")
print("will perform on truly new data.")

Each of those five outer-fold scores represents a model that was tuned entirely on data that didn’t include the test fold. The average is an honest estimate. It will almost certainly be lower than what you’d get if you ran GridSearchCV on the full dataset and reported its best_score_—but it’s the truth, and the truth is what your stakeholders need.

Nested CV is computationally expensive. With 5 outer folds, 3 inner folds, and 9 hyperparameter combinations, you’re training 135 models just to get one honest performance number. For large datasets, this can be prohibitive. But if you’re reporting a performance claim in a paper, a regulatory submission, or a stakeholder presentation where trust matters, this is the gold standard.

The Decision Map: Which Splitter Do I Use?

We’ve covered a lot of ground. Let’s synthesize everything into a practical decision framework you can apply to any new dataset.

Ask yourself these questions, in order:

1. Are your rows independent? If multiple rows come from the same patient, user, device, store, or any other grouping, you need GroupKFold. If your classes are also imbalanced, reach for StratifiedGroupKFold.

2. Is your data a time series? If the order of rows matters and you’re predicting the future, use TimeSeriesSplit. Set a gap that matches your real forecast horizon. Consider max_train_size if old data becomes irrelevant.

3. Are your classes imbalanced? If you’re doing classification and some classes are rare, use StratifiedKFold. This ensures every fold has a meaningful number of examples from each class.

4. If none of the above apply, plain KFold with shuffle=True is probably fine. Your data is approximately i.i.d., and the standard tool works.

5. Are you also tuning hyperparameters AND reporting a final score? Wrap everything in nested cross-validation. The inner loop tunes; the outer loop evaluates. Never report the inner loop’s best score as your final performance estimate.

6. Regardless of splitter choice: always put preprocessing inside a Pipeline inside the CV loop. Fit on train, transform on test, every fold, no exceptions.

Here’s a quick-reference table:

Situation	i.i.d. Violation	Splitter	sklearn Class
Data is i.i.d.	None (assumed)	Shuffled K-Fold	KFold(shuffle=True)
Rare classes	Identically distributed	Stratified K-Fold	StratifiedKFold
Grouped rows (patients, users)	Independent	Group K-Fold	GroupKFold
Grouped + imbalanced classes	Both	Stratified Group K-Fold	StratifiedGroupKFold
Time series	Both + temporal order	Time Series Split	TimeSeriesSplit
Tuning + evaluation	Selection bias	Nested CV	cross_val_score(GridSearchCV(...))

The core insight is this: every specialized splitter exists because a specific i.i.d. assumption is violated. Diagnose the violation first—by looking at your data’s structure, not its values—and the right splitter will be obvious.

Recap and What’s Next

Let’s lock this in:

• Plain K-Fold assumes your data is i.i.d. Shuffle it. If your data is sorted, unshuffled K-Fold can create folds with zero examples of a class.
• Stratified K-Fold fixes class imbalance by preserving class proportions in every fold. Essential for fraud detection, rare disease diagnosis, and any problem where one class is scarce.
• Group K-Fold fixes non-independent rows by keeping all rows from the same group (patient, user, store) together. This is the most common CV mistake in industry, and the leakage is invisible until production.
• Time Series Split fixes temporal dependence by ensuring training always precedes testing. Use a gap to simulate your real forecast horizon.
• Preprocessing leakage kills even the best splitter choice. Always wrap scaling, imputation, and feature selection in a Pipeline inside your CV loop.
• Nested CV gives honest performance estimates when you’re also tuning hyperparameters. It’s expensive but it’s the truth.

In the next article, we’ll tackle what happens when your model’s performance metric itself is misleading. Accuracy, precision, recall, F1, ROC AUC—which one actually answers your business question? You’ll learn to pick metrics the same way you now pick splitters: by diagnosing what your data and your problem actually demand.

Check Your Understanding

Remember: What does i.i.d. stand for, and what does each part mean in plain English?

Understand: Why does StratifiedKFold matter most when classes are rare? Explain what happens in a plain K-Fold fold that gets zero examples of the minority class.

Apply: You have a dataset of 5,000 product reviews from 500 users (10 reviews per user). You want to predict whether a review is positive or negative. Which splitter should you use, and why?

Analyze: A colleague runs TimeSeriesSplit on daily sales data with gap=0 and reports an R² of 0.95. You change the gap to 30 and the R² drops to 0.60. What explains this drop?

Evaluate: A data scientist standardizes their features before calling cross_val_score and reports an AUC of 0.85. They did not use a Pipeline. Should you trust this number? Why or why not?

Create: Design a cross-validation strategy for a medical study with 200 patients, 5 MRI scans per patient, a rare disease (only 15 patients affected), and the goal of both tuning a neural network’s architecture and reporting a final AUC. Describe every component of your strategy.