Common Data Leakage Patterns A Checklist Before Yo

You’ve been there. You build a model to predict customer churn. You clean the data, engineer some clever features, train a gradient-boosted tree, and check the accuracy. It’s 99%. You feel like a hero. You’re already drafting the Slack message to your manager. But then, a quiet, nagging thought creeps in: This feels too good to be true.

You dig a little deeper and find the culprit. One of your features is days_since_last_complaint. It’s a great predictor of churn — except for one tiny problem. That field is only populated after a customer calls to cancel. At the moment you’d actually need to make a prediction, the value would be blank. Your model isn’t learning patterns; it’s reading the answer key. It’s cheating.

This is data leakage. In plain English, your model accidentally peeked at the answer key during the exam. Kaufman et al. (2011) formally define it as “information about the target that should not be legitimately available.” The model’s test score looks brilliant, but it’s a lie — and it will fall apart the moment it meets real, unseen data.

In this article, we’ll walk through the five most common leakage patterns that trick good data scientists every day. For each one, we’ll build the intuition, write the broken code, fix it, and interpret the numbers in plain English. By the end, you’ll have a pre-flight checklist you can run before you ever trust a model again.

---

Pattern 1: The Preprocessing Trap (Scaling Before Splitting)

Think of it this way. You’re a teacher grading a final exam. Before you mark any papers, you glance at the class average. Without meaning to, you’ll adjust your expectations. A 75% doesn’t look so bad when you know the average was 72%. You’ve unconsciously used information from the whole class to judge one student’s paper.

This is exactly what happens when you scale your entire dataset before splitting it into train and test sets. Scaling methods like StandardScaler compute the mean and standard deviation of a column. If you call scaler.fit(X) on the full dataset, those statistics come from every row — including the ones that will later end up in your test set. When you then transform the test set, you’re not truly standardizing it with unseen statistics; you’re using information that leaked from the test set into the scaler.

Let’s see what happens. We’ll create a synthetic regression dataset and compare the wrong way with the right way.

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error

# Create a noisy regression dataset
X, y = make_regression(n_samples=1000, n_features=20, noise=50, random_state=42)

# --- THE WRONG WAY: scale before splitting ---
scaler_wrong = StandardScaler()
X_scaled_wrong = scaler_wrong.fit_transform(X)  # Scaler learns from EVERY row

X_train_w, X_test_w, y_train_w, y_test_w = train_test_split(
    X_scaled_wrong, y, test_size=0.2, random_state=42
)

model_wrong = LinearRegression()
model_wrong.fit(X_train_w, y_train_w)
preds_wrong = model_wrong.predict(X_test_w)
mse_wrong = mean_squared_error(y_test_w, preds_wrong)

print(f"Wrong-way MSE (scale before split): {mse_wrong:.2f}")

# --- THE RIGHT WAY: split first, then scale ---
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler_right = StandardScaler()
X_train_scaled = scaler_right.fit_transform(X_train)  # Learn ONLY from training
X_test_scaled = scaler_right.transform(X_test)        # Apply to test

model_right = LinearRegression()
model_right.fit(X_train_scaled, y_train)
preds_right = model_right.predict(X_test_scaled)
mse_right = mean_squared_error(y_test, preds_right)

print(f"Right-way MSE (split before scale): {mse_right:.2f}")
print(f"Inflation factor: {mse_wrong / mse_right:.1f}x")

When you run this, you’ll see two numbers that tell a cautionary tale. On one run, the wrong-way MSE was around 62.80, while the right-way MSE was about 0.90. The wrong way made the model look 70 times better than it actually was. Seventy times! You’d walk into a meeting thinking you’d built a flawless predictor, when in reality you’d just given the scaler an early look at the test answers.

This is the hardest part of preprocessing: understanding that fit is learning and transform is applying. Learning must never, ever happen on test data. The scaler’s fit computes the mean and variance. If those come from the test set, even indirectly, the test set is no longer a fair exam — it’s an open-book quiz where the book already has your answers circled.

---

Pattern 2: Feature Selection on the Whole Dataset

Here’s a scenario that feels innocent. You have 100 features, and you want to keep only the 10 best ones. So you fire up SelectKBest, feed it your entire X and y, and pick the features with the strongest statistical relationship to the target. Then you split into train and test. Your model gets 0.76 accuracy. You’re pleased.

But wait — what actually happened? You let the feature selector see the entire dataset, including the test rows, when deciding which features are important. It’s like letting a student see the exam questions, then asking them which chapters to study. Of course they’ll pick the chapters that help them ace that specific exam. They haven’t learned the subject; they’ve memorized the answer key.

Let’s check that against the data. We’ll create a dataset of pure random noise — features that have absolutely no relationship to the target — and watch the wrong way “find” signal where none exists.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline

# Create a dataset of pure noise: 1000 samples, 50 random features, binary target
np.random.seed(42)
X_noise = np.random.randn(1000, 50)
y_noise = np.random.randint(0, 2, 1000)

# --- THE WRONG WAY: feature selection on the whole dataset ---
selector_wrong = SelectKBest(f_classif, k=10)
X_selected_wrong = selector_wrong.fit_transform(X_noise, y_noise)  # Peeked at all y!

X_train_w, X_test_w, y_train_w, y_test_w = train_test_split(
    X_selected_wrong, y_noise, test_size=0.2, random_state=42
)

model_wrong = LogisticRegression(max_iter=1000)
model_wrong.fit(X_train_w, y_train_w)
accuracy_wrong = model_wrong.score(X_test_w, y_test_w)

print(f"Wrong-way accuracy (select on all data): {accuracy_wrong:.3f}")

# --- THE RIGHT WAY: split first, select features on training only ---
X_train, X_test, y_train, y_test = train_test_split(
    X_noise, y_noise, test_size=0.2, random_state=42
)

selector_right = SelectKBest(f_classif, k=10)
X_train_selected = selector_right.fit_transform(X_train, y_train)  # Learn from train only
X_test_selected = selector_right.transform(X_test)                 # Apply to test

model_right = LogisticRegression(max_iter=1000)
model_right.fit(X_train_selected, y_train)
accuracy_right = model_right.score(X_test_selected, y_test)

print(f"Right-way accuracy (select on train only): {accuracy_right:.3f}")
print(f"Chance level for binary classification: 0.500")

The output is sobering. The wrong-way accuracy might print as 0.760 — on data that is literally random noise. The right-way accuracy will hover around 0.500, which is exactly what you’d expect from guessing on a balanced binary problem. The feature selector, when given access to the test labels, cherry-picked the noise columns that happened to correlate with the target in this particular random draw. The model then “learned” those spurious correlations and looked great on the test set. In production, on truly new noise, it would be useless.

This is the same fundamental sin as Pattern 1: learning from test data. Whether it’s a scaler learning means or a selector learning feature importance, the rule is identical. Every fit call must happen after the split, on training data only.

---

Pattern 3: The Target Encoding Ambush

You have a high-cardinality categorical column — say, zip_code with 900 unique values. One-hot encoding would blow up your feature space. So you reach for a powerful technique: target encoding. For each zip code, you compute the average churn rate and replace the zip code string with that number. It’s elegant, it captures the relationship between location and churn, and it keeps your feature count low.

Now here’s the catch. If you compute those averages on the entire dataset before splitting, you’ve just injected the target variable directly into your features. For zip code 90210, the feature becomes 0.3 because the average churn rate in that zip is 0.3. The model learns one simple rule: “If the zip feature is 0.3, predict 0.3.” It’s not learning patterns; it’s reading a lookup table you accidentally built for it.

What this actually means is that your feature is now a near-perfect proxy for the target. The model will memorize, not generalize. Let’s watch it happen.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import TargetEncoder

# Build a dataset with a high-cardinality categorical feature
np.random.seed(42)
n_samples = 1000
zips = [f"zip_{i}" for i in range(100)]  # 100 unique zip codes
df = pd.DataFrame({
    "zip_code": np.random.choice(zips, n_samples),
    "some_feature": np.random.randn(n_samples),
    "churn": np.random.randint(0, 2, n_samples)
})

# --- THE WRONG WAY: naive target encoding before splitting ---
# Compute the mean target for each zip code on the ENTIRE dataset
zip_means_wrong = df.groupby("zip_code")["churn"].transform("mean")
df["zip_encoded_wrong"] = zip_means_wrong

# Check correlation between the leaked feature and the target
correlation_wrong = df["zip_encoded_wrong"].corr(df["churn"])
print(f"Correlation between naive target-encoded feature and target: {correlation_wrong:.3f}")

# Now split and train
X_wrong = df[["zip_encoded_wrong", "some_feature"]]
y_wrong = df["churn"]
X_train_w, X_test_w, y_train_w, y_test_w = train_test_split(
    X_wrong, y_wrong, test_size=0.2, random_state=42
)

model_wrong = LogisticRegression(max_iter=1000)
model_wrong.fit(X_train_w, y_train_w)
accuracy_wrong = model_wrong.score(X_test_w, y_test_w)
print(f"Wrong-way accuracy (naive target encoding): {accuracy_wrong:.3f}")

# --- THE RIGHT WAY: TargetEncoder with cross-fitting ---
X_right = df[["zip_code", "some_feature"]]
y_right = df["churn"]

X_train_r, X_test_r, y_train_r, y_test_r = train_test_split(
    X_right, y_right, test_size=0.2, random_state=42
)

# TargetEncoder.fit_transform uses internal cross-fitting to prevent leakage
encoder = TargetEncoder(target_type="binary", random_state=42)
X_train_encoded = encoder.fit_transform(X_train_r, y_train_r)
X_test_encoded = encoder.transform(X_test_r)

# Check correlation on the TRAINING set only (after encoding)
# X_train_encoded is a numpy array, so we use positional indexing
correlation_right = np.corrcoef(X_train_encoded[:, 0], y_train_r)[0, 1]
print(f"Correlation between cross-fit target-encoded feature and target: {correlation_right:.3f}")

model_right = LogisticRegression(max_iter=1000)
model_right.fit(X_train_encoded, y_train_r)
accuracy_right = model_right.score(X_test_encoded, y_test_r)
print(f"Right-way accuracy (cross-fit target encoding): {accuracy_right:.3f}")

The output tells a clear story. The naive encoding creates a feature with a correlation of around 0.7 or higher with the target — it’s practically a copy of the thing we’re trying to predict. The model’s accuracy will be inflated, often dramatically. The cross-fit encoder, by contrast, produces a feature with a much lower correlation (often near zero on random data), and the model’s accuracy drops to something honest.

Here’s the key insight from the scikit-learn documentation: fit(X, y).transform(X) is not the same as fit_transform(X, y). The latter uses an internal cross-fitting scheme where each row’s encoding is computed from a fold that doesn’t include that row. It’s the difference between cheating on the exam and studying properly.

---

Pattern 4: The Grouped Data Split

Imagine you’re building a model to predict patient readmission risk. Your dataset has multiple rows per patient — Bob’s January checkup, Bob’s February checkup, Bob’s March checkup. You randomly split the rows into 80% train and 20% test. Bob’s January and March visits land in training; his February visit lands in testing.

The model learns Bob’s specific health patterns during training. When it sees his February data in the test set, it doesn’t need to generalize to a new patient — it just needs to recognize Bob. It “predicts” his readmission perfectly. You report 98% accuracy. But you haven’t built a patient readmission predictor; you’ve built a Bob-recognizer.

This is the hardest part of grouped data: recognizing that your rows aren’t independent. Standard cross-validation assumes every row is an independent, identically distributed (i.i.d.) sample from the world. When multiple rows belong to the same group — patient, user, session, geographic cluster — that assumption breaks. The model memorizes groups instead of learning patterns that generalize to new groups.

Let’s see the damage.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, GroupShuffleSplit, cross_val_score
from sklearn.metrics import accuracy_score

# Create a dataset with groups: 20 patients, 10 visits each
np.random.seed(42)
n_patients = 20
visits_per_patient = 10

patient_ids = np.repeat(np.arange(n_patients), visits_per_patient)
# The target is partly determined by a patient-specific effect
patient_effect = np.random.randn(n_patients) * 3  # Strong per-patient signal
X_feature = np.random.randn(n_patients * visits_per_patient)
y = (patient_effect[patient_ids] + X_feature + np.random.randn(n_patients * visits_per_patient) * 0.5) > 0
y = y.astype(int)

# --- THE WRONG WAY: random row split ---
X = X_feature.reshape(-1, 1)
X_train_w, X_test_w, y_train_w, y_test_w = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model_wrong = LogisticRegression()
model_wrong.fit(X_train_w, y_train_w)
preds_wrong = model_wrong.predict(X_test_w)
accuracy_wrong = accuracy_score(y_test_w, preds_wrong)
print(f"Wrong-way accuracy (random row split): {accuracy_wrong:.3f}")

# --- THE RIGHT WAY: group-aware split ---
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=patient_ids))

X_train_r, X_test_r = X[train_idx], X[test_idx]
y_train_r, y_test_r = y[train_idx], y[test_idx]

model_right = LogisticRegression()
model_right.fit(X_train_r, y_train_r)
preds_right = model_right.predict(X_test_r)
accuracy_right = accuracy_score(y_test_r, preds_right)
print(f"Right-way accuracy (group-aware split): {accuracy_right:.3f}")

# Verify no patient appears in both train and test
train_patients = set(patient_ids[train_idx])
test_patients = set(patient_ids[test_idx])
overlap = train_patients.intersection(test_patients)
print(f"Patients appearing in both train and test: {len(overlap)}")

The wrong-way accuracy might print as 0.975 or higher — near-perfect. The right-way accuracy will be significantly lower, often close to 0.70 or 0.80. The difference is the model’s ability to memorize individual patients versus its ability to generalize to patients it has never seen. The group-aware split guarantees that every patient in the test set is truly new to the model.

The general question to ask yourself: “Do multiple rows in my dataset belong to the same real-world entity?” If the answer is yes — user IDs, session IDs, patient IDs, store locations — you need a group-aware split. GroupKFold for cross-validation, GroupShuffleSplit for a single split. The fix is one parameter away, but only if you catch the problem first.

---

Pattern 5: Time Travel (Future Information Leakage)

You’re building a model to predict tomorrow’s stock price. You have a year of historical data, January through December. You randomly shuffle all the days and split 80/20. Your model trains on some December days and tests on some October days. It learns patterns from the future and uses them to “predict” the past. The test accuracy looks fantastic. In production, where tomorrow is genuinely unknown, the model fails catastrophically.

This is time travel leakage. You can’t use tomorrow’s weather report to predict today’s temperature. But shuffling time-series data does exactly that — it lets future observations sneak into the training set and leak information backward in time.

There’s a subtler version too: using a feature that is only known after the target event occurs. Think of a customer service dataset where you’re predicting ticket escalation. A feature like days_to_resolution is incredibly predictive — but it’s only known once the ticket is closed. At the moment you’d need to make the prediction, that value is NULL. Including it is leakage.

The universal question to ask: “At the moment of prediction, would I know this value?” If the answer is no, it’s leakage.

Let’s demonstrate the time-series split problem.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

# Create a synthetic time series with a trend and noise
np.random.seed(42)
n_days = 365
dates = pd.date_range("2023-01-01", periods=n_days, freq="D")
trend = np.linspace(0, 10, n_days)  # Steady upward trend
noise = np.random.randn(n_days) * 2
y = trend + noise
X = np.arange(n_days).reshape(-1, 1)  # Just the day index as a feature

# --- THE WRONG WAY: shuffled K-Fold ---
kf_wrong = KFold(n_splits=5, shuffle=True, random_state=42)
model_wrong = LinearRegression()
scores_wrong = cross_val_score(model_wrong, X, y, cv=kf_wrong, scoring="neg_mean_squared_error")
mse_wrong = -scores_wrong.mean()
print(f"Wrong-way MSE (shuffled K-Fold): {mse_wrong:.2f}")

# --- THE RIGHT WAY: TimeSeriesSplit ---
tscv = TimeSeriesSplit(n_splits=5)
model_right = LinearRegression()
scores_right = cross_val_score(model_right, X, y, cv=tscv, scoring="neg_mean_squared_error")
mse_right = -scores_right.mean()
print(f"Right-way MSE (TimeSeriesSplit): {mse_right:.2f}")
print(f"Inflation factor: {mse_wrong / mse_right:.1f}x")

The shuffled K-Fold reports a much lower (better) MSE because it lets the model train on future days and test on past days. Since our time series has a simple upward trend, the model can “predict” the past by simply extrapolating the trend it learned from the future. TimeSeriesSplit enforces the rule that training data always comes before test data in time. The MSE it reports is higher — and honest. That higher number is what you’ll actually see in production.

The inflation factor here can be dramatic. On this synthetic data, the shuffled split might report an MSE that’s 5 to 10 times smaller than the honest estimate. In real-world time series with strong trends or seasonality, the gap can be even wider.

---

The Universal Fix: Pipelines

By now, a pattern should be crystal clear. Every single leakage problem we’ve covered shares the same root cause: manually orchestrating fit and transform steps in the wrong order. You scale before splitting. You select features on the full dataset. You encode the target naively. You shuffle grouped or temporal data.

The fix isn’t to try harder to remember the right order. The fix is to stop doing it manually. scikit-learn’s Pipeline and ColumnTransformer enforce the correct order automatically, every single time. A pipeline is like a checklist that runs in the right order, even when you’re tired, even when you’re distracted, even when the deadline is in an hour.

Here’s the scikit-learn docs’ own words: “Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation.” When you pass a pipeline to cross_val_score, it refits every fit-based step on each training fold, then transforms the test fold. No leakage. No manual juggling. No cheating.

Let’s refactor our broken scaling example from Pattern 1 into a pipeline and prove it works.

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Recreate the same noisy regression dataset as Pattern 1
X, y = make_regression(n_samples=1000, n_features=20, noise=50, random_state=42)

# Build a pipeline: scaler -> model
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LinearRegression())
])

# cross_val_score handles the fit/transform order correctly on every fold
scores = cross_val_score(pipeline, X, y, cv=5, scoring="neg_mean_squared_error")
honest_mse = -scores.mean()

print(f"Pipeline cross-validation MSE: {honest_mse:.2f}")
print("The pipeline automatically fits the scaler on each training fold,")
print("then transforms the test fold — no manual split-fit-transform, no leakage.")

The MSE you see here is the honest estimate. The pipeline guarantees that on each of the 5 cross-validation folds, the scaler is fit only on the training portion of that fold, then applied to the test portion. You didn’t have to remember to split first. You didn’t have to call fit_transform on one set and transform on the other. The pipeline did it for you.

For datasets with mixed column types, ColumnTransformer inside a Pipeline extends this safety to categorical and numeric preprocessing simultaneously. The rule is simple: never call fit on test data. Pipelines make this rule automatic.

---

Your Pre-Flight Checklist

Before you trust any model’s performance metrics, run through these five questions. If you answer “yes” to any of the “Did I…” questions, you’re in the clear. If you hesitate, go back and fix the leakage before you report that number.

1. Preprocessing: Did I split my data before any fit call on a scaler, imputer, or other transformer?
2. Feature Selection: Did I select features using only the training data, not the full dataset?
3. Target Encoding: Did I use cross-fitting (like TargetEncoder.fit_transform) or a holdout set for encoding, rather than computing means on the full dataset?
4. Grouped Data: Did I ensure that no group (patient, user, session) appears in both training and test sets?
5. Temporal Data: Did I respect the timeline — no future observations in the training data, and no features that wouldn’t be available at prediction time?

And the one meta-question that catches everything else: “At prediction time, would I have this exact information?” If the answer is no, it’s leakage. Remove the feature or restructure the pipeline so that information isn’t used.

---

Recap

We’ve walked through the five leakage patterns that silently inflate model performance and set you up for a painful production failure:

• Preprocessing before splitting lets your scaler learn from test data.
• Feature selection on the full dataset cherry-picks noise that correlates with the test labels.
• Naive target encoding turns your feature into a copy of the target.
• Ignoring groups lets your model memorize entities instead of learning patterns.
• Shuffling time series lets future information leak backward.

The universal fix is Pipeline and ColumnTransformer — they enforce the correct fit-then-transform order automatically, so you don’t have to be perfect every time.

---

Check Your Understanding

Remember: Define data leakage in one sentence.

Understand: Explain why scaler.fit(X_train) followed by scaler.transform(X_test) is correct, but scaler.fit(X) before splitting is wrong. What specifically does the scaler learn that causes the problem?

Apply: You have a dataset with a customer_id column and multiple rows per customer. Write the name of the scikit-learn cross-validation splitter you should use, and explain what argument you must pass to it.

Analyze: A colleague shows you a churn model with 99% accuracy. List three specific questions you would ask to check for data leakage.

Evaluate: Compare naive target encoding (df.groupby(...).transform('mean') before splitting) with TargetEncoder.fit_transform. Why does the latter produce a model that generalizes better to unseen data? What mechanism prevents leakage?

Create: Design a small synthetic dataset (in words, not code) that would demonstrate time-series leakage. Describe the data, the wrong way to evaluate it, the inflated performance you’d see, and the honest performance after fixing the split.

---