Data Leakage: Why Your 'Perfect' Model is Probably Lying to You

The Exam Paper with the Answers Written on the Back

Imagine sitting in a classroom, about to take a difficult math exam. You turn the page over, and there it is: the answers to every question, printed in light gray text on the back of the paper. You breeze through the test, circle all the right answers, and hand it in. You get 100%.

But do you actually know math? No. Drop into a real engineering job the next day and you’d fail immediately—those “answers on the back” don’t exist in the real world.

That’s the frustration of Data Leakage. It’s the most common reason machine learning projects fail after they leave the lab. Your model looks brilliant during training, then the moment it goes live, the performance vanishes. The model saw information it shouldn’t have had access to. It cheated, and you didn’t even realize you were giving it the answers.

Building a ‘Too Good to Be True’ Model

We’ll build a model to predict Customer Churn — whether a customer will cancel their subscription. To keep it realistic, the features include things like how long they’ve been a member and how much they spend.

Here’s the catch. We’ll also include a feature called refund_amount. Think about that for a second. If a customer gets a “Cancellation Refund,” that information only exists after they’ve decided to leave. Including it in training data hands the model a crystal ball into the future.

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# 1. Create a synthetic dataset
np.random.seed(42)
n_samples = 1000

data = pd.DataFrame({
    'tenure_months': np.random.randint(1, 60, n_samples),
    'monthly_spend': np.random.uniform(20, 150, n_samples),
    'emails_opened': np.random.randint(0, 50, n_samples),
    'churn': np.random.choice([0, 1], n_samples)
})

# Here is the 'Leaky' feature: 
# If they churned, we give them a refund. If not, refund is 0.
# In the real world, we wouldn't know this at the time of prediction!
data['refund_amount'] = data['churn'] * np.random.uniform(10, 50, n_samples)
data.loc[data['churn'] == 0, 'refund_amount'] = 0

# 2. Split the data
X = data.drop('churn', axis=1)
y = data['churn']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Train a simple model
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

# 4. Check the score
predictions = model.predict(X_test)
print(f"Model Accuracy: {accuracy_score(y_test, predictions) * 100:.2f}%")

The 99% Accuracy Trap: Interpreting the Numbers

Run the code above and you’ll see an accuracy score of 100.00% (or very close to it).

A 100% accuracy score isn’t a reason to celebrate. It’s a reason to panic. Your model hasn’t learned the complex behavior of your customers — it has simply figured out that if refund_amount > 0, the answer is always “Churned.” This is the hardest part of being a data scientist: resisting the urge to show your boss that perfect score. Let’s look at which features the model thought were most important.

import matplotlib.pyplot as plt

importances = model.feature_importances_
feature_names = X.columns

# Let's see what the model relied on
for name, importance in zip(feature_names, importances):
    print(f"{name}: {importance:.4f}")

# The output will show 'refund_amount' dominating everything else.

These numbers tell us refund_amount has an importance score near 1.0, while tenure_months and monthly_spend sit near 0. The model is telling us: “I don’t care about customer behavior; I’m just looking at the refund column.” That’s useless for predicting future churn, because you won’t know the refund amount until the customer is already gone.

The Two Flavors of Leakage: Target and Train-Test

So there are two ways this usually happens.

1. Target Leakage This is what we just did. Your predictors include data that is a direct result of the target. Predicting whether a patient has pneumonia? If your features include “Took Pneumonia Medication,” that’s target leakage. The medication is a result of the diagnosis, not a cause.

2. Train-Test Contamination More subtle. This is when information from your test set “leaks” into your training process during data preparation.

Here’s the thing: if you calculate the average monthly spend of all customers and use that to fill in missing values before you split your data, your training set now knows the test set’s average. It has “peeked” at the future.

# THE WRONG WAY (Contamination)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# Peeking: fitting on the WHOLE dataset before splitting
X_scaled = scaler.fit_transform(X) 
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

# THE RIGHT WAY (Clean)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Only learn from training data
X_test_scaled = scaler.transform(X_test)      # Apply learned rules to test data

How to Leak-Proof Your Workflow

How do we keep these leaks out? Three rules help.

Rule 1: The Timeline Test. Before you include a feature, ask: “Would I know this value at the exact second I need to make a prediction?” If the answer is “No” or “Only sometimes,” drop it.

Rule 2: Use Scikit-Learn Pipelines. Pipelines act as a firewall. They keep your scaling, math, and modeling in the correct order, and they stop the test set from ever talking to the training set.

from sklearn.pipeline import Pipeline

# This is the gold standard for safety
safe_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier())
])

# The pipeline handles the 'fit' on train and 'transform' on test automatically!
safe_pipeline.fit(X_train.drop('refund_amount', axis=1), y_train)

Rule 3: Check your correlations. If a single feature has a correlation of 0.95 or higher with your target, don’t assume you’ve found a miracle variable. Assume it’s a ghost from the future and investigate it immediately.

Summary: Staying Paranoid is a Superpower

A healthy dose of skepticism will save your career in data science. Remember:

• If a model looks perfect, it’s almost certainly broken.
• Leakage doesn’t throw an error code. It just gives you a wrong answer that looks right.
• Always split your data before any math or preprocessing.
• Use Pipelines to keep training and testing logic strictly separated.

So you know how to keep your models honest. In the next guide, we’ll look at Data Drift—what happens when the world changes but your model stays the same.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the difference between Target Leakage and Train-Test Contamination?

Understand In your own words, explain why refund_amount is a case of target leakage, using the article’s “answers only exist after the decision” framing.

Apply Using the article’s “Timeline Test” (“would I know this value at the exact second I need to make a prediction?”), evaluate a proposed feature for the churn model: days_since_last_support_ticket. Would this pass or fail the Timeline Test, and why?

Analyze The article’s “WRONG WAY” code fits StandardScaler on the full dataset before splitting. Walk through exactly what numeric information about the test set leaks into the training process through the scaler’s learned mean_ and scale_ parameters, even though the test set’s target values (y_test) are never directly touched.

Evaluate The article’s Rule 3 says a feature correlated 0.95+ with the target should be treated as “a ghost from the future” and investigated. Critique this as an automatic red flag: can you think of a case where a very high correlation is legitimate signal rather than leakage, and how would you tell the difference before throwing the feature out?

Create Design a leak-proof feature-engineering checklist for a new prediction task: predicting whether a job applicant will be hired, using features scraped from their application (resume length, years of experience, referral source, interview_score). Identify which feature(s) would fail the Timeline Test and explain why, using the article’s medication/diagnosis analogy as a model.

---