How to Detect and Handle Data Drift in Production Models

1. The Seasonal Shift: Why Your Model Stops Working

Last time, Dev deployed a new recommender version and watched conversion rates plummet — users were getting snow boots in July. He reached for the previous version and realized he had experiment tracking but no way to revert. So he built proper model versioning: every model saved with metadata (version, metrics, data hash, git commit, environment), registered in MLflow, and rollbacks reduced to a one-line config change that swaps the active version in seconds. But a rollback only fixes a bad deploy. What happens when the model goes wrong and nobody pushed anything?

Dev’s product-recommendation engine was humming along. Click-through rates were solid, conversions were up, and the model had been serving ten million users without a hiccup for weeks. He’d trained it on summer shopping data: browsing patterns from June, July, and August. It was brilliant.

Then the holiday season hits. Click-through rates start dropping. Users are browsing gifts, winter coats, and home goods — but the recommender is still pushing the summer products it learned to love. The model is still running the same code, the servers are healthy, and the database isn’t down.

The world changed, but the model stayed frozen in summer. This is data drift. The input distribution (what users browse, when they shop, what categories they visit) shifted so much that the patterns the model learned are no longer true.

Here’s the thing: the model is a map. If the landscape changes—a new road is built or a bridge collapses—the map isn’t “broken,” but it is no longer useful. In production, silent failures are worse than loud ones. A crashed server triggers an alarm; a drifting model just quietly gives you wrong answers while you think everything is fine.

2. Data Drift vs. Model Drift: What’s Actually Different?

Before fixing it, we need to know what we’re looking at. Two main culprits cause a model to start failing.

• Data Drift (Feature Drift): The inputs changed. In math terms, this is $P(X)$ changing. Maybe you started marketing to a younger demographic, so the average age of your users dropped from 45 to 25. The relationship between age and buying might be the same, but the “mix” of people is different.
• Model Drift (Concept Drift): The relationship between input and output changed. This is $P(Y|X)$ shifting. During a global pandemic, “high travel history” might have shifted from predicting “frequent flyer” to predicting “quarantine risk.” The input is the same, but the meaning has flipped.

Here’s the catch: Data drift is usually fixable by feeding the model newer data. Model drift is harder because your underlying assumptions about the world no longer hold.

3. Spotting Drift: Four Detection Strategies

How do we know if our data is drifting? We can’t just wait for customers to complain. We need statistical smoke detectors.

1. Compare Distributions: Run the Kolmogorov-Smirnov (KS) test to check whether two datasets share the same distribution.
2. Track Summary Stats: If a feature’s mean or standard deviation shifts noticeably, something is up.
3. Watch Confidence: When your model’s probability score for its top choice starts dropping, it’s seeing things it doesn’t recognize.
4. Windowing: Compare a “Reference Window” (your training data) against a “Sliding Window” (the last 24 hours of production data).

Here’s what drift detection looks like in Python:

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp

# Create 'Training' data (Normal distribution)
np.random.seed(42)
train_feature = np.random.normal(loc=100, scale=15, size=1000)

# Create 'Production' data (Shifted slightly higher)
prod_feature = np.random.normal(loc=110, scale=15, size=1000)

# Run the KS Test
statistic, p_value = ks_2samp(train_feature, prod_feature)

print(f"KS Statistic: {statistic:.4f}")
print(f"P-Value: {p_value:.4f}")

if p_value < 0.05:
    print("Drift Detected: The distributions are significantly different.")
else:
    print("No Drift: The distributions look similar.")

This is the drift detector Dev wishes he’d been running before the holiday shopping season blindsided his recommender — a two-sample KS test that compares the feature distribution the model was trained on against what it’s seeing in production today.

• import numpy as np — NumPy, the numerical array library. The training and production data here are synthetic NumPy arrays; in Dev’s real pipeline, train_feature would be a column from his training dataset (say, average session duration) and prod_feature would be the same column from yesterday’s production logs.
• import pandas as pd — pandas, imported here for later use (the DriftMonitor class in Section 4 wraps DataFrames). It’s not used in this block but is needed downstream.
• from scipy.stats import ks_2samp — the two-sample Kolmogorov-Smirnov test from SciPy’s stats module. This is the workhorse: it compares two empirical distributions and returns a statistic (how different they are) and a p-value (how likely that difference is due to chance).
• np.random.seed(42) — fixes the random seed so the synthetic data is reproducible. In production, Dev wouldn’t need this — he’d load real data.
• train_feature = np.random.normal(loc=100, scale=15, size=1000) — generates 1000 samples from a normal distribution with mean 100 and standard deviation 15. This simulates a feature from Dev’s summer training set.
• prod_feature = np.random.normal(loc=110, scale=15, size=1000) — generates 1000 samples from the same shape distribution but with mean 110. The mean has shifted by 10 units — the kind of shift Dev sees when holiday shoppers behave differently from summer shoppers.
• statistic, p_value = ks_2samp(train_feature, prod_feature) — runs the KS test and unpacks both return values. statistic is the maximum gap between the two cumulative distribution functions (a number between 0 and 1). p_value is the probability that two samples this different could have come from the same underlying distribution.
• print(f"KS Statistic: {statistic:.4f}") — prints the KS statistic to 4 decimal places. The seeded run above prints 0.3110, meaning the two CDFs differ by up to about 31% at some point along the x-axis.
• print(f"P-Value: {p_value:.4f}") — prints the p-value. A near-zero p-value means it’s virtually impossible these two samples came from the same distribution.
• if p_value < 0.05: — the standard 0.05 significance threshold. If the p-value falls below it, the code declares drift. The 0.05 threshold means “there’s less than a 5% chance this difference is random noise.”
• print("Drift Detected: ...") / print("No Drift: ...") — prints a human-readable verdict. In Dev’s production system, the “Drift Detected” branch would also fire a Slack alert or create a ticket.

What this actually means: The KS statistic is 0.3110. That’s the maximum gap between the two cumulative distribution curves. The p-value sits near zero — well below 0.05 — so we can be confident the production data has drifted from our training data.

That p_value < 0.05 check is fine for a one-off script like this one, but Section 4’s production-ready DriftMonitor deliberately replaces it with a threshold on the statistic itself. At Dev’s scale, the p-value would fire on shifts too small to matter — the next section explains why.

Kolmogorov-Smirnov test statistic:

$D_{n,m} = \sup_x \left| F_n(x) - F_m(x) \right|$

where $F_n(x)$ and $F_m(x)$ are the empirical cumulative distribution functions (eCDFs) of the training sample (size $n$) and the production sample (size $m$). The KS statistic $D$ is the maximum vertical gap between the two CDF curves — the wider the gap at any point, the more the two distributions disagree.

The p-value is computed from the Kolmogorov distribution and answers: “if both samples truly came from the same distribution, what’s the probability of observing a gap this large?”

Wasserstein distance (1D, between empirical distributions):

$W_1(P, Q) = \int_{-\infty}^{\infty} \left| F_P(x) - F_Q(x) \right| \, dx$

Unlike the KS statistic (which takes the maximum gap), the Wasserstein distance integrates the entire area between the two CDF curves. This makes it smoother and less sensitive to a single sharp crossing point.

Plain English	Statistical symbol	Python equivalent
Maximum vertical gap between the two CDF curves	$D = \sup_x \|F_{\text{train}}(x) - F_{\text{prod}}(x)\|$	ks_2samp(train, prod).statistic
Probability that two samples this different came from the same distribution	$p$	ks_2samp(train, prod).pvalue
Significance threshold (reject “same distribution” if $p < \alpha$)	$\alpha$ (typically $0.05$)	0.05
Total area between the two CDF curves (Earth Mover’s Distance)	$W_1$	wasserstein_distance(train, prod)

Four drift-detection strategies — when to reach for which:

Strategy	What it catches	Sensitivity	False-alarm risk	When to use
Distribution comparison (KS test)	Any shape change in a feature’s distribution	High — catches subtle shifts in distributional shape, not just the mean	Medium — on very large samples, even a 0.1% shift becomes statistically significant (p < 0.05) even when it’s practically irrelevant	First-line detection on numeric features
Summary stats (mean, std)	Large shifts in central tendency or spread	Low — misses shape changes (e.g., a bimodal shift where the mean stays the same)	Low — only fires on big, obvious moves	Quick health check, dashboard-friendly metrics
Confidence tracking	Model uncertainty increasing (inputs it doesn’t recognize)	Medium — indirect signal; tells you the model is uncomfortable but not which feature caused it	Medium — confidence can drop for reasons other than drift (e.g., legitimate edge cases, rare-but-valid inputs)	When you care about model output quality, not just input distributions
Windowing (reference vs. sliding)	Smoothed drift over time, filtering out brief anomalies	Depends on window size — short windows catch fast drift but are noisier; long windows catch slow drift but lag	Lower than point-in-time comparisons — a 7-day window smooths out day-of-week seasonality that a 1-day window would flag as drift	Essential for any production system with seasonal or day-of-week patterns

The sensitivity vs. false-alarm tradeoff:

The KS test is the most sensitive of the four — it catches any distributional difference — but that’s also its weakness. On a sample of 100,000 users, a 0.1% mean shift will be statistically significant (p < 0.05) even though it’s practically irrelevant. Dev doesn’t want his phone buzzing at 3 AM because the average session duration moved from 8.00 minutes to 8.01 minutes.

Summary stats are the opposite: cheap and intuitive, but they miss distributional shape changes. If Dev’s users split into two camps (short sessions and long sessions) while the overall mean stays the same, summary stats see nothing wrong.

Confidence tracking is an indirect signal — it tells you the model is uncomfortable, but not why. Useful as a complement alongside a distribution test, not as a replacement.

Windowing isn’t really a separate strategy — it’s a smoothing layer you apply on top of any of the above. Where it matters most is on data with day-of-week or weekday-vs-holiday cycles: comparing today’s data against a 7-day rolling reference window filters out cycles that would otherwise look like drift. A feature with no such cycle doesn’t need it.

Choosing among these: the KS test alone becomes noisy at high traffic volumes — that’s the case for thresholding on the statistic rather than the p-value once sample sizes are large enough that trivial shifts become “significant.” Summary stats are worth the low cost as a dashboard-friendly complement, but not a replacement, whenever shape changes (not just mean shifts) are plausible for a feature. Confidence tracking adds value once there’s an existing distribution test to pair it with, since on its own it can’t say which feature moved. Windowing is worth the extra bookkeeping specifically when the data has a known cycle — daily, weekly, or seasonal — and can be skipped otherwise.

4. Building a Drift Monitor: Code That Runs in Production

You don’t want to run manual scripts every day. A monitor that watches your back makes more sense. Let’s build a simple class you can integrate into a pipeline.

class DriftMonitor:
    def __init__(self, reference_data, threshold=0.1):
        self.reference_data = reference_data
        self.threshold = threshold
        
    def check_drift(self, new_batch):
        drift_report = {}
        for column in self.reference_data.columns:
            stat, p_val = ks_2samp(self.reference_data[column], new_batch[column])
            # We use the statistic here as a 'drift score'
            is_drifted = stat > self.threshold
            drift_report[column] = {
                "drift_score": round(stat, 3),
                "is_drifted": is_drifted
            }
        return drift_report

# Example usage — a reference with enough rows for the statistic to
# actually discriminate between "stable" and "drifted" (see the note above)
np.random.seed(0)
df_train = pd.DataFrame({'age': np.random.normal(loc=40, scale=8, size=500)})

df_prod_stable = pd.DataFrame({'age': np.random.normal(loc=40, scale=8, size=500)})   # same population
df_prod_shifted = pd.DataFrame({'age': np.random.normal(loc=55, scale=8, size=500)})  # a real shift

monitor = DriftMonitor(df_train)
print("Stable batch:", monitor.check_drift(df_prod_stable))
print("Shifted batch:", monitor.check_drift(df_prod_shifted))

This is the reusable drift monitor Dev can drop into his recommender’s daily pipeline — instantiate it once with the training data as a reference, then call check_drift on every new batch of production data.

• class DriftMonitor: — a class that wraps the KS test into a production-ready monitor. Dev instantiates this once (pointing it at the training data) and then calls check_drift on each new batch of production features.
• def __init__(self, reference_data, threshold=0.1): — the constructor takes two arguments: reference_data (a DataFrame containing the training features to compare against) and threshold (the KS statistic above which a feature is flagged as drifted). The default of 0.1 means “flag drift if the two distributions differ by more than 10% at their widest point.”
• self.reference_data = reference_data / self.threshold = threshold — stores both as instance attributes so check_drift can access them without re-passing them every call.
• def check_drift(self, new_batch): — the main method. Takes a new batch of production data (a DataFrame with the same columns as the reference) and checks every column for drift.
• drift_report = {} — initializes an empty dictionary to collect per-column results. The keys will be column names; the values will be dicts with drift_score and is_drifted.
• for column in self.reference_data.columns: — iterates over every column in the reference (training) data. For Dev’s recommender, these might be session_duration, items_viewed, time_of_day, price_range, etc.
• stat, p_val = ks_2samp(self.reference_data[column], new_batch[column]) — runs the two-sample KS test on that one column, comparing the training distribution to the production distribution. Returns the KS statistic (stat) and p-value (p_val). Note: only stat is used for the drift decision; p_val is computed but discarded.
• is_drifted = stat > self.threshold — a boolean flag. Note the deliberate choice to threshold on the statistic rather than the p-value. Unlike the p-value, which shrinks with sample size and fires on trivially small shifts at large scales, the statistic converges toward the same population quantity regardless of how much data you have — but only asymptotically. At small sample sizes it has a resolution floor of roughly 1 / min(n, m): with a 5-row reference, the smallest possible statistic is 0.2, which is already above this class’s 0.1 threshold. That’s why the example below uses hundreds of reference rows rather than a handful.
• drift_report[column] = {"drift_score": round(stat, 3), "is_drifted": is_drifted} — stores the drift score (rounded to 3 decimal places for readability) and the boolean flag under the column name.
• return drift_report — returns the full report: a dictionary mapping each column name to its drift score and flag. Dev can iterate over this to find which features drifted and by how much.
• df_train = pd.DataFrame({'age': np.random.normal(loc=40, scale=8, size=500)}) — 500 rows of reference (training) data for one feature (age), centered on 40. 500 rows is enough for the KS statistic to have real resolution — with only 5 reference rows, as noted above, the smallest possible statistic is 0.2, already above the threshold.
• df_prod_stable — a second batch of 500 rows drawn from the same distribution as training. This is what a healthy production week looks like: no real shift, just sampling noise.
• df_prod_shifted — 500 rows drawn from a distribution centered on 55 instead of 40 — a genuine, 15-year shift.
• monitor = DriftMonitor(df_train) — creates a DriftMonitor with the training DataFrame as the reference. The default threshold (0.1) is used.
• monitor.check_drift(df_prod_stable) / monitor.check_drift(df_prod_shifted) — runs the drift check against each batch. The stable batch’s KS statistic lands around 0.05 — below the 0.1 threshold, so is_drifted is False. The shifted batch’s statistic lands around 0.69 — well above it, so is_drifted is True.

In this output, the stable batch reports is_drifted: False with a drift score around 0.05 — the monitor correctly recognizes data from the same population as not drifted. The shifted batch reports is_drifted: True with a drift score around 0.69 — a real, large shift. Having both outcomes side by side is the point: a monitor that can only ever say “drifted” isn’t telling you anything.

5. What to Do When You Detect Drift: The Response Playbook

Once the alarm goes off, what’s the move?

• Step 1: Investigate. Is it a data quality issue? A broken sensor sending zeros, say. If so, fix the sensor — not the model.
• Step 2: Retrain. If the world has genuinely changed, the standard fix is to retrain on a mix of old data and recent production data.
• Step 3: Adjust Thresholds. Say you’re a bank and drift has made your model slightly less certain. You might temporarily require a higher confidence score before approving a loan.

6. Measuring Drift: KS Test, Wasserstein, and PSI

These three metrics answer slightly different questions, and each fits some situations better than others.

1. KS Test: Best for catching whether any difference exists at all.
2. Wasserstein Distance: Often called the “Earth Mover’s Distance.” It measures how much “work” it takes to transform one distribution into another. Smoother and less twitchy than KS.
3. PSI (Population Stability Index): Widely used in finance and credit scoring, where regulators expect a bucketed, auditable metric. It buckets the data into deciles and checks whether the share of observations in each bucket has shifted.

from scipy.stats import wasserstein_distance

w_dist = wasserstein_distance(train_feature, prod_feature)
print(f"Wasserstein Distance: {w_dist:.2f}")

This is the complementary drift metric Dev runs alongside the KS test — instead of asking “is there any difference?” (KS), the Wasserstein distance asks “how much work would it take to morph the training distribution into the production distribution?”

• from scipy.stats import wasserstein_distance — imports the 1D Wasserstein distance from SciPy. Also known as the Earth Mover’s Distance: imagine the training distribution as a pile of dirt and the production distribution as the target shape — the Wasserstein distance is the minimum amount of dirt (times distance moved) needed to reshape one into the other.
• w_dist = wasserstein_distance(train_feature, prod_feature) — computes the distance between the two NumPy arrays defined in Section 3. For two normal distributions with means 100 and 110 and the same standard deviation (15), the Wasserstein distance equals the difference in means: approximately 10.
• print(f"Wasserstein Distance: {w_dist:.2f}") — prints the distance to 2 decimal places. Unlike the KS statistic (which is always between 0 and 1 and therefore unitless), the Wasserstein distance is in the same units as the feature itself. A value of ~10.77 means “on average, production values are about 11 units away from where training says they should be” — a number Dev can interpret directly as “session durations shifted by about 11 seconds” or “prices shifted by about 11 dollars.”

A Wasserstein distance of 10.77 means the values in our production data sit roughly 11 units away from where they “should” be, on average, relative to training. That gives you a physical sense of the shift magnitude.

7. Multivariate Drift: When Multiple Features Shift Together

Here’s the tricky part: sometimes each individual feature looks fine, but their combination is off. Say you have a model that predicts health. “Running 10 miles” is normal. “Having a 102-degree fever” is normal — occasionally. But “Running 10 miles while having a 102-degree fever” is a massive outlier.

One way to catch this is PCA (Principal Component Analysis). We project the data into a lower dimension and check whether the reconstruction error goes up. If the model struggles to “reconstruct” the new data from the old patterns, the relationships between features have drifted. It’s a named option worth knowing about, not a full walkthrough here — the tradeoff to be aware of is that reconstruction error is a single aggregate number, so it tells you that the combination is off without saying which features are responsible.

8. Drift in Classification vs. Regression

In classification, there’s an extra concern: Label Drift. Say your model detects spam, and suddenly 90% of incoming emails are spam instead of the usual 10%. Precision and recall will shift — even if the content of the spam hasn’t changed. So, keep an eye on your class balance in production.

9. Real-World Traps: False Alarms and Silent Failures

Don’t be the boy who cried wolf.

• Seasonality: If your model sees more traffic on weekends, a Friday-to-Saturday comparison will look like drift. It’s not. It’s a cycle. Use a 7-day reference window to smooth this out.
• Sample Size: Only 10 data points today? The KS test might flag drift from random noise alone. Wait for a statistically significant batch size.

10. Putting It Together: A Minimal Production Drift System

A production-ready system comes down to three pieces: Storage (to keep track of history), Computation (to run the tests), and Alerting (to tell you when to wake up).

def production_pipeline(new_data, reference_data):
    monitor = DriftMonitor(reference_data)
    report = monitor.check_drift(new_data)
    
    for feature, results in report.items():
        if results['is_drifted']:
            print(f"ALERT: Feature '{feature}' drifted (Score: {results['drift_score']})")
            # In a real system, you'd trigger a Slack message or an email here

# Simulate a production run — with a batch large enough to trust the result.
# Section 9 warns that a handful of points can flag drift from noise alone;
# this uses 40 rows, above the 30-row minimum the project brief enforces.
np.random.seed(1)
prod_batch = pd.DataFrame({'age': np.random.normal(loc=55, scale=8, size=40)})
production_pipeline(prod_batch, df_train)

This is the glue function that ties the DriftMonitor into a production pipeline — Dev wraps the drift check in a function that iterates the report and fires an alert for every flagged feature.

• def production_pipeline(new_data, reference_data): — a function that takes new production data and the reference (training) data, runs the full drift check, and alerts on any flagged features. In Dev’s setup, this would be called on a schedule (e.g., hourly via a cron job or Airflow task).
• monitor = DriftMonitor(reference_data) — instantiates a DriftMonitor (the class from Section 4) with the reference data. In production, Dev would load the training data snapshot that was saved alongside the model version in MLflow (from the previous article’s versioning workflow).
• report = monitor.check_drift(new_data) — runs the drift check and gets back a dictionary mapping each feature name to its drift_score and is_drifted flag.
• for feature, results in report.items(): — iterates over the report, unpacking each feature name and its results dictionary. report.items() yields (key, value) pairs from the dictionary.
• if results['is_drifted']: — checks whether this particular feature was flagged as drifted (i.e., its KS statistic exceeded the threshold).
• print(f"ALERT: Feature '{feature}' drifted (Score: {results['drift_score']})") — prints an alert with the feature name and its drift score. In Dev’s production system, this is where he’d call a Slack webhook (requests.post(slack_webhook_url, json={"text": f"Drift alert: {feature}"})), send a PagerDuty event, or write to a monitoring dashboard.
• # In a real system, you'd trigger a Slack message or an email here — the comment marks the insertion point for real alerting logic. The print statement is a stand-in; production would use an actual notification service.
• prod_batch = pd.DataFrame({'age': np.random.normal(loc=55, scale=8, size=40)}) — simulates a production run with 40 rows, not 2. A 2-row batch would trip the same small-sample-noise problem Section 9 warns about — the KS test can look dramatic on a handful of points without being statistically meaningful. With 40 rows drawn from a distribution genuinely centered 15 years above training, the statistic lands at 0.726 with a p-value far below 0.05: real, well-supported drift, and the alert fires.

11. What’s Next: Continuous Learning

Drift detection is your early warning system. Once you can spot it, the next step is Automated Retraining. This sets up a feedback loop where the model learns from its mistakes in real-time. One caveat: if you automate retraining on bad data (say, a data entry error), you’ll just teach your model to be wrong faster.

12. Recap and Debugging Checklist

• Check Accuracy: Is it actually dropping, or a false alarm?
• Univariate Check: Run a KS test on every input feature.
• Label Check: Did the class mix (e.g., Spam vs. Not Spam) shift?
• Data Quality: Rule out broken sensors or null values first.
• Retrain: If the drift is real, update your training set with recent data.

Monitoring drift is what separates a model that works once from one that keeps delivering. Start simple, interpret the numbers, and stay alert.

---

Dev can now detect that the recommender’s input distributions have shifted. The KS test fires. The drift score climbs. The alert lands in his Slack. But detecting drift after the fact — running a batch comparison on yesterday’s data — isn’t the same as knowing the model is failing right now. He has drift detection. What he lacks is real-time monitoring: no live accuracy feed, no latency dashboard, no error-rate alert that fires the moment something breaks. The drift detector tells him the world changed. What he needs is something that tells him the model is broken this minute. Next article, we build that monitoring layer.

---

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the difference between Data Drift and Model (Concept) Drift, using the article’s $P(X)$ vs $P(Y|X)$ notation?

Understand In your own words, explain why the article says “Data drift is usually fixable” while “Model drift is harder”—what’s different about what broke in each case?

Apply Using the article’s DriftMonitor class (threshold=0.1, drift flagged when KS statistic > threshold), would a feature with a KS statistic of 0.08 be flagged as drifted?

Analyze The article’s Section 9 warns that a Friday-to-Saturday comparison can look like drift when it’s really just weekly seasonality, and recommends a 7-day reference window to smooth this out. Walk through why comparing against a 7-day window specifically (rather than, say, a 1-day or 30-day window) addresses this particular false-alarm pattern.

Evaluate The article recommends PCA reconstruction error to catch multivariate drift (features that are individually normal but abnormal in combination, like the “running 10 miles with a 102°F fever” example). Critique this approach: PCA reconstruction error is itself a single aggregate number — what nuance does it lose compared to being able to point to which feature combination is unusual?

Create Design a drift-monitoring plan for a new production model: a real estate price estimator that was trained on 2023 listings. Specify which detection strategy from the article (KS test, summary stats, confidence tracking, PSI) you’d use for the square_footage feature versus the neighborhood categorical feature, and explain why the same test isn’t appropriate for both.

---

Related articles

• Model Versioning and Rollbacks: Treating Models Like Code) — Dev can roll back a bad deploy now, but what if the model degrades without anyone pushing a new version? Data drift is the silent killer that versioning alone can’t catch.
• Monitoring Model Performance in Production (Without Ground Truth)) — Dev can detect drift after the fact, but detecting drift isn’t the same as knowing the model is failing right now. The next step is real-time monitoring without waiting for ground-truth labels.

References & Further reading

• Gama, J., Žliobaitė, I., Pechenizkiy, A., & Bouchachia, A. (2014). A Survey on Concept Drift Adaptation. ACM Computing Surveys, 46(4), Article 44, 1–37. — the canonical survey distinguishing data drift ($P(X)$) from concept drift ($P(Y|X)$) and cataloging detection and adaptation strategies for streaming and production environments.
• SciPy ks_2samp documentation — https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html — official reference for the two-sample Kolmogorov-Smirnov test, including the statistic definition, p-value computation, asymptotic behavior at large sample sizes, and the alternative parameter for one-sided tests.

---