When Should You Retrain? Building a Simple Retraining Trigger

Last time, Dev built a monitoring layer that catches the recommender’s silent failures in near-real-time — the KS test fires when input distributions shift, the prediction-distribution check catches model collapse, and the health log surfaces problems before they compound. But every alert he built is reactive: the alarm goes off after the model has already started degrading, after conversion rates have already dipped, after users have already been getting bad recommendations. He was always reacting, never anticipating. What he wants now is a trigger that tells him when to retrain — not on a fixed Monday schedule, but the moment the recommender’s data has actually shifted enough to need a refresh.

Imagine you built a fraud detection model in 2020. On day one, it was a superstar, catching 99% of scammers. You felt like a hero. But by 2024, that same model is letting half the scammers through the door. What happened?

The code didn’t break. The server didn’t slow down. The world changed, but your model is still living in 2020. This is model decay

1. The Problem: Your Model Gets Worse Over Time

Most people picture a machine learning model like a car — build it once, fill it with gas, go. A model is closer to a garden. Stop tending it and weeds move in.

Models carry one big assumption: tomorrow will look like yesterday. But data keeps shifting. A fraud model trained before mobile payments caught on won’t recognize how scammers exploit digital wallets today. That gap is called drift.

When performance decays, there’s no crash and no error message. The model fails quietly. It keeps producing answers, but those answers get steadily worse. Revenue slips. Users grow frustrated. Trust in your AI erodes. You can’t train once and walk away.

2. What Actually Changes? Data Drift vs. Concept Drift

To fix this, we need to understand the two ways the world changes.

Data Drift is when your inputs change. Say you have a house price predictor. If you start seeing far more 5-bedroom mansions in your data than you did during training, that’s data drift. The shape of your data has shifted.

Concept Drift is the subtler one. This is when the relationship between things changes. Imagine a model that predicts whether someone will buy a luxury watch based on their income. In a booming economy, a $100k salary might make them a likely buyer. In a recession, that same $100k earner might be saving every penny. The input is the same, but the meaning has changed.

Data drift is like the players on a field changing; concept drift is like the rules of the game changing. Either way, you need to retrain.

3. Retraining on a Fixed Schedule

Many teams start by retraining every Monday morning. Simple enough, right?

It has a real cost: if your data hasn’t changed, you’re burning cloud compute for no reason, and if a market shift hits Tuesday, your model stays stale until next Monday. But it also has a real benefit that’s easy to undervalue — bounded staleness (never more than a week out of date), predictable compute cost, and zero dependency on a monitoring system that might itself be broken or unmonitored. Whether that tradeoff is worth it depends on how expensive false alarms and missed drift are for your specific model.

4. Performance-Based Triggers

What’s the most direct way to know if a model is failing? Look at its score.

For a classification model, track its accuracy or F1-score over time. Set a threshold—say, 85%—and trigger a retrain when the score drops below it. This approach is intuitive because it measures exactly what we care about: results.

The Hardest Part: Calculating a score requires “ground truth” (the actual answer). Labels often arrive late. If you predict a customer will churn, you might wait 30 days to find out if you were right. Performance-based triggers are useful, but this delay means they often react too slowly.

5. Detecting Drift Without Waiting for Labels

Since we can’t always wait for labels, we look at the data itself. Compare what the model is seeing now to what it saw during training.

Statistical tests check whether these two groups of data come from the same distribution. The Kolmogorov-Smirnov (KS) test and Wasserstein distance are two common tools for this.

Say your average user age was 25 during training, and now it’s 45. The KS test will flag that. Here’s how this looks in Python.

import numpy as np
from scipy.stats import ks_2samp, wasserstein_distance

np.random.seed(42)

# Let's simulate our training data (Age of users)
train_age = np.random.normal(loc=30, scale=5, size=1000)

# Scenario A: New data looks just like training data
new_data_stable = np.random.normal(loc=30, scale=5, size=1000)

# Scenario B: New data has drifted (users are older)
new_data_drifted = np.random.normal(loc=35, scale=5, size=1000)

# Let's run the KS test
# A high p-value means the distributions are likely the same.
# A tiny p-value (usually < 0.05) means they have drifted.
stable_p = ks_2samp(train_age, new_data_stable).pvalue
drift_p = ks_2samp(train_age, new_data_drifted).pvalue

print(f"Stable data p-value: {stable_p:.4f}") # 0.2635 — well above 0.05
print(f"Drifted data p-value: {drift_p:.4f}") # 0.0000

# Wasserstein distance measures the 'cost' of turning one distribution into another.
# 0 means identical; larger numbers mean more drift.
dist_stable = wasserstein_distance(train_age, new_data_stable)
dist_drift = wasserstein_distance(train_age, new_data_drifted)

print(f"Stable distance: {dist_stable:.4f}") # 0.3294 — small
print(f"Drifted distance: {dist_drift:.4f}") # 4.9325 — much larger

This is the drift-detection check Dev runs on his recommender’s input features — the same KS test and Wasserstein distance he used in his DriftMonitor class, now repurposed as the backbone of his retraining trigger.

• import numpy as np — NumPy for generating synthetic data distributions. In Dev’s real pipeline, train_age would be the saved training-data snapshot for a feature like session_duration or items_viewed, not a synthetic normal distribution.
• from scipy.stats import ks_2samp, wasserstein_distance — imports the two statistical tools Dev uses: the two-sample KS test (compares empirical CDFs) and the Wasserstein distance (measures the “cost” of morphing one distribution into another).
• np.random.seed(42) — fixes the random seed. Without one, the “stable” p-value falls below 0.05 (a false alarm) by chance about one run in twenty — worth knowing, since nothing in a real drift check protects you from that either.
• train_age = np.random.normal(loc=30, scale=5, size=1000) — synthetic training data: 1,000 samples from a normal distribution with mean 30 and standard deviation 5. This represents the feature distribution the model was trained on.
• new_data_stable = np.random.normal(loc=30, scale=5, size=1000) — Scenario A: production data that looks just like training data. Same mean, same spread. No drift.
• new_data_drifted = np.random.normal(loc=35, scale=5, size=1000) — Scenario B: production data where the mean has shifted from 30 to 35. The 5-unit shift is the drift signal — small enough to be realistic, large enough for the KS test to catch.
• stable_p = ks_2samp(train_age, new_data_stable).pvalue — runs the KS test on the stable pair. Prints 0.2635 — well above 0.05, meaning “these distributions are probably the same.” No drift.
• drift_p = ks_2samp(train_age, new_data_drifted).pvalue — runs the KS test on the drifted pair. With a 5-unit mean shift and 1,000 samples, the p-value prints 0.0000 — the test is virtually certain these distributions differ.
• dist_stable = wasserstein_distance(train_age, new_data_stable) — Wasserstein distance for the stable pair. Prints 0.3294 — close to 0, the distributions overlap almost perfectly.
• dist_drift = wasserstein_distance(train_age, new_data_drifted) — Wasserstein distance for the drifted pair. Prints 4.9325 — the “cost” of shifting the mean by 5 units. Dev uses this as his drift metric because, unlike the p-value (which becomes hyper-sensitive at large sample sizes), the Wasserstein distance scales with the magnitude of the shift, making it easier to set a meaningful threshold.

In the output, a drift_p of 0.0000 means there’s effectively zero chance these two datasets are the same. The model is seeing a different world now.

Which retraining trigger should Dev reach for? — three approaches, each catching drift at a different stage:

Trigger	What it catches	What it misses	When to use it
Fixed schedule (every Monday)	Guarantees bounded staleness (never more than a week out of date), predictable compute cost, no monitoring dependency	Doesn’t react to anything — wastes compute on a quiet week, stays stale until next Monday during a fast-moving one	When you have zero monitoring infrastructure and need a baseline safety net
Performance-based (accuracy/F1 drops below threshold)	Actual model degradation — the thing you ultimately care about	Slow: requires ground-truth labels, which arrive late (30+ days for churn, months for lending). By the time the score drops, the damage is done	When labels arrive quickly (ad clicks, immediate feedback) and you can afford to wait for the score to move
Distribution-based (KS test / Wasserstein distance on features)	Input drift — the world changed before the model’s performance reflects it	Concept drift where inputs look the same but the input→output relationship changed; also can’t tell you if the drift actually hurts performance	When labels arrive late and you need an early-warning signal — exactly Dev’s situation with the recommender

The key tradeoff: Performance-based triggers measure what you care about (results) but react slowly. Distribution-based triggers react quickly but measure a proxy (input shape), which can fire false alarms — not every drift hurts the model. When both signals are available, one can offset the other’s weakness: distribution drift as an early warning, a later performance drop as confirmation that it actually mattered. Section 6 shows what combining them looks like in code.

6. Building Your First Retraining Trigger in Code

Let’s build a simple monitoring system — a class that tracks both performance and drift, and decides when a retrain is due.

class RetrainingTrigger:
    def __init__(self, performance_threshold=0.85, drift_threshold=2.0):
        self.performance_threshold = performance_threshold
        self.drift_threshold = drift_threshold

    def should_retrain(self, current_performance, train_dist, current_dist):
        # 1. Check Performance (if labels are available — current_performance
        # may be None while ground truth is still pending, so this check is
        # skipped rather than crashing, and control falls through to drift)
        if current_performance is not None and current_performance < self.performance_threshold:
            return True, f"Performance dropped to {current_performance}"

        # 2. Check Data Drift (using Wasserstein Distance)
        drift_score = wasserstein_distance(train_dist, current_dist)
        if drift_score > self.drift_threshold:
            return True, f"Drift detected: score {drift_score:.4f}"

        return False, "Model is healthy"

# Usage — one call on drifted data, one on stable data, so both verdicts show up
trigger = RetrainingTrigger(performance_threshold=0.85, drift_threshold=2.0)

fire_alarm, reason = trigger.should_retrain(0.88, train_age, new_data_drifted)
print(f"Drifted batch — should we retrain? {fire_alarm}. Reason: {reason}")

fire_alarm, reason = trigger.should_retrain(0.88, train_age, new_data_stable)
print(f"Stable batch — should we retrain? {fire_alarm}. Reason: {reason}")

This is the RetrainingTrigger class Dev builds for the recommender — a single object that combines both signals (performance and drift) and returns a yes/no decision plus a human-readable reason.

• class RetrainingTrigger: — defines the trigger class. Dev instantiates this once and calls should_retrain() on each batch of production data.
• def __init__(self, performance_threshold=0.85, drift_threshold=2.0): — constructor with two thresholds. performance_threshold is the minimum acceptable model performance (below this, retrain), matching the 85% figure from Section 4. drift_threshold is the maximum acceptable Wasserstein distance (above this, retrain). 2.0 isn’t an arbitrary starting point — Section 8 shows the calibration: run the drift check on stable data alone, and the “noise floor” (the distance you get from sampling variation with no real drift) tops out well under 1.0 for this feature, so 2.0 sits safely above it while still catching the ~5.0 distance a real shift produces.
• self.performance_threshold = performance_threshold / self.drift_threshold = drift_threshold — stores the thresholds as instance attributes.
• def should_retrain(self, current_performance, train_dist, current_dist): — the core method. Takes the current model performance (if labels are available — otherwise None), the training distribution, and the current production distribution. Returns a tuple of (bool, string).
• if current_performance is not None and current_performance < self.performance_threshold: — Signal 1: if performance is known and has dropped below the threshold, fire immediately. The is not None guard matters: labels often aren’t in yet, and comparing None < 0.85 raises a TypeError — without the guard, the “labels haven’t arrived, fall back to drift-only” behavior described in Section 10 would crash instead.
• return True, f"Performance dropped to {current_performance}" — fires the retrain with a clear reason message. The reason string is important — when the alarm goes off at 2 a.m., Dev needs to know why without reading the code.
• drift_score = wasserstein_distance(train_dist, current_dist) — Signal 2: computes the Wasserstein distance between the training and current distributions. Dev chose Wasserstein over the KS p-value because the distance scales with the magnitude of the shift (a 5-unit shift gives a distance of ~5), making it easier to set a meaningful threshold than a p-value that collapses to 0 at large sample sizes.
• if drift_score > self.drift_threshold: — if the drift score exceeds the threshold, fire the retrain. This is the proactive signal — it catches drift before the performance metric has time to reflect it.
• return True, f"Drift detected: score {drift_score:.4f}" — fires with the drift score in the reason message.
• return False, "Model is healthy" — if neither signal fires, the model is healthy. No retrain needed.
• trigger.should_retrain(0.88, train_age, new_data_drifted) — current performance 0.88 (above the 0.85 threshold, so the performance signal doesn’t fire) against the drifted data (Wasserstein distance ~4.93, well above the 2.0 threshold), so the drift signal fires and the trigger returns True.
• trigger.should_retrain(0.88, train_age, new_data_stable) — same performance, but against the stable data (Wasserstein distance ~0.33, below the 2.0 threshold). Neither signal fires, so the trigger returns False, "Model is healthy" — the verdict a nightly job should see on most nights.

The two calls show both verdicts side by side: performance held at 0.88 in both, but the drifted batch’s Wasserstein distance (~4.93) is far above the 2.0 threshold, while the stable batch’s (~0.33) is comfortably below it. The trigger flags the first for retraining and clears the second — a monitor that can only ever say “retrain” isn’t telling you anything useful.

7. Putting It All Together: A Real-World Scenario

Consider a Telecom Churn model.

• Month 1: Accuracy holds steady at 90%.
• Month 3: The company launches a major marketing campaign targeting seniors. The “Age” feature drifts.
• Month 4: The model wasn’t trained on many seniors, so it starts missing churners. Accuracy drops to 70%.

Our trigger would likely catch the drift in Month 3, letting us retrain before accuracy tanks in Month 4. That keeps the company from losing customers to an outdated model.

8. The Hard Part: Choosing Your Thresholds

This is where the science becomes an art. Set your drift threshold too low and your model retrains every time a single user logs in. Set it too high and you’ll miss a total market collapse.

There’s no magic number. You have to look at your historical data. Run your drift tests on the last six months of data and see what “normal” noise looks like. Set your threshold just above that noise.

Here’s that calibration on the article’s own train_age reference — repeatedly compare stable samples against it and see how large the Wasserstein distance gets from sampling noise alone, with no real drift:

np.random.seed(42)
train_age = np.random.normal(loc=30, scale=5, size=1000)

noise_scores = []
for _ in range(200):
    stable_sample = np.random.normal(loc=30, scale=5, size=1000)
    noise_scores.append(wasserstein_distance(train_age, stable_sample))
noise_scores = np.array(noise_scores)

print(f"Observed noise floor over 200 stable comparisons: max={noise_scores.max():.4f}, mean={noise_scores.mean():.4f}")

This prints a max of 0.4990 and a mean of 0.2531 — pure sampling noise never gets close to 1.0 for this feature. That’s why RetrainingTrigger’s default drift_threshold is 2.0, not the 0.1 you’d use for a KS statistic: it sits about 4x above the observed noise ceiling, comfortably clear of false alarms, while still well below the ~5.0 a real shift produces.

9. When Performance Labels Arrive Late

In credit lending, you might not know whether a loan went bad for a year or more. When the wait is that long, distribution monitoring is often the only signal available on a useful timescale — proxy labels, sampled human review, and holdout backtesting are the other options, and all of them have their own lag or cost. The thinking goes: “I don’t know if I’m wrong yet, but the people I’m seeing today look nothing like the people I saw when I was learning, so I should probably update my knowledge.” Worth remembering from the tradeoffs above, though: a distribution shift on its own doesn’t tell you whether it actually hurt performance — it’s a reason to investigate, not automatically a reason to retrain.

10. Automating the Trigger: From Decision to Action

Once should_retrain exists, you won’t want to run it by hand. Wrap it in a short script that runs nightly.

# Pseudocode for an automated pipeline
def nightly_check():
    data = fetch_recent_production_data()
    performance = calculate_latest_accuracy() # if available
    
    trigger = RetrainingTrigger()
    fire, reason = trigger.should_retrain(performance, train_data, data)
    
    if fire:
        print(f"ALARM: {reason}. Starting Airflow Pipeline...")
        # kick_off_retraining_job()
    else:
        print("All good. See you tomorrow.")

This is the nightly automation Dev wraps around his RetrainingTrigger — the script that turns a decision into an action without anyone having to press a button.

• def nightly_check(): — the entry point for the scheduled job. In Dev’s setup, this runs via a cron job or an Airflow DAG at midnight.
• data = fetch_recent_production_data() — pulls the last day’s (or last hour’s) production data from the feature store or the prediction logs. In Dev’s recommender pipeline, this would fetch the last 24 hours of user features.
• performance = calculate_latest_accuracy() — computes the model’s recent performance, if ground-truth labels are available. For the recommender, this might be the click-through rate on the last day’s recommendations. If labels haven’t arrived yet, this returns None or a default value, and the trigger falls back to drift-only.
• trigger = RetrainingTrigger() — instantiates the trigger with the calibrated defaults from Section 8 (performance_threshold=0.85, drift_threshold=2.0). This matters here specifically: the nightly job takes whatever the default is, so if that default sat below the feature’s own noise floor, this script would fire “ALARM” every single night regardless of whether anything actually drifted — which is exactly why Section 8’s calibration exists. In production, Dev would still pass in thresholds tuned per feature rather than relying on the class default for every column.
• fire, reason = trigger.should_retrain(performance, train_data, data) — calls the trigger. train_data is the saved training-data snapshot (stored alongside the model version). data is the fresh production batch.
• if fire: / print(f"ALARM: {reason}. Starting Airflow Pipeline...") — if the trigger fires, the script logs the alarm and kicks off the retraining pipeline. The commented-out # kick_off_retraining_job() would trigger the Airflow DAG or CI/CD pipeline that retrains the model on recent data.
• else: print("All good. See you tomorrow.") — if the trigger doesn’t fire, the script exits quietly. No wasted compute, no unnecessary retraining.

Apache Airflow or GitHub Actions can schedule the check and spin up a new training job on your cloud provider automatically.

11. Recap and Next Steps

We’ve moved from “set it and forget it” to a reactive system. Here’s what we learned:

• Models decay because the world is dynamic.
• Data drift is a change in inputs; Concept drift is a change in the rules.
• Performance triggers are the most accurate but often delayed.
• Drift triggers (like KS tests) are early warning systems.
• Automation is what keeps your models current without constant manual oversight.

Dev’s trigger fires one night — the Wasserstein distance on session_duration crosses 2.0, the nightly script logs “ALARM: Drift detected,” and the Airflow pipeline kicks off a retraining job on fresh data. By morning, Dev has a new candidate model in his experiment tracker. But now he faces a new question: is this freshly retrained model actually better than the one currently live? The drift trigger told him when to retrain. It didn’t tell him whether the retrain worked. Before he ships the new model to his ten million users, he needs to know — rigorously and statistically — that the candidate beats the incumbent. That’s the next question.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the difference between Data Drift and Concept Drift, using the article’s luxury-watch-buyer example for Concept Drift?

Understand In your own words, explain why performance-based triggers “often react too slowly” for something like churn prediction, using the article’s 30-day label-delay example.

Apply Using the article’s RetrainingTrigger.should_retrain logic, if current_performance=0.90 (above the 0.85 threshold) but drift_score=3.5 (above the 2.0 drift_threshold), would the trigger fire, and what reason would it report?

Analyze The article’s Telecom Churn scenario has drift appearing in Month 3 (from a senior-targeted marketing campaign) but accuracy not dropping until Month 4. Walk through why a drift-based trigger catches this a full month earlier than a performance-based trigger would, given the same underlying root cause.

Evaluate The article’s Section 8 says there’s “no magic number” for the drift threshold and recommends looking at “the last six months of data” to find normal noise. Critique this calibration method: what happens if that six-month baseline period itself contained an undetected drift event — would the resulting threshold be too strict, too lenient, or is it impossible to tell without more information?

Create Design a retraining trigger for a new scenario: a job-recommendation model where ground-truth labels (did the user get hired) arrive 2-6 months late, but you have daily access to feature distributions (user skills, job postings). Using the article’s dual-trigger pattern (performance + drift), describe how you’d weight the two signals given the long label delay, and what drift signal you’d monitor as an early proxy.

---

Related articles

• Monitoring Model Performance in Production Without a PhD) — Dev built the monitoring layer that catches silent failures in near-real-time. This article builds the proactive retraining trigger on top of that monitoring.
• Is Your Model Actually Better? A Plain-English Guide to Statistical Significance) — Dev’s trigger fires and the recommender retrains. But is the new model actually better than the one currently live? The next article shows how to check, rigorously, before shipping.

References & Further reading

• Gama, J., Žliobaitė, I., Pechenizkiy, A., & Bouchachia, A. (2014). A Survey on Concept Drift Adaptation. ACM Computing Surveys, 46(4), Article 44. — the definitive survey on concept drift, data drift, and adaptation strategies. Covers the taxonomy of drift types (incremental, sudden, gradual, recurring) and the detection methods (performance-based, distribution-based) that Dev’s retraining trigger builds on.
• Hulten, G. (2020). Building Machine Learning Systems That Work: Continuous Training. In Machine Learning Systems Design. — covers the engineering patterns for automating retraining pipelines, including trigger design, data validation gates, and pipeline orchestration with tools like Airflow and TFX.
• SciPy wasserstein_distance documentation — https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wasserstein_distance.html — official reference for the Wasserstein distance metric used in the retraining trigger.

---