Is Your Model Actually Better? A Plain-English Guide to Statistical Significance

Last time, Dev built a retraining trigger for his recommender — a dual-signal system that fires when the model’s performance drops below a threshold, or when the Wasserstein distance on input features crosses a drift threshold. One night, the drift signal trips. The Wasserstein distance on session_duration crosses 2.0, the nightly script logs “ALARM: Drift detected,” and the Airflow pipeline kicks off a retraining job on fresh data. By morning, Dev has a new candidate model in his experiment tracker. The trigger told him when to retrain — but not whether the retrain worked. So now he needs to know, rigorously and statistically, whether the candidate actually beats the incumbent before he ships it to ten million users.

The ‘Lucky Guess’ Problem

Dev has two versions of his recommender to compare. Model A—the live one—hits 85% accuracy on the holdout set. Model B, freshly retrained, hits 86%. The urge to deploy Model B right away is understandable. But is Model B actually smarter, or did it just catch a favorable split in the test set?

This is the “Lucky Guess” problem—sampling error, in data science terms. We test models on a slice of the real world, so the scores we see are estimates. Shuffle the data and test again, and the numbers could flip.

Think of it like a coin toss. Flip a coin 10 times, get 6 heads—you wouldn’t conclude the coin is rigged. You’d assume those 10 tosses just happened to favor heads. Same logic applies here: that 1% lead could be structural, or it could be noise. Let’s see what happens when we look at the uncertainty around these scores.

import numpy as np
from sklearn.metrics import accuracy_score
from scipy import stats

# Let's simulate 1000 predictions for two models
# Model A is our baseline, Model B is slightly 'better'
np.random.seed(42)
y_true = np.random.randint(0, 2, 1000)

# Model A: mask sets a 15% error rate (intended ~85% accuracy)
pred_a = y_true.copy()
mask_a = np.random.choice([True, False], size=1000, p=[0.15, 0.85])
pred_a[mask_a] = 1 - pred_a[mask_a]

# Model B: mask sets a 13.5% error rate (intended ~86.5% accuracy)
pred_b = y_true.copy()
mask_b = np.random.choice([True, False], size=1000, p=[0.135, 0.865])
pred_b[mask_b] = 1 - pred_b[mask_b]

acc_a = accuracy_score(y_true, pred_a)
acc_b = accuracy_score(y_true, pred_b)

print(f"Model A Accuracy: {acc_a:.3f}")
print(f"Model B Accuracy: {acc_b:.3f}")
# Model A Accuracy: 0.848
# Model B Accuracy: 0.858
#
# Neither number matches the mask's own probability exactly (0.85 and 0.865)
# -- with 1,000 rows, the realized sample accuracy wobbles around the
# population rate you coded in. That wobble is the "Lucky Guess" problem
# from the intro, showing up before a single test has even been run.

# Calculate a simple 95% Confidence Interval for Model A
stderr = np.sqrt((acc_a * (1 - acc_a)) / 1000)
ci_low, ci_high = acc_a - 1.96 * stderr, acc_a + 1.96 * stderr
print(f"Model A 95% CI: [{ci_low:.3f}, {ci_high:.3f}]")
# Model A 95% CI: [0.826, 0.870]

This is the simulation Dev runs to put uncertainty bars around his two recommenders’ accuracy scores — the same Model A (incumbent) vs. Model B (retrained candidate) comparison he needs to make before shipping.

• np.random.seed(42) — locks the random number generator so the simulation is reproducible. Without this, every run would produce different synthetic data and different accuracy numbers.
• y_true = np.random.randint(0, 2, 1000) — generates 1,000 ground-truth labels, each randomly 0 or 1. In Dev’s real pipeline, these would be the holdout set labels — did the user click the recommended product or not?
• pred_a = y_true.copy() — starts Model A’s predictions as a perfect copy of the ground truth. The next line will inject errors to simulate a realistic ~85% accuracy.
• mask_a = np.random.choice([True, False], size=1000, p=[0.15, 0.85]) — creates a boolean mask where ~15% of entries are True. These are the rows where Model A will get the prediction wrong.
• pred_a[mask_a] = 1 - pred_a[mask_a] — flips the prediction wherever the mask is True. If the true label was 1, it becomes 0, and vice versa. This simulates a model that gets ~85% right.
• mask_b = np.random.choice([True, False], size=1000, p=[0.135, 0.865]) — Model B’s error mask uses a 13.5% error rate, simulating a slightly better model (~86.5% accuracy).
• acc_a = accuracy_score(y_true, pred_a) — computes Model A’s accuracy by comparing predictions to ground truth. The fraction of correct predictions.
• stderr = np.sqrt((acc_a * (1 - acc_a)) / 1000) — the standard error of a proportion. For a binomial accuracy metric, the standard error is $\sqrt{p(1-p)/n}$ where $p$ is the accuracy and $n$ is the sample size. This tells you how much the accuracy would wiggle if you drew a different test set of the same size.
• ci_low, ci_high = acc_a - 1.96 * stderr, acc_a + 1.96 * stderr — constructs a 95% confidence interval using the normal approximation. The value 1.96 is the z-score that captures the middle 95% of a standard normal distribution. The interpretation: if you repeated this experiment many times with different test sets, about 95% of those intervals would contain the model’s true accuracy.

Confidence interval for a proportion (Model A’s accuracy):

$CI_{95\%} = \hat{p} \pm 1.96 \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

McNemar test statistic (with continuity correction):

$\chi^2 = \frac{(|b - c| - 1)^2}{b + c}$

Where $b$ is the count of rows where Model A is correct and Model B is wrong, and $c$ is the count where Model B is correct and Model A is wrong. The $-1$ is the continuity correction (Yates’ correction) that adjusts for the discrete nature of the count data.

Plain English	Statistical symbol	Python equivalent
Model’s observed accuracy	$\hat{p}$	acc_a
Number of test samples	$n$	1000 / len(y_true)
Standard error of accuracy	$\sqrt{\hat{p}(1-\hat{p})/n}$	np.sqrt((acc_a * (1 - acc_a)) / 1000)
95% z-score (normal distribution)	$z_{0.975}$	1.96
CI lower bound	$\hat{p} - 1.96 \cdot SE$	acc_a - 1.96 * stderr
CI upper bound	$\hat{p} + 1.96 \cdot SE$	acc_a + 1.96 * stderr
A-right, B-wrong (discordant cell)	$b$	tb[0, 1]
B-right, A-wrong (discordant cell)	$c$	tb[1, 0]
McNemar test statistic	$\chi^2$	chi2 from mcnemar(ary=tb, corrected=True)
p-value from McNemar test	$p$	p from mcnemar(ary=tb, corrected=True)

the output shows Model A at 84.8% (95% CI: 82.6% to 87.0%) and Model B at 85.8%. Model B’s point estimate sits right inside Model A’s interval, which feels like it should mean “we can’t be confident B is better” — but that reasoning has a real gap, and it’s worth naming before relying on it: A’s CI describes how much A’s own score would wiggle on a different test set, not whether B is different from A. The two models were scored on the same 1,000 rows, so their errors are correlated in a way a single model’s CI can’t see. It’s a fast, useful sanity check, but not a substitute for a test built to compare two models directly — which is exactly what the next section does.

Meet the McNemar Test: The ‘Tie-Breaker’ for Classifiers

To know whether Model B is actually better, final scores aren’t enough. We need the disagreements.

Picture a scoreboard for mistakes. Rows where both models are right tell us nothing about which one is better — same for rows where both are wrong. The only rows that count are the ones where Model A is right and Model B is wrong, or the reverse.

This is the McNemar Test. It builds a ‘contingency table’ out of those disagreements. If Model B beats Model A about as often as Model A beats Model B, the models are essentially the same. A large tilt in one direction points to a winner.

from mlxtend.evaluate import mcnemar
from mlxtend.evaluate import mcnemar_table

# Create the table of disagreements
# Row 0, Col 0: Both correct
# Row 0, Col 1: A correct, B wrong
# Row 1, Col 0: B correct, A wrong
# Row 1, Col 1: Both wrong
tb = mcnemar_table(y_true, pred_a, pred_b)
print("Contingency Table:")
print(tb)
# [[734 114]
#  [124  28]]

chi2, p = mcnemar(ary=tb, corrected=True)
print(f"p-value from McNemar: {p:.4f}")
# p-value from McNemar: 0.5596

This is the contingency table Dev builds for his two recommenders — the same Model A vs. Model B from the accuracy simulation above, now broken down by where they disagree rather than just by aggregate score.

• from mlxtend.evaluate import mcnemar — imports the McNemar test implementation from mlxtend, a companion library to scikit-learn that provides extra evaluation utilities. Dev could also implement the test by hand (it’s a one-liner chi-squared calculation), but mlxtend handles the edge cases and corrections.
• from mlxtend.evaluate import mcnemar_table — imports the helper that builds the 2×2 disagreement matrix from the ground-truth labels and the two models’ predictions.
• tb = mcnemar_table(y_true, pred_a, pred_b) — builds the 2×2 table. The four cells are:
  • Row 0, Col 0 — both models correct (agreement, ignored by McNemar)
  • Row 0, Col 1 — Model A correct, Model B wrong (Model A “wins” this row)
  • Row 1, Col 0 — Model B correct, Model A wrong (Model B “wins” this row)
  • Row 1, Col 1 — both models wrong (agreement, ignored by McNemar)
• chi2, p = mcnemar(ary=tb, corrected=True) — runs the McNemar test with continuity correction (Yates’ correction). The correction subtracts 1 from $|b - c|$ before squaring, which prevents the test from being overly aggressive on small sample sizes. The chi2 output is the test statistic; p is the p-value. A small p-value (typically < 0.05) means the disagreements are significantly skewed — one model is systematically beating the other, not just trading wins by chance.

Which model-comparison method should Dev reach for?

Method	What it tells you	What it misses	When to use it
Eyeball the accuracy delta	Which model scored higher on the test set	Whether the difference is real or noise; gives zero uncertainty estimate; a 1% gap on 500 rows is meaningless	Quick sanity checks on toy problems where the stakes are low and you just need a rough ranking
Confidence intervals	The range of plausible values for each model’s accuracy — how much would the score wiggle with a different test set?	Overlapping CIs don’t prove equivalence — two models can have overlapping CIs and still be significantly different. “B’s point estimate falls inside A’s CI” is a rule of thumb, not a proper hypothesis test	When you need a quick uncertainty estimate and don’t want to set up a formal pairwise test; good for dashboards and monitoring
McNemar test	Whether the disagreements between two classifiers are systematically skewed — the rigorous pairwise comparison that uses the structure of which rows each model gets wrong	Only works for classification (not regression); requires both models to predict on the exact same test set; doesn’t tell you the magnitude of the improvement (you need effect size for that)	When you’re making a shipping decision and need to know if the new model is statistically better, not just numerically better — exactly Dev’s situation with the recommender

The key tradeoff: Eyeballing is fast but tells you nothing about uncertainty. Confidence intervals give you a range, but “the other model’s point estimate falls inside my CI” is a heuristic, not a proper significance test — it can miss real differences because it ignores the correlation between the two models’ errors (both are tested on the same rows). The McNemar test is the most rigorous of the three because it uses the structure of the disagreements — which specific rows did the models disagree on? — rather than just comparing aggregate scores. For Dev’s shipping decision, where a false positive means deploying a model that isn’t actually better and a false negative means leaving a better model on the shelf, the rigor is worth the extra three lines of code.

In practice, we’re ignoring the ties and focusing on the cases where the models disagreed. When the top-right and bottom-left numbers in our table are close, the test reports no real difference — and that’s exactly what happens here: 114 rows where A was right and B was wrong, against 124 the other way. Close enough that McNemar’s own p-value comes back at 0.56, not the 0.05 or lower you’d want to call this a real difference.

The Hardest Part: What a p-value actually says

Here’s the hardest part of model comparison to internalize: the p-value. Most people think a p-value of 0.05 means there’s a 95% chance their model is better. That is wrong.

Think of the p-value as a ‘Surprise Meter.’ We start with the Null Hypothesis, which is just the ‘Nothing Special’ assumption — we assume both models are identical.

A p-value of 0.03 means: “If these models were actually identical, we’d see a result this extreme only 3% of the time by pure luck.”

It measures how strange our data looks under the assumption that nothing has changed. When the p-value drops low enough (usually below 0.05), we say “Okay, this is too weird to be a fluke,” and reject the ‘Nothing Special’ assumption.

def interpret_p(p_val):
    threshold = 0.05
    if p_val < threshold:
        return f"p={p_val:.4f}: The difference is likely real (Statistically Significant)."
    else:
        return f"p={p_val:.4f}: The difference could just be luck (Not Significant)."

print(interpret_p(p))

This is the helper Dev writes to turn the McNemar test’s raw p-value into a human-readable verdict — the same function he’ll call when comparing his retrained recommender against the incumbent.

• def interpret_p(p_val): — a simple function that takes a p-value and returns a plain-English string. Dev calls this right after running mcnemar() so the output is immediately interpretable in his nightly report.
• threshold = 0.05 — the conventional significance level (alpha). This means “I’m willing to accept a 5% chance of being wrong — of declaring the models different when they’re actually the same.” Dev could tighten this to 0.01 for high-stakes shipping decisions or loosen it to 0.10 for exploratory checks.
• if p_val < threshold: — if the p-value is below the threshold, the observed disagreement pattern is unlikely under the null hypothesis (both models identical). Dev concludes the difference is probably real.
• return f"p={p_val:.4f}: The difference is likely real (Statistically Significant)." — the “significant” verdict, with the actual p-value printed to 4 decimal places so Dev can see how significant — a p of 0.049 is barely past the line, while a p of 0.0001 is overwhelming.
• return f"p={p_val:.4f}: The difference could just be luck (Not Significant)." — the “not significant” verdict. Important: this does not mean the models are the same. It means the evidence isn’t strong enough to rule out luck. The difference could be real but small, or the test set could be too small to detect it.
• print(interpret_p(p)) — calls the function with p, the p-value from the McNemar test in the previous code block. For Dev’s recommender comparison, this is the final line in his model-evaluation notebook: the statistical verdict on whether Model B is actually better than Model A.

A p-value of 0.40 means: if the two models were truly identical, you’d see a gap in your disagreement counts at least this large 40% of the time from noise alone. That’s a coin flip’s worth of “this could easily be nothing,” which is exactly Dev’s actual result above — not evidence the models are the same, but nowhere near enough evidence to call B the winner either. You wouldn’t bet a million-dollar marketing budget on a result that unremarkable, would you?

Comparing Regressions with the Diebold-Mariano Test

What if you aren’t doing classification? If you’re predicting house prices or stock trends, there’s no “right” or “wrong”—just errors. Residuals, more precisely.

To compare two regression models, we use the Diebold-Mariano test. No contingency table here. We look at the “error series” instead—subtracting Model B’s error from Model A’s error for each prediction. If Model B is truly better, its errors should be consistently smaller.

This is how we check whether a more complex model—say, a massive Neural Network—is actually worth the extra compute over a simple Linear Regression.

def diebold_mariano_test(y_true, p1, p2):
    # Calculate errors (Absolute Error)
    e1 = np.abs(y_true - p1)
    e2 = np.abs(y_true - p2)
    
    # The difference in errors
    d = e1 - e2
    
    # Mean difference divided by its standard deviation
    # This is a simplified version of the DM statistic
    dm_stat = np.mean(d) / (np.std(d, ddof=1) / np.sqrt(len(d)))
    p_value = stats.norm.cdf(-np.abs(dm_stat)) * 2
    return dm_stat, p_value

# Simulate regression errors. 1,500 predictions, not 100 -- at 100 rows this
# specific comparison (sigma 50 vs. 45) is underpowered, reaching p < 0.05
# only about 17% of the time across repeated draws, regardless of which
# model is genuinely better. 1,500 rows is comfortably enough for a 10%
# reduction in error spread to show up reliably.
y_real = np.linspace(100, 1000, 1500)
reg_a = y_real + np.random.normal(0, 50, 1500)
reg_b = y_real + np.random.normal(0, 45, 1500)

stat, p_reg = diebold_mariano_test(y_real, reg_a, reg_b)
print(f"DM Statistic: {stat:.4f}, p-value: {p_reg:.4f}")
# DM Statistic: 4.2221, p-value: 0.0000

This is the Diebold-Mariano test Dev would use if his recommender were scored on a regression metric (like predicted click probability) rather than binary classification — the same “is Model B actually better?” question, adapted for continuous errors.

• def diebold_mariano_test(y_true, p1, p2): — defines the DM test function. y_true is the ground truth, p1 and p2 are the two models’ predictions. In Dev’s case, these would be the incumbent’s and the retrained model’s predicted click probabilities.
• e1 = np.abs(y_true - p1) — absolute errors for Model A: how far off each prediction is from the true value. The DM test can also use squared errors ((y_true - p1)**2) or other loss functions; absolute error is a robust default.
• e2 = np.abs(y_true - p2) — absolute errors for Model B.
• d = e1 - e2 — the error difference series. For each prediction, this is positive when Model A’s error is larger (Model B is better on that row) and negative when Model B’s error is larger. If Model B is consistently better, d will be predominantly positive.
• dm_stat = np.mean(d) / (np.std(d, ddof=1) / np.sqrt(len(d))) — the DM test statistic: the mean error difference divided by its standard error. This is structurally identical to a one-sample t-statistic — it measures how many standard deviations the mean error difference is from zero. ddof=1 uses the sample standard deviation (dividing by $n-1$ rather than $n$).
• p_value = stats.norm.cdf(-np.abs(dm_stat)) * 2 — computes a two-tailed p-value from the standard normal distribution. np.abs(dm_stat) takes the absolute value of the statistic (so the test is symmetric — it doesn’t matter which model is better, just whether one is). stats.norm.cdf(-...) gives the left tail probability, and * 2 doubles it for a two-tailed test. A small p-value means the mean error difference is too large to be explained by random variation alone.
• y_real = np.linspace(100, 1000, 1500) — generates 1,500 evenly spaced ground-truth values from 100 to 1000. This simulates a regression target with a smooth trend.
• reg_a = y_real + np.random.normal(0, 50, 1500) — Model A’s predictions: true values plus Gaussian noise with standard deviation 50. Model A has larger errors.
• reg_b = y_real + np.random.normal(0, 45, 1500) — Model B’s predictions: same true values but noise with standard deviation 45. Model B is slightly better — its errors are consistently ~10% smaller.
• stat, p_reg = diebold_mariano_test(y_real, reg_a, reg_b) — runs the test. A positive stat with a low p_reg means Model B’s errors are significantly smaller than Model A’s — the improvement is real, not noise.

A positive DM statistic with a low p-value means Model B’s errors are significantly smaller. Here, stat = 4.22 and p_reg < 0.0001 — decisively significant. If the p-value is high, the “improvement” is just wiggling in the noise, which is what the 100-row version of this exact comparison mostly shows instead (p < 0.05 only about 17% of the time across repeated draws).

When to Stop: The ‘Good Enough’ Rule

Statistical significance is not the same as practical importance.

More rows push p-values down for a fixed effect, but “more rows” doesn’t make every effect detectable, and it’s worth seeing where the line actually falls. At 10 million rows and a realistic 10% disagreement rate between two models, a 0.0001% accuracy improvement still isn’t significant (p ≈ 0.99 — indistinguishable from noise even at that scale). Push the improvement to 0.1% and the picture flips completely: p drops below 1e-8. Somewhere between those two lies a threshold where a genuinely tiny effect crosses into “detectable,” and once it does, size still doesn’t imply importance. If the new model costs $50,000 in server fees and saves $5 in fraud losses, a detected-but-tiny effect is still a bad trade.

Check the Effect Size. It measures the magnitude of the improvement. Cohen’s d is one standard approach — it counts how many standard deviations better the new model performs.

def cohens_d(x, y):
    nx, ny = len(x), len(y)
    var_x, var_y = np.var(x, ddof=1), np.var(y, ddof=1)
    pooled_sd = np.sqrt(((nx - 1) * var_x + (ny - 1) * var_y) / (nx + ny - 2))
    return (np.mean(x) - np.mean(y)) / pooled_sd

# Comparing the errors of our regression models
d_val = cohens_d(np.abs(y_real - reg_a), np.abs(y_real - reg_b))
print(f"Effect Size (Cohen's d): {d_val:.4f}")
# Effect Size (Cohen's d): 0.1553

This is the effect-size check Dev runs after the significance test — the “okay, it’s statistically real, but is it big enough to matter?” question that determines whether the retrained recommender is worth shipping.

• def cohens_d(x, y): — defines Cohen’s d, the most common effect-size metric. x and y are the two samples being compared — in Dev’s case, the absolute errors of Model A and Model B. A positive d means Model A has larger errors (Model B is better); a negative d means the opposite.
• nx, ny = len(x), len(y) — the sample sizes. Both are 1,500 in this simulation. Cohen’s d accounts for unequal sample sizes through the pooled standard deviation.
• var_x, var_y = np.var(x, ddof=1), np.var(y, ddof=1) — sample variances for each model’s errors. ddof=1 means divide by $n-1$ (sample variance, unbiased estimator) rather than $n$ (population variance). This is the standard choice when computing effect sizes from sample data.
• pooled_sd = np.sqrt(((nx - 1) * var_x + (ny - 1) * var_y) / (nx + ny - 2)) — the pooled standard deviation, which combines the two samples’ variances into a single spread estimate, weighted by sample size. The formula is $\sqrt{((n_x - 1)s_x^2 + (n_y - 1)s_y^2) / (n_x + n_y - 2)}$. This is the denominator of Cohen’s d — it standardizes the mean difference so the result is on a “standard deviation” scale.
• return (np.mean(x) - np.mean(y)) / pooled_sd — Cohen’s d: the difference in means divided by the pooled standard deviation. This tells you the improvement in terms of standard deviations — a dimensionless number that’s comparable across datasets and metrics.
• d_val = cohens_d(np.abs(y_real - reg_a), np.abs(y_real - reg_b)) — calls the function with Model A’s absolute errors as x and Model B’s absolute errors as y. If Model B is better (smaller errors), d_val will be positive — the mean of Model A’s errors minus the mean of Model B’s errors, divided by their pooled spread.
• print(f"Effect Size (Cohen's d): {d_val:.4f}") — prints the effect size. Dev checks this number alongside the p-value: a significant p-value with a tiny d means the improvement is real but practically negligible — probably not worth the deployment risk.

• A Cohen’s d of 0.2 is small.
• 0.5 is medium.
• 0.8 or higher is large.

The regression demo above lands at d = 0.155 — below even the “small” line, despite a p-value under 0.0001. That pairing is worth sitting with: the reduction in error is real and statistically detectable at 1,500 rows, but it is also genuinely modest. A significant p-value with a small effect size doesn’t mean you’ve found nothing; it means you’ve found something real that may or may not be worth the cost of shipping it — which is a judgment call about the business, not a statistical one.

What we covered:

1. A higher score might just be a ‘Lucky Guess’ due to sampling error.
2. The McNemar Test checks whether classifier disagreements are balanced.
3. The p-value is a ‘Surprise Meter’ — it tells you how likely a result is a fluke.
4. The Diebold-Mariano Test checks whether error reductions in regression are consistent.
5. Effect Size tells you whether a ‘significant’ finding matters to the bottom line.

Next time you see a leaderboard, don’t just glance at the top number. Ask for the p-value.

Dev runs the McNemar test on his two recommenders: p = 0.56. Not significant — with 1,000 test rows and disagreement counts of 114 versus 124, the gap is well within what noise alone would produce. He can’t call Model B the winner, but he can’t rule it out either; a 1% accuracy bump on 1,000 rows is simply too small a sample to settle. Rather than ship on a hunch or scrap a retrain that might be genuinely better, he does what the numbers are actually telling him to do: gather more evidence before committing all ten million users. That’s what shadow traffic and a canary deployment are for — not a formality after the statistics already said yes, but a way to keep accumulating disagreement counts on real production traffic until the McNemar test has enough to say something definitive.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What does the McNemar Test focus on, and why does it ignore the rows where both models agree?

Understand In your own words, explain what a p-value of 0.03 actually means, using the article’s “Surprise Meter” framing—and what common misinterpretation the article warns against.

Apply Using the article’s interpret_p function logic (threshold=0.05), would a p-value of 0.049 be classified as “Statistically Significant” or “Not Significant”?

Analyze The article’s “Good Enough” Rule section says with 10 million rows, even a 0.0001% improvement can be statistically significant. Walk through why sample size specifically drives p-values down for a fixed effect size, and why this makes Effect Size (like Cohen’s d) a necessary companion metric rather than an optional extra.

Evaluate The article’s Model A/B confidence interval example shows Model A’s CI [82.6%, 87.0%] contains Model B’s 85.8% accuracy. Critique the reasoning “B’s point estimate falls inside A’s CI, so we can’t be sure it’s actually better”: is that actually the correct test for whether two models differ significantly, or does it conflate two different statistical questions?

Create Design a model-comparison report for a stakeholder: your team’s new fraud model has p=0.001 (highly significant) but a Cohen’s d of 0.03 (tiny effect) when compared to the current production model. Write the one-paragraph recommendation you’d give, using the article’s distinction between statistical significance and practical importance.

---

Related articles

• When Should You Retrain? Building a Simple Retraining Trigger) — Dev’s retraining trigger fired and produced a candidate model. This article shows how he checks whether that candidate is actually better before shipping it.
• A/B Testing Deployed Models: Shadow Deployments and More) — Dev’s McNemar test came back inconclusive on 1,000 rows. Here’s how he uses shadow traffic, canary deployments, and controlled A/B tests to keep gathering evidence on real production traffic without betting all 10 million users on an unproven model.

References & Further reading

• McNemar, Q. (1947). “Note on the sampling error of the difference between correlated proportions or percentages.” Psychometrika, 12(2), 153–157. — the original paper introducing the McNemar test for comparing two classifiers evaluated on the same test set. This is the foundation of the statistical comparison Dev uses to decide whether his retrained recommender is genuinely better than the incumbent.
• Dietterich, T. G. (1998). “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms.” Neural Computation, 10(7), 1895–1923. — the canonical reference on statistical significance testing for ML model comparison, covering the McNemar test, the 5×2 cross-validated paired t-test, and practical guidance on when each test is appropriate for a shipping decision.

---

Apply What You Learned

Topic: How sampling error creates uncertainty in model scores, and the pipeline of statistical tests (McNemar for classification, Diebold-Mariano for regression) and effect size (Cohen’s d) that turns a raw accuracy delta into a defensible ship/no-ship decision.

Draw a mindmap (paper, Excalidraw, Miro — anything) with at least these nodes:

• Sampling error (“Lucky Guess” problem)
• Confidence interval (95% CI with z = 1.96)
• McNemar test (contingency table, discordant cells b and c)
• Diebold-Mariano test (error difference series for regression)
• p-value (“Surprise Meter” under the null hypothesis)
• Effect size (Cohen’s d: 0.2 small, 0.5 medium, 0.8 large)
• Practical importance (“Good Enough” Rule: significance ≠ business value)

and at least these edges:

• Sampling error → Confidence interval
• McNemar test → p-value
• p-value → Effect size

Rubric: all 7 named nodes present; the 3 required edges drawn; one extra edge of your own with a one-sentence justification of why you added it.

Your team lead reviews Dev’s retrained recommender and says: “Model A hits 84.8% with a 95% CI of [82.6%, 87.0%], and Model B hits 85.8% — that falls inside A’s CI, so we can’t conclude B is better.” Write a 200–400 word stakeholder memo that either critiques or defends this reasoning.

Deliverable: A memo (plain text, 200–400 words) addressed to your team lead, taking a clear position on whether “Model B’s point estimate inside Model A’s CI” is sufficient evidence to block shipping.

Rubric:

• Cites the specific numbers from the article: Model A at 84.8%, CI [82.6%, 87.0%], Model B at 85.8%, n = 1000 test rows
• Identifies the statistical flaw: CI-overlap is a heuristic, not a proper pairwise hypothesis test — it treats the two models’ scores as independent when both are evaluated on the same 1000-row test set
• Names the McNemar test as the proper alternative and explains what it does: compares the discordant cells b (A right, B wrong) and c (B right, A wrong) rather than aggregate scores
• Correctly characterizes what a p-value does not mean — it is not “the probability that Model B is better” or “95% chance the difference is real”; it is the probability of seeing data this extreme if the models were identical
• Takes a clear position (defend or critique the team lead’s reasoning) and supports it with at least one specific argument from the article’s “Alternatives & Tradeoffs” comparison

Dev’s retraining trigger fires and produces a candidate model every time the Wasserstein distance on session_duration crosses 2.0. He needs a reusable model-comparison service that any nightly Airflow job can call to get a structured ship/no-ship verdict — not just “B scored higher” but a statistically defensible decision with effect-size gating.

Starter: projects/is-your-model-actually-better-a-plain-english-guid/model_comparison.py

Deliverable: Complete the model_comparison.py stub so that every # TODO function body is filled in and the __main__ smoke test runs end-to-end without errors. The module must export two entry points: ship_decision(y_true, pred_a, pred_b) for classification and diebold_mariano_test(y_true, pred_a, pred_b) for regression.

Rubric:

• compute_confidence_interval(0.85, 1000) returns approximately (0.828, 0.872) — uses the normal approximation with z = 1.96 and sqrt(p*(1-p)/n)
• build_contingency_table produces a 2×2 array where [0,1] = A-correct/B-wrong and [1,0] = B-correct/A-wrong (the discordant cells the McNemar test operates on)
• run_mcnemar calls mcnemar(ary=tb, corrected=True) — the continuity correction (Yates’) is applied, not skipped
• interpret_p_value returns “Statistically Significant” when p < 0.05 and “Not Significant” otherwise, and does not return any string claiming “95% chance the model is better”
• compute_cohens_d uses the pooled standard deviation with ddof=1 and returns a dimensionless float interpretable on the 0.2 / 0.5 / 0.8 scale
• ship_decision returns a dict with keys accuracy_a, accuracy_b, ci_a, p_value, effect_size, and verdict — where verdict is "SHIP" only when both p < 0.05 and |Cohen’s d| ≥ 0.2, "DO NOT SHIP — negligible effect" when p < 0.05 but d < 0.2, and "INCONCLUSIVE" when p ≥ 0.05
• diebold_mariano_test runs on the regression simulation (y_real, reg_a, reg_b) and produces a positive DM statistic with a two-tailed p-value < 0.05, confirming Model B’s errors are significantly smaller than Model A’s