ANOVA, Explained for People Who Just Want to Compare Three Groups

1. Let’s get real for a second

You have three landing page designs — A, B, and C — and you need to know which one gets the highest click-through rate. You’ve already run a few hundred visitors through each page. Now you’re staring at three sets of numbers, wondering if the differences you see are real or just random noise.

The obvious move, the one most people try first, is to run a t-test for each pair. A versus B. A versus C. B versus C. Three t-tests, three p-values, you pick the ones below 0.05, and you’re done. Right?

Each t-test has a 5% chance of crying wolf — that’s what the 0.05 significance level means. But run three tests, and the chance that at least one of them cries wolf on its own jumps to about 14%. Here’s the exact math: 1 - (0.95)^3 ≈ 0.143. That’s nearly a 1 in 7 chance of finding a difference that isn’t there.

And it gets worse. If you were comparing four designs and ran six t-tests, the probability of at least one false positive climbs to 1 - (0.95)^6 ≈ 0.265. Six comparisons at the 0.05 level gives you a 26.5% chance of at least one Type I error. Three tests is a milder version of the same trap.

Let’s see this happen in practice. We’ll simulate three groups where the true means are identical — no real difference at all — then run three t-tests and count how often we get a false positive.

import numpy as np
from scipy import stats

np.random.seed(42)  # so you get the same results
n_reps = 3000
false_positives = 0

for _ in range(n_reps):
    # Three groups with the same true mean (50) and same standard deviation (10)
    group_a = np.random.normal(50, 10, 30)
    group_b = np.random.normal(50, 10, 30)
    group_c = np.random.normal(50, 10, 30)
    
    # Three t-tests
    p_ab = stats.ttest_ind(group_a, group_b).pvalue
    p_ac = stats.ttest_ind(group_a, group_c).pvalue
    p_bc = stats.ttest_ind(group_b, group_c).pvalue
    
    # If any p-value is below 0.05, that's a false positive
    if p_ab < 0.05 or p_ac < 0.05 or p_bc < 0.05:
        false_positives += 1

print(f"False positive rate with three t-tests: {false_positives / n_reps:.3f}")
print(f"Expected rate from 1 - (0.95)^3: {1 - (0.95)**3:.3f}")

Go ahead, run that code. You’ll see the false positive rate bounce around 14% — right where the math says it should be.