The Winner's Curse: Why Underpowered Experiments Inflate Your Results

1. The Mystery of the ‘Successful’ Failure

Imagine testing a new energy drink. You give it to 5 friends. By total chance, 4 of them feel a massive surge of energy. You run a quick statistical test. The p-value is 0.04. Success! It’s “statistically significant.” So, you tell the world you’ve found a miracle cure for tiredness.

Would you trust that result as much as a study of 5,000 people showing a smaller, steadier improvement? Probably not. Intuition says the small study feels “lucky.”

In data science, this is the Winner’s Curse. When an experiment has very few participants (low power), it doesn’t just miss real effects. It only catches the ones wildly exaggerated by random noise. If a small study finds a “significant” result, the effect size it reports is almost certainly a lie. Here’s what happens when we simulate this.

import numpy as np
import pandas as pd
from scipy import stats

def run_experiment(sample_size, true_effect=0.05):
    # Simulate two groups: Control and Treatment
    # We assume a small true improvement of 5%
    control = np.random.normal(loc=1.0, scale=0.5, size=sample_size)
    treatment = np.random.normal(loc=1.0 + true_effect, scale=0.5, size=sample_size)
    
    t_stat, p_val = stats.ttest_ind(control, treatment)
    return p_val, (treatment.mean() - control.mean())

# Small study: 10 people
np.random.seed(42)
small_results = [run_experiment(10) for _ in range(1000)]
significant_small = [res[1] for res in small_results if res[0] < 0.05]

# Large study: 1000 people
large_results = [run_experiment(1000) for _ in range(1000)]
significant_large = [res[1] for res in large_results if res[0] < 0.05]

print(f"Small study avg measured effect: {np.mean(significant_small):.3f}")
print(f"Large study avg measured effect: {np.mean(significant_large):.3f}")

In the code above, the true effect is 0.05. But in the small study, the average “significant” result will likely be around 0.40 or higher. That is nearly 8 times the truth! The large study stays much closer to 0.05. The hard part to swallow is this: a “significant” p-value in a tiny test is often a sign of high noise, not a high-quality discovery.

2. What is Power, Really? (The Smoke Detector Analogy)

We talk a lot about p-values — the risk of a False Positive — but Statistical Power gets less attention. Think of it as the sensitivity setting on a smoke detector.

• High Power: The detector is sensitive. It catches even a small, smoldering fire in the toaster. You won’t miss a fire.
• Low Power: The detector is dusty and old. It only triggers once the entire kitchen is engulfed in flames. A medium-sized fire, and it stays silent.

Power is the probability your test actually detects an effect when one truly exists. At 80% power, you’d catch the winner 80 times out of 100 runs and miss it the other 20. That 20% chance of missing the fire is what we call a Type II Error.

3. The Three Knobs You Can Turn

Power isn’t a random number that falls out of thin air. It comes from three specific knobs you can turn before running your experiment.

1. Knob 1: Sample Size. The simplest one. More data means more power — like upgrading to a sharper pair of glasses when you’re trying to make out something far away.
2. Knob 2: Effect Size. Are you chasing a 50% lift in sales or a 0.1% nudge? Big effects are easy to spot. Small ones take work.
3. Knob 3: Noise (Variance). High-variance data buries the signal. A whisper is easy enough to catch in a library. At a rock concert, not so much.

So let’s see how these knobs interact, using statsmodels.

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()

# How many people do we need to detect a 'small' effect (0.2) with 80% power?
sample_needed = analysis.solve_power(effect_size=0.2, alpha=0.05, power=0.8)
print(f"Sample size needed for small effect: {round(sample_needed)}")

# How many people for a 'large' effect (0.8)?
sample_needed_large = analysis.solve_power(effect_size=0.8, alpha=0.05, power=0.8)
print(f"Sample size needed for large effect: {round(sample_needed_large)}")

The output tells you that a small effect needs 393 people per group, while a large effect needs just 26. Translation: on a tight budget, you can only reliably detect the big wins.

4. Let’s See What Happens: The Power Curve

Visualizing power shows us where diminishing returns set in. We usually aim for 80% power. Why 80%? It’s an industry convention—one that balances the cost of collecting data against the risk of missing a win.

import matplotlib.pyplot as plt

sample_sizes = np.arange(5, 100)
effect_size = 0.5 # A medium effect

powers = [analysis.solve_power(effect_size=effect_size, nobs1=n, alpha=0.05) for n in sample_sizes]

plt.figure(figsize=(10, 5))
plt.plot(sample_sizes, powers, color='teal', lw=3)
plt.axhline(0.8, color='red', linestyle='--', label='80% Power Threshold')
plt.title("The Power Curve")
plt.xlabel("Sample Size (per group)")
plt.ylabel("Statistical Power")
plt.legend()
plt.grid(alpha=0.3)
plt.show()

The curve starts steep, then flattens. Going from 10 to 30 users gives you a big boost in power. Going from 80 to 100 gives you much less “bang for your buck.” This curve tells you exactly when to stop spending on more data.

5. The Danger Zone: Why Underpowered Tests Lie

This is the most dangerous part of data science. When power is low, we suffer from Magnitude Error (M-Error).

The test is ‘insensitive.’ The only way it reaches the threshold of ‘statistical significance’ is if random noise happens to push the result in the same direction as the effect. You end up reporting a ‘300% increase’ for a feature that actually only improved things by 5%.

This is why startups often see ‘miracle’ results in their first week of A/B testing — results that completely vanish once the test runs for a full month. The first week was underpowered. The ‘win’ was just a fluke.

6. Your Pre-Flight Checklist

Before you click ‘Start’ on an experiment, run through this workflow:

1. Estimate Baseline: What’s your current conversion rate or average? Say, 10% conversion.
2. Pick your MDE: What’s the Minimum Detectable Effect that actually matters? If a change comes in under 2%, is it worth the engineering time?
3. Calculate Time: Run a power calculator to see how long you’ll need to hit the required sample size.

Here’s a helper function for your own projects:

def power_check(baseline_mu, expected_lift_percent, std_dev):
    analysis = TTestIndPower()
    # Calculate effect size (Cohen's d)
    target_mu = baseline_mu * (1 + expected_lift_percent)
    effect_size = (target_mu - baseline_mu) / std_dev
    
    required_n = analysis.solve_power(effect_size=effect_size, power=0.8, alpha=0.05)
    
    print(f"--- Pre-Flight Report ---")
    print(f"To detect a {expected_lift_percent*100}% lift...")
    print(f"You need {int(np.ceil(required_n))} users per group.")
    print(f"If you have fewer, your results will be unreliable.")

# Example: Baseline 100 sales, looking for 5% lift, standard deviation of 20
power_check(100, 0.05, 20)

What if the required sample size is too big? Don’t run an underpowered test and hope for the best. A few options:

• Run it longer: Collect data for two weeks instead of one.
• Reduce Noise: Try CUPED (using historical data to ‘level’ the playing field), or target a more specific user segment to lower the variance.

Recap:

• Power is your experiment’s sensitivity.
• Underpowered tests give you the ‘Winner’s Curse’—exaggerated, lying results.
• Aim for at least 80% power before trusting a ‘significant’ p-value.
• If you can’t hit the sample size, reduce noise or look for larger effects.

Next in this series: ‘Multiple Comparisons’—what happens when you test 20 different things at once until something finally looks significant by accident.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three “knobs” that determine statistical power, according to the article?

Understand In your own words, explain the “Winner’s Curse” using the article’s smoke detector analogy — why does a low-power test that does trigger tend to report an exaggerated effect size?

Apply Using the article’s solve_power results (393 people needed for a small effect of 0.2, 26 for a large effect of 0.8), what does this tell you about the relationship between how subtle an effect is and how much data you need to reliably detect it?

Analyze The article says the power curve “starts steep and then flattens out” — moving from 10 to 30 users gives a big power boost, but 80 to 100 gives little. Walk through why this diminishing-returns shape happens mathematically (in terms of how standard error shrinks with sample size) rather than being an arbitrary property of the curve.

Evaluate The article recommends aiming for “at least 80% power” as an industry convention. Critique treating 80% as a universal target: for a medical trial where missing a real, harmful side effect (a Type II error) is far more costly than a false alarm, would 80% power still be an appropriate bar, or should the target shift?

Create Design a pre-flight power check (using the article’s power_check function pattern) for a new scenario: a mobile app team wants to detect a 3% lift in a baseline 40% retention rate, with a standard deviation of 15. Walk through what values you’d plug into the formula and what you’d do if the resulting required sample size were larger than your entire user base.

---