You’ve designed a new ‘Buy Now’ button. It’s a vibrant shade of sunset orange, and you’re convinced it will outperform the old gray one. You launch an A/B test, and by lunchtime on Day 2, the orange button is winning by 20%. You’re thrilled—ready to call the race, declare victory, and head to happy hour.

But wait. This is the ‘Just Run It’ trap. Checking results every day is like peeking at a cake in the oven every two minutes. The heat escapes, and the top might look done while the middle is still liquid. In data science, this is called the Peeking Problem.

Stop a test the moment you see a ‘winner’ and you’re likely falling for a False Positive—seeing a pattern in what is actually random noise. Run a test that’s too small, though, and you might miss a billion-dollar idea because the data wasn’t loud enough to be heard over the static. That’s a False Negative. So before you hit the start button, you need to know exactly how much data to collect.

Think of Sample Size as a Microscope

What’s actually going on here? Think of your sample size as the lens of a microscope.

If you’re trying to spot an elephant (a massive 50% increase in sales), you don’t need much magnification. You’ll see that elephant even with a blurry, low-resolution view. But if you’re hunting for a tiny bacterium (a 0.5% increase in conversion), you need an expensive microscope with a massive lens.

In A/B testing, that lens is your sample size. The smaller the change you want to detect, the more users you need. This is where Minimum Detectable Effect (MDE) comes in—the smallest win you actually care about. If a change is so small it wouldn’t cover the cost of the developer’s time to build it, there’s no point measuring it.

Here’s the catch: we don’t aim for infinite samples because time is money. Every day spent testing is a day you aren’t testing something else. We need the Goldilocks zone—enough data to be confident, but not so much that we waste months.

The Four Knobs You Can Turn

Before touching any code, it helps to understand the four “knobs” on our testing machine. Forget the Greek letters for a moment — here’s what each one actually controls.

1. Baseline Conversion: Your starting point. If 10% of people currently click your button, 0.10 is your baseline.
2. MDE (Minimum Detectable Effect): The lift that makes the work worth doing. Do you care about a 5% improvement or a 20% improvement?
3. Significance Level (Alpha): Your strictness against being fooled by luck. Usually set to 0.05, meaning you accept a 5% chance of being wrong.
4. Statistical Power (1 - Beta): Your sensitivity. It’s the probability that if there is a winner, you’ll actually find it. Industry standard is 0.80 (80%).

Here’s what these look like as a Python dictionary:

# Our experimental setup
test_params = {
    'baseline_conversion': 0.10,  # 10% of users currently buy
    'mde': 0.02,                 # We want to detect at least a 2% absolute lift (to 12%)
    'alpha': 0.05,               # 5% chance of a false alarm
    'power': 0.80                # 80% chance of catching a real winner
}

print(f"Testing for a lift from {test_params['baseline_conversion']*100}% "
      f"to {(test_params['baseline_conversion'] + test_params['mde'])*100}%")

Let’s See What Happens: The Python Way

This is the part that trips people up. The math. But here’s the thing—hardly any data scientists do this by hand. We use a library called statsmodels to handle it.

We’re going to use something called TTestIndPower. Don’t let the name scare you; it’s just a calculator that asks, “How many people do I need?”

from statsmodels.stats.power import TTestIndPower
import statsmodels.stats.proportion as prop

# 1. Initialize the calculator
analysis = TTestIndPower()

# 2. Calculate 'Effect Size'
# This is a way to combine our baseline and MDE into one number for the math
effect_size = prop.proportion_effectsize(0.10, 0.12)

# 3. Calculate the sample size
sample_size = analysis.solve_power(
    effect_size=effect_size, 
    alpha=0.05, 
    power=0.80, 
    alternative='two-sided'
)

print(f"Required sample size per group: {round(sample_size)}")
print(f"Total users needed: {round(sample_size) * 2}")

What this actually means: If the output is 3,835, you need 3,835 users to see the old button and 3,835 users to see the new button. If your site gets 1,000 visitors a day, your test will take about 8 days. If the number was 50,000, you’d be looking at a 100-day test. So now you can tell your boss exactly how long to wait.

The ‘What If’ Game: Visualizing Trade-offs

Let’s check that against the data. A common mistake is assuming that detecting a 1% lift instead of a 2% lift takes just twice as much data.

The relationship is actually exponential. To detect smaller and smaller changes, the amount of data you need climbs steeply. Here’s the plot.

import numpy as np
import matplotlib.pyplot as plt

# Range of MDEs from 1% lift to 10% lift
mdes = np.linspace(0.01, 0.10, 10)
sizes = []

for m in mdes:
    es = prop.proportion_effectsize(0.10, 0.10 + m)
    n = analysis.solve_power(effect_size=es, alpha=0.05, power=0.80)
    sizes.append(n)

plt.figure(figsize=(10, 6))
plt.plot(mdes * 100, sizes, marker='o', color='#ff5733')
plt.title("The Cost of Greed: Sample Size vs. MDE")
plt.xlabel("Minimum Detectable Effect (%)")
plt.ylabel("Sample Size Required (Per Group)")
plt.grid(True, alpha=0.3)
plt.show()

That curve tells the story. Detecting a 10% lift requires very few users—around 200. But as you try to detect a 1% lift, the line shoots up toward 15,000 users. This visual helps stakeholders understand why we can’t test every tiny tweak. Some things just aren’t worth the time they take to measure.

Recap and Next Steps

You’ve just moved from guessing to science. Calculate your sample size upfront, and the daily temptation to peek at your numbers fades.

Here’s your new workflow:

1. Define your MDE: Decide the smallest win that actually matters to the business.
2. Turn the Knobs: Use 0.05 for Alpha and 0.80 for Power unless you have a very good reason not to.
3. Calculate ‘n’: Use Python to find your required sample size.
4. Set Expectations: Tell your team exactly how many days the test will run, and do not stop early.

Next up in this series: what to do once the test is over. We’ll walk through how to read a p-value without getting a headache.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the “Peeking Problem,” and why does stopping a test the moment you see a winner increase your risk of a false positive?

Understand In your own words, explain the “microscope” analogy — why does detecting a smaller MDE require a larger sample size than detecting a bigger one?

Apply Using the article’s four knobs (baseline conversion, MDE, alpha, power), if a team changes their power target from 0.80 to 0.90 without changing anything else, would you expect the required sample size to go up or down?

Analyze The article’s curve shows sample size exploding as MDE shrinks — roughly 200 users for a 10% lift versus roughly 15,000 for a 1% lift, a ~75x increase for a 10x smaller effect. Walk through why this relationship is not linear (i.e., why halving the MDE doesn’t just double the required sample size).

Evaluate The article’s workflow says to “use 0.05 for Alpha and 0.80 for Power unless you have a very good reason not to.” Critique this default for a high-stakes scenario: a company testing a pricing change where a false positive (rolling out a price increase that actually hurts revenue) would be very costly. Would you keep alpha at 0.05, or adjust it, and why?

Create Design a sample-size calculation for a new test: a checkout page redesign where the current baseline conversion is 25%, and the team only cares about lifts of 3 percentage points or more. Using the article’s proportion_effectsize + solve_power pattern, describe the calculation steps you’d run, and what you’d tell the team if the required sample size exceeds their monthly traffic.

---