Designing A Valid A B Test Common Mistakes That In

You’ve been there. The product team is buzzing about a new checkout button. They run a two-week A/B test, see a ‘significant’ 5% lift in conversions, and ship it to everyone. Revenue drops. What happened? The test was doomed before the first user saw it — not because the button was bad, but because the experiment’s design was broken from the start.

What if your experiment was invalid before it even started? How would you even know? That’s the scary part: most design flaws leave no obvious red flag. The numbers look fine. The p-value is small. But the result is a mirage.

In this article, we’ll walk through four pre-test design mistakes that silently invalidate A/B tests. By the end, you’ll have a practical checklist you can use to validate any experiment before you launch. No math degree required — just honest, careful thinking.

Here’s what we’ll cover:

• Mistake #1: Testing without a clear hypothesis
• Mistake #2: Running a test without planning the sample size
• Mistake #3: A broken control group
• Mistake #4: Ignoring multiple comparisons

Let’s dive in.

Mistake #1: Testing Without a Clear Hypothesis — ‘Let’s Just See What Happens’

This is the most common pre-test mistake. A team decides to ‘test the new button’ without specifying what success looks like. They’ll measure everything and see what sticks. This is like wandering around in the dark hoping to find treasure — you might stumble on something, but you won’t know if it’s real.

The intuition: hypothesis testing in plain English

Every A/B test is built on two competing ideas:

• Null hypothesis (H₀): The boring world where nothing changed. The new button and the old button perform the same.
• Alternative hypothesis (H₁): The interesting world where something real happened. The new button is different (better or worse).

A good hypothesis is a prediction you can prove wrong. It’s specific, falsifiable, and tied to a single metric.

Bad hypothesis: ‘The new button will do better.’

• Better how? Click-through rate? Conversion? Revenue per user? No direction.

Good hypothesis: ‘The new button will increase click-through rate by at least 2 percentage points compared to the old button.’

• Specific metric (click-through rate), minimum effect size (2 percentage points), direction (increase).

Why does this matter? Without a clear H₁, you can’t calculate the sample size you need (Mistake #2). You also open the door to p-hacking — running many analyses and reporting only the one that looks significant. As FigPii notes, ‘A hypothesis is what gives your test direction.’ [1]

This is one of the hardest parts for beginners: resisting the urge to ‘just explore.’ But exploration belongs in discovery phases, not in confirmatory experiments.

Let’s see what happens when we formalize a hypothesis in code

Here’s a simple Python function that checks whether the observed data supports the alternative hypothesis. It’s not a full statistical test — just a way to make the hypothesis concrete.

import numpy as np

def check_hypothesis(control_clicks, control_visitors, treatment_clicks, treatment_visitors, min_lift=0.02):
    """
    Check if the treatment group's click-through rate is at least `min_lift` higher than control.
    
    Parameters:
    - control_clicks, control_visitors: data from control group
    - treatment_clicks, treatment_visitors: data from treatment group
    - min_lift: minimum absolute increase in click-through rate (e.g., 0.02 = 2 percentage points)
    
    Returns:
    - True if the observed lift meets or exceeds min_lift, False otherwise.
    """
    # Calculate click-through rates
    ctr_control = control_clicks / control_visitors
    ctr_treatment = treatment_clicks / treatment_visitors
    
    # Observed lift
    observed_lift = ctr_treatment - ctr_control
    
    # Check if the lift meets the minimum threshold
    if observed_lift >= min_lift:
        return True, f"Observed lift {observed_lift:.3f} meets or exceeds {min_lift}. Hypothesis supported."
    else:
        return False, f"Observed lift {observed_lift:.3f} is below {min_lift}. Hypothesis not supported."

# Example: control has 100 clicks out of 2000 visitors, treatment has 150 clicks out of 2000 visitors
result, message = check_hypothesis(100, 2000, 150, 2000, min_lift=0.02)
print(message)
# Output: Observed lift 0.025 meets or exceeds 0.02. Hypothesis supported.

What’s happening here? We’re making our hypothesis concrete: we expect at least a 2 percentage point lift. The function checks if the observed data meets that threshold. This is a simplified version of what a proper hypothesis test does — it compares an observed effect to a pre-specified minimum meaningful effect.

But notice: we haven’t accounted for random variation. That’s where sample size planning comes in.

Mistake #2: Running a Test Without Planning the Sample Size — The Power Trap

You launch a test with 500 users per group. After two weeks, you see a 10% lift in conversions — and it’s statistically significant! Great, right? Not so fast. Underpowered tests (tests with too few users) don’t just fail to detect real effects — they actively mislead you by exaggerating the size of any effect they do detect.

Think of it like trying to hear a whisper in a noisy room. If you only listen for a second, you might think you heard something — but you’re likely wrong. If you listen for a longer time, you can be more confident. In A/B testing, ‘listening time’ is sample size.

What is statistical power?

Statistical power is the probability that your test will detect a real effect, if one exists. It’s like the sensitivity of your experiment. A power of 0.8 means you have an 80% chance of catching a real effect of a given size.

To calculate the sample size you need, you need four things:

1. Effect size — how big a change matters to your business (e.g., a 2% lift in conversion)
2. Significance level (α) — usually 0.05 (5% chance of false positive)
3. Power (1 - β) — usually 0.8 (80% chance of detecting a real effect)
4. Sample size per group — what we’re solving for

Here’s the dangerous part: if your test is underpowered, you’re more likely to miss real effects (false negatives). But even worse, the effects you do detect are likely to be overestimated. This is called the ‘winner’s curse’ — the significant results you see are inflated because only the largest effects survive the noise. Omniconvert explains: ‘Underpowered tests are not just inconclusive — they are misleading.’ [3]

Statsig adds: ‘Power analysis helps you determine the minimum detectable effect (MDE) — the smallest effect your test can reliably detect.’ [4]

Let’s calculate the required sample size with Python

We’ll use statsmodels to compute the sample size needed for a given effect size, alpha, and power.

import statsmodels.stats.power as smp

# Set parameters
effect_size = 0.02  # We want to detect a 2% absolute lift in conversion rate (e.g., from 10% to 12%)
alpha = 0.05        # Significance level (5% chance of false positive)
power = 0.8         # Desired power (80% chance of detecting the effect if real)

# Use the normal approximation for proportions
# We need to convert the absolute lift to Cohen's h effect size for proportions
# Cohen's h = 2 * arcsin(sqrt(p1)) - 2 * arcsin(sqrt(p2))
# But statsmodels has a convenience function: proportion_effectsize
from statsmodels.stats.proportion import proportion_effectsize

# Assume baseline conversion rate is 10% (0.10), and we want to detect a lift to 12% (0.12)
baseline = 0.10
target = baseline + effect_size  # 0.12

# Calculate Cohen's h
cohens_h = proportion_effectsize(baseline, target)
print(f"Cohen's h effect size: {cohens_h:.3f}")

# Now compute required sample size per group
# Using TTestIndPower (independent samples t-test) as an approximation for proportions
from statsmodels.stats.power import TTestIndPower
analysis = TTestIndPower()
sample_size_per_group = analysis.solve_power(
    effect_size=cohens_h,
    alpha=alpha,
    power=power,
    alternative='two-sided'
)

print(f"Required sample size per group: {int(np.ceil(sample_size_per_group))}")
# Output: Required sample size per group: 6204

What does this mean? To have an 80% chance of detecting a 2 percentage point lift (from 10% to 12% conversion), you need about 6,204 users in each group. If you only have 500 per group, you’d need a much larger effect (like 4-5 percentage points) to have the same power. That means you’d miss smaller but still valuable improvements.

VWO notes: ‘Sample size calculation is not optional — it’s the foundation of a valid test.’ [5]

Mistake #3: A Broken Control Group — The Silent Invalidator

The control group is the bedrock of an A/B test. It’s your baseline — what would have happened without the treatment. If the control group isn’t truly comparable to the treatment group, the test is invalid from the start.

What makes a control group valid?

• Random assignment: Every user has an equal chance of being in control or treatment.
• Simultaneous exposure: Both groups experience the test at the same time.
• Identical conditions: The only difference between groups is the treatment itself.

Common failures

1. Time-based splits during a holiday: If you show the treatment only during business hours and the control runs 24/7, you’re comparing different time periods — not the same conditions.
2. Selection bias: Showing the test only to logged-in users while the control includes everyone. Logged-in users may behave differently.
3. Novelty effect: Users behave differently because they know they’re in a test. This can inflate the treatment effect temporarily.

The A/A test: a diagnostic tool

An A/A test is when you run the same variant on both groups. If your randomization engine works correctly, you should see no significant difference between the two groups. If you do, something is broken — the randomization is flawed, or there’s a time-of-day bias, or some other confound.

FigPii recommends: ‘Run an A/A test to validate your testing tool before any real experiment.’ [1]

Let’s simulate a broken randomization

Here’s a Python simulation that shows how non-random assignment creates bias. We’ll simulate a scenario where the treatment group is shown during peak hours and the control during off-peak hours.

import numpy as np
from scipy import stats

np.random.seed(42)

# Simulate 10,000 users per group
n = 10000

# Control group: baseline conversion rate 10%, but during off-peak hours (lower traffic, same conversion)
# Treatment group: same baseline, but during peak hours when conversion is actually 12% due to higher intent
# This is a confound: the time of day affects conversion, not the treatment

# Let's create a biased assignment: treatment gets peak-hour users, control gets off-peak
# Peak hour conversion rate = 12%, off-peak = 10%
control_conversions = np.random.binomial(1, 0.10, n)
treatment_conversions = np.random.binomial(1, 0.12, n)  # But this 2% lift is due to time, not treatment

# Now run a two-sample proportion test
from statsmodels.stats.proportion import proportions_ztest

count = np.array([treatment_conversions.sum(), control_conversions.sum()])
nobs = np.array([n, n])

stat, p_value = proportions_ztest(count, nobs, alternative='two-sided')
print(f"Z-statistic: {stat:.3f}")
print(f"P-value: {p_value:.4f}")
print(f"Treatment conversion: {treatment_conversions.mean():.3f}")
print(f"Control conversion: {control_conversions.mean():.3f}")

What’s happening? The treatment group has a higher conversion rate, but it’s because they were shown during peak hours — not because of the treatment. The p-value is likely very small, suggesting a ‘significant’ effect. But it’s a false positive caused by a broken control group.

This is the hardest part: you can’t fix a bad control group after the data is collected. You have to get it right before the test starts.

Mistake #4: Ignoring Multiple Comparisons — The ‘Just Check Everything’ Trap

You’ve launched your test. You measure click rate, conversion, time on page, bounce rate, scroll depth, and 15 other metrics. You find that two of them show a significant p-value. Victory! But wait — the more things you check, the more likely you are to find something that looks real but isn’t.

The family-wise error rate

If you test 20 metrics at α=0.05, the chance of at least one false positive is:

FWER = 1 - (1 - α)^k

For k=20, α=0.05: FWER = 1 - (0.95)^20 ≈ 0.64. That’s a 64% chance of seeing at least one false positive!

MetricGate explains: ‘If you test 20 metrics, there’s a 40% chance of at least one false positive (actually 64% with α=0.05).’ [2]

The Bonferroni correction

A simple fix: divide α by the number of tests. If you’re testing 20 metrics, use α’ = 0.05 / 20 = 0.0025. This controls the family-wise error rate, but it’s conservative — it reduces power (more false negatives).

The best fix: pre-register a single primary metric

Before the test starts, decide what ‘success’ means. Pick one metric that matters most. If you must measure multiple metrics, pre-specify which ones and apply a correction.

Let’s simulate the multiple comparisons problem

import numpy as np
from scipy import stats

np.random.seed(123)

# Simulate 20 metrics with no real effect (all null hypotheses are true)
n_metrics = 20
n_users = 1000  # per group

# Generate data: control and treatment are identical (mean=0, std=1)
control_data = np.random.normal(0, 1, (n_users, n_metrics))
treatment_data = np.random.normal(0, 1, (n_users, n_metrics))

# Run t-tests for each metric
p_values = []
for i in range(n_metrics):
    t_stat, p_val = stats.ttest_ind(control_data[:, i], treatment_data[:, i])
    p_values.append(p_val)

# Count how many are significant at α=0.05
alpha = 0.05
false_positives = sum(p < alpha for p in p_values)
print(f"Number of false positives (no correction): {false_positives} out of {n_metrics}")
print(f"Expected false positives: {alpha * n_metrics:.2f}")

# Apply Bonferroni correction
alpha_bonferroni = alpha / n_metrics
false_positives_bonf = sum(p < alpha_bonferroni for p in p_values)
print(f"Number of false positives (Bonferroni correction): {false_positives_bonf} out of {n_metrics}")

What does this mean? Without correction, we expect about 1 false positive (0.05 * 20 = 1). In this run, we might see 1 or 2. With Bonferroni, we rarely see any false positives. But the trade-off is that we might miss a real effect if it’s small.

Putting It All Together: A Pre-Test Validation Checklist

Before you launch your next A/B test, run through this 5-minute sanity check. Each item addresses one of the mistakes we covered.

1. Is the hypothesis specific, falsifiable, and tied to a single primary metric?

   • Example: ‘The new checkout button will increase conversion rate by at least 1 percentage point.’
   • Source: FigPii [1]

2. Did we calculate the required sample size using power analysis? Do we have enough traffic to reach it?

   • Use statsmodels to compute sample size for your expected effect size, α=0.05, power=0.8.
   • Sources: Omniconvert [3], Statsig [4], VWO [5]

3. Is the control group truly random and simultaneous? Have we run an A/A test to verify?

   • Check for time-of-day, day-of-week, or user-segment biases.
   • Source: FigPii [1]

4. Have we pre-registered the primary metric and any corrections for multiple comparisons?

   • If measuring multiple metrics, apply Bonferroni or another correction.
   • Source: MetricGate [2]

5. Have we set a stopping rule (minimum sample size and fixed duration) to avoid peeking?

   • Don’t check results daily and stop early. Decide the sample size and duration beforehand.
   • Source: MetricGate [2]

A Python validation function

Here’s a simple function that checks these items and returns a pass/fail status.

def validate_ab_test_design(hypothesis_specific, sample_size_calculated, control_group_valid, 
                            primary_metric_preregistered, stopping_rule_set):
    """
    Validate an A/B test design against the pre-test checklist.
    All parameters are boolean flags indicating whether the condition is met.
    Returns a dictionary with overall status and details.
    """
    checks = {
        'Specific hypothesis': hypothesis_specific,
        'Sample size calculated': sample_size_calculated,
        'Valid control group': control_group_valid,
        'Primary metric pre-registered': primary_metric_preregistered,
        'Stopping rule set': stopping_rule_set
    }
    
    passed = all(checks.values())
    
    result = {
        'status': 'PASS' if passed else 'FAIL',
        'checks': checks,
        'message': 'All checks passed. Ready to launch!' if passed else \
                   'Some checks failed. Review the details below.'
    }
    return result

# Example usage: a test that fails on sample size and multiple comparisons
result = validate_ab_test_design(
    hypothesis_specific=True,
    sample_size_calculated=False,  # Not calculated
    control_group_valid=True,
    primary_metric_preregistered=False,  # Measuring 10 metrics without correction
    stopping_rule_set=True
)

print(result['status'])
print(result['message'])
for check, passed in result['checks'].items():
    status = '✓' if passed else '✗'
    print(f"  {status} {check}")

What’s happening? This function makes the checklist actionable. You can run it before every test to catch common mistakes.

Recap: What You Learned

Let’s quickly recap the four mistakes and how to avoid them:

• Mistake #1: No clear hypothesis → Write a specific, falsifiable hypothesis tied to one metric.
• Mistake #2: No sample size planning → Use power analysis to determine the required number of users.
• Mistake #3: Broken control group → Ensure random, simultaneous assignment; run an A/A test.
• Mistake #4: Ignoring multiple comparisons → Pre-register a primary metric or apply a correction.

Use the pre-test validation checklist before every experiment. It won’t guarantee a perfect test, but it will catch the most common traps that invalidate results before data collection begins.

Check Your Understanding

Test yourself with these questions, organized by Bloom’s taxonomy.

Remember: List the four mistakes that invalidate A/B tests before they start.

Understand: Explain in your own words why an underpowered test is dangerous, not just ‘inconclusive.’

Apply: A team wants to test a new feature with 500 users per group and 10 metrics. Identify which mistakes are present.

Analyze: Compare two test designs — one with a clear hypothesis and power analysis, one without — and explain which is more trustworthy.

Evaluate: A colleague says ‘We don’t need to calculate sample size; we’ll just run the test for two weeks and see.’ Write a short response explaining why this is risky.

Create: Design a valid A/B test for a hypothetical product change. Write the hypothesis, calculate the required sample size (using a Python library), describe the control group setup, and specify the primary metric and any corrections.