Reference: Hypothesis Testing

Roster

The Neyman-Pearson ritual in five steps

Every frequentist hypothesis test — no matter the test statistic — follows the same five-step recipe. Understanding the procedure is more important than memorising any particular formula, because the procedure is what guarantees the error rates you signed up for.

1. Pick a null hypothesis (H₀). This is the “baseline” or “boring” world: no difference, no effect, coefficient equals zero, model A and model B perform identically. The null is never “proven” — it is the thing you try to disprove.

2. Pick an alternative hypothesis (H₁). This is what you are looking for. A one-sided alternative (“B > A”) is more powerful but requires a pre-registered justification; a two-sided alternative (“B ≠ A”) is the default unless you have a strong directional prior.

3. Pick α. Before touching data, decide the false-positive rate you will tolerate. The convention is α = 0.05, but the right value depends on the cost of a false positive. If a false positive triggers a costly deployment, use α = 0.01 or lower.

4. Compute the test statistic from data. Reduce your sample to a single scalar (t, z, χ², F, D) using the formula dictated by your test choice.

5. Compare the statistic to its sampling distribution under H₀. The tail area beyond your statistic is the p-value. If p ≤ α, reject H₀; otherwise, fail to reject. The word “fail” is deliberate — you do not “accept” H₀, you merely withhold judgment.

Decision tree: which test for which question

The single biggest source of hypothesis-testing errors in practice is using the wrong test for the question. The tree below maps data shape and question type to the correct test.

• Is the question about a mean (or difference of means)?

  • One group, comparing to a known value → One-sample t-test (data approximately normal, n < 30) or z-test (n ≥ 30 or σ known).
  • Two independent groups → Welch’s two-sample t-test (unequal variances; the safe default) or Student’s t-test (if you verified equal variances with Levene’s test).
  • Two paired/matched groups (before vs after, same subjects) → Paired t-test.
  • Three or more groups → One-way ANOVA (if assumptions hold) or Kruskal-Wallis (non-parametric).
  • Data is heavily skewed or ordinal → Mann-Whitney U (independent) or Wilcoxon signed-rank (paired).

• Is the question about variances or spread?

  • Comparing two variances → F-test (if normal) or Levene’s test / Bartlett’s test (more robust).
  • Testing a single variance against a constant → Chi-square test for variance.

• Is the question about the shape of a distribution?

  • Comparing to a named theoretical distribution → Kolmogorov-Smirnov (KS) test (continuous distributions).
  • Comparing two empirical distributions → Two-sample KS test.
  • Better sensitivity in tails → Anderson-Darling test.

• Is the question about independence or association between categorical variables?

  • Two categorical variables, contingency table → Pearson’s chi-square test of independence.
  • Small samples or expected cell counts < 5 → Fisher’s exact test.
  • Ordinal categories → Mantel-Haenszel test.

• Is the question about whether residuals are autocorrelated (time series)?

  • Testing residual autocorrelation at multiple lags → Ljung-Box test (preferred; tests a group of lags jointly).
  • Testing autocorrelation at lag 1 only → Durbin-Watson test.
  • Testing for unit root (stationarity) → Augmented Dickey-Fuller (ADF) test.

• Is the question about whether two models have different performance?

  • Paired accuracy/AUC across folds or segments → Paired t-test or McNemar’s test (binary outcomes) or 5×2 cross-validated paired t-test.

Worked example: a two-sample t-test with scipy.stats

The most common hypothesis test in data science is the two-sample t-test: “Is the mean of group A different from the mean of group B?” Below is a complete, runnable example.

import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=42)

# Two groups: control (mean=50) and treatment (mean=53)
control  = rng.normal(loc=50, scale=10, size=100)
treatment = rng.normal(loc=53, scale=10, size=100)

# Welch's two-sample t-test (does NOT assume equal variances)
result = stats.ttest_ind(treatment, control, equal_var=False)

print(f"Mean control:    {control.mean():.3f}")
print(f"Mean treatment:  {treatment.mean():.3f}")
print(f"t-statistic:     {result.statistic:.4f}")
print(f"p-value:         {result.pvalue:.5f}")
print(f"95% CI for diff: {result.confidence_interval(0.95)}")

Expected output (will vary slightly with the RNG but reproducible at seed 42):

Mean control:    49.497
Mean treatment:  52.894
t-statistic:     2.7170
p-value:         0.00720
95% CI for diff: (0.930, 5.862)

With α = 0.05, since p = 0.007 ≤ 0.05, we reject the null that the means are equal. The 95% confidence interval for the difference does not include zero, which is consistent with the rejection.

Worked example: Bonferroni vs. Benjamini-Hochberg

When you run m hypothesis tests simultaneously, the probability that at least one is a false positive balloons. If each test has a 5% false-positive rate, running 20 independent tests gives a 1 − (0.95)²⁰ ≈ 64% chance of at least one false positive. This is the multiple comparisons problem.

Two corrections are standard:

• Bonferroni divides α by m and tests each p-value against α/m. It controls the family-wise error rate (FWER) — the probability of any false positive among all m tests. Extremely conservative: as m grows, almost nothing survives.

• Benjamini-Hochberg (BH) controls the false-discovery rate (FDR) — the expected proportion of false positives among the rejected tests. Sort p-values ascending, and reject all tests where $p_{(i)} \le \frac{i}{m} \cdot q$, with q the desired FDR (typically 0.05 or 0.10).

import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=7)

# 100 tests, 10 true alternatives, 90 nulls
m = 100
true_effects = 10

# Generate p-values: ~Beta(alpha,1) for true effects, Uniform(0,1) for nulls
p_nulls   = rng.uniform(0, 1, size=m - true_effects)
p_trues   = rng.beta(1, 20, size=true_effects)  # skewed small
pvals = np.concatenate([p_trues, p_nulls])

# --- Bonferroni ---
alpha_bonf = 0.05 / m
bonf_reject = pvals <= alpha_bonf
print(f"Bonferroni threshold: {alpha_bonf:.5f}")
print(f"Bonferroni rejections: {bonf_reject.sum()}")

# --- Benjamini-Hochberg ---
q = 0.05
ranked = np.sort(pvals)
thresholds = (np.arange(1, m + 1) / m) * q
# Find the largest k where p_(k) <= k/m * q, reject all up to k
k_max = np.max(np.where(ranked <= thresholds)[0]) if np.any(ranked <= thresholds) else 0
bh_reject = pvals <= ranked[k_max] if k_max > 0 else np.zeros(m, dtype=bool)
print(f"BH threshold (largest surviving p): {ranked[k_max]:.5f}")
print(f"BH rejections: {bh_reject.sum()}")

Expected output:

Bonferroni threshold: 0.00050
Bonferroni rejections: 1
BH threshold (largest surviving p): 0.00099
BH rejections: 3

BH finds more discoveries (3 vs. 1) because it tolerates a small fraction of false positives among the rejections rather than guarding against any false positive. For exploratory data analysis, genomics, or any high-throughput setting, BH is almost always the better choice. Bonferroni is appropriate when a single false positive is catastrophic (e.g., regulatory submissions, safety-critical systems).

from statsmodels.stats.multitest import multipletests

# Bonferroni
_, p_bonf, _, _ = multipletests(pvals, alpha=0.05, method='bonferroni')
print(f"Bonferroni rejections: {(p_bonf <= 0.05).sum()}")

# Benjamini-Hochberg
_, p_bh, _, _ = multipletests(pvals, alpha=0.05, method='fdr_bh')
print(f"BH rejections: {(p_bh <= 0.05).sum()}")

The power curve: why underpowered experiments lie

Power is the probability of detecting an effect that actually exists. It depends on three things: the effect size, the sample size n, and α. The power curve plots power against n for a fixed effect size and α, showing how detection probability climbs as you collect more data.

The classic failure mode: an underpowered experiment (n too small for the effect you’re trying to detect) produces a non-significant result, and you conclude “no effect.” But the test was never capable of detecting that effect in the first place. Worse, among the significant results that do survive, effect sizes are inflated (the “winner’s curse” or “type M error”).

import numpy as np
from scipy import stats

def compute_power(effect_size, n_per_group, alpha=0.05, two_sided=True):
    """Power of a two-sample t-test given n_per_group observations in each group."""
    # Non-centrality parameter
    ncp = effect_size * np.sqrt(n_per_group / 2)
    df = 2 * n_per_group - 2
    if two_sided:
        crit = stats.t.ppf(1 - alpha / 2, df)
        power = 1 - stats.nct.cdf(crit, df, ncp) + stats.nct.cdf(-crit, df, ncp)
    else:
        crit = stats.t.ppf(1 - alpha, df)
        power = 1 - stats.nct.cdf(crit, df, ncp)
    return power

# Power curves for three effect sizes (n = observations per group)
ns = np.arange(10, 501, 10)
for d, label in [(0.2, "small (d=0.2)"), (0.5, "medium (d=0.5)"), (0.8, "large (d=0.8)")]:
    powers = [compute_power(d, n) for n in ns]
    n_80 = next(n for n, p in zip(ns, powers) if p >= 0.80)
    print(f"{label}: need n={n_80} per group for 80% power")

Expected output:

small (d=0.2): need n=400 per group for 80% power
medium (d=0.5): need n=70 per group for 80% power
large (d=0.8): need n=30 per group for 80% power

The pattern is unambiguous: small effects require enormous samples. A Cohen’s d of 0.2 (a “small” effect by convention) needs ~400 observations per group (~800 total, since both arms need the same size) to reach 80% power. Many real-world A/B tests are attempting to detect effects of 0.01–0.05 in relative conversion rate — which demands tens of thousands of users per arm.

Edge cases and common mistakes

1. “The p-value is the probability that H₀ is true.” No. The p-value is computed assuming H₀ is true. It is P(data | H₀), not P(H₀ | data). Confusing these is the base-rate fallacy and is the single most common statistical error in published research.

2. Using a two-sided test when a one-sided test was pre-registered (or vice versa). One-sided tests have more power but require a directional hypothesis declared before data inspection. Switching to one-sided after seeing the data doubles the false-positive rate. Always pre-register your sidedness.

3. Assuming equal variances without checking. Student’s t-test (the default in many stats packages) assumes both groups share the same population variance. If they don’t, Type I error inflates. Use Welch’s t-test (equal_var=False) unless you have a reason not to.

4. Interpreting failure to reject as “no effect.” Non-significance does not mean the effect is zero — it means your test lacked the precision to distinguish the effect from zero. Always report the confidence interval alongside the p-value; a wide CI around zero tells you the test was underpowered, not that the effect is absent.

5. Multiple comparisons without correction. Running 20 A/B metrics without correction gives a ~64% chance of at least one false positive. If you are checking many outcomes, use BH (for discovery) or Bonferroni (for confirmatory).

6. Peeking at data and stopping early. Repeatedly checking p-values and stopping when p < 0.05 inflates the false-positive rate far above α. This is optional stopping, and it is one of the most damaging mistakes in online experimentation. Use sequential testing procedures (e.g., always-valid p-values, group sequential designs) if you need to peek.

7. Using the wrong test for the data shape. Applying a t-test to severely skewed data with n = 8. The t-test assumes the sampling distribution of the mean is approximately normal; for small n and skewed data, this fails. Use a non-parametric test (Mann-Whitney, Wilcoxon) or transform the data.

8. Confusing practical significance with statistical significance. With n = 1,000,000, any tiny difference is statistically significant. A 0.001% lift in conversion rate can yield p < 0.001 but may not justify the engineering cost of shipping the change. Always pair p-values with effect-size estimates and business-impact calculations.

9. Testing residuals for autocorrelation with the wrong lag structure. Using Durbin-Watson when you care about multiple lags; Durbin-Watson only tests lag-1 autocorrelation. Use Ljung-Box for joint multi-lag testing.

10. Forgetting that chi-square requires expected cell counts ≥ 5. With sparse contingency tables, Pearson’s chi-square approximation breaks down. Use Fisher’s exact test (2×2) or simulate the p-value (simulate_p_value=True in scipy.stats.chi2_contingency).

Cross-references

• What does a p-value actually mean and why practiti)
• Type I vs Type II errors how to actually manage th)
• Statistical power why underpowered experiments lie
• The liar s paradox in data why testing too much le)
• Stop guessing how to calculate a b test sample siz
• Is your model actually better a plain english guid)
• A B testing deployed models shadow deployments and)
• Think like a Bayesian a guide for frequentists who)

Further reading

Foundational papers:

• Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd. — The original articulation of significance testing and the p-value.
• Neyman, J., & Pearson, E. S. (1933). “On the problem of the most efficient tests of statistical hypotheses.” Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337. — The Neyman-Pearson lemma; the formal framework of α, β, and the rejection region.
• Benjamini, Y., & Hochberg, Y. (1995). “Controlling the false discovery rate: a practical and powerful approach to multiple testing.” Journal of the Royal Statistical Society, Series B, 57(1), 289–300. — The BH procedure; the most-cited paper in multiple-comparisons statistics.
• Wasserstein, R. L., & Lazar, N. A. (2016). “The ASA statement on p-values: context, process, and purpose.” The American Statistician, 70(2), 129–133. — The community’s official response to p-value misuse.

Library documentation:

• scipy.stats — Hypothesis tests (t-tests, KS, chi-square, Ljung-Box via statsmodels)
• statsmodels.stats.multitest — Multiple testing corrections (Bonferroni, BH, Holm, etc.)
• statsmodels.stats.power — Power and sample size (TTestIndPower, GofChisquarePower, etc.)

Kaggle competition / dataset:

• Kaggle: A/B Testing datasets — A collection of A/B test datasets ideal for practising two-sample tests, power calculations, and multiple-comparison corrections on realistic conversion-rate data.