Reference: Significance Tests

This is the catalog of individual significance tests — what each one tests, the statistic it uses, when it applies, and what breaks when its assumptions don’t hold. For the framework that wraps around these tests (null vs. alternative, Type I/II errors, power, p-values as a concept), see Reference: Hypothesis Testing. This file does not re-explain that framework; it assumes you know what a p-value is and want to know which test produces it.

A significance test has three parts: a null hypothesis (H0) — the boring world you’re trying to reject; a test statistic — a single number computed from your sample; and a sampling distribution for that statistic under H0 — the values it would take if you reran the experiment infinitely many times with H0 true. The choice of test is almost entirely determined by the shape of your data (paired? independent? categorical? continuous?) and the assumptions you can defend (normality? equal variances? large sample?). Get those two questions right and the test picks itself.

Roster

Test	Null hypothesis	Statistic → sampling distribution	When to use	Used in corpus
z-test	$\bar{x} = \mu_0$ (with $\sigma$ known)	$z = \frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}}$ → $N(0,1)$	One sample, $\sigma$ known (rare in practice)	Confidence Intervals)
One-sample t-test	$\bar{x} = \mu_0$ (with $\sigma$ unknown)	$t = \frac{\bar{x}-\mu_0}{s/\sqrt{n}}$ → $t(n-1)$	One sample, $\sigma$ unknown, roughly normal	What does a p-value actually mean)
Student’s t-test	$\mu_1 = \mu_2$	$t = \frac{\bar{x}_1 - \bar{x}_2}{s_p\sqrt{1/n_1 + 1/n_2}}$ → $t(n_1{+}n_2{-}2)$	Two independent groups, equal variances	Is your model actually better?)
Welch’s t-test	$\mu_1 = \mu_2$	$t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}$ → $t(\nu)$	Two independent groups, unequal variances (prefer this unless you can confidently defend equal variances)	Is your model actually better?)
Paired t-test	$\bar{d} = 0$	$t = \frac{\bar{d}}{s_d/\sqrt{n}}$ → $t(n-1)$	Same units measured twice (pre/post, two models on the same test set)	Is your model actually better?)
F-test (variances)	$\sigma_1^2 = \sigma_2^2$	$F = s_1^2 / s_2^2$ → $F(n_1{-}1,\, n_2{-}1)$	Comparing two normal-sample variances; pre-check for Student’s t	—
F-test (regression)	All $q$ excluded coefficients = 0	$F = \frac{(SSR_r - SSR_u)/q}{SSR_u/(n-k)}$ → $F(q, n{-}k)$	Nested linear-model comparison, normal errors	ARIMA & SARIMA
Wald test	$\theta = \theta_0$	$W = (\hat\theta-\theta_0)^\top V^{-1}(\hat\theta-\theta_0)$ → $\chi^2(q)$	Large-sample coefficient restrictions in MLE models	ARIMA & SARIMA
Ljung-Box	No autocorrelation up to lag $m$	$Q = n(n+2)\sum_{k=1}^m \frac{\hat\rho_k^2}{n-k}$ → $\chi^2(m)$	Time-series residual whiteness check	ARIMA & SARIMA
Chi-square (independence)	Two categorical variables independent	$\chi^2 = \sum\frac{(O-E)^2}{E}$ → $\chi^2((r{-}1)(c{-}1))$	Counted data in a contingency table	Detecting data drift)
Mann-Whitney U	Two independent samples share a distribution	$U = \sum \mathbb{1}_{x_i < y_j}$ → normal approx	Nonparametric alternative to two-sample t-test	Is your model actually better?)
Wilcoxon signed-rank	Median of paired differences = 0	$W = \sum \text{rank}(\lvert d_i\rvert)\,\text{sign}(d_i)$ → normal approx	Nonparametric alternative to paired t-test	Is your model actually better?)
KS (one-sample)	Sample CDF = reference $F_0$	$D = \sup_x \lvert F_n(x) - F_0(x)\rvert$ → Kolmogorov dist.	Distribution comparison against a known CDF	Detecting data drift)
KS (two-sample)	Two sample CDFs equal	$D = \sup_x \lvert F_{n_1}(x) - F_{n_2}(x)\rvert$	Comparing two empirical distributions	Detecting data drift)
Permutation test	Group labels exchangeable	Any statistic; distribution built by relabeling	Exact test when no parametric assumption is defensible	The Bootstrap

Where the test statistics come from

Every test in this catalog has the same skeleton:

$$\text{statistic} \;=\; \frac{\text{signal}}{\text{noise}}$$

where “signal” is the discrepancy between your data and the null, and “noise” is the standard error of that signal under H0. The distribution the statistic follows depends on what you assume about the noise.

From z to t

If $\sigma$ is known, the sample mean’s standard error is $\sigma/\sqrt{n}$ and

$$z \;=\; \frac{\bar x - \mu_0}{\sigma/\sqrt{n}} \;\sim\; N(0,1) \quad\text{under } H_0: \mu = \mu_0.$$

When $\sigma$ is unknown we plug in $s$ (the sample standard deviation). The plug-in is no longer normal because $s$ itself is random:

$$t \;=\; \frac{\bar x - \mu_0}{s/\sqrt{n}} \;=\; z \cdot \frac{\sigma}{s} \;\sim\; t(n-1).$$

The $t$ distribution has heavier tails than $N(0,1)$ — that is the price of estimating $\sigma$ from the same data. At $n=30$ the two are nearly indistinguishable when $s\approx\sigma$; when the realized sample standard deviation differs from the assumed $\sigma$, the two statistics themselves diverge regardless of $n$ — see worked example 1 below, where that happens at $n=30$.

Welch-Satterthwaite degrees of freedom

For two samples with unequal variances the standard error

$$\text{SE} \;=\; \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$$

is a weighted sum of $\chi^2$ variables, not a single $\chi^2$. Welch-Satterthwaite matches the first two moments of this sum to a $t(\nu)$ distribution by choosing

$$\nu \;=\; \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{(s_1^2/n_1)^2}{n_1 - 1} + \frac{(s_2^2/n_2)^2}{n_2 - 1}}.$$

$\nu$ falls between $\min(n_1, n_2)-1$ and $n_1 + n_2 - 2$, recovering the Student’s t-test degrees of freedom exactly when $s_1^2/n_1 = s_2^2/n_2$.

The trinity for coefficient restrictions

Wald, likelihood-ratio, and score tests all ask the same question — “does imposing this restriction make the fit materially worse?” — but evaluate it at different points. All three converge to $\chi^2(q)$ under $H_0$, where $q$ is the number of restrictions.

Plain English	Statistical symbol	Python equivalent
Parameter estimate minus hypothesized value	$\hat\theta - \theta_0$	params - theta_0
Estimated covariance of the parameter	$V = \text{Cov}(\hat\theta)$	model.cov_params()
Wald statistic (squared standardized distance)	$W = (\hat\theta-\theta_0)^\top V^{-1}(\hat\theta-\theta_0)$	(theta - theta_0) @ inv(V) @ (theta - theta_0)
Likelihood-ratio statistic	$LR = -2(\ell_r - \ell_u)$	2 * (llf_u - llf_r)
Score statistic	$S = U^\top I^{-1} U$	(rarely computed by hand; exposed by some statsmodels methods)

Decision tree: which test for which data shape

Start at the top. Every fork is a question you can answer by looking at your data.

• One group, one number you’re comparing against (e.g., “is our sample mean different from 100?”)
  • $\sigma$ known and data normal and $n$ large enough → z-test (rare in practice; $\sigma$ is usually unknown)
  • $\sigma$ unknown, data roughly symmetric and unimodal → one-sample t-test
  • Data skewed or ordinal → Wilcoxon signed-rank (symmetric differences) or permutation test on the mean
• Two independent groups, comparing their locations (treatment vs. control; model A vs. model B on disjoint test sets)
  • Both normal, variances equal → Student’s t-test
  • Both normal, variances clearly unequal → Welch’s t-test
  • Not normal but ordinal/continuous, similar shapes → Mann-Whitney U
  • Nothing defensible → permutation test on the mean difference
• Two paired/repeated measurements on the same units (pre/post; same test set evaluated by two models)
  • Differences $d_i = y_i - x_i$ roughly normal → paired t-test
  • Differences not normal → Wilcoxon signed-rank
  • Categorical 0/1 outcome paired → McNemar’s test (not in this catalog)
• Comparing the full distribution, not just the mean (data drift, two-sample goodness of fit)
  • One sample vs. a known reference CDF → KS one-sample
  • Two samples vs. each other → KS two-sample
  • Categorical distributions → chi-square (independence for two samples; goodness of fit for one)
• More than two groups
  • Normal, equal variances → one-way ANOVA (a regression F-test in disguise)
  • Non-normal → Kruskal-Wallis (not in this catalog)
• Variance comparison specifically (deciding between Student’s t and Welch’s t)
  • Two normal samples → F-test on variances (fragile; many skip it and just use Welch)
• Time-series diagnostics (residuals from a fitted model)
  • “Are the residuals white?” → Ljung-Box at a chosen lag $m$
• Model coefficient restrictions (“drop these 3 features — does the fit get worse?”)
  • Linear regression, normal errors, finite sample → F-test (nested-model form)
  • Large-sample MLE (logistic, ARIMA, etc.) → Wald test when re-fitting the restricted model is expensive or you’re testing many restrictions against one fit; likelihood-ratio test when the restricted model is cheap to fit and small-sample reliability matters more than convenience
• None of the assumptions feel safe → permutation test with a statistic chosen to match the question (mean, median, AUC — anything computable)

Worked examples

The code uses scipy.stats, statsmodels, and numpy. The blocks share one rng — seeded once, at the top of block 1 — and run in reading order: each block’s random draws depend on every draw made in the blocks before it. Copy them top to bottom in one session, or copy the two-line import-and-seed preamble from block 1 into whichever block you want to run standalone. The numbers shown below every block are the actual, verified output of running all eleven blocks once, in order.

1. z-test and one-sample t-test

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)
sample = rng.normal(loc=102, scale=5, size=30)  # true mean 102, sd 5

# --- z-test (sigma known = 5, from historical data) ---
xbar = sample.mean()
z = (xbar - 100) / (5 / np.sqrt(len(sample)))
p_z = 2 * (1 - stats.norm.cdf(abs(z)))
print(f"z = {z:.3f}, p = {p_z:.4f}")
# z = 2.283, p = 0.0224

# --- one-sample t-test (sigma unknown, estimated from sample) ---
t_stat, p_t = stats.ttest_1samp(sample, popmean=100)
print(f"t = {t_stat:.3f}, p = {p_t:.4f}, df = {len(sample)-1}")
# t = 2.939, p = 0.0064, df = 29

The two p-values do not match here (0.0224 vs. 0.0064), even though $n=30$ is normally large enough that $t(29)\approx N(0,1)$. The reason is that the z-test’s assumed $\sigma=5$ and the sample’s actual standard deviation (3.88) differ: the z-test treats $\sigma=5$ as known and pays no penalty for that assumption being wrong, while the t-test estimates $s=3.88$ from the data — a smaller denominator, a larger statistic, a smaller p-value. A known variance is an assumption with teeth, not a formality: get it wrong and the z-test’s p-value is wrong too, silently, with no warning in the output.

2. Student’s t-test vs. Welch’s t-test

group_A = rng.normal(loc=10, scale=2, size=50)   # variance ~ 4
group_B = rng.normal(loc=11, scale=4, size=50)   # variance ~ 16 -- 4x unequal

# Student's t-test (assumes equal variances)
t_stu, p_stu = stats.ttest_ind(group_A, group_B, equal_var=True)
print(f"Student's t = {t_stu:.3f}, p = {p_stu:.4f}")
# Student's t = -0.472, p = 0.6376

# Welch's t-test (allows unequal variances)
t_welch, p_welch = stats.ttest_ind(group_A, group_B, equal_var=False)
print(f"Welch's t = {t_welch:.3f}, p = {p_welch:.4f}")
# Welch's t = -0.472, p = 0.6381

The two test statistics coincide (both −0.472) — not because the group means are the same (they aren’t: 10.062 vs. 10.306), but because the two sample sizes are equal ($n_1=n_2=50$), which makes the pooled and unpooled standard errors algebraically identical. Only the degrees of freedom differ: Welch’s $\nu \approx 69.85$ against Student’s exact $n_1+n_2-2=98$, which is why the p-values diverge slightly (0.6376 vs. 0.6381, a gap of about 0.0004) even though the statistic doesn’t. With unequal sample sizes the two statistics themselves would diverge — and that’s exactly when the Student-vs-Welch choice starts to matter.

scipy.stats.ttest_ind(a, b, equal_var=False) does four things in one call:

1. Computes the two sample means $\bar x_1, \bar x_2$.
2. Computes the standard error $\sqrt{s_1^2/n_1 + s_2^2/n_2}$ without pooling.
3. Computes the Welch-Satterthwaite degrees of freedom $\nu$.
4. Looks up the two-sided p-value from $t(\nu)$.

In the example, the two samples have true variances of 4 and 16 — a 4× ratio. With equal_var=True, scipy pools them into $s_p^2$ as if they were equal. With equal_var=False, the heavier-tailed $t(\nu)$ with $\nu \approx 69.85$ (below $n_1+n_2-2=98$, reflecting the unequal variances) correctly accounts for the extra uncertainty in the variance estimate. Here the two group sizes are equal, so the statistics themselves land on the same value (−0.472) and only $\nu$ — and therefore the p-value — differs, by about 0.0004. With unequal sample sizes as well as unequal variances, the two statistics diverge too, and the gap between the two tests’ conclusions can grow large enough to flip a decision.

3. Paired t-test

# Same 100 customers, predicted spend under model_old and model_new
model_old = rng.normal(loc=50, scale=10, size=100)
model_new = model_old + rng.normal(loc=2.0, scale=3, size=100)  # +2 lift on average

t_paired, p_paired = stats.ttest_rel(model_new, model_old)
print(f"paired t = {t_paired:.3f}, p = {p_paired:.4g}")
# paired t = 6.526, p = 2.903e-09

The paired test is dramatically more powerful than a two-sample test on the same data because it removes the between-customer variance. You’re testing the mean of the differences, not the difference of the means.

4. Mann-Whitney U and Wilcoxon signed-rank

# Mann-Whitney needs two INDEPENDENT samples -- draw the second group
# fresh, not from the first. (Deriving group 2 from group 1 and running
# an independent-samples test on it is exactly the mistake the edge-case
# list below warns about.)
g1 = rng.lognormal(mean=1, sigma=0.8, size=80)
g2_indep = rng.lognormal(mean=1.3, sigma=0.9, size=80)

u_stat, p_mw = stats.mannwhitneyu(g1, g2_indep, alternative="two-sided")
print(f"U = {u_stat:.0f}, p = {p_mw:.4f}")
# U = 2643, p = 0.0575

# Wilcoxon needs genuinely PAIRED data -- construct a separate paired
# array rather than reusing g2_indep.
g2_paired = g1 + rng.normal(loc=0.5, scale=1.0, size=80)
d = g2_paired - g1
w_stat, p_w = stats.wilcoxon(d)
print(f"W = {w_stat:.0f}, p = {p_w:.4g}")
# W = 870, p = 0.0003216

Mann-Whitney U ranks both groups together and compares the rank sums. It tests whether a random observation from one group is equally likely to exceed a random observation from the other — not whether the medians differ, unless you also assume the two distributions have the same shape. Run on two genuinely independent groups here, it does not reject at the conventional 0.05 level (p = 0.0575) — the two lognormal populations really do overlap heavily (their correlation is 0.003, i.e. none). Run on the deliberately paired array a few lines later, the same-flavored Wilcoxon test rejects overwhelmingly (p = 0.0003), because pairing removes the between-unit variance the same way the paired t-test did in example 3. That contrast — not an accidental one this time — is the power difference the edge-case list below is warning you about.

5. Kolmogorov-Smirnov

# One-sample: does this sample come from N(0, 1)?
sample_for_ks = rng.normal(0, 1, size=200)
d_1, p_1 = stats.kstest(sample_for_ks, "norm")
print(f"D = {d_1:.3f}, p = {p_1:.4f}")
# D = 0.079, p = 0.1551  -> don't reject; consistent with N(0,1)

# Two-sample: do two distributions differ?
s1 = rng.normal(0, 1, size=300)
s2 = rng.normal(0.3, 1, size=300)
d_2, p_2 = stats.ks_2samp(s1, s2)
print(f"D = {d_2:.3f}, p = {p_2:.4g}")
# D = 0.167, p = 0.0004689  -> reject; the distributions differ

KS is sensitive to any difference in distributions (mean, variance, shape), which is why it’s the workhorse of data-drift detection. It is, however, underpowered for differences that show up only in the tails — Anderson-Darling weights tails more heavily and is often a better choice there.

6. Chi-square test of independence

# A 2x3 contingency table: rows = device type, cols = churn bucket
table = np.array([[120, 30, 10],
                  [80, 60, 40]])
chi2, p_chi, dof, expected = stats.chi2_contingency(table)
print(f"chi2 = {chi2:.3f}, dof = {dof}, p = {p_chi:.4g}")
# chi2 = 34.944, dof = 2, p = 2.582e-08

print("expected counts under independence:")
print(expected.round(1))
# [[ 94.1  42.4  23.5]
#  [105.9  47.6  26.5]]

The expected-count matrix is what the table would look like if rows and columns were independent — each row and column must sum to the same margins as the observed table (row totals 160/180, column totals 200/90/50). The test statistic is the sum of $(O-E)^2/E$ over all cells. Watch for cells with $E < 5$ — the $\chi^2$ approximation degrades badly there.

7. F-test (variance comparison)

# Two normal samples; test whether their variances are equal
a = rng.normal(0, 2, size=50)
b = rng.normal(0, 3, size=60)
f_stat = a.var(ddof=1) / b.var(ddof=1)
# Two-sided p-value (F is asymmetric, so we double the smaller tail)
p_left = stats.f.cdf(f_stat, dfn=49, dfd=59)
p_f = 2 * min(p_left, 1 - p_left)
print(f"F = {f_stat:.3f}, p = {p_f:.4f}")
# F = 0.430, p = 0.0029

The F-test for variances is extremely sensitive to non-normality — even mild skew inflates the false-positive rate. Many practitioners skip it and just default to Welch.

8. F-test for nested regression models

import statsmodels.api as sm

n = 200
X_full = rng.normal(size=(n, 4))
y = 1.0 * X_full[:, 0] - 0.5 * X_full[:, 2] + rng.normal(scale=1, size=n)

X_restricted = sm.add_constant(X_full[:, [0, 1]])      # drop columns 2 and 3 (X_full[:,2], X_full[:,3])
X_unrestricted = sm.add_constant(X_full)
r_model = sm.OLS(y, X_restricted).fit()
u_model = sm.OLS(y, X_unrestricted).fit()

# After add_constant, the exog names are const, x1, x2, x3, x4, mapping to
# X_full's columns 0-3 in order -- so X_full[:,2] and X_full[:,3] are named
# x3 and x4, not x2 and x3. Name the restriction to match what was dropped.
print(u_model.f_test("x3 = 0, x4 = 0"))
# <F test: F=19.178, p=2.493e-08, df_denom=195, df_num=2>

# Same statistic, from the sum-of-squares form this comparison is built on --
# using the restricted model that was already fit above, not discarding it:
q = 2                              # number of restrictions
k = X_unrestricted.shape[1]        # params in the unrestricted model
f_manual = ((r_model.ssr - u_model.ssr) / q) / (u_model.ssr / (n - k))
print(f"manual F from SSR = {f_manual:.3f}")
# manual F from SSR = 19.178

This is the regression F-test in its native form: comparing a “restricted” model (which imposes the null) to an “unrestricted” model. The test asks “how much sum-of-squares did we give up by dropping these variables, relative to the degrees of freedom lost?” — and the manual computation above, using r_model’s and u_model’s residual sums of squares directly, reproduces statsmodels’ own F-statistic exactly (19.178 both ways).

9. Ljung-Box (time-series residual autocorrelation)

from statsmodels.stats.diagnostic import acorr_ljungbox
import statsmodels.api as sm

# arma_generate_sample defaults to np.random.standard_normal for its noise,
# which ignores the rng object used everywhere else on this page and makes
# the block non-reproducible run to run. Pass distrvs=rng.standard_normal
# to route its randomness through the same seeded generator.
y = sm.tsa.arma_generate_sample([1, -0.6], [1, 0.0], nsample=500, distrvs=rng.standard_normal)
model = sm.tsa.arima.ARIMA(y, order=(1, 0, 0)).fit()
lb = acorr_ljungbox(model.resid, lags=[10], return_df=True)
print(lb)
#    lb_stat  lb_pvalue
# 10  6.642     0.7587  -> residuals look white at lag 10

A significant Ljung-Box on residuals means there’s still autocorrelation your model hasn’t captured — usually a sign you need more AR or MA terms, or a seasonal component.

10. Wald test

# A logistic regression; testing a linear restriction on coefficients
import statsmodels.formula.api as smf
import pandas as pd

df = pd.DataFrame({
    "y":  rng.binomial(1, 0.4, size=300),
    "x1": rng.normal(size=300),
    "x2": rng.normal(size=300),
    "x3": rng.normal(size=300),
})
logit = smf.logit("y ~ x1 + x2 + x3", data=df).fit(disp=0)
# H0: x2 - x3 = 0 (the two coefficients are equal)
# scalar=True silences a FutureWarning about wald_test's return type
# changing after statsmodels 0.14, and gives the scalar chi2 form below.
print(logit.wald_test("x2 = x3", scalar=True))
# <Wald test (chi2): statistic=0.264, p-value=0.6073, df_denom=1>

The Wald test compares $\hat\theta$ to $\theta_0$ scaled by the inverse of the estimated covariance matrix. It’s asymptotic — valid in large samples where the MLE is approximately normal.

11. Permutation test (the assumption-light fallback)

# Two groups with a weird, non-normal distribution and unequal variances
g1 = rng.lognormal(0, 1.2, size=40)
g2 = rng.lognormal(0.4, 1.5, size=40)

observed = g1.mean() - g2.mean()
combined = np.concatenate([g1, g2])
n1 = len(g1)

# Build the null distribution by relabeling 10,000 times
perm_stats = np.empty(10_000)
for i in range(10_000):
    idx = rng.permutation(len(combined))
    perm_stats[i] = combined[idx[:n1]].mean() - combined[idx[n1:]].mean()

p_perm = np.mean(np.abs(perm_stats) >= abs(observed))
print(f"observed diff = {observed:.3f}, p (permutation) = {p_perm:.4f}")
# observed diff = -7.304, p (permutation) = 0.0212

The only assumption is exchangeability under the null — that group labels carry no information if H0 is true. The p-value is the fraction of relabelings that produced a statistic at least as extreme as the one you observed.

The permutation test builds the null distribution by brute force:

1. Pool the two samples into one array of $n_1 + n_2$ values.
2. Randomly split the pooled array into two groups of sizes $n_1$ and $n_2$ — this is a relabeling under the null that group membership doesn’t matter.
3. Compute the mean difference for this random split.
4. Repeat 10,000 times to approximate the null distribution.
5. Compare: the p-value is the fraction of permuted statistics whose absolute value is at least as large as the observed one.

The single rng.permutation(len(combined)) call per iteration is the key — it produces one shuffle that partitions the pooled array into two halves. Calling rng.permutation(80) twice (a common bug) would produce two independent shuffles and break the logic.

For a two-sided p-value, take np.mean(np.abs(perm_stats) >= abs(observed)). For one-sided, drop the np.abs and pick the direction that matches your alternative. For statistics other than the mean (e.g., median, trimmed mean, AUC), just swap the mean() calls — the permutation framework is identical.

The scipy.stats.permutation_test function (added in SciPy 1.8) wraps this exact logic and supports a vectorized mode for speed; the hand-rolled version above is for transparency.

Edge cases and common mistakes

• Using the wrong t-test. Welch’s t-test costs a small amount of power when variances are genuinely equal, and is far more robust when they aren’t. If you can confidently defend equal variances — a controlled experiment, a known data-generating process, or a prior variance test — Student’s test is slightly more powerful; if you can’t defend it, default to Welch. R’s t.test defaults to Welch for exactly this reason.
• Misreading Mann-Whitney U. The null is not “the medians are equal” — it’s “a random draw from group 1 is equally likely to exceed a random draw from group 2.” Medians differ under H0 only if the shapes are equal; if shapes differ the medians can match while the test still rejects. Read it as a stochastic-dominance test, not a median test.
• Ignoring paired structure. Running ttest_ind (or Mann-Whitney) on paired data is the single most common power loss in practice. If you measured the same units twice (or evaluated two models on the same test set), the right test is paired — see worked example 4 above, where the same 80 units run through both an (independent) Mann-Whitney test and a (paired) Wilcoxon test with very different power.
• Chi-square with small expected counts. If any cell has $E < 5$, the $\chi^2$ approximation is unreliable. For $2 \times 2$ tables, switch to Fisher’s exact test (scipy.stats.fisher_exact). For larger tables, scipy.stats.chi2_contingency has no built-in Monte Carlo option — it takes only observed, correction, and lambda_ — so hand-roll a permutation test on the contingency table instead: shuffle the category labels, recompute the statistic each time, and compare to the observed value.
• KS with ties. The classical KS distribution assumes continuous data; ties make the test conservative. For discrete or heavily-tied data, Anderson-Darling or a chi-square on binned counts is often more appropriate.
• F-test for variances on non-normal data. The test is fragile — skew inflates the false-positive rate to 10-15% even at moderate sample sizes. Use Levene’s or Brown-Forsythe if you actually need a variance test, or skip the test and just use Welch.
• Ljung-Box at the wrong lag. A small lag $m$ can miss seasonality at longer lags; a large $m$ relative to $n$ dilutes power. A common rule of thumb is $m \approx \ln(n)$, or check several lags and look for systematic structure rather than relying on a single threshold.
• Wald vs. likelihood-ratio vs. score. These three are the classical trinity of asymptotic tests. Wald is the easiest to compute (you only need the unrestricted model) but is the least reliable in small samples — the covariance estimate can be unstable, and the statistic is not invariant to reparameterization. Prefer Wald when re-fitting the restricted model is expensive or you’re testing many different restrictions against one fit; prefer the likelihood-ratio test when the restricted model is cheap to fit and small-sample reliability matters more.
• One-sided vs. two-sided. The two-sided p-value is roughly twice the one-sided — but you must decide before looking at the data. Switching to one-sided after seeing the sign inflates your Type I error.
• Multiple testing. Running many tests on the same data inflates the family-wise error rate. The naive fix is Bonferroni (multiply p-values by the number of tests); the better fix when tests are correlated is Benjamini-Hochberg (control the false-discovery rate).
• Permutation tests are not assumption-free. You still need exchangeability under the null. If your groups have different variances and you’re testing means, a permutation test on the raw mean difference can be miscalibrated — consider permuting residuals from a model that captures the variance structure, or using a studentized statistic.
• Power before sample size. Every test in this catalog has a power formula. If you’re designing a study, do the power calculation before collecting data — statsmodels.stats.power has helpers for t-tests, F-tests, and chi-square.

Cross-references

• What does a p-value actually mean, and why practitioners misread it)
• Is your model actually better? A plain-English guide to A/B testing models)
• How to detect and handle data drift in production)
• ARIMA and SARIMA intuitively — when classical forecasting still wins
• Confidence intervals: what “95% confident” actually means)
• The bootstrap: estimating uncertainty without assumptions
• Reference: Hypothesis Testing — the framework this catalog sits inside

Further reading

• Lehmann, E. L., and Joseph P. Romano. Testing Statistical Hypotheses (Springer, 3rd ed.). The classical graduate text; covers the Wald/LR/score trinity and the decision-theoretic framing in depth.
• Conover, W. J. Practical Nonparametric Statistics (Wiley, 3rd ed.). The reference for Mann-Whitney, Wilcoxon, KS, and the rest of the rank-based family.
• Library docs:
  • scipy.stats — the standard one-sample, two-sample, and rank-based tests.
  • statsmodels.stats.diagnostic — Ljung-Box, Breusch-Pagan, White, and other model-diagnostic tests.
  • statsmodels.stats.weightstats — Welch’s t-test variants and the CompareMeans interface.
  • statsmodels.stats.anova — the regression F-test and ANOVA tables.
  • scipy.stats.permutation_test (SciPy 1.8+) — a flexible permutation-test interface for arbitrary user-supplied statistics.