The 'Liar's Paradox' in Data: Why Testing Too Much Leads to False Discoveries

Picture this: you’re a scientist trying to prove that jelly beans cause acne. You run a study, look at the data, and find no link. But you don’t stop there. You test twenty different colors. Purple? No link. Brown? No link. Green? Aha! The data shows a statistically significant link with a p-value of 0.05.

So did you just make a medical breakthrough? Probably not. You likely fell victim to the “Liar’s Paradox” of statistics, also known as the Multiple Testing Problem.

1. The Jelly Bean Problem: How to Lie with Statistics by Accident

If you flip a coin 20 times and get 20 heads, you’d call it a miracle. But have 1,000,000 people each flip a coin 20 times and someone will get 20 heads by pure luck. Almost certainly.

In data science, every time you check a new feature, run a new A/B test variant, or look at a new subgroup, you’re flipping that coin. Use the standard 5% threshold (p < 0.05) to decide if something is “real.” You’re saying you’re willing to be wrong 1 out of every 20 times.

Run 20 independent tests where there’s absolutely no real effect. Your chance of seeing at least one “fake” result is 1 - (0.95^20). That’s about 64%. More likely than not, you find a lie. This is the Family-Wise Error Rate (FWER).

Here’s what that looks like in Python. We’ll generate 20 variables of pure random noise and count how many “significant” results turn up.

import numpy as np
import pandas as pd
from scipy import stats

# Set seed for reproducibility
np.random.seed(42)

# Create 100 rows of data
# 'target' is random noise. 'features' are 20 columns of random noise.
n_rows = 100
n_features = 20
target = np.random.normal(0, 1, n_rows)
features = np.random.normal(0, 1, (n_rows, n_features))

p_values = []
for i in range(n_features):
    # Calculate correlation and p-value for each feature
    _, p_val = stats.pearsonr(features[:, i], target)
    p_values.append(p_val)

results = pd.DataFrame({'feature': range(n_features), 'p_value': p_values})
significant = results[results['p_value'] < 0.05]

print(f"Total tests run: {n_features}")
print(f"Number of 'significant' results found: {len(significant)}")
print(significant)

What this actually means: Run it enough times with different seeds and you’ll eventually see it happen. The numbers are completely random — no real relationship — yet roughly 1 in 20 tests will cross the p < 0.05 line by chance. Across 20 tests, that’s a good chance of at least one “significant” result showing up even though nothing real is there. A junior analyst might report a lucky hit like that as a “key driver” to their boss. It’s just noise.

2. The Bonferroni Correction: The Strict Parent of Statistics

How do we stop these ghosts? The simplest way is the Bonferroni Correction. Think of it as the strict parent who says, “If you’re going to ask 20 questions, you better be 20 times more certain about the answers.”

To apply it, take your target alpha (usually 0.05) and divide it by the number of tests you ran.

• New Threshold = 0.05 / 20 = 0.0025

Now a result only counts as “real” if its p-value drops below 0.0025. That’s effective at stopping false positives (Type I errors). The tradeoff: it’s so strict that real discoveries often get caught in the same net. Statisticians call this “low power.”

Here’s how it looks on our previous results, using the statsmodels library.

from statsmodels.stats.multitest import multipletests

# Apply Bonferroni correction
rejected, p_adjusted, _, _ = multipletests(p_values, alpha=0.05, method='bonferroni')

results['bonferroni_significant'] = rejected
print(f"Significant results after Bonferroni: {rejected.sum()}")

Interpretation: Whatever “significant” results slipped through the uncorrected threshold get wiped out — Bonferroni’s 0.0025 bar is hard for chance alone to cross. This correction makes sense when the cost of a mistake is high, like a medical trial where a false positive could put a dangerous drug on the market.

3. The Benjamini-Hochberg Method: A Smarter Way to Filter

Bonferroni is often too mean. In most data science work, we can tolerate a few mistakes if it means we don’t miss the big picture. This is where the False Discovery Rate (FDR) and the Benjamini-Hochberg (BH) method come in.

Instead of trying to eliminate all false positives, BH ensures that, among the results you call significant, only a small fraction (say 5%) are actually fake.

It works on a sliding scale:

1. Rank your p-values from smallest to largest.
2. The smallest p-value gets the strictest threshold.
3. As the p-values grow, the threshold loosens slightly.

This preserves real signals that Bonferroni might have crushed. Let’s see it on a dataset where we’ve hidden some real signals among the noise.

# Create 100 features: 90 are noise, 10 have a real (but modest) effect on the target
np.random.seed(42)
real_effect = target.reshape(-1, 1) * 0.25 + np.random.normal(0, 1, (n_rows, 10))
noise = np.random.normal(0, 1, (n_rows, 90))
all_features = np.hstack([real_effect, noise])

p_vals_mixed = [stats.pearsonr(all_features[:, i], target)[1] for i in range(100)]

# Apply BH correction
rejected_bh, _, _, _ = multipletests(p_vals_mixed, alpha=0.05, method='fdr_bh')
rejected_bonf, _, _, _ = multipletests(p_vals_mixed, alpha=0.05, method='bonferroni')

print(f"Real signals found by Bonferroni: {sum(rejected_bonf[:10])}")
print(f"Real signals found by Benjamini-Hochberg: {sum(rejected_bh[:10])}")

Interpretation: With this seed, Bonferroni catches only 1 of the 10 real signals. Benjamini-Hochberg catches 4. BH usually finds more of the real signals than Bonferroni because it doesn’t demand the same extreme certainty for every test. You get a better list of leads to investigate.

4. Which One Should You Use? A Decision Guide

Picking a correction method is a business call, not just a math one.

• Use Bonferroni if: You’re doing “Confirmatory” research. One shot to prove a hypothesis, and a false positive is a disaster (e.g., launching a new engine design).
• Use Benjamini-Hochberg if: You’re doing “Exploratory” research. You’re sifting through 500 marketing variables to find the top 10. You’d rather have 12 leads — 2 of them might be wrong — than only 2.

Method	Strictness	Goal	Best For
No Correction	None	Find everything	Almost never (unless only 1 test)
Bonferroni	Extreme	Zero False Positives	Medical, Engineering, Legal
Benjamini-Hochberg	Moderate	Balance	Marketing, Genomics, A/B Testing

What we learned today:

1. The more you test, the more the data will lie to you (Multiple Testing Problem).
2. Bonferroni is the safest but harshest fix, dividing your alpha by the number of tests.
3. Benjamini-Hochberg is the “data scientist’s choice,” controlling the percentage of errors while keeping your model’s power high.

Next time you’re staring at a big table of p-values, remember: just because it’s significant doesn’t mean it’s true. Correct your p-values, or you’re just hunting ghosts.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the Family-Wise Error Rate (FWER), and what formula does the article use to calculate it for 20 independent tests?

Understand In your own words, explain why Bonferroni is described as having “low power” — what’s the tradeoff it makes to guarantee almost zero false positives?

Apply Using the article’s Bonferroni formula (alpha / number_of_tests), what would the corrected significance threshold be if you ran 200 tests instead of 20, starting from the same alpha of 0.05?

Analyze The article’s Benjamini-Hochberg example ranks p-values and applies a “sliding scale” threshold instead of one fixed cutoff for every test. Walk through why the smallest p-value in the batch gets the strictest threshold while larger p-values get a more relaxed one — what is this sliding scale trying to control that a single fixed threshold (like Bonferroni’s) doesn’t?

Evaluate The article’s decision table recommends Bonferroni for “Medical, Engineering, Legal” and BH for “Marketing, Genomics, A/B Testing.” Critique this categorization: is the right choice really determined by the industry, or by something else the article mentions earlier (the cost of a false positive vs. a missed discovery) that could point either way within the same industry depending on the specific decision being made?

Create Design a multiple-testing correction plan for a new scenario: a growth team ran 50 different subject-line variants in an email test and wants to identify which ones significantly beat the control. Given that acting on a false winner costs a few days of a mediocre subject line (low cost) but missing a real winner means leaving money on the table, would you recommend Bonferroni or Benjamini-Hochberg, and why?

---

Apply What You Learned

Scenario: Your team ran 50 feature-importance tests on a churn-prediction model. A junior analyst reported 8 “significant” features using uncorrected p < 0.05. Your VP asks: “Are these real, or did we just test enough things to get lucky?” Write a 200–400 word stakeholder memo defending your choice of Benjamini-Hochberg over both no correction and Bonferroni for this analysis.

Deliverable: A memo (200–400 words) that quantifies the false-positive risk of 50 uncorrected tests, explains why Bonferroni is too conservative here, and justifies BH with evidence from the article — all written for a non-technical VP.

Rubric:

• Computes FWER for 50 tests using the article’s formula 1 - (0.95^n) and states the resulting probability (~92%)
• Names the Bonferroni corrected threshold for 50 tests (0.05 / 50 = 0.001) and explains why this “low power” hurts this analysis
• References the article’s concrete evidence: Bonferroni caught only 1 of 10 real signals while BH caught 4 — as proof that BH’s sliding-scale threshold preserves more real discoveries
• Frames the recommendation as a business tradeoff (cost of chasing a false lead vs cost of missing a real churn driver), not just a statistical preference
• Stays within 200–400 words and is comprehensible to a reader who does not know what a p-value is