What Does a P-Value Actually Mean? (And Why Practitioners Keep Misusing It)

If you have ever looked at a data report, you have likely seen the term “p-value.” Usually it comes with a number like 0.05 or 0.001. Most people treat it like a “truth detector.” Small number, the result is real. Big number, noise.

But here is the catch: that is not what a p-value measures. Misinterpreting this single number is behind a great deal of bad science and poor business decisions. So let’s look at what it actually says.

1. The Ice Cream Problem: Why Your Intuition About P-Values Is Probably Wrong

Imagine you’re an analyst for a city. You notice that on days when ice cream sales go up, drowning deaths also rise. You run a statistical test, and the p-value comes back at 0.001.

In most offices, a p-value that small would be treated as “proof” that ice cream causes drowning. That’s absurd, of course. Both ice cream sales and swimming — which raises drowning risk — climb with summer heat. Heat is the confounder.

Here’s what the data looks like.

import pandas as pd
import numpy as np
from scipy import stats

# Create a fake dataset where Temperature drives both Ice Cream and Drowning
np.random.seed(42)
days = 100
temp = np.random.normal(75, 10, days)
ice_cream_sales = 20 + 2.5 * temp + np.random.normal(0, 10, days)
drownings = 1 + 0.1 * temp + np.random.normal(0, 2, days)

# Compute correlation and p-value
corr, p_val = stats.pearsonr(ice_cream_sales, drownings)

print(f"Correlation: {corr:.3f}")
print(f"P-value: {p_val:.5f}")

The correlation sits around 0.7, and the p-value is well below 0.05. If you looked only at the p-value, you might ban ice cream to save lives. This is the mistake practitioners make every day. They assume a small p-value means their specific theory is correct. It doesn’t. A small p-value just means the data is unlikely to have arisen by pure random chance, under a very specific set of assumptions.

2. What a P-Value Actually Is (In One Sentence)

Here’s the formal definition: a p-value is the probability of seeing data this extreme if the null hypothesis is true.

Think of the null hypothesis as the boring version of reality where nothing interesting is happening — say, ice cream has zero effect on drowning. The p-value asks: if we actually live in that boring world, how often would we see a result this weird just by luck?

It is NOT the probability that your hypothesis is true. It’s a conditional probability, full stop.

A coin flip helps build intuition. If I claim a coin is “fair” (the null hypothesis), but you flip it 100 times and get 65 heads, how surprised should you be?

# Simulate 10,000 experiments of 100 fair coin flips
n_flips = 100
simulations = 10000
results = np.random.binomial(n_flips, 0.5, simulations)

# How many times did we get 65 or more heads?
extreme_results = np.sum(results >= 65)
p_value = extreme_results / simulations

print(f"P-value for 65 heads: {p_value:.5f}")

The result is roughly 0.0018. So if the coin is fair, you’d only see 65+ heads about 0.18% of the time. That makes the “fair coin” theory look unlikely — but it doesn’t prove the coin is weighted. Someone could have just gotten very lucky (or unlucky).

3. The Threshold Trap: Why 0.05 Is Arbitrary (And Dangerous)

Why do we use 0.05? In the 1920s, a statistician named Ronald Fisher proposed 1 in 20 (0.05) as a convenient cutoff for being “fairly confident.”

Not a law of nature. Just a rule of thumb.

Today, that rule has hardened into a false binary. A p-value of 0.049 counts as a “discovery.” A p-value of 0.051 gets called “noise.” The difference between those two is almost nothing. This “Threshold Trap” drives the replication crisis — researchers chase a lucky 0.05, then can’t reproduce their own results.

# Simulate 100 experiments where there is NO real effect
# We expect ~5 of them to show p < 0.05 just by luck.
significant_count = 0
for _ in range(100):
    group_a = np.random.normal(0, 1, 100)
    group_b = np.random.normal(0, 1, 100)
    _, p = stats.ttest_ind(group_a, group_b)
    if p < 0.05:
        significant_count += 1

print(f"Number of 'significant' results found by accident: {significant_count}")

Even when there is absolutely no difference between groups, you’ll find “significant” results about 5% of the time. Run enough tests, and you’re guaranteed a p-value < 0.05.

4. P-Hacking and Garden of Forking Paths

This behavior is called p-hacking. Run 20 different analyses, and suppose only one yields a p-value of 0.04. You might publish just that one.

Statisticians call this the “Garden of Forking Paths.” Every time you drop an outlier or focus on a specific subgroup (like “men over 50”), you’re running a new test. Leave those hidden tests unaccounted for, and your p-value is a lie.

5. What P-Values Can’t Tell You (The Hard Part)

Here’s the part that trips people up: a small p-value does not mean the effect matters.

Give yourself a massive sample — say, 1 million users — and even a useless difference will register a p-value of 0.00001.

# Large sample size makes tiny effects 'significant'
n = 1000000
control = np.random.normal(100, 10, n)
test = np.random.normal(100.05, 10, n) # Only a 0.05% difference!

_, p = stats.ttest_ind(control, test)
print(f"P-value for tiny effect: {p:.5f}")

The p-value is microscopic. The effect is practically zero. Don’t mistake statistical significance for business significance.

6. Effect Size and Confidence Intervals: What You Should Report Instead

Report the Effect Size and Confidence Interval (CI) rather than stopping at “p < 0.05.”

• Effect Size: How big is the difference? (e.g., “We sold 5 more units per day”)
• Confidence Interval: What’s the range of likely values? (e.g., “We’re 95% sure the increase falls between 3 and 7 units”)

# Calculating a Confidence Interval
mean_diff = np.mean(test) - np.mean(control)
std_err = np.sqrt(np.var(test)/n + np.var(control)/n)
ci_lower = mean_diff - 1.96 * std_err
ci_upper = mean_diff + 1.96 * std_err

print(f"Effect Size (Mean Diff): {mean_diff:.4f}")
print(f"95% CI: [{ci_lower:.4f}, {ci_upper:.4f}]")

Now we get a much fuller picture. The CI shows the effect is small (around 0.05), even though the p-value cleared the “significant” bar.

7. Bayesian Thinking: An Alternative to P-Values

Some practitioners prefer Bayesian statistics. Instead of asking “How weird is the data?”, Bayesians flip the question: “Given this data, what’s the probability my hypothesis is true?”

You start by stating a Prior — what you believed before the experiment. The subjectivity bothers some people. But the approach is more intuitive. You get a Credible Interval, which you can read at face value: “There is a 95% chance the true effect falls in this range.”

8. A Checklist: How to Use P-Values Responsibly

To sidestep these traps, work through this checklist:

1. Report the Effect Size: A result can be statistically significant without being practically useful. Report the magnitude.
2. Show Confidence Intervals: Give a range, not just a binary “yes/no.”
3. Disclose All Tests: Don’t hide the 19 tests that failed to surface the 1 that worked.
4. Check for Confounders: Could something else — the summer heat, for instance — explain this?

9. Recap: What You’ve Learned

• A p-value is the probability of observing your data if the Null Hypothesis is true.
• It is not the probability that your theory is correct.
• The 0.05 threshold is arbitrary; don’t treat it as a hard boundary.
• Large samples can make tiny, meaningless effects look “significant.”
• Look at Effect Sizes and Confidence Intervals too — they give the fuller picture.

Next up: Causal Inference — how to move beyond correlation and pin down what actually causes what.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the precise, technical definition of a p-value, according to the article — and what common misinterpretation does it explicitly rule out?

Understand In your own words, explain the ice cream/drowning example — why does a tiny p-value on the correlation between the two NOT mean ice cream causes drowning?

Apply Using the article’s coin-flip simulation logic, if flipping a coin 100 times produced 58 heads instead of 65, would you expect the resulting p-value to be larger or smaller than the article’s ~0.0018 for 65 heads? Why?

Analyze The article’s Section 5 shows a sample of 1,000,000 users producing a “significant” p-value for a real effect of only 0.05%. Walk through why increasing sample size shrinks the p-value for a fixed effect size, even though the effect itself hasn’t gotten any more practically meaningful.

Evaluate The article’s checklist item 3 says “Disclose All Tests: Don’t hide the 19 tests that failed to show the 1 that worked.” Critique how realistic this is as a norm: what incentive structure in research and business reporting makes p-hacking attractive even when everyone agrees, in the abstract, that disclosure is the right thing to do?

Create Design a reporting template (following the article’s Section 6 recommendation) for a marketing team’s A/B test results, that forces the team to report Effect Size and Confidence Interval alongside the p-value, rather than just a “significant / not significant” verdict. Sketch what fields the template would require and why each one closes off one of the article’s specific misuse patterns.

---