Frequentist Vs Bayesian Statistics The Debate Expl

You just ran an A/B test. Your p-value is 0.04. Your stakeholder leans over your shoulder and says, “Great, so we’re 96% sure the treatment works.”

You know that’s wrong. But why exactly? And what would a Bayesian say instead?

This moment — the p-value misinterpretation — is the single most common statistical error in industry. A 2016 paper in Nature found that over half of published p-values in psychology papers were misinterpreted as “the probability that the null hypothesis is true.” This isn’t just an academic issue; it happens in boardrooms every day.

Here’s the thing: both Frequentists and Bayesians would tell your stakeholder they’re wrong. But they’d give very different reasons. And they’d offer very different alternatives.

This article won’t pick a side. Instead, it will show you what each framework actually asks, so you can use both where they fit. By the end, you’ll understand why the debate exists, what each approach is good for, and — most importantly — how to answer the question your stakeholder is really asking.

What does ‘probability’ even mean?

The entire debate boils down to one question: what is probability?

A Frequentist says: probability is a long-run frequency. If you flip a fair coin a million times, about half will be heads. The probability of heads is the proportion you’d see if you repeated the experiment forever.

A Bayesian says: probability is a degree of belief. Before you flip the coin, you have some belief about it being fair. After you see some flips, you update that belief. Probability is how confident you are, given what you’ve seen.

Neither is “right.” They’re different definitions for different jobs. Let’s see what this means with actual code.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

# Set seed for reproducibility
np.random.seed(42)

# --- Frequentist simulation: coin flips converge to 0.5 ---
print("=== Frequentist: Long-run frequency ===")

# Simulate 10,000 coin flips (0 = tails, 1 = heads)
flips = np.random.binomial(1, 0.5, 10000)

# Compute running proportion of heads
cumulative_heads = np.cumsum(flips)
proportions = cumulative_heads / np.arange(1, 10001)

# Show the last few proportions — they should be close to 0.5
print(f"After 100 flips: {proportions[99]:.4f}")
print(f"After 1,000 flips: {proportions[999]:.4f}")
print(f"After 10,000 flips: {proportions[9999]:.4f}")
print()

# --- Bayesian simulation: update belief after 10 flips ---
print("=== Bayesian: Updating belief ===")

# We start with a prior: Beta(1, 1) — uniform, meaning we have no strong belief
# This is a "flat prior" — all probabilities equally likely
prior_a, prior_b = 1, 1

# We flip a coin 10 times and get 7 heads, 3 tails
n_flips = 10
n_heads = 7
n_tails = n_flips - n_heads

# Posterior is Beta(prior_a + heads, prior_b + tails)
posterior_a = prior_a + n_heads
posterior_b = prior_b + n_tails

# Sample from the posterior to see our updated belief
posterior_samples = np.random.beta(posterior_a, posterior_b, 10000)

# The mean of the posterior is our best guess for the coin's bias
mean_bias = np.mean(posterior_samples)
print(f"Prior: Beta({prior_a}, {prior_b}) — uniform, no strong belief")
print(f"Data: {n_heads} heads, {n_tails} tails out of {n_flips} flips")
print(f"Posterior: Beta({posterior_a}, {posterior_b})")
print(f"Mean of posterior (our best guess for bias): {mean_bias:.3f}")

# 95% credible interval: the range where 95% of our belief lies
lower = np.percentile(posterior_samples, 2.5)
upper = np.percentile(posterior_samples, 97.5)
print(f"95% credible interval: [{lower:.3f}, {upper:.3f}]")

What did we just see? The Frequentist simulation showed that as we flip more coins, the proportion of heads converges to 0.5. In plain English, this means that over a long enough period, the random noise averages out and we see the true underlying frequency of the event.

The Bayesian simulation showed something different. We started with a flat prior (no strong belief), saw 7 heads out of 10 flips, and updated our belief. The posterior distribution tells us directly: “given what I believed and what I saw, I think the coin’s bias is around 0.667, and I’m 95% sure it’s between 0.39 and 0.89.”

Notice the difference. The Frequentist answer is about the procedure: “if I repeated this experiment forever, the proportion would converge to 0.5.” The Bayesian answer is about your belief: “given what I’ve seen, this is how confident I am.”

To put it simply, a frequentist asks: “How often would this happen if we repeated the experiment forever?” while a Bayesian asks: “Given what I’ve seen, how confident should I be?”

The frequentist toolbox: p-values and confidence intervals

Now let’s walk through a concrete A/B test the frequentist way. We’ll use a small sample so the limitations are obvious.

Imagine you’re testing a new website button color. The control group (blue button) had 12 conversions out of 100 visitors. The treatment group (red button) had 18 conversions out of 100 visitors. Did the red button actually perform better?

import numpy as np
import statsmodels.stats.proportion as smp

# Data: conversions and total visitors for each group
control_conversions = 12
control_total = 100
treatment_conversions = 18
treatment_total = 100

# Run a two-proportion z-test (frequentist)
# This tests the null hypothesis: the true conversion rates are equal
z_stat, p_value = smp.proportions_ztest(
    count=[treatment_conversions, control_conversions],
    nobs=[treatment_total, control_total],
    alternative='larger'  # one-sided: treatment > control
)

# Compute the 95% confidence interval for the difference in proportions
# (treatment - control)
ci_lower, ci_upper = smp.proportions_diff_confint_ind(
    treatment_conversions, treatment_total,
    control_conversions, control_total,
    alpha=0.05
)

print("=== Frequentist A/B Test Results ===")
print(f"Control conversion rate: {control_conversions/control_total:.1%}")
print(f"Treatment conversion rate: {treatment_conversions/treatment_total:.1%}")
print(f"Observed difference (treatment - control): {(treatment_conversions/control_total - control_conversions/control_total):.1%}")
print()
print(f"Z-statistic: {z_stat:.3f}")
print(f"p-value (one-sided): {p_value:.4f}")
print(f"95% confidence interval for the difference: [{ci_lower:.4f}, {ci_upper:.4f}]")
print()

# Plain English interpretation
print("=== What this actually means ===")
print(f"The p-value of {p_value:.4f} means: if the true conversion rates were equal,")
print(f"there's a {p_value*100:.1f}% chance of seeing a difference this large (or larger)")
print("just due to random chance.")
print()
print("This does NOT mean there's a 96% chance the treatment works.")
print("It means the data is somewhat unusual under the null hypothesis.")
print()
print(f"The 95% confidence interval [{ci_lower:.3f}, {ci_upper:.3f}] means:")
print("if we repeated this experiment 100 times, about 95 of the intervals")
print("would contain the true difference. We don't know if THIS one does.")

Here’s the hard part: these numbers do NOT tell you the probability that your hypothesis is true. They tell you about the procedure, not the parameter.

Think of it this way: a p-value is like asking “how weird would this be if nothing were going on?” A p-value of 0.04 says “this would happen only 4% of the time if the null were true.” That’s useful, but it’s not the same as “there’s a 96% chance the treatment works.”

And the confidence interval? It’s about the procedure, not this specific interval. If you ran the experiment 100 times, 95 of those intervals would contain the true effect. But you can’t say there’s a 95% chance this interval contains the truth. That’s a subtle but crucial difference.

The Bayesian toolbox: posterior distributions and credible intervals

Now let’s answer the same question the Bayesian way. The intuition is different: you start with a prior belief, update it with data, and get a posterior distribution that directly tells you what you want to know.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

# Set seed for reproducibility
np.random.seed(42)

# Data: same A/B test
control_conversions = 12
control_total = 100
treatment_conversions = 18
treatment_total = 100

# Prior: we use a weak prior (Beta(1,1) for each group)
# This says "before seeing data, any conversion rate is equally likely"
prior_a, prior_b = 1, 1

# Posterior for control group: Beta(prior_a + conversions, prior_b + non-conversions)
control_posterior_a = prior_a + control_conversions
control_posterior_b = prior_b + (control_total - control_conversions)

# Posterior for treatment group
treatment_posterior_a = prior_a + treatment_conversions
treatment_posterior_b = prior_b + (treatment_total - treatment_conversions)

# Sample from the posteriors
control_samples = np.random.beta(control_posterior_a, control_posterior_b, 100000)
treatment_samples = np.random.beta(treatment_posterior_a, treatment_posterior_b, 100000)

# Compute the difference (treatment - control) for each sample
difference_samples = treatment_samples - control_samples

# 95% credible interval for the difference
lower = np.percentile(difference_samples, 2.5)
upper = np.percentile(difference_samples, 97.5)

# Probability that treatment > control
prob_treatment_better = np.mean(difference_samples > 0)

print("=== Bayesian A/B Test Results ===")
print(f"Control posterior mean: {np.mean(control_samples):.3f}")
print(f"Treatment posterior mean: {np.mean(treatment_samples):.3f}")
print(f"Mean difference (treatment - control): {np.mean(difference_samples):.3f}")
print()
print(f"95% credible interval for the difference: [{lower:.3f}, {upper:.3f}]")
print(f"Probability that treatment > control: {prob_treatment_better:.3f}")
print()

print("=== What this actually means ===")
print(f"The 95% credible interval [{lower:.3f}, {upper:.3f}] means:")
print("given our prior and the data, there's a 95% probability that")
print("the true difference lies in this range.")
print()
print(f"The probability that treatment > control is {prob_treatment_better:.3f}.")
print("That's a direct answer to 'how sure are we that the treatment works?'")

This is the key difference. A Bayesian says: “based on what I believed and what I saw, I’m 95% sure the effect is between -0.03 and 0.15.” That’s exactly what your stakeholder wanted to know.

But here’s the catch: the prior is subjective. Different priors give different answers. If you started with a strong prior that red buttons are terrible, you’d get a different posterior. That’s a feature, not a bug — it makes your assumptions explicit.

The same data, two different answers: a worked comparison

Let’s run both methods on the exact same data and compare them side by side. Note that in this specific comparison, we will reverse the results to see how the frameworks handle a case where the treatment group performs worse (12 conversions) than the control group (18 conversions).

import numpy as np
import statsmodels.stats.proportion as smp
from scipy.stats import beta

np.random.seed(42)

# Data
control_conversions = 18
control_total = 100
treatment_conversions = 12
treatment_total = 100

# --- Frequentist analysis ---
z_stat, p_value = smp.proportions_ztest(
    count=[treatment_conversions, control_conversions],
    nobs=[treatment_total, control_total],
    alternative='two-sided'
)

ci_lower, ci_upper = smp.proportions_diff_confint_ind(
    treatment_conversions, treatment_total,
    control_conversions, control_total,
    alpha=0.05
)

# --- Bayesian analysis ---
prior_a, prior_b = 1, 1

control_posterior_a = prior_a + control_conversions
control_posterior_b = prior_b + (control_total - control_conversions)
treatment_posterior_a = prior_a + treatment_conversions
treatment_posterior_b = prior_b + (treatment_total - treatment_conversions)

control_samples = np.random.beta(control_posterior_a, control_posterior_b, 100000)
treatment_samples = np.random.beta(treatment_posterior_a, treatment_posterior_b, 100000)
difference_samples = treatment_samples - control_samples

bayes_lower = np.percentile(difference_samples, 2.5)
bayes_upper = np.percentile(difference_samples, 97.5)
prob_treatment_better = np.mean(difference_samples > 0)

# --- Comparison table ---
print("=" * 70)
print(f"{'Metric':<35} {'Frequentist':<20} {'Bayesian':<20}")
print("=" * 70)
print(f"{'Control conversion rate':<35} {'18.0%':<20} {'18.0%':<20}")
print(f"{'Treatment conversion rate':<35} {'12.0%':<20} {'12.0%':<20}")
print(f"{'Observed difference':<35} {'-6.0%':<20} {'-6.0%':<20}")
print(f"{'p-value / P(treatment > control)':<35} {f'{p_value:.4f}':<20} {f'{prob_treatment_better:.3f}':<20}")
print(f"{'95% interval (lower)':<35} {f'{ci_lower:.3f}':<20} {f'{bayes_lower:.3f}':<20}")
print(f"{'95% interval (upper)':<35} {f'{ci_upper:.3f}':<20} {f'{bayes_upper:.3f}':<20}")
print("=" * 70)
print()

print("=== Plain English Interpretation ===")
print("Frequentist: The p-value of {:.4f} means that if the true rates were equal,".format(p_value))
print("there's a {:.1f}% chance of seeing a difference this extreme by random chance.".format(p_value*100))
print("The 95% CI says: if we repeated this experiment 100 times, about 95 intervals")
print("would contain the true difference.")
print()
print("Bayesian: Given our weak prior and the data, there's a {:.1f}% probability".format(prob_treatment_better*100))
print("that the treatment is actually worse than control.")
print("The 95% credible interval says: we're 95% sure the true difference is")
print("between {:.3f} and {:.3f}.".format(bayes_lower, bayes_upper))

Notice what happened. With this small sample, both methods agree: the treatment looks worse, but we’re not very certain. The Frequentist p-value is 0.24 — not significant. The Bayesian posterior says there’s only a 12% chance the treatment is better — consistent with the p-value.

But the interpretation is different. The Frequentist says “we can’t reject the null.” The Bayesian says “we’re 88% sure the treatment is worse.” Same data, different answers to different questions.

When to use which: a practical decision guide

So when should you use each framework? Here’s a simple guide:

Use frequentist when:

• You need a standardized, regulator-approved method (e.g., drug trials)
• You have very large samples (both methods converge anyway)
• You want to control long-run error rates (e.g., “only 5% of our experiments will be false positives”)

Use Bayesian when:

• You have prior information you want to incorporate (e.g., previous experiments)
• You need a direct probability statement about your hypothesis (e.g., “how sure are we this works?”)
• You have small samples and need to be honest about uncertainty

The honest answer? Many practitioners use both, depending on the problem. The debate is philosophical, not practical — in most real-world cases, both methods give similar answers with enough data.

Recap: what you actually learned

Let’s summarize the key takeaways:

1. Frequentists define probability as long-run frequency; Bayesians define it as degree of belief.
2. A p-value is NOT the probability that your hypothesis is true. It’s the probability of seeing data this extreme if the null were true.
3. A credible interval IS the probability that the parameter lies in that range (given your prior).
4. Both frameworks are valid. The right one depends on what question you’re asking.

Check Your Understanding

Remember: What is the frequentist definition of probability?

Understand: Explain in your own words why a p-value is not the probability that the null hypothesis is true.

Apply: Given a small A/B test dataset (e.g., 15 conversions out of 80 vs 22 out of 80), compute both the frequentist p-value and the Bayesian posterior probability that treatment > control.

Analyze: Compare the two results from the Apply question. Why might they differ? Which one would you present to a stakeholder who wants to know “how sure are we that this works?”

Evaluate: In a regulated industry (e.g., drug trials), which framework is typically required and why?

Create: Design a simple simulation that shows how the frequentist and Bayesian answers converge as sample size increases.