Bayesian A B Testing An Alternative To P Values Fo

You ran an A/B test. The p-value came back 0.04 — statistically significant! Your product manager is thrilled. But your senior engineer is skeptical: “The lift is tiny. Are we sure this isn’t a fluke?” The designer chimes in: “The new button looks better, right?” Everyone looks at you. The room goes quiet. You stare at that 0.04 and realize… you can’t actually answer the question they’re all asking: What’s the probability that Version B is actually better than A?

Sound familiar? You’re not alone. P-values tell you “the probability of seeing data this extreme if there were no real difference.” But that’s a twisted way of saying nothing useful. What your team wants to know is: “Given the data we saw, what’s the chance B really wins?”

Bayesian A/B testing gives you that number directly. Instead of a confusing p-value, you get a clean probability — like “P(B > A) = 0.91” — that your whole team can understand. By the end of this tutorial, you’ll have a working Bayesian A/B test in Python that answers the real question.

---

Why P-Values Leave Your Team Stuck in ‘Maybe’

Let’s poke the wound a little. A p-value is the probability of seeing results at least as extreme as yours, if the null hypothesis (no difference) were true. That’s a mouthful. Here’s the problem: it’s not the question your team asked. Your team asked: “Should we ship B?” The p-value answers: “How weird would our data be if nothing were going on?” Those are completely different things.

Confidence intervals are slightly better, but still misunderstood. A 95% confidence interval does not mean there’s a 95% chance the true effect is inside it — that’s a common misinterpretation. In reality, if you repeated the experiment 100 times, 95 of the intervals would contain the true effect. But for this experiment, the interval either does or doesn’t contain the true effect.

Worse: if you peek at your p-value halfway through the experiment, you inflate your false-positive rate dramatically. Frequentist methods assume you collect all your data before analyzing. Check early, and you’re cheating — even accidentally.

Let’s see what happens when we simulate a frequentist A/B test where we know the true effect:

import numpy as np
from scipy.stats import chi2_contingency

# Set a seed for reproducibility
np.random.seed(42)

# True conversion rates: Control = 5%, Variant = 6%  (true effect = +1%)
true_control_rate = 0.05
true_variant_rate = 0.06
n_per_group = 1000

# Simulate one experiment
trials = np.ones(n_per_group, dtype=int)
control_conversions = np.random.binomial(1, true_control_rate, n_per_group)
variant_conversions = np.random.binomial(1, true_variant_rate, n_per_group)

# Build contingency table for chi-squared test
contingency_table = np.array([
    [control_conversions.sum(), n_per_group - control_conversions.sum()],
    [variant_conversions.sum(), n_per_group - variant_conversions.sum()]
])

chi2, p_value, dof, expected = chi2_contingency(contingency_table)

# Estimate effect and approximate 95% CI
control_rate = control_conversions.mean()
variant_rate = variant_conversions.mean()
lift = variant_rate - control_rate
se = np.sqrt(control_rate * (1 - control_rate) / n_per_group + variant_rate * (1 - variant_rate) / n_per_group)
ci_lower = lift - 1.96 * se
ci_upper = lift + 1.96 * se

print(f"Control conversion rate: {control_rate:.3f}  (true: {true_control_rate})")
print(f"Variant conversion rate: {variant_rate:.3f}  (true: {true_variant_rate})")
print(f"Observed lift: {lift:.3f}  (true lift: {true_variant_rate - true_control_rate:.3f})")
print(f"P-value: {p_value:.4f}")
print(f"95% CI for lift: [{ci_lower:.4f}, {ci_upper:.4f}]")

What did we get? Let’s interpret:

• The p-value is around 0.35 (it varies each run). That’s not significant — but we know there’s a real effect! The experiment just didn’t have enough data to detect it.
• The confidence interval includes zero. A frequentist would say “we can’t reject the null.” But the null is false.
• This happens more often than you’d think: underpowered tests hide real improvements.

Now imagine this at work: you spent weeks on a feature, the p-value is 0.04 — “statistically significant!” — but the team can’t agree on shipping. The frequentist method told you the data is unlikely under the null, but it didn’t tell you the probability B is better. That’s the gap Bayesian thinking fills.

---

Bayesian Thinking: You Already Do This Every Day

You already use Bayesian thinking without realizing it. Imagine you’re a product manager who’s run 50 experiments. Most features show small positive effects — maybe a 0.5% uplift on average. A new test shows a 10% uplift on Day 1 with only 100 users. Do you believe it? Probably not — your prior experience says huge early results are usually noise. You want to see more data before trusting it.

That’s Bayesian reasoning: start with a prior belief (based on experience), then update it with data. The result is a posterior — what you believe after seeing the evidence.

The math behind this is surprisingly simple for conversion rates. A conversion event (click, purchase, signup) is a Bernoulli trial: 0 or 1. A common “prior” for a conversion rate is a Beta distribution. The Beta is defined by two parameters, alpha and beta. Think of them as “prior successes” and “prior failures.”

Here’s the magic: when you use a Beta prior and observe Bernoulli data, the posterior is also a Beta — just with updated parameters. This is called a conjugate prior. No simulation needed for the math; just addition.

Let’s see it in action. We’ll start with a flat prior — Beta(1,1) — which says “I have no prior belief; every conversion rate from 0% to 100% is equally likely.” Then we’ll update it with data from 100 visitors and 5 conversions.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

# Parameters for the Beta distribution
alpha_prior = 1
beta_prior = 1

# Observed data: 100 visitors, 5 conversions
n_visitors = 100
n_conversions = 5

# Posterior is Beta(prior_alpha + successes, prior_beta + failures)
alpha_post = alpha_prior + n_conversions
beta_post = beta_prior + (n_visitors - n_conversions)

# Generate x values for plotting
x = np.linspace(0, 1, 500)

# PDFs
prior_pdf = beta.pdf(x, alpha_prior, beta_prior)
# For the likelihood, we'll plot the scaled probability of observing the data
# given each possible rate (not normalized, just for visualization)
likelihood = (x ** n_conversions) * ((1 - x) ** (n_visitors - n_conversions))
# Scale likelihood to have similar height as prior/posterior for visualization
likelihood_scaled = likelihood / likelihood.max() * prior_pdf.max()
posterior_pdf = beta.pdf(x, alpha_post, beta_post)

plt.figure(figsize=(10, 6))
plt.plot(x, prior_pdf, label='Prior: Beta(1,1)', linestyle='--', color='gray')
plt.plot(x, likelihood_scaled, label='Likelihood (scaled)', linestyle=':', color='orange')
plt.plot(x, posterior_pdf, label=f'Posterior: Beta({alpha_post},{beta_post})', color='steelblue')
plt.xlabel('Conversion Rate')
plt.ylabel('Density')
plt.title('Bayesian Update: Prior → Likelihood → Posterior')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

Notice what happened: the posterior (blue line) sits between the flat prior and the likelihood peak. The data “pulled” our belief toward the observed rate (5%). But because we started with no prior information, the posterior’s peak matches the observed rate exactly. The spread shows our uncertainty: we’re not sure the true rate is exactly 5%; it could be anywhere from ~2% to ~10%.

This is the hardest part of Bayesian thinking: a distribution — not a single number — represents your belief. You’re not saying “the conversion rate is 5%.” You’re saying “based on what I’ve seen, I believe the rate is around here, with this much uncertainty.” That uncertainty is powerful information for decision-making.

---

What’s the Probability That Variant Actually Wins?

Now let’s answer the question that started this whole journey. We have two groups — Control and Variant — each with their own Beta posterior. We want to know: given what we now believe, what’s the probability that Variant’s true conversion rate is higher than Control’s?

The method is straightforward: draw random samples from each posterior distribution, count how often the Variant sample is larger than the Control sample, and divide by the total number of draws. This is called Monte Carlo simulation.

Let’s create a realistic A/B test scenario and compute this directly.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

# Simulate experiment data
np.random.seed(42)
n_control = 1500  # Visitors in control
n_variant = 1500  # Visitors in variant

# True rates (hidden from us in real life)
true_rate_control = 0.055
true_rate_variant = 0.065

# Generate observations
control_successes = np.random.binomial(1, true_rate_control, n_control).sum()
variant_successes = np.random.binomial(1, true_rate_variant, n_variant).sum()

print(f"Control: {control_successes} conversions out of {n_control} visitors")
print(f"         = {control_successes / n_control:.4f} ({control_successes / n_control * 100:.2f}%)")
print(f"Variant: {variant_successes} conversions out of {n_variant} visitors")
print(f"         = {variant_successes / n_variant:.4f} ({variant_successes / n_variant * 100:.2f}%)")

# Flat prior for both
alpha_prior, beta_prior = 1, 1

# Posterior parameters
alpha_control = alpha_prior + control_successes
beta_control = beta_prior + (n_control - control_successes)
alpha_variant = alpha_prior + variant_successes
beta_variant = beta_prior + (n_variant - variant_successes)

# Monte Carlo: draw 100,000 samples from each posterior
n_samples = 100_000
samples_control = beta.rvs(alpha_control, beta_control, size=n_samples)
samples_variant = beta.rvs(alpha_variant, beta_variant, size=n_samples)

# Compute P(B > A)
prob_b_wins = (samples_variant > samples_control).mean()

# Compute expected uplift
expected_uplift = (samples_variant - samples_control).mean()

print(f"\nP(Variant > Control) = {prob_b_wins:.4f}")
print(f"                     = {prob_b_wins * 100:.2f}%")
print(f"Expected uplift: {expected_uplift:.4f} ({expected_uplift * 100:.2f}%)")

# Plot the posteriors
x = np.linspace(0.02, 0.10, 500)
plt.figure(figsize=(10, 6))
plt.plot(x, beta.pdf(x, alpha_control, beta_control), label='Control', color='gray')
plt.plot(x, beta.pdf(x, alpha_variant, beta_variant), label='Variant', color='steelblue')
plt.axvline(true_rate_control, color='gray', linestyle=':', alpha=0.7)
plt.axvline(true_rate_variant, color='steelblue', linestyle=':', alpha=0.7)
plt.xlabel('Conversion Rate')
plt.ylabel('Density')
plt.title('Posterior Distributions: Control vs Variant')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

Here’s what that number means: P(Variant > Control) = 0.86 means that, based on our data and our (flat) prior belief, Variant beats Control in about 86 out of 100 possible “parallel universes” of how the data could have played out. This is exactly the number your team wanted: the probability that B is actually better.

Compare this to a p-value. A p-value of 0.04 would mean “there’s a 4% chance of seeing data this extreme if the null were true.” That’s not the same as “96% chance B wins.” The Bayesian number — 86% — is lower than 96% because it accounts for uncertainty conservatively. It’s honest about what we don’t know.

What about the expected uplift? That’s another number you can put in front of stakeholders: “We expect a 0.85% absolute lift, but we have a lot of uncertainty around that estimate.” That’s a much more useful sentence than “p < 0.05.”

---

The Superpower: Peeking Without Penalty

One of the biggest practical wins of Bayesian A/B testing: you can check results early without breaking your statistics. How? Because Bayesian inference is inherently sequential. Your posterior after 100 visitors is just your belief given those 100 visitors. It’s valid on its own. Your posterior after 500 visitors is your belief given all 500. Each update is a clean, honest representation of what you know at that point.

With frequentist methods, every time you peek at your p-value, you increase your false-positive rate. It’s like checking your cards repeatedly in a game of poker — you’re not supposed to do it! But Bayesian methods have no such penalty because there’s no concept of “repeated testing” that needs correcting.

Let’s watch the posterior narrow over time as more data comes in.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

np.random.seed(42)

# True conversion rates
true_rate = 0.06

# Simulate sequential data (2000 visitors total)
n_total = 2000
data = np.random.binomial(1, true_rate, n_total)

# Prior
alpha_prior, beta_prior = 1, 1

# Checkpoints to visualize
checkpoints = [50, 200, 500, 2000]
colors = ['lightblue', 'cornflowerblue', 'steelblue', 'darkblue']

x = np.linspace(0, 0.15, 500)
plt.figure(figsize=(10, 6))

for i, n in enumerate(checkpoints):
    successes = data[:n].sum()
    failures = n - successes
    
    alpha_post = alpha_prior + successes
    beta_post = beta_prior + failures
    
    posterior_pdf = beta.pdf(x, alpha_post, beta_post)
    plt.plot(x, posterior_pdf, label=f'n={n} (rate={successes/n:.3f})', color=colors[i], linewidth=2 - i*0.3)

plt.axvline(true_rate, color='red', linestyle='--', alpha=0.5, label=f'True rate ({true_rate})')
plt.xlabel('Conversion Rate')
plt.ylabel('Density')
plt.title('Posterior Distribution Narrows as Data Accumulates')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

Notice how the posterior distribution gets tighter and taller as n increases. At n=50 (light blue), the curve is wide and flat — a lot of uncertainty. By n=2000 (dark blue), it’s narrow and peaked — we’re pretty confident the true rate is around 6%. The peak shifts slightly each time as new evidence comes in, but it never “breaks” because of peeking.

One caution: Bayesian methods let you peek without penalty, but you still shouldn’t stop an experiment the instant the posterior crosses some threshold. If you stop as soon as P(B > A) > 0.95, you’re still running a kind of sequential test that can inflate false positives. The solution? Precommit to a decision rule: “I’ll check after every 500 visitors, and I’ll stop when P(B > A) > 0.95 and I have at least 1000 visitors per group.” Guardrails like this keep you honest while still giving you flexibility.

---

What About the Prior? (And How to Not Sweat It)

Here’s the elephant in the room: the prior. Many people distrust Bayesian analysis because “you can make it say anything by choosing the right prior.” There’s some truth there: the prior does influence results. But the fix isn’t to avoid Bayesian methods — it’s to check how sensitivity your results are to different, reasonable priors.

Let’s run the same experiment with three different priors, to see if our conclusion changes.

import numpy as np
from scipy.stats import beta

# Same data as before (repeating for self-contained code block)
np.random.seed(42)
n_control, n_variant = 1500, 1500
true_rate_control, true_rate_variant = 0.055, 0.065
control_successes = np.random.binomial(1, true_rate_control, n_control).sum()
variant_successes = np.random.binomial(1, true_rate_variant, n_variant).sum()

# Define three different priors
priors = {
    'Flat (Beta(1,1))': (1, 1),
    'Weakly Skeptical (Beta(2,10))': (2, 10),  # Prior belief: low conversion rate ~ 17%
    'Optimistic (Beta(10,2))': (10, 2)  # Prior belief: high conversion rate ~ 83%
}

n_samples = 100_000

print("Prior Sensitivity Analysis")
print("-" * 60)
print(f"Observed control rate: {control_successes / n_control:.4f}")
print(f"Observed variant rate: {variant_successes / n_variant:.4f}")
print()

for prior_name, (alpha_pr, beta_pr) in priors.items():
    # Posterior for control
    alpha_control = alpha_pr + control_successes
    beta_control = beta_pr + (n_control - control_successes)
    
    # Posterior for variant
    alpha_variant = alpha_pr + variant_successes
    beta_variant = beta_pr + (n_variant - variant_successes)
    
    # Monte Carlo
    samples_control = beta.rvs(alpha_control, beta_control, size=n_samples)
    samples_variant = beta.rvs(alpha_variant, beta_variant, size=n_samples)
    
    prob_b_wins = (samples_variant > samples_control).mean()
    expected_lift = (samples_variant - samples_control).mean()
    
    print(f"{prior_name}:")
    print(f"  P(B > A) = {prob_b_wins:.4f}  ({prob_b_wins * 100:.1f}%)")
    print(f"  Expected lift = {expected_lift:.4f}  ({expected_lift * 100:.2f}%)")
    print()

Look at the results. With a flat (neutral) prior, P(B > A) is around 86%. With a weakly skeptical prior (which expects low conversion rates), the probability drops slightly — but it’s still above 80%. With an optimistic prior, it rises a bit. In all three cases, the conclusion is the same: Variant is more likely to be better. The exact number changes, but the decision doesn’t. That’s a robust result.

The prior does influence results. But the fix isn’t to avoid Bayesian methods — it’s to check how sensitive your results are to different, reasonable priors.

When would a prior matter a lot? When you have very little data. If you only had 50 visitors per group, the skeptical prior might pull P(B > A) well below 50%, while the optimistic prior keeps it high. In that case, you’d need to discuss the prior assumptions with your team before committing to a decision. Document your prior choice and show the sensitivity analysis — it’s a sign of rigor, not weakness.

For most product experiments (thousands of visitors per group), a flat prior is a safe, defensible default. It lets the data speak for itself while giving you the direct probability your team wants.

---

What You’ve Learned — And Where Bayesian Takes You Next

Let’s recap the journey:

1. P-values answer the wrong question. They tell you the probability of the data under the null — not the probability your variant is better.
2. Bayesian thinking is intuitive. You already update beliefs based on new evidence in daily life. Using distributions to represent uncertainty is the hardest part, but it is more informative than a single number.
3. Beta-Bernoulli conjugate priors make the math simple. Posterior = Beta(prior_alpha + successes, prior_beta + failures). No simulation needed.
4. Monte Carlo sampling gives you P(B > A). Draw from both posteriors, count where variant > control, divide by total draws. This answers exactly the question your team asked.
5. Sequential monitoring works without penalty. Check results early, update your posterior, and keep going — no inflation of false-positive rates.
6. Prior sensitivity analysis builds trust. Show that your conclusion holds across reasonable priors, and document your assumptions.

You now have a working Bayesian A/B test in Python. It’s not as intimidating as it sounds — it’s just counting successes and failures, updating a Beta distribution, and comparing samples.

Where does this pattern take you next? Part 2 of this series will show you how to use historical experiments as informed priors — potentially reducing your sample sizes by 30-50%. We’ll also explore Bayes Factor for comparing models and expected loss for making business decisions. That’s where Bayesian thinking becomes a true superpower for product teams.

For now, you can answer the question that started this whole conversation: “What’s the probability that B is actually better?” And you can back it up with clean, interpretable code.

---

Check Your Understanding

1. Remember: What are the two parameters of a Beta distribution, and what do they conceptually represent in the context of conversion rates?

2. Understand: In your own words, explain the difference between a p-value and P(B > A) — which one answers the question “should we ship the variant?”

3. Apply: If you observe 30 conversions out of 1000 visitors for Control and 45 conversions out of 1000 for Variant, what are the posterior parameters for each group (using a flat prior)?

4. Analyze: You run a sensitivity analysis with three different priors and get P(B > A) values of 0.72, 0.68, and 0.91. What does this tell you about the robustness of your result? Would you ship the variant?

5. Evaluate: A colleague argues that flat priors are “unfair” because they ignore historical data. How would you respond, and when might a weakly informative prior be better than a flat one?

---