Simpson's Paradox: When Aggregated Data Tells the Opposite Story

Imagine you’re comparing success rates for two medical treatments. Treatment A outperforms Treatment B across all patients combined. You’d recommend Treatment A. Simple.

But then you split the data by kidney stone size. Treatment B does better for small stones and better for large stones.

How can Treatment B win in every subgroup yet lose overall? This isn’t a calculation error — it’s Simpson’s Paradox. A real phenomenon, and one of the trickiest traps in data science, because the data is accurate but the decision it leads you to is wrong.

1. The Paradox That Breaks Your Intuition

In 1973, the University of California, Berkeley had a problem. Their admission data showed men being admitted at a significantly higher rate than women. On the surface, it looked like a clear case of gender bias.

Here are the raw numbers. We’ll use a simplified version of this famous dataset to see what happened.

import pandas as pd

# Creating the Berkeley-style dataset
data = {
    'Department': ['A', 'A', 'B', 'B'],
    'Gender': ['Male', 'Female', 'Male', 'Female'],
    'Applied': [800, 100, 100, 800],
    'Admitted': [500, 80, 10, 80]
}

df = pd.DataFrame(data)

# 1. Calculate Aggregate Admission Rates
aggregate = df.groupby('Gender').sum(numeric_only=True)
aggregate['Rate'] = aggregate['Admitted'] / aggregate['Applied']
print("--- Aggregate Admission Rates ---")
print(aggregate[['Applied', 'Admitted', 'Rate']])

# 2. Calculate Department-level Rates
df['Rate'] = df['Admitted'] / df['Applied']
print("\n--- Department-level Rates ---")
print(df[['Department', 'Gender', 'Rate']])

What this actually means: Looking at the Aggregate Admission Rates, men have a 56.6% success rate (510/900), while women have 17.7% (160/900). That’s a big gap.

But look at the Department-level Rates. In Department A, women have an 80% rate vs men’s 62.5%. In Department B, women have a 10% rate vs men’s 10%. In both departments, women did as well or better than men. The aggregate story is the exact opposite of the subgroup story.

2. What’s Actually Going On: The Intuition

How can the same data tell two opposite stories? This is the hardest part to wrap your head around, but it comes down to weights.

The aggregate rate is a “weighted average.” In our Berkeley example, Department A is very easy to get into, and Department B is very hard.

• Most men applied to Department A (the easy one).
• Most women applied to Department B (the hard one).

Because the men were mostly in the “easy” pool, their overall average looks high. Because the women were mostly in the “hard” pool, their overall average looks low. The “Department” variable is the hidden driver here. In statistics, we call this a confounder.

3. Seeing It in Code: A Step-by-Step Walkthrough

Python can show us how these trends reverse. The synthetic dataset below compares “Study Hours” against “Test Scores” for two classes.

import matplotlib.pyplot as plt
import numpy as np

# Create two groups of students
# Group 1: High achievers who study a lot but have a hard test
np.random.seed(42)
class_a_study = np.random.normal(10, 2, 50)
class_a_score = 0.5 * class_a_study + 50  # Positive trend

# Group 2: Struggling students who study less and have an easy test
class_b_study = np.random.normal(5, 2, 50)
class_b_score = 0.5 * class_b_study + 70  # Positive trend

plt.figure(figsize=(10, 6))
plt.scatter(class_a_study, class_a_score, label='Class A (Hard Test)', alpha=0.6)
plt.scatter(class_b_study, class_b_score, label='Class B (Easy Test)', alpha=0.6)

# Plot the aggregate trend line
all_study = np.concatenate([class_a_study, class_b_study])
all_score = np.concatenate([class_a_score, class_b_score])
m, b = np.polyfit(all_study, all_score, 1)
plt.plot(all_study, m*all_study + b, color='red', label='Aggregate Trend')

plt.xlabel('Study Hours')
plt.ylabel('Test Score')
plt.title("Simpson's Paradox: Individual vs. Aggregate Trends")
plt.legend()
plt.show()

print(f"Aggregate Correlation Slope: {m:.2f}")

Checking that against the data: Within each class, the slope is +0.5 — more study, higher score. But the red line (the aggregate) has a negative slope. Look only at that line and you’d conclude studying lowers your score. Here’s why: Class B has higher scores overall (the easy test) but lower study hours. Those points pull the left side of the graph up, tilting the whole line downward.

4. Why This Happens: The Role of Confounding

Simpson’s Paradox flags a confounder in your data. A confounder is a variable that influences both the “cause” and the “effect.”

In the Berkeley case:

1. Gender (Cause) influenced which Department people applied to.
2. Department (Confounder) influenced the Admission Rate (Effect).

Ignore the Department, and you mix the “easy” and “hard” results together. The result is a spurious (fake) correlation. To fix it, stratify—which just means looking at each group separately.

5. The Danger: When Simpson’s Paradox Leads to Bad Decisions

This isn’t just a math puzzle. It’s a trap, and the costs are measured in lives and dollars.

• Medical Treatments: In a well-known kidney stone study, Treatment A looked worse overall. But stratifying by stone size told a different story: Treatment A was actually better for both small and large stones. Going by the aggregate alone, doctors would have abandoned the more effective treatment.
• Business: A company sees their overall conversion rate dropping and panics. They change the website. But stratify by device and the picture flips — conversion is rising on both Mobile and Desktop. The overall drop was because the share of Mobile users, who convert at a lower rate, went up.

6. How to Spot Simpson’s Paradox Before It Catches You

Here’s a quick checklist to keep handy:

1. Check group sizes. Are your subgroups wildly different in size — say, 800 versus 100?
2. Watch for confounders. Is there a variable like “Department” or “Device Type” tied to both your input and your output?
3. Stratify, always. Don’t report just the aggregate number. Break it down by the key subgroups.

def check_reversal(df, group_col, outcome_col, stratify_col):
    # Calculate aggregate
    agg = df.groupby(group_col)[outcome_col].mean()
    agg_direction = agg.iloc[0] > agg.iloc[1]
    
    # Calculate stratified
    strat = df.groupby([stratify_col, group_col])[outcome_col].mean().unstack()
    
    print("--- Analysis Report ---")
    print(f"Aggregate Trend: {agg.index[0]} > {agg.index[1]} is {agg_direction}")
    
    for val in strat.index:
        strat_direction = strat.loc[val].iloc[0] > strat.loc[val].iloc[1]
        if strat_direction != agg_direction:
            print(f"!!! WARNING: Trend reversed in {stratify_col} == {val} !!!")
        else:
            print(f"Trend consistent in {stratify_col} == {val}")

7. Simpson’s Paradox and Causal Inference: The Big Picture

The practical takeaway: correlation is not causation.

Simpson’s Paradox is the first step into the world of Causal Inference. Data alone can’t tell the whole story. You need a “causal model”—a mental map of how variables affect each other. If Department affects Admission, you must control for it.

8. Practice: Finding the Paradox

Try this: Load a dataset of your choice — the Titanic survival data works well. Compare survival rates between men and women. Then stratify by class (1st, 2nd, 3rd). Does the gap narrow, widen, or flip?

9. Recap and What’s Next

What we covered:

• Simpson’s Paradox occurs when a trend appears in subgroups but disappears or reverses in the aggregate.
• The culprits are unequal group sizes and confounding variables.
• The fix is to stratify your data and reason about what’s actually driving the effect.

Next in this series, we’ll get into Causal Diagrams (DAGs) — drawing the maps that tell us exactly which variables to control for, so a paradox doesn’t catch us off guard again.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is a confounder, and what role did “Department” play in the Berkeley admissions example?

Understand In your own words, explain why the aggregate admission rate favored men even though women had equal-or-better rates in every individual department, using the article’s “weighted average” explanation.

Apply Using the article’s Berkeley-style numbers, if Department A had 800 male applicants with a 62.5% admit rate and Department B had 100 male applicants with a 10% admit rate, verify the aggregate male admission count (500 + 10 = 510) matches what the article reports for the combined 900 male applicants.

Analyze The article’s check_reversal function compares the direction of the aggregate trend to the direction within each stratified subgroup. Walk through why comparing directions (which group is bigger) rather than comparing raw rates is the right check for detecting a Simpson’s Paradox reversal, and what a false negative from this function might look like (a case it would miss).

Evaluate The article’s checklist item 1 says to “check group sizes” as a warning sign, since 800 vs. 100 is suspicious. Critique this heuristic: can Simpson’s Paradox occur even when subgroups are roughly equal in size, or is unequal group size actually a necessary condition for the reversal to happen?

Create Design a stratification check for a new scenario: a company sees that its overall customer satisfaction score dropped this quarter, and suspects Simpson’s Paradox. Propose a plausible confounding variable (like the article’s “Device Type” business example) and describe what you’d need to see in the stratified data to conclude the drop is real versus a mix-shift artifact.

---