Propensity Scores in Plain English: Matching, Weighting, and When They Fail

Last Time on the 90-Day Policy

Maya — the new data scientist at Blue Harbor Hotels — has been investigating the 90-day policy (a fee on bookings made more than 90 days out). Last tutorial, she reached for an Instrumental Variable — a random nudge (a website glitch) that wiggles the treatment without touching the hidden confounder. It worked. It was also overkill.

Here’s what shifts the picture for Maya: most of the time, we can measure our confounders. The stress example was the extreme case. In everyday business data, the problem isn’t a missing variable — it’s that the two groups we’re comparing look nothing alike.

Maya has a list of guest traits: loyalty status, room type, past cancellations. She knows these affect both how early someone books (lead_time) and whether they cancel. So how do you fairly compare a “long lead time” guest to a “short lead time” guest when they’re basically different species of customer?

This tutorial is the answer: Propensity Scores — the everyday workhorse Maya will reach for when she builds her board deck.

What you’ll learn here:

• Why comparing “apples to oranges” breaks naive comparisons (the selection bias problem)
• What a propensity score is
• How matching finds “data twins”
• How weighting creates a “synthetic randomized trial”
• The Common Support trap that makes both methods lie
• How to choose between them and refute the result

Prerequisites: Tutorials 1–5, or comfort with DoWhy, DAGs, confounders, and the basic workflow.

1. The ‘Apple-to-Oranges’ Problem

Comparing people who book 6 months in advance to people who book 2 days out is like comparing gym-goers to non-gym-goers on health. People who join a gym are already more health-conscious. If they turn out healthier, was it the gym — or were they just “healthier types” to begin with?

In Blue Harbor’s data, “Long Lead Time” guests (our treated group) are often families planning summer vacations. “Short Lead Time” guests (our control group) are often business travelers. Fundamentally different people. So if Maya just compares their cancellation rates, she isn’t comparing the effect of lead time. She’s comparing families to business travelers. This is Selection Bias: the bias that arises when treated and control groups are selected (or self-select) into their groups based on traits that also affect the outcome.

The raw comparison shows how bad the bias gets:

import pandas as pd
import numpy as np
from dowhy import CausalModel
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt

# Create a biased dataset
np.random.seed(42)
n = 5000

# Confounder: Is the guest a 'Planner' (Loyal/Family)?
planner = np.random.binomial(1, 0.5, n)

# Planners are much more likely to have long lead times
# Treatment: 1 if lead_time > 30 days, 0 otherwise
treatment_prob = 0.2 + 0.6 * planner
treatment = np.random.binomial(1, treatment_prob)

# Planners are also naturally less likely to cancel
# Outcome: is_cancelled
cancel_prob = 0.3 - 0.2 * planner + 0.05 * treatment
is_cancelled = np.random.binomial(1, np.clip(cancel_prob, 0, 1))

df = pd.DataFrame({"is_planner": planner, "treatment": treatment, "is_cancelled": is_cancelled})

# Raw comparison
raw_treated = df[df['treatment'] == 1]['is_cancelled'].mean()
raw_control = df[df['treatment'] == 0]['is_cancelled'].mean()
print(f"Raw Difference: {raw_treated - raw_control:.4f}")

Interpret every number: the raw difference might show a negative number (e.g., -0.05). That suggests booking early reduces cancellations. But we programmed the data so that treatment actually increases cancellation by 5% (0.05). The “Planner” trait is strong enough to hide the true effect. Apples and oranges.

Here’s the gut-check: if Maya had presented that -0.05 to Delgado, she’d have told him the 90-day policy helps — when it actually hurts by 5%. The wrong comparison leads to the wrong business decision.

2. What is a Propensity Score, Anyway?

To fix this, Maya needs the groups to look identical. She needs a “control” person who matches a “treated” person. But with 20 confounders, a perfect match is impossible — no two humans line up across 20 traits.

The Propensity Score solves this. Instead of matching on 20 traits, we compress them into a single number: the probability of getting the treatment.

Think of it as a “tendency” score. If a business traveler and a family both have a 60% chance of booking early based on their traits, they’re “balanced” — equally likely to land in either group, so comparable.

Let’s calculate this score by hand to demystify it:

# We use Logistic Regression to predict treatment based on confounders
X = df[['is_planner']]
y = df['treatment']
ps_model = LogisticRegression().fit(X, y)

# The propensity score is the probability of being in the '1' (treated) group
df['propensity_score'] = ps_model.predict_proba(X)[:, 1]

print(df[['is_planner', 'treatment', 'propensity_score']].head())

What this means in practice: every row now has a score between 0 and 1. A score of 0.8 means “based on your traits, you had an 80% chance of booking early.” The logistic regression is just a statistical tool that learns the probability of a yes/no outcome — here, “did they book early?” — from the traits we feed it.

The payoff: two people with the same propensity score are, statistically speaking, interchangeable. One happened to book early; the other didn’t. The difference in their outcomes is then attributable to the treatment, not to who they are.

3. Method 1: Matching (Finding Your Data Twin)

Matching is the most intuitive way to use the score. For every person who did book early (treated), Maya searches the database for someone who didn’t (control) with a nearly identical propensity score.

Think of it as creating “twins.” If the twins match in their tendency to book early, but only one actually did, then the difference in their cancellation has to come from the booking time itself.

Here’s the catch: sometimes a treated person is unusual enough that no control “twin” exists. We throw that data out. So you’re trading match quality against sample size — better matches, less data.

# Using DoWhy to perform Propensity Score Matching
model = CausalModel(
    data=df,
    treatment='treatment',
    outcome='is_cancelled',
    common_causes=['is_planner']
)

identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)

# We use 'matching' and set a 'caliper' (how close the twins must be)
estimate_match = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.propensity_score_matching",
    method_params={'distance_metric': 'minkowski', 'p': 2}
)

print(f"Causal Effect (Matching): {estimate_match.value:.4f}")

Reading the result: a value of 0.051 means booking early raises the cancellation chance by 5.1%. That’s much closer to the truth (we programmed 0.05) than the raw comparison. The caliper is a tolerance dial — it sets how close two scores must be to qualify as “twins.” Too tight, and you discard too much data; too loose, and your “twins” aren’t really twins.

4. Method 2: Weighting (The ‘Synthetic Randomized Trial’)

What if Maya doesn’t want to throw data away? She uses Inverse Probability Weighting (IPW).

Picture a “Planner” who booked at the last minute and still landed in the control group. That’s unusual. In IPW, we give this person more weight — they count for more in our average. A rare planner who booked late is highly informative, because they’re a control who looks exactly like a treated family.

By weighting everyone by the inverse of their propensity score, we build a pseudo-population — an imaginary world where Planners and Non-Planners are distributed equally across both groups. It’s as if our messy observational data became a randomized controlled trial (the gold standard experiment where groups are assigned by a coin flip).

estimate_weight = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.propensity_score_weighting"
)

print(f"Causal Effect (Weighting): {estimate_weight.value:.4f}")

Interpret every number: if Weighting gives 0.049, that’s also very close to the true 0.05 effect. Weighting is often more efficient since it uses all the data. But it can be sensitive when some scores sit very close to 0 or 1 — dividing by a tiny probability produces a gigantic weight, and one giant weight can dominate everything.

5. The Common Support Trap

This is the hardest part of propensity scores: Common Support.

Say all “Planners” book early (100% probability) and all “Non-Planners” book late (0% probability). You can’t match them. There’s no overlap between the groups. The math will still give you a number — but that number is a lie. You can’t compare groups that share no common ground.

The technical term is a positivity violation: some group has essentially zero probability of receiving the treatment (or the control), so no comparison is possible for them.

We check this by plotting the distribution of scores:

# Plotting Common Support
plt.hist(df[df['treatment']==1]['propensity_score'], alpha=0.5, label='Treated')
plt.hist(df[df['treatment']==0]['propensity_score'], alpha=0.5, label='Control')
plt.title("Propensity Score Distribution (Common Support Check)")
plt.legend()
plt.show()

What to look for: the two histograms should overlap. If “Treated” clusters at 0.9 and “Control” at 0.1, that’s a positivity violation. Don’t trust the causal estimate — you’d be extrapolating into empty space.

Think of two mountain peaks with no valley between them. You can imagine a bridge, but you have no evidence about what the valley looks like.

6. Which One Should You Trust?

So which method does Maya put in her board deck?

• Matching communicates well. Telling a stakeholder “we found 1,000 pairs of identical customers” lands instantly — non-technical people get “twins.”
• Weighting handles large, complex datasets where discarding samples would cost statistical power.

To test which is more robust, we run a Placebo Treatment refutation (from Part 3 — trying to break your own result on purpose). We swap the treatment for random noise. A sound model should find no effect.

refute = model.refute_estimate(identified_estimand, estimate_weight, 
                               method_name="placebo_treatment_refuter")
print(refute)

Interpret every number: the “New Effect” should land near 0.00. If it does, the weighting method holds up — not just chasing noise.

What we learned today:

• Selection Bias arises when treated and control groups differ in fundamental ways.
• A Propensity Score is one number that summarizes how likely someone was to receive treatment.
• Matching finds “twins” and discards the rest.
• Weighting re-balances the scales so rare cases count for more.
• Common Support is the essential check that your groups actually overlap.

Maya now has a defensible average: booking early increases cancellation by about 5%. She’s ready to present to Delgado.

But while building the deck, a strange pattern catches her eye. The average is 5% — yet the individual guests respond wildly differently. Some families cancel far more than 5% more often. Some loyal business travelers cancel less.

One number is hiding a lot of stories. That’s the problem the next tutorial is about to expose — in a way that will make Delgado very uncomfortable.

In this tutorial, you learned:

• The selection-bias problem behind apples-to-oranges comparisons
• What a propensity score is and why it summarizes 20 traits into one number
• How matching creates data twins, and how weighting rebalances rare cases
• The common-support trap and positivity violations
• How to refute a result with a placebo test

Next up: Tutorial 7 — Beyond the Average: finding which customers the treatment actually helps.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is a propensity score, and what single quantity does it summarize?

Understand In your own words, explain why the raw comparison at the start of the tutorial (raw_treated - raw_control) showed a negative number even though the true, programmed effect of treatment was +0.05.

Apply The tutorial reports Matching gives 0.051 and Weighting gives 0.049, versus a true effect of 0.05. Using these three numbers, which method’s estimate is closer to the true effect, and by how much?

Analyze The tutorial says Matching “discards the rest” when no twin exists, while Weighting “uses more data” by reweighting instead. Walk through a scenario using the Common Support section: if a ‘Planner’ has a propensity score of 0.98 and there are almost no ‘Non-Planners’ anywhere near that score, explain what happens differently to that data point under Matching versus under Weighting.

Evaluate The tutorial recommends Matching for stakeholder communication (“we found 1,000 pairs of identical customers”) and Weighting for large, complex datasets. Critique this recommendation: is “easier to explain to a stakeholder” a good reason to pick a statistical method, or could it lead a team to choose the less accurate option for the wrong reasons?

Create Design a Common Support check for a new scenario: a subscription company wants to know if “upgrading to premium support” reduces churn, using company_size and contract_length as confounders. Describe what a positivity violation would look like in this specific context (what group would have almost no representation at certain propensity scores), and what business reality might cause it.

---