Beyond the Average: Finding Which Customers the Treatment Actually Helps

Last Time on the 90-Day Policy

Maya — the data scientist at Blue Harbor Hotels — has built a solid case around the 90-day policy (a fee on bookings made more than 90 days out). She drew the DAG, estimated the effect, stress-tested it against hidden confounders, and applied propensity scores to compare apples to apples by balancing who got treated. Her conclusion: booking early increases cancellation by about 5%.

She has one clean, defensible average. And that’s exactly the problem.

As Maya builds the board deck, she breaks the numbers down by customer type. What she finds makes her stomach drop: the “5% average” is hiding two opposite stories. New guests respond well to the policy nudge. But Blue Harbor’s most loyal guests — the Platinum members — are being hurt by it.

In Part 3, we calculated the Average Treatment Effect (ATE) — one single number describing how a change affects everyone. But in the real world, “everyone” is a myth. This tutorial is where Maya (and you) learn to stop trusting the average and start asking the question that actually matters: who does this treatment help, and who does it hurt?

What you’ll learn here:

• Why the average treatment effect hides opposite reactions (heterogeneity)
• What a Conditional Average Treatment Effect (CATE) is
• The Fundamental Problem of Causal Inference (why individual effects are invisible)
• How EconML’s Double Machine Learning estimates individual effects
• How to build and read an uplift curve to target the right customers

Prerequisites: Tutorials 1–6, or comfort with DoWhy, ATE, and the basic causal workflow.

1. The Average is a Lie: Why One-Size-Fits-All Fails

Say Blue Harbor offers a $20 discount to prevent hotel cancellations. For new travelers, this might be the nudge they need to stay. But for the most loyal Platinum guests, a deep discount might make the brand feel “cheap” — lowering their perceived value of the service and making them more likely to leave.

If the new guests improve by 10% and the loyal guests get 5% worse, the ATE might show a “positive 2.5%.” Look only at that average, and you’ll keep sending the discount to everyone — unknowingly damaging your relationship with your best customers.

This phenomenon is called Heterogeneity. It just means different people react in different ways. When we only look at the average, we’re blind to two dangerous groups:

• “Sleeping Dogs” — people who react poorly to the treatment (it backfires).
• “Sure Things” — people who would have stayed anyway, making the treatment a waste of money.

Let’s simulate this in the hotel dataset. We’ll create a scenario where lead_time (booking early) affects repeat guests differently than new guests:

import pandas as pd
import numpy as np
from dowhy import CausalModel
import matplotlib.pyplot as plt

# Set seed for reproducibility
np.random.seed(42)
n = 10000

# Features
is_repeat_guest = np.random.binomial(1, 0.4, n)

# Treatment: Booking more than 30 days in advance
treatment = np.random.binomial(1, 0.5, n)

# Outcome: Cancellation probability
# New guests (is_repeat_guest=0) are HELPED by lead time (cancellation drops)
# Repeat guests (is_repeat_guest=1) are HURT by lead time (cancellation rises)
base_cancel = 0.2
effect_new = -0.10  # Lead time reduces cancellation by 10%
effect_repeat = 0.05 # Lead time increases cancellation by 5%

cancel_prob = base_cancel + (1 - is_repeat_guest) * treatment * effect_new + (is_repeat_guest) * treatment * effect_repeat
y = np.random.binomial(1, np.clip(cancel_prob, 0, 1))

df = pd.DataFrame({"is_repeat_guest": is_repeat_guest, "treatment": treatment, "is_cancelled": y})

ate = df[df['treatment']==1]['is_cancelled'].mean() - df[df['treatment']==0]['is_cancelled'].mean()
print(f"Average Treatment Effect (ATE): {ate:.4f}")

Interpret every number: the ATE might come out to roughly -0.04 (a 4% reduction in cancellations). If Maya stops here, she concludes booking early is always good. But she knows the truth: it’s great for new guests, bad for repeat guests. The average is hiding the danger.

Here’s the arithmetic behind the lie. The population is 60% new guests (helped by -10%) and 40% repeat guests (hurt by +5%). The weighted average: (0.6 × -0.10) + (0.4 × +0.05) = -0.06 + 0.02 = -0.04. Both effects are real, both are strong, and the average of them looks mild and harmless. The average isn’t wrong — it’s just useless for deciding what to do.

2. Meet the ‘Conditional’ Average Treatment Effect (CATE)

To fix this, we move from ATE to CATE. The “C” stands for Conditional. Just the average effect for a specific slice of the data.

Think of it this way: “Given that a customer is a Repeat Guest, what is the effect of the treatment?”

So we’re no longer just asking whether it works. We’re asking who it works for. That shift is the foundation of Uplift Modeling — predicting how individuals respond differently, so you can target only the ones who benefit.

Here’s how the effects look when we slice the data manually:

# Manual CATE calculation by segmenting the data
for guest_type in [0, 1]:
    subset = df[df['is_repeat_guest'] == guest_type]
    cate_manual = subset[subset['treatment']==1]['is_cancelled'].mean() - subset[subset['treatment']==0]['is_cancelled'].mean()
    label = "New Guest" if guest_type == 0 else "Repeat Guest"
    print(f"CATE for {label}: {cate_manual:.4f}")

Interpret every number: the New Guest CATE sits around -0.10 (good — booking early reduces their cancellations by 10%), while the Repeat Guest CATE is around +0.05 (bad — it increases theirs by 5%). Same treatment, opposite effects. This is the exact pattern Maya suspects with the 90-day policy.

3. The Hardest Part: You Can’t See the Counterfactual

In the example above, we cheated. We grouped people by a known variable (is_repeat_guest). In real life, response patterns hide across dozens of variables — age, spend, location, booking channel, and more.

This is the Fundamental Problem of Causal Inference: for any single person, we only see one outcome. If Jane booked early and cancelled, we don’t know whether she would have stayed had she booked late. We can’t subtract “Outcome if Treated” from “Outcome if Control” for one individual, because one of those outcomes is a counterfactual — a thing that never happened.

A counterfactual is the road not taken: the outcome that would have happened in the alternative universe. We never observe it directly for a real person.

To solve this, we need Machine Learning to guess the missing outcome. We look for “twins” in the data to fill in the blanks — if we find someone exactly like Jane who booked late, we use their outcome as a proxy for Jane’s counterfactual. Same “twin” idea as propensity matching, but now we need individual guesses, not just balanced group averages.

4. Enter EconML: The Power Tool for Causal ML

DoWhy handles the assumption mapping — the DAG — but Maya needs a proper compute engine for the matching math across many variables. That’s EconML, a library built to pair with DoWhy.

EconML relies on Meta-Learners. These wrap around standard ML tools like XGBoost or LightGBM and redirect them toward causal effect estimation rather than plain prediction. Maya will focus on Double Machine Learning (DML).

DML works as a two-stage cleaning process:

1. ML predicts the treatment and the outcome from confounders, then subtracts those predictions to leave only the residuals — the leftover noise. A residual is what remains after the model’s prediction is removed: the part of the variable the confounders couldn’t explain.
2. It then measures the relationship between the “leftover” treatment and “leftover” outcome, isolating the true causal signal from background noise.

Why does this work? The confounders’ influence sits in the predicted part, which got subtracted away. What’s left is the part of the treatment that isn’t explained by confounders — the part we can safely compare against the leftover outcome.

5. Step-by-Step: Estimating Uplift with DML

DoWhy’s EconML integration lets us estimate individual effects. We define is_repeat_guest as an Effect Modifier — a feature that changes how the treatment works. Regular confounders affect who gets treated; effect modifiers affect whether the treatment helps. The same variable can do both.

from econml.dml import LinearDML
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier

# Define the model
model = CausalModel(
    data=df,
    treatment='treatment',
    outcome='is_cancelled',
    common_causes=['is_repeat_guest'],
    effect_modifiers=['is_repeat_guest']
)

identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)

# Estimate CATE using Double Machine Learning
estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.econml.dml.LinearDML",
    method_params={
        "init_params": {
            'model_y': RandomForestClassifier(),
            'model_t': RandomForestClassifier(),
            'discrete_treatment': True
        },
        "fit_params": {}
    }
)

# Get the individual effects (CATE) for every row in our data
conditional_effects = estimate.evaluator.effect(df)
print(f"Average estimated effect: {np.mean(conditional_effects):.4f}")
print(f"Range of effects: {conditional_effects.min():.4f} to {conditional_effects.max():.4f}")

The numbers tell a fuller story. The average effect still sits around -0.04, but the range runs from -0.10 to +0.05. The model identified that for some people the effect is negative (good), and for others it’s positive (bad). The average was true but useless. The range is where the actionable information lives.

6. Visualizing the Truth: The Uplift Curve

How does Maya know if these individual guesses are actually right? She uses an Uplift Curve (also called a Cumulative Gain chart).

The concept is straightforward: rank all customers from “most likely to respond well” to “least likely,” then calculate the cumulative benefit of treating the top X% versus treating people at random. A good model bends sharply — the top-ranked customers really do respond better.

# Add the predictions to our dataframe
df['uplift_score'] = conditional_effects

# Sort by uplift score ascending (lower is better for cancellation)
df = df.sort_values('uplift_score', ascending=True) # lower is better for cancellation

def get_uplift_curve(df):
    # Calculate cumulative gain
    df['cum_treated'] = df['treatment'].cumsum()
    df['cum_control'] = (1 - df['treatment']).cumsum()
    df['cum_y_treated'] = (df['is_cancelled'] * df['treatment']).cumsum()
    df['cum_y_control'] = (df['is_cancelled'] * (1 - df['treatment'])).cumsum()
    
    # Calculate lift
    lift = (df['cum_y_treated'] / df['cum_treated']) - (df['cum_y_control'] / df['cum_control'])
    return lift.values

uplift_values = get_uplift_curve(df)

plt.plot(np.linspace(0, 100, len(uplift_values)), uplift_values)
plt.axhline(y=ate, color='r', linestyle='--', label='Average Effect')
plt.title("Uplift Curve: Targeting the Most Responsive Guests")
plt.xlabel("Percentage of Population Treated")
plt.ylabel("Estimated Causal Effect")
plt.legend()
plt.show()

What to look for: if the line starts well below the red dashed ATE line (more negative), your model is finding the right people. The “bonus” area between your curve and that flat ATE line is the money you save by not treating the wrong people. A flat line means the model learned nothing. A bending line means it learned who responds — which is how you prove the targeting works.

7. Summary and Next Steps

Maya found the smoking gun. The 90-day policy’s average looked fine — but that average was hiding something. Blue Harbor was penalizing the customers it values most. Loyal repeat guests were cancelling more, not less, because of the booking-time penalty.

This guide moved past simple averages to find the stories hiding inside the data:

• The ATE is a Lie: it masks groups where the treatment fails or backfires.
• CATE is the Key: conditioning on traits reveals the “Uplift” for specific segments.
• Double ML: EconML estimates these effects even when the counterfactual stays hidden.
• Uplift Curves: ranking customers by responsiveness shows the model works.

Maya can now predict who will respond. So what should Blue Harbor do about it? Before she can answer that — before she finishes typing up her findings — the dashboard lights up red. Cancellations spiked 20% on a single Tuesday. The policy debate just became an emergency.

What you learned:

• Why the ATE hides opposite reactions (heterogeneity)
• What CATE is and how to compute it by slicing data
• The Fundamental Problem of Causal Inference
• How Double Machine Learning estimates individual effects
• How to build and interpret an uplift curve

Next up: Tutorial 8 — Root Cause Analysis: why did cancellations suddenly spike 20%?

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What does CATE stand for, and how is it different from ATE?

Understand In your own words, explain the “Fundamental Problem of Causal Inference”—why can’t we ever directly observe both “what happened to Jane” and “what would have happened to Jane” under the other treatment?

Apply The tutorial’s simulation sets New Guests (40% of the population) to have an effect of -0.10 and Repeat Guests (60%) to have an effect of +0.05. Using these proportions and per-group effects, calculate the weighted-average ATE and check it against the tutorial’s reported value of roughly -0.04.

Analyze Double ML’s first stage predicts treatment and outcome from confounders and works with the residuals (leftover noise) rather than the raw variables. Walk through why using residuals instead of raw treatment/outcome values is what lets DML isolate the causal signal from confounding.

Evaluate The tutorial’s uplift curve example treats “who to give a discount to” purely as a cancellation-minimization problem. Critique this framing: the article itself opens by noting that discounts can make Platinum guests feel the brand is “cheap.” What business cost is being left out of the uplift curve’s optimization, and how might that change which guests you’d actually want to target?

Create Design an effect-modifier variable (other than is_repeat_guest) that you’d expect to create real heterogeneity in how a $20 cancellation-prevention discount works, for this hotel scenario. Explain which subgroup you’d expect to respond well and which subgroup might respond poorly or backfire.

---