Counterfactual Reasoning: What Would Have Happened?

Last Time on the 90-Day Policy

Maya — the data scientist at Blue Harbor Hotels — just solved a fake crisis. Cancellations spiked 20% on a Tuesday, but her GCM (Graphical Causal Model — a “digital twin” that models how every variable is generated) traced the spike to a booking-vendor bug, not real guests cancelling. She saw the whole system for the first time.

Now she has a dangerous idea. If she can model how every variable is generated — if she truly has a digital twin of the hotel — she can ask the question data science keeps circling back to: “What if?”

Maya pulls up the record of a specific guest who cancelled under the 90-day policy. “What if,” she whispers, “we hadn’t charged them the fee? Would they have stayed?”

That question is a counterfactual — the highest rung on a ladder Maya has been climbing all series.

What you’ll learn here:

• What counterfactual reasoning is and why it’s the “Sliding Doors” of data science
• The Fundamental Problem of Causal Inference (why we can’t see the road not taken)
• How a Structural Causal Model (SCM) powers the counterfactual API
• Judea Pearl’s Ladder of Causation and why counterfactuals sit on top
• How to use counterfactuals for Causal Attribution — who was actually “savable”

Prerequisites: Tutorials 1–8, or comfort with DoWhy, DAGs, GCMs, ATE, and the full causal workflow.

1. The ‘Sliding Doors’ of Data Science

Have you ever missed a bus by ten seconds and thought, “If I hadn’t hit snooze, I’d be at work by now”? That’s counterfactual thinking — imagining a version of the past that didn’t happen to understand why the present did.

Standard data science usually focuses on what did happen. You look at a spreadsheet of 10,000 hotel bookings and see who cancelled. Causal inference asks what could have happened instead. This is the Fundamental Problem of Causal Inference: we can never observe the alternative reality for the same person.

A guest either cancelled or they didn’t. We never see the version of Guest #42 who had a different lead time but was identical in every other way. That unobserved alternative — what would have happened — is the counterfactual, literally “against the facts.”

So for the hotel, the stakes are concrete. If Maya sees a guest cancelled after booking 100 days in advance, she wants to know: did the long lead time cause the cancellation, or would this person have cancelled anyway because they’re naturally indecisive? Answering that is the key to real personalization — and to knowing whether the 90-day policy actually saved anyone.

2. Average Effects vs. Individual Realities

In Part 3, Maya learned about the Average Treatment Effect (ATE) — a single number for the whole population. The ATE is like climate: on average, longer lead times increase the probability of cancellation. Counterfactuals are more like the weather on a specific Tuesday, for a specific person.

Statisticians have notation for exactly this:

• $Y(1)$ is the outcome if the guest is treated (e.g., long lead time).
• $Y(0)$ is the outcome if the guest is not treated (e.g., short lead time).

For any single guest, we only observe one of these. Booked early? We see $Y(1)$. The value of $Y(0)$ stays a “missing” counterfactual. The ATE averages the difference across everyone; counterfactual reasoning tries to estimate both values for one specific individual.

That’s the real shift. Averages hide individual stories — and after Part 7, Maya knows exactly how much damage a hidden story can do.

3. The Counterfactual API in DoWhy

To ask “what if” about a specific row of data, Maya can’t just use a simple DAG. She needs a Structural Causal Model (SCM). Think of it as a set of recipes. Instead of merely stating “lead time affects cancellations,” an SCM defines the exact mathematical formula — including the random noise — that produced that specific guest’s behavior.

This is the same GCM API from Part 8, the digital twin. But now Maya uses it differently. Instead of explaining a spike across many rows, she’ll change one thing about one guest and watch what happens.

import pandas as pd
import numpy as np
import networkx as nx
from dowhy import gcm

# 1. Setup our familiar hotel data
np.random.seed(42)
n = 1000
planner = np.random.binomial(1, 0.5, n)
lead_time = 50 * planner + np.random.normal(20, 10, n)
# Longer lead time raises cancellation risk directly — the same positive
# sign this whole series has been building toward since Part 2
cancel_prob = 0.05 + (0.009 * lead_time) - (0.15 * planner)
is_cancelled = np.random.binomial(1, np.clip(cancel_prob, 0, 1))

df = pd.DataFrame({"is_planner": planner, "lead_time": lead_time, "is_cancelled": is_cancelled})

# 2. Define the Structural Causal Model (SCM) — the *invertible* variant,
# so DoWhy can abduct a specific guest's own noise from their observed row
causal_graph = nx.DiGraph([('is_planner', 'lead_time'), 
                           ('is_planner', 'is_cancelled'), 
                           ('lead_time', 'is_cancelled')])

scm = gcm.InvertibleStructuralCausalModel(causal_graph)
scm.set_causal_mechanism('is_planner', gcm.EmpiricalDistribution())
scm.set_causal_mechanism('lead_time', gcm.AdditiveNoiseModel(gcm.ml.create_linear_regressor()))
# is_cancelled is a yes/no outcome, but the counterfactual API needs an
# *invertible* mechanism to reconstruct this guest's own noise. A Discrete
# Additive Noise Model (round a regression output to whole numbers) stays
# invertible where a plain classifier-based mechanism can't be — Part 8
# hit this exact requirement when it built its own digital twin.
scm.set_causal_mechanism('is_cancelled', gcm.DiscreteAdditiveNoiseModel(gcm.ml.create_linear_regressor()))

# 3. Fit the model to learn the 'mechanisms'
gcm.fit(scm, df)

# 4. Pick a specific guest who cancelled — programmatically, from the real data
cancelled_idx = df[df["is_cancelled"] == 1].index[0]
specific_guest = df.loc[[cancelled_idx]]
print(f"Guest index: {cancelled_idx}")
print(f"Original Lead Time: {specific_guest['lead_time'].values[0]:.2f}")
print(f"Original is_planner: {specific_guest['is_planner'].values[0]}")
print(f"Original Cancellation: {specific_guest['is_cancelled'].values[0]}")

# 5. Counterfactual: what if their lead time had been just 10 days instead?
cf_data = gcm.counterfactual_samples(scm, 
                                    {'lead_time': lambda x: 10}, 
                                    observed_data=specific_guest)

print(f"Counterfactual Cancellation: {cf_data['is_cancelled'].values[0]}")

Running this prints:

Guest index: 0
Original Lead Time: 21.78
Original is_planner: 0
Original Cancellation: 1
Counterfactual Cancellation: 1

Interpret every number: Guest 0 (the first row in df where is_cancelled == 1) originally had a lead time of 21.78 days and cancelled. When Maya forces their lead time down to just 10 days — a much shorter booking window than they actually used — the counterfactual answer is still 1: cancelled. Not every cancellation traces back to lead time. Guest 0’s own abducted “noise” — whatever it is about this specific person (their mood, their plans, something the model doesn’t observe) — is what drove the outcome, and that noise carries into the counterfactual world unchanged. Shrinking their lead time doesn’t touch it.

Read that carefully. The counterfactual asks: if this person — with all their same quirks, same mood, same luck — had booked closer to their stay, would they have cancelled? It’s not saying “people who book early cancel less.” It’s asking whether this specific human, in the alternate timeline, would have behaved differently — and for Guest 0, the honest, deterministic answer is no. That’s not a failure of the method; it’s the method telling you something real: some cancellations are about the lead-time policy, and some are about the guest. Section 5 shows both kinds side by side, with real guests.

4. The Hardest Part: The ‘Ladder of Causation’

This is the hardest concept in the series. Judea Pearl, a pioneer in causal inference, describes a Ladder of Causation with three rungs:

1. Seeing (Association): What does a symptom tell me about a disease? This is standard ML — finding patterns. (Rung 1.)
2. Doing (Intervention): What happens if I take the aspirin? This is the ATE and randomized trials — changing the world. (Rung 2.)
3. Imagining (Counterfactuals): Was it the aspirin that stopped my headache, or would it have gone away anyway? (Rung 3.)

Counterfactuals sit at the top. To reach this rung you need more than data — you need a model you trust completely. Here’s why: to imagine a different past, you have to “fix” the noise. You assume the specific random things that happened to that guest (their mood, the weather that day) stay exactly the same, while only the treatment changes.

To imagine a different past, you must believe your model captures the “why” of the world perfectly. That’s a much stronger assumption than a plain ATE requires. The ATE just needs your confounders handled. A counterfactual needs your entire mechanism to be correct, including how noise enters the system.

5. Interpreting the ‘What-If’ Number

How does Maya put this to work? Say she has 100 guests who cancelled. She can run a counterfactual for each one: “If I had offered this guest a 10% discount, would they have stayed?”

• Guest A: Real outcome = Cancelled. Counterfactual with discount = Cancelled. (Don’t spend the discount — they were leaving anyway.)
• Guest B: Real outcome = Cancelled. Counterfactual with discount = Stayed. (This is a “savable” guest. The missing discount caused the cancellation.)

This is Causal Attribution — pinpointing the specific cause responsible for an outcome, one individual at a time.

Here’s the same idea, this time with real guests from the data and the lead-time lever the SCM already has a mechanism for, instead of a hypothetical discount:

# Select the first 5 guests who cancelled
cancelled_guests = df[df['is_cancelled'] == 1].head(5)

# Ask: what if their lead time had been half as long?
cf_results = gcm.counterfactual_samples(scm, 
                                       {'lead_time': lambda x: x / 2}, 
                                       observed_data=cancelled_guests)

# Compare outcomes
for i in range(5):
    idx = cancelled_guests.index[i]
    orig_lt = cancelled_guests.iloc[i]['lead_time']
    cf_lt = cf_results['lead_time'].iloc[i]
    cf_outcome = cf_results['is_cancelled'].iloc[i]
    label = 'Cancelled' if cf_outcome == 1 else 'Stayed'
    print(f"Guest {idx}: lead_time {orig_lt:.1f} -> {cf_lt:.1f}, Original=Cancelled, CF={label}")

Running this prints:

Guest 0: lead_time 21.8 -> 10.9, Original=Cancelled, CF=Cancelled
Guest 2: lead_time 73.8 -> 36.9, Original=Cancelled, CF=Stayed
Guest 3: lead_time 76.1 -> 38.1, Original=Cancelled, CF=Stayed
Guest 7: lead_time 74.6 -> 37.3, Original=Cancelled, CF=Stayed
Guest 8: lead_time 69.3 -> 34.6, Original=Cancelled, CF=Stayed

What the numbers show: Guest 0 — the same guest from Section 3 — is a lost cause twice over. Halving their lead time (down to about 11 days) doesn’t save them either, matching what forcing it all the way down to 10 already showed. But Guests 2, 3, 7, and 8 all flip from Cancelled to Stayed. All four happen to be planners with long original lead times (69–76 days); cut that in half and the model now predicts they’d have stayed. That’s Causal Attribution in action: Guest 0’s cancellation was really about them, not their booking window — the Guest A pattern above. Guests 2, 3, 7, and 8’s cancellations were about the booking window — they were savable, the Guest B pattern.

6. Summary and Next Steps

Counterfactuals are the closest thing to time travel in data science. They let Maya move past averages and into individual stories — the last piece of her 90-day-policy puzzle.

Here’s what she learned:

• Counterfactuals ask “what if” about specific past events.
• The Ladder of Causation puts counterfactuals at the top, because they require a functional model of the world (an SCM).
• Unit-level reasoning helps identify which specific customers were actually affected by a treatment versus those who were “lost causes.”

Maya now has every tool she needs. She knows:

• whether long lead times cause cancellations (Tutorials 2–3),
• whether hidden ghosts could break that answer (Tutorial 4),
• how to get around unmeasurable ghosts (Tutorial 5),
• how to compare apples to apples (Tutorial 6),
• who the policy helps and who it hurts (Tutorial 7),
• how to find the root cause when things break (Tutorial 8),
• and which specific guests were actually “savable” (this tutorial).

Time to write the memo. One end-to-end case study — from “the naive number” to “here’s what Blue Harbor should actually do with the 90-day policy.” Mr. Delgado is waiting.

In this tutorial, you learned:

• What counterfactuals are and how they differ from averages
• The Fundamental Problem of Causal Inference
• How an SCM powers the counterfactual API
• Pearl’s Ladder of Causation (seeing → doing → imagining)
• How causal attribution identifies which guests were “savable”

Next up: Tutorial 10 — the full end-to-end case study: does lead time actually cause cancellations, and what should Blue Harbor do about the 90-day policy?

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three rungs of Judea Pearl’s “Ladder of Causation,” and which one do counterfactuals occupy?

Understand In your own words, explain why $Y(1)$ and $Y(0)$ can never both be observed for the same guest, and how that relates to the “Fundamental Problem of Causal Inference” mentioned earlier in the series.

Apply Using the tutorial’s Guest A / Guest B framework (real outcome vs. counterfactual-with-discount outcome), classify a third guest, Guest C: real outcome = Stayed, counterfactual-without-the-discount-they-received = Stayed. What business conclusion would you draw about whether the discount was worth giving to Guest C?

Analyze The tutorial says reaching the “Imagining” rung requires you to “fix” the noise—assuming a guest’s mood, the weather, and other random factors stay the same while only the treatment changes. Walk through why this assumption is much stronger (and harder to justify) than the assumptions needed for a plain ATE estimate from Part 3.

Evaluate The tutorial’s gcm.counterfactual_samples() step returns a single deterministic answer per guest — Section 3’s Guest 0 gets a flat “still cancelled,” no matter how aggressively lead time is cut. Critique treating that hard yes/no as the full picture: what nuance does a single deterministic label hide compared to a probability, and what would you have to change about the is_cancelled mechanism (hint: think about the classifier-based mechanism Part 8 originally tried and had to abandon for this exact API) to get an actual probability back out?

Create Design a counterfactual-attribution workflow for a customer-support scenario: for customers who churned after a slow support response, propose what you’d hold fixed (like the tutorial “fixes” mood and weather) and what you’d intervene on (like the tutorial intervenes on lead time), to figure out which churned customers were actually “savable” with a faster response.

---