The End-to-End Case Study: What Should Blue Harbor Do With the 90-Day Policy?

Last Time on the 90-Day Policy

For nine tutorials, Maya — the data scientist at Blue Harbor Hotels — has been investigating the 90-day policy: a fee on any booking made more than 90 days before arrival, shipped by a previous team to cut cancellations. She has:

• drawn the DAG (causal picture) and estimated the effect (Tutorials 2–3),
• stress-tested it against hidden confounders (Tutorial 4),
• found a side door around unmeasurable ghosts (Tutorial 5),
• balanced her groups with propensity scores (Tutorial 6),
• discovered the average was hiding a split: new guests helped, loyal guests hurt (Tutorial 7),
• traced a fake crisis to a vendor bug with root cause analysis (Tutorial 8),
• and identified which specific guests were actually “savable” with counterfactuals (Tutorial 9).

Now it’s time to write the memo. Every tool she learned goes into one end-to-end investigation — from a naive correlation to a business decision. The complete causal workflow, on one problem, start to finish. You’re going to run it right alongside her, on the exact same dataset the whole series has been using.

What you’ll learn here:

• How to run the full causal workflow end to end on one real problem
• Why the naive answer overstates the effect — and why a per-day slope can hide the real story of a threshold policy
• How to move through the five steps: Why → DAG → Identify → Estimate → Refute
• How to turn a causal estimate into a business recommendation
• The honest limits of what the analysis can and can’t tell you

Prerequisites: Tutorials 1–9. This is the capstone — everything comes back.

The $10,000 Question: Does Lead Time Actually Cause Cancellations?

Maya is the revenue analyst for Blue Harbor Hotels. She’s spotted a clear pattern: guests who book far in advance cancel far more often than those who book two days before arrival.

Her instinct — and the instinct of the team that shipped the 90-day policy — is to treat long lead times as the enemy. But does the long lead time cause the cancellation, or is it just a marker for something else? Perhaps people who book far in advance are simply “vacation dreamers” who change their minds, regardless of when they book.

If Blue Harbor gets this wrong, it scares away loyal customers with a policy that doesn’t fix the problem. Predictive models can tell us that long lead times correlate with cancellations. They’re silent on the why. To change a policy, Maya needs to move from prediction to causation — the whole point of this series.

So we’ll start with the naive approach — the way most analysts get trapped. And rather than build a new dataset for the finale, Maya reaches for the same one this whole series has been running on: the 3,000-row synthetic hotel-booking table from Tutorial 2, seeded once with np.random.seed(0).

import pandas as pd
import numpy as np
from dowhy import CausalModel
import networkx as nx
import statsmodels.api as sm

# This is the exact dataset from Tutorial 2 — same seed, same n, same
# generating equations. Nothing gets rebuilt from scratch for the case
# study; Maya is running the full workflow on the table Tutorials 2-9
# have all been probing.
np.random.seed(0)
n = 3000

# Confounders: is_repeat_guest and market_segment each influence BOTH
# lead_time and is_cancelled below — that's what actually makes them
# confounders, not just a comment saying so.
is_repeat_guest = np.random.binomial(1, 0.3, n)
market_segment = np.random.choice(
    ["Direct", "Corporate", "Online TA", "Groups"], n,
    p=[0.2, 0.25, 0.45, 0.1]
)

# lead_time (Treatment): repeat guests need less runway (they know the
# property already), Groups need lead time to coordinate, Corporate
# trips get booked close to the date.
segment_lead_scale = np.select(
    [market_segment == "Corporate", market_segment == "Groups"],
    [35, 90],
    default=60,
)
lead_time_scale = np.clip(segment_lead_scale - 45 * is_repeat_guest, 10, None)
lead_time = np.random.exponential(scale=lead_time_scale).astype(int)

# special_requests: a mediator on the lead_time -> is_cancelled path
# (Tutorial 2, Section 2) — carried over unchanged, not used in today's
# graph, but still part of the same generating process.
segment_request_base = np.select(
    [market_segment == "Corporate", market_segment == "Groups"],
    [0.3, 1.1],
    default=0.7,
)
special_requests_rate = segment_request_base + 0.01 * lead_time
special_requests = np.random.poisson(special_requests_rate, n)

# adr: the collider from Tutorial 2 — also carried over, also excluded
# from every graph below, for the same reason it was excluded there.
adr_base = np.where(market_segment == "Corporate", 120, 95)
adr = adr_base + lead_time * 0.1 + np.random.normal(0, 15, n)

# is_cancelled (Outcome): true per-day causal effect of lead_time is
# 0.003 — the same figure locked in back in Tutorial 2 and confirmed in
# every tutorial since.
segment_cancel_shift = np.select(
    [market_segment == "Corporate", market_segment == "Groups"],
    [-0.05, 0.08],
    default=0.0,
)
cancel_prob = (
    0.2
    + 0.003 * lead_time
    - 0.3 * is_repeat_guest
    - 0.03 * special_requests
    + segment_cancel_shift
    + np.random.normal(0, 0.08, n)
)
is_cancelled = np.random.binomial(1, np.clip(cancel_prob, 0, 1))

df = pd.DataFrame({
    "lead_time": lead_time,
    "is_repeat_guest": is_repeat_guest,
    "market_segment": market_segment,
    "special_requests": special_requests,
    "adr": adr.round(2),
    "is_cancelled": is_cancelled,
})

# The Naive Approach: Simple Correlation
correlation = df['lead_time'].corr(df['is_cancelled'])
print(f"Naive Correlation: {correlation:.4f}")

# Simple Linear Regression (Naive Estimate)
X = sm.add_constant(df['lead_time'])
model_naive = sm.OLS(df['is_cancelled'], X).fit()
print(f"Naive Regression Coefficient: {model_naive.params['lead_time']:.6f}")

Naive Correlation: 0.3954
Naive Regression Coefficient: 0.003295

The naive regression tells Maya that for every extra day of lead time, the probability of cancellation goes up by about 0.0033 — close to Tutorial 2’s true 0.003/day figure, but that closeness is a coincidence of this particular naive slice, not proof the number is trustworthy. The real pollution is hiding one level down: is_repeat_guest is doing most of the damage. Repeat guests book roughly 16 days out on average versus 53 days for everyone else, and they cancel at 2.3% versus 31.7% for first-timers — a variable that swings both the treatment and the outcome that hard, sitting outside the naive regression entirely, is exactly the confounding shape Tutorial 1 warned about. market_segment adds a second layer on top: Groups book furthest out and cancel the most (a positive +0.08 shift in cancellation risk baked into the DGP), while Corporate books closest in and cancels the least (-0.05). Both segments push the naive correlation in the same direction it’s already leaning.

A quick vocabulary reminder: a confounder (here, is_repeat_guest and market_segment) is a variable that influences both the treatment (lead time) and the outcome (cancellation), faking part of the connection between them. That’s the trap Maya warned about back in Tutorial 1 — the “ice cream and drowning” mistake, hiding inside a real business decision.

One more wrinkle, and it’s the one that actually matters for the memo. The 90-day policy isn’t a per-day slope — it’s a threshold. It fires once, the moment a booking crosses 90 days out. So the number Blue Harbor’s VP actually needs isn’t “how much does each extra day matter,” it’s “how much does crossing the 90-day line change the odds of cancellation.” Maya defines that threshold directly:

df["is_long_lead"] = (df["lead_time"] > 90).astype(int)
print(f"Share flagged by the 90-day policy: {df['is_long_lead'].mean():.1%}")

naive_gap = (
    df.loc[df.is_long_lead == 1, "is_cancelled"].mean()
    - df.loc[df.is_long_lead == 0, "is_cancelled"].mean()
)
print(f"Naive mean gap (long-lead minus short-lead cancel rate): {naive_gap:.4f}")

Share flagged by the 90-day policy: 12.9%
Naive mean gap (long-lead minus short-lead cancel rate): 0.3936

12.9% of bookings actually trip the fee. And the raw, uncontrolled gap is enormous: guests flagged by the policy cancel 56.8% of the time, versus 17.5% for everyone else — a 39.4 percentage-point gap with no adjustment for anything. That’s the number that would have justified the policy on its face back when it was written. It’s also, per the last nine tutorials, almost certainly too big — because is_repeat_guest and market_segment are baked into who crosses that 90-day line in the first place. This is the number the rest of today’s workflow exists to correct.

Step 1: Mapping the Chaos (The DAG)

To fix this, Maya needs to draw a map of her assumptions: a Directed Acyclic Graph (DAG). Think of it as saying, “I believe X causes Y, and Z causes both.” Arrows point from cause to effect. Nothing points backward in time.

This is the hardest part of causal inference. You sit down with domain experts and ask: “What else could be driving this?” In the hotel case, we know market_segment affects how early people book and how likely they are to flake. We also know is_repeat_guest matters — loyal guests book earlier and cancel less, and the data-generating process above confirms it (lead_time_scale and cancel_prob both depend on it directly).

Let’s build this map in DoWhy, explicitly telling the model that market_segment and is_repeat_guest are confounders of the policy threshold is_long_lead:

# Define the causal graph
G = nx.DiGraph()
G.add_nodes_from(["is_long_lead", "is_cancelled", "market_segment", "is_repeat_guest"])
G.add_edges_from([
    ("market_segment", "is_long_lead"),
    ("market_segment", "is_cancelled"),
    ("is_repeat_guest", "is_long_lead"),
    ("is_repeat_guest", "is_cancelled"),
    ("is_long_lead", "is_cancelled"),
])

model = CausalModel(
    data=df,
    treatment="is_long_lead",
    outcome="is_cancelled",
    graph=G
)

# Visualize the graph to ensure it matches our intuition
model.view_model()

What this means: Maya is acknowledging backdoor paths. There’s a path from is_long_lead back through market_segment to is_cancelled, for instance. It creates a fake relationship that she needs to block by holding market_segment constant. (A backdoor path, from Tutorial 3, is any route from treatment to outcome that goes against the causal arrow and then with it — carrying bias along the way.)

special_requests and adr are still sitting in df — they came along with the rest of Tutorial 2’s dataset — but they don’t appear in today’s graph. Tutorial 2 already worked out their roles (mediator, collider) and the case study doesn’t need to relitigate that; today’s question is specifically about the two backdoor confounders and the 90-day threshold.

Step 2: Can We Even Solve This? (Identification)

Before doing any math, Maya asks: “Given this graph, is it even possible to find the answer?” This is Identification — proving whether your causal question can be answered from the data you have.

Think of it as a sanity check. If Maya forgot to measure a major confounder, DoWhy will warn her. Here, it looks for a Backdoor Adjustment Set — the variables we need to control for to isolate the true effect of crossing the 90-day line.

identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)
print(identified_estimand)

Estimand type: EstimandType.NONPARAMETRIC_ATE

### Estimand : 1
Estimand name: backdoor
Estimand expression:
       d
───────────────(E[is_cancelled|market_segment,is_repeat_guest])
d[is_long_lead]
Estimand assumption 1, Unconfoundedness: If U→{is_long_lead} and U→is_cancelled then
P(is_cancelled|is_long_lead,market_segment,is_repeat_guest,U) =
P(is_cancelled|is_long_lead,market_segment,is_repeat_guest)

### Estimand : 2
Estimand name: iv
No such variable(s) found!

### Estimand : 3
Estimand name: frontdoor
No such variable(s) found!

DoWhy confirms she must adjust for market_segment and is_repeat_guest to get the causal effect. No instrumental-variable or front-door estimand exists here — that’s expected, since this graph has no instrument and no mediator in play. If the confounders weren’t in the data, DoWhy would tell her the effect is unidentifiable — no amount of data would fix it. That check alone keeps her from publishing a broken result.

Step 3: Calculating the Number (Estimation)

Now the actual calculation. Maya uses Propensity Score Stratification — the technique from Tutorial 6. Rather than a plain regression, this method groups similar guests together and compares a guest who crossed the 90-day line to one who didn’t, sharing the same market segment and loyalty status. By keeping the comparison apples to apples, it strips away the bias. (This is also why is_long_lead has to be a yes/no flag rather than the raw day count — propensity-score methods stratify on the probability of receiving a binary treatment, and the 90-day policy is naturally binary anyway: you either cross the line or you don’t.)

# Estimate the causal effect
causal_estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.propensity_score_stratification",
    target_units="ate"  # Average Treatment Effect
)

print(f"Causal Estimate (ATE): {causal_estimate.value:.6f}")

pct_diff = (causal_estimate.value - naive_gap) / naive_gap * 100
print(f"Difference from Naive gap: {pct_diff:.2f}%")

Causal Estimate (ATE): 0.300018
Difference from Naive gap: -23.77%

So how does that hold up against the data? Just like the last nine tutorials predicted: the causal estimate is lower than the naive one. Crossing the 90-day line does cause a jump in cancellation risk — a real 30.0 percentage-point increase — just not the full 39.4-point gap Blue Harbor’s original team was staring at. About a quarter of that raw gap was is_repeat_guest and market_segment riding along, not the threshold itself.

The naive number would have justified the 90-day policy at close to its full face value. The honest number is meaningfully smaller. That means the policy was over-reacting to a signal that’s real, but partly borrowed.

Step 4: Trying to Break Our Result (Refutation)

In science, we don’t just accept a number — we try to break it. DoWhy calls this Refutation. Maya runs two tests:

1. Placebo Treatment: replace is_long_lead with random noise. If the logic holds, the “causal effect” of that noise should be zero. If it isn’t, the model is picking up ghosts.
2. Data Subset Refuter: drop a random 20% of the data, then re-run the analysis. The estimate should stay roughly the same. If it swings wildly, the result was a fluke of that particular sample.

# Refute 1: Placebo Treatment
refute_placebo = model.refute_estimate(
    identified_estimand,
    causal_estimate,
    method_name="placebo_treatment_refuter",
    placebo_type="permute"
)
print(refute_placebo)

# Refute 2: Subset Refuter
refute_subset = model.refute_estimate(
    identified_estimand,
    causal_estimate,
    method_name="data_subset_refuter",
    subset_fraction=0.8
)
print(refute_subset)

Refute: Use a Placebo Treatment
Estimated effect: 0.3000175068833426
New effect: 0.0020554890636689978
p value: 0.98

Refute: Use a subset of data
Estimated effect: 0.3000175068833426
New effect: 0.28860994738270884
p value: 0.46

Here’s the thing, though: check the p-values. A p-value is a number from 0 to 1 measuring how likely your result would be if there were no real effect. A tiny one — say, 0.01 — means the result would be very unlikely under pure chance. A big one, like 0.5, means it could easily happen by chance.

For a refutation, a high p-value is what you want. It means:

• In the placebo test, we cannot prove the effect of pure noise is real — the “effect” of a randomly permuted treatment collapsed to 0.0021, essentially nothing, with p = 0.98. (If that p-value drops below ~0.05, the model found a “causal effect” for random noise — a red flag that the DAG is wrong.)
• In the subset test, dropping 20% of the rows moved the estimate from 0.3000 to 0.2886 — a small shift, with p = 0.46, meaning that shift is well within what random sampling noise would produce.

Maya’s results: no real effect of noise on the placebo, and a stable estimate on the subset. Her number survives its own stress test.

From Data to Decision: What Now?

Maya started with a business question: should Blue Harbor keep penalizing long lead times?

The causal workflow gave her some answers:

• The true causal effect of crossing the 90-day line is smaller than the raw comparison suggested — 30.0 points, not 39.4.
• A meaningful share of the raw gap is driven by who books long-lead (loyal-guest status, market segment), not the threshold itself.
• The result is robust to noise and to data subsets.
• The naive per-day regression coefficient (0.0033) happened to land close to Tutorial 2’s true 0.003/day figure — but that was luck, not vindication. The actual policy variable, the 90-day threshold, is where the bias shows up clearly.

The full series adds another layer. From Tutorial 7, she knows the effect isn’t uniform — the average hides that the policy hurts loyal repeat guests, the exact customers a hotel chain can’t afford to lose. From Tutorial 9, she knows only some cancelling guests are “savable” by changing the booking conditions.

Business Recommendation: Instead of a blanket non-refundable policy for all long lead times, Blue Harbor should consider a targeted policy — applying scrutiny to the Groups segment specifically, since Groups both cross the 90-day line most often (27.9% of Groups bookings) and cancel at the highest rate of any segment (33.6%), or offering small incentives for guests to confirm their stay 30 days out. A blanket fee mostly catches Direct guests who simply like to plan ahead (16.4% of Direct bookings cross the line) and repeat guests who almost never trip it in the first place (only 1.2% of repeat-guest bookings cross 90 days, since loyal guests already book close in). Those are the customers the average was hiding.

Worth noting: this analysis proves why cancellations happen. It does not prove that a non-refundable deposit changes guest behavior. That’s a different intervention with its own confounders — does a fee scare off dreamers or just push them to a competitor? The causal workflow tells you what to fix; it doesn’t guarantee the fix works. You’d want to test the actual policy next.

Your Causal Checklist:

1. Start with the ‘Why’: what policy are you trying to change?
2. Draw the DAG: don’t skip this — it’s where the science happens.
3. Identify: let DoWhy check your logic.
4. Estimate: use propensity scores to compare apples to apples.
5. Refute: try to break your result before your boss does.

Priya reads the memo and smiles. Delgado, for once, is quiet. Maya has gone from “the email slide was a lie” to “here’s the truth about a policy that’s been charging our best customers for nine months” — and she can defend every step.

That’s the arc of this series. From spotting a naive correlation, through drawing a DAG, identifying and estimating the effect, stress-testing it, finding who it helps and hurts, and finally turning it all into a decision — the full causal workflow, end to end, run start to finish on the same 3,000-row dataset since Tutorial 2.

What you learned:

• How the naive correlation overstates a causal effect
• The five-step causal workflow: Why → DAG → Identify → Estimate → Refute
• Why a threshold policy needs a threshold treatment variable, not just a per-day slope
• How to interpret p-values in refutation (high is good there!)
• How to turn an estimate into a business recommendation
• The honest gap between “why cancellations happen” and “this policy will fix it”

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the five steps in the tutorial’s “Causal Checklist”?

Understand In your own words, explain what a “high p-value” means in the context of the Placebo Treatment refuter, and why a high p-value is the good outcome here (unlike in most hypothesis testing, where a low p-value is the exciting result).

Apply The tutorial computes “Difference from Naive” as a percentage: (causal_estimate - naive_estimate) / naive_estimate * 100. If the naive regression coefficient were 0.0015 and the causal (propensity-stratified) estimate were 0.0009, what would this formula report as the percentage difference?

Analyze The tutorial’s Data Subset Refuter deletes 20% of the data and re-runs the analysis, expecting the effect to “stay roughly the same.” Walk through why a wildly different result after removing a random 20% subset would specifically suggest the original estimate was “a fluke of the specific sample,” rather than suggesting the DAG itself is wrong (which is what the Placebo Treatment refuter tests for).

Evaluate The tutorial’s final business recommendation suggests a targeted policy for the “Groups” segment instead of a blanket non-refundable policy. Critique this recommendation: the whole case study only tested whether crossing the 90-day line causes cancellation — it never causally tested whether a non-refundable deposit changes guest behavior. What’s the gap between “we understand why cancellations happen” and “we know this specific policy will fix it”?

Create Design your own end-to-end causal case study outline (following the tutorial’s checklist: Why → DAG → Identify → Estimate → Refute) for a different business question: “does offering free shipping cause higher order values?” Name the treatment, outcome, at least one confounder, and one refutation test you’d run before trusting the result.

---