Instrumental Variables: When You Can't Measure the Confounder Directly

Last Time on the 90-Day Policy

Maya — the new data scientist at Blue Harbor Hotels — has been investigating whether Blue Harbor’s 90-day policy (a fee on bookings made more than 90 days out) is justified. Last tutorial, she stress-tested her model against unmeasured confounders — the “ghost” variables like guest loyalty or stress that aren’t in the database. She measured exactly how strong such a ghost would need to be to break her result.

Then Mr. Delgado, the VP of Revenue, pushed further: “What if the ghost isn’t a maybe? What if we KNOW there’s a hidden factor, like a guest’s stress level, and we simply cannot measure it — no matter what?”

Most data scientists would give up here and fall back on simple correlation. There’s a way around it, though — Instrumental Variables (IV). Think of it as a side door. When the front door (direct measurement) is locked and the back door (controlling for confounders) is blocked by ghosts, you look for a side window to climb through.

This tutorial is Maya’s lesson in finding that window — and it might be the most fun trick in the whole series.

What you’ll learn here:

• When a confounder can’t be measured at all, and why the backdoor stays open
• What an Instrumental Variable (IV) is, with the three rules it must satisfy
• How to identify an IV estimand in DoWhy
• How to estimate with Two-Stage Least Squares (2SLS)
• Why IV estimates are more accurate but less precise — and when that tradeoff is worth it

Prerequisites: Tutorials 1–4, or comfort with DoWhy, DAGs, estimands, and the idea of unmeasured confounders.

1. The Ghost in the Data: When Backdoors Stay Open

Back to the hotel dataset — the same 3,000-booking dataset from Tutorial 2, rebuilt with the same np.random.seed(0). Maya wants to know if longer lead_time causes more cancellations. In the previous parts, she controlled for is_repeat_guest and market_segment. But a guest’s personal stress or travel intent plays a big role here, and she hasn’t accounted for that yet.

If a guest is highly stressed, they might book very early to feel prepared (high lead time) — but they’re also more likely to cancel because their plans are fragile (high cancellation). Blue Harbor can’t put a “Stress-o-meter” on customers, so this variable U stays unobserved (not measured, not a column in df — that’s what makes it a ghost rather than just another confounder like is_repeat_guest).

What this actually means: the backdoor is open. From Part 2: a backdoor path is a route from treatment to outcome that runs against the causal arrow, creating a fake correlation. Any relationship Maya sees between lead time and cancellation might partly be the “Ghost of Stress” haunting both variables.

Here’s the full data-generating process. It matters that it’s built in this order — U and a second variable, system_delay, are drawn first, before lead_time even exists, and cancel_prob isn’t computed until lead_time has already absorbed everything that’s going to act on it:

import pandas as pd
import numpy as np
from dowhy import CausalModel

np.random.seed(0)
n = 3000

# U is our ghost variable (guest stress) — never added to df, since it's
# unmeasured by definition. If Blue Harbor could put a Stress-o-meter on
# every guest, U wouldn't be a ghost anymore.
u = np.random.normal(0, 1, n)

# system_delay is drawn independently of U, right here at the top. This
# ordering matters: it's what lets system_delay genuinely act on lead_time
# below, instead of being bolted on after cancel_prob is already computed.
system_delay = np.random.binomial(1, 0.5, n)

# --- Same base variables as Tutorials 2-4 (same seed, same 3,000 bookings) ---
is_repeat_guest = np.random.binomial(1, 0.3, n)
market_segment = np.random.choice(
    ["Direct", "Corporate", "Online TA", "Groups"], n,
    p=[0.2, 0.25, 0.45, 0.1]
)
segment_lead_scale = np.select(
    [market_segment == "Corporate", market_segment == "Groups"],
    [35, 90],
    default=60,
)
lead_time_scale = np.clip(segment_lead_scale - 45 * is_repeat_guest, 10, None)
lead_time = np.random.exponential(scale=lead_time_scale).astype(int)

# Ghost stress U nudges lead_time up: stressed guests over-plan and lock in
# a room earlier than they otherwise would.
lead_time = lead_time + (15 * u).astype(int)

# system_delay nudges lead_time up too. Blue Harbor migrated its booking
# engine mid-quarter; bookings routed through the still-stabilizing new
# checkout flow took guests noticeably longer to finish. Many abandoned the
# flow partway through and came back to rebook weeks later, once IT had
# patched it — which is why the effect below is measured in weeks, not days.
lead_time = lead_time + system_delay * 25
lead_time = np.clip(lead_time, 0, None)

# special_requests / adr, unchanged from Tutorial 2, built off the FINAL
# lead_time — after both the ghost and the migration have already acted on it.
segment_request_base = np.select(
    [market_segment == "Corporate", market_segment == "Groups"],
    [0.3, 1.1],
    default=0.7,
)
special_requests_rate = segment_request_base + 0.01 * lead_time
special_requests = np.random.poisson(special_requests_rate, n)

adr_base = np.where(market_segment == "Corporate", 120, 95)
adr = adr_base + lead_time * 0.1 + np.random.normal(0, 15, n)

# cancel_prob is computed LAST, from the final lead_time — same 0.003
# per-day coefficient as every other tutorial in this series — plus the
# ghost's own direct pull on cancellation (the confounding channel: U pushes
# lead_time up AND pushes cancel_prob up on its own). Notice system_delay
# does NOT appear anywhere in this formula — it only reaches is_cancelled
# by way of lead_time. That's the exclusion restriction, built in by
# construction rather than asserted.
segment_cancel_shift = np.select(
    [market_segment == "Corporate", market_segment == "Groups"],
    [-0.05, 0.08],
    default=0.0,
)
cancel_prob = (
    0.2
    + 0.003 * lead_time
    - 0.3 * is_repeat_guest
    - 0.03 * special_requests
    + segment_cancel_shift
    + 0.15 * u
    + np.random.normal(0, 0.08, n)
)
is_cancelled = np.random.binomial(1, np.clip(cancel_prob, 0, 1))

df = pd.DataFrame({
    "lead_time": lead_time,
    "is_repeat_guest": is_repeat_guest,
    "market_segment": market_segment,
    "special_requests": special_requests,
    "adr": adr.round(2),
    "is_cancelled": is_cancelled,
    "system_delay": system_delay,
})

print(f"Cancellation rate: {df['is_cancelled'].mean():.1%}")

# Define the graph with an unobserved variable 'U' — system_delay isn't in
# this graph yet. It's sitting in df, but Maya hasn't earned the right to
# call it an instrument until Section 2 checks it against all three rules.
gml_graph = """graph [
    directed 1
    node [ id "lead_time" label "lead_time" ]
    node [ id "is_cancelled" label "is_cancelled" ]
    node [ id "U" label "U" ]
    edge [ source "U" target "lead_time" ]
    edge [ source "U" target "is_cancelled" ]
    edge [ source "lead_time" target "is_cancelled" ]
]"""

model = CausalModel(data=df, treatment="lead_time", outcome="is_cancelled", graph=gml_graph)
model.view_model()

Cancellation rate: 27.0%

There’s a column in df we haven’t touched yet: system_delay. Hold that thought — it’s the whole point of Section 2.

If Maya tries to use the backdoor method here, DoWhy will warn her: there is no way to close the path through U, because U isn’t in the data. She’s stuck.

That warning is the whole point. DoWhy just told her that the front door and the back door are both sealed.

2. The ‘Instrument’ Intuition: A Lever for Change

Priya teaches Maya the trick. Her line: “You need a ‘lever’ that can wiggle the lead time without wiggling the stress.”

That’s exactly what system_delay is. Blue Harbor’s booking-engine migration didn’t know or care whether a guest was stressed or relaxed — it was a technical rollout, full stop. But it did push lead time up for the bookings caught in it, because guests routed through the shaky new checkout flow abandoned it and rebooked weeks later.

To count as a valid instrument, system_delay has to satisfy three rules:

1. Relevance: It actually changes the treatment. (The migration must actually increase lead time.)
2. Exclusion: It only affects the outcome through the treatment. (The migration shouldn’t make people cancel for any reason other than the extra lead time it caused.)
3. Independence: It isn’t affected by the ghost confounder. (The migration rollout happened on Blue Harbor’s own schedule — not because stressed people were on the site.)

The hard part: a real-world variable that clears all three is rare. Find one, though, and it’s worth the hunt — you get to physically wiggle the treatment while holding the ghost still.

Talk is cheap, though. Here’s what “checking the rules” actually looks like against the data Maya already has:

import statsmodels.api as sm

# Update the graph to include the instrument (Z)
gml_graph_iv = """graph [
    directed 1
    node [ id "lead_time" label "lead_time" ]
    node [ id "is_cancelled" label "is_cancelled" ]
    node [ id "U" label "U" ]
    node [ id "system_delay" label "system_delay" ]
    edge [ source "system_delay" target "lead_time" ]
    edge [ source "U" target "lead_time" ]
    edge [ source "U" target "is_cancelled" ]
    edge [ source "lead_time" target "is_cancelled" ]
]"""

model = CausalModel(data=df, treatment="lead_time", outcome="is_cancelled", graph=gml_graph_iv)

# --- Relevance, checked (not asserted): does system_delay actually move lead_time? ---
X = sm.add_constant(df["system_delay"])
first_stage = sm.OLS(df["lead_time"], X).fit()
print(f"First-stage coefficient (system_delay -> lead_time): {first_stage.params['system_delay']:.4f}")
print(f"First-stage F-statistic: {first_stage.fvalue:.2f}")

# --- Exclusion, sanity-checked: system_delay against is_cancelled directly, no controls ---
reduced_form = sm.OLS(df["is_cancelled"], X).fit()
print(f"Reduced-form coefficient (system_delay -> is_cancelled): {reduced_form.params['system_delay']:.4f}")

First-stage coefficient (system_delay -> lead_time): 23.6724
First-stage F-statistic: 135.58
Reduced-form coefficient (system_delay -> is_cancelled): 0.0742

Now the three rules are honest, not decorative:

• Relevance — the first-stage F-statistic is 135.58. The rule of thumb for a strong instrument is F ≫ 10; this clears it by more than 13x. system_delay moves lead_time by about 23.7 days, on average, and that coefficient is nowhere near zero (t ≈ 11.6). This is exactly the check that would fail if the instrument were a simulation artifact rather than a real lever — and here, it doesn’t.
• Exclusion — the reduced form (is_cancelled regressed on system_delay alone, no lead_time in sight) comes out to 0.0742. That’s not zero, but it’s not supposed to be — Exclusion says the instrument reaches the outcome only through the treatment, not that it has zero raw correlation with the outcome. Here’s the arithmetic check: system_delay moves lead_time by ~23.7 days, and each day of lead_time moves cancel_prob by 0.003, so the implied indirect effect is 23.7 × 0.003 ≈ 0.071 — close to the 0.0742 the reduced form actually measures, with the small gap explained by sampling noise. That match is what Exclusion holding looks like in real numbers, not an assumption taken on faith.
• Independence — u and system_delay were drawn from two separate, unrelated np.random calls at the very top of the script, before either lead_time or market_segment existed. Their sample correlation is -0.0065 — indistinguishable from zero. The migration rollout has nothing to do with any individual guest’s stress level.

Look at the picture again: system_delay points only at lead_time. The ghost U doesn’t touch it, and it doesn’t touch is_cancelled directly. One clean arrow, now backed by three checks instead of three assertions.

3. Step 2 Revisited: Identifying the IV Estimand

Maya puts the question to DoWhy: “Since the backdoor is blocked by U, is there any other way?”

Front door locked, backdoor blocked. So what’s left? A side window. When Maya runs identify_effect(), DoWhy spots the system_delay variable and reaches for Instrumental Variable logic.

ident = model.identify_effect()
print(ident)

Check the output — there’s a section labeled Instrumental Variables. DoWhy identified system_delay as a valid instrument for the path between lead_time and is_cancelled. The side window worked.

This is a different recipe (estimand) than the backdoor formula from Part 3. There, you hold a list of confounders steady. Here, the confounder is invisible. The recipe uses the random nudge as its key.

4. Estimation via 2SLS: The Two-Step Dance

So how do we actually calculate the number? We use Two-Stage Least Squares (2SLS). The name sounds intimidating, but it’s just two steps:

1. Stage 1: Use the instrument (system_delay) to predict lead_time. This gives us a “clean” version of lead time — one based only on the migration, not on the ghost U. The predicted lead time captures pure “migration effect,” since stress played no role in causing the migration rollout schedule.
2. Stage 2: Use this clean lead time to predict is_cancelled.

The two stages are literally two ordinary regressions run back to back — nothing more exotic than that:

# Stage 1: predict lead_time using only the instrument
X = sm.add_constant(df["system_delay"])
stage1 = sm.OLS(df["lead_time"], X).fit()
lead_time_hat = stage1.predict(X)

# Stage 2: predict is_cancelled using the CLEAN, predicted lead_time
X_hat = sm.add_constant(lead_time_hat.rename("lead_time_hat"))
stage2 = sm.OLS(df["is_cancelled"], X_hat).fit()

print(f"Causal Effect (IV / 2SLS): {stage2.params['lead_time_hat']:.6f}")
print(f"95% CI: {stage2.conf_int().loc['lead_time_hat'].values}")

Causal Effect (IV / 2SLS): 0.003135
95% CI: [0.001796 0.004474]

Interpret every number: the estimate comes out to 0.003135. That means for every 1-day increase in lead time caused by the migration, the probability of cancellation rises by about 0.31 percentage points. That’s within a hair of the 0.003 true per-day effect this whole series has been built around since Part 2 — the same number Parts 3 and 4 estimate with backdoor adjustment. 2SLS, using a completely different identification strategy (a lever instead of a list of controls), lands almost exactly on the same target.

The phrase “caused by the migration” matters — this is the effect for the compliers: people whose behavior actually changed because of the instrument. A complier is someone who ends up booking later than they otherwise would have because the checkout flow they hit was the buggy one. The people who were booking three months out anyway don’t count — they never “complied.”

5. The Catch: Why Instruments Are Not Magic Bullets

Instruments have a catch. If the nudge from your instrument is too small, your estimate goes wild. This is called Weak Instrument Bias. It’s like trying to turn a ship with a teaspoon — you’re technically steering, but the water does whatever it wants. (Maya’s F-statistic of 135.58 means she’s nowhere near that cliff edge — but it’s worth knowing where the cliff is.)

Now compare the IV estimate to a naive regression that never accounts for U at all:

from sklearn.linear_model import LinearRegression

naive_model = LinearRegression().fit(df[['lead_time']], df['is_cancelled'])
print(f"Naive (Biased) Effect: {naive_model.coef_[0]:.6f}")

Naive (Biased) Effect: 0.003476

Here’s the honest comparison, run on the same data:

	Naive OLS	IV / 2SLS
Estimate	0.003476	0.003135
Standard error	0.000128	0.000683
95% CI width	≈0.0005	≈0.0027

The naive model says 0.003476; the IV says 0.003135. That’s about 11% apart — the naive estimate is roughly 1.11× the IV estimate, not the dramatic multiple you might expect from an IV horror story. The Ghost of Stress is real (it’s baked into the DGP: U pushes both lead_time and cancel_prob), but here its distortion of the naive number is modest, not catastrophic — U explains only a slice of the variance in either variable.

What is dramatic is the width of the uncertainty. The naive model’s 95% CI is barely 0.0005 wide; the IV’s is about 5.3× wider, at roughly 0.0027. That’s the real tradeoff Maya has to internalize:

• Accurate means “pointing at the right target” (no bias). Here, IV (0.003135) lands closer to the series’ established 0.003 truth than naive (0.003476) does.
• Precise means “narrow spread” (tight uncertainty). Naive wins here by a wide margin — its CI is a fifth the width of the IV’s.
• The naive model is precise-but-slightly-wrong: a confident, tight estimate that’s biased upward by the ghost.
• The IV is closer-to-right-but-noisier: a much wider range, but centered nearer the truth.

When the business decision is big — like changing a cancellation policy across a whole hotel chain — I’d still lean toward the number that’s honest about the confounder, even at the cost of a wider range.

What we learned today:

• When you can’t measure a confounder, the backdoor is blocked.
• An Instrumental Variable is a random nudge that affects your treatment but not the outcome directly — and that has to be true by construction in your data-generating process, not just asserted afterward.
• 2SLS is the two-step process of cleaning your treatment variable using that nudge.
• Here, IV recovered an estimate (0.003135) very close to the series’ canonical 0.003 effect, while the naive estimate was biased by about 11% and roughly 5x more precise (narrower) than the IV’s.

Maya has now found the side door. But she’s realizing something uncomfortable: most of the time, Blue Harbor actually can measure its confounders. The stress example was extreme. The everyday problem is different — they have the measurements, but the treated and control groups look nothing alike, which makes comparisons unfair.

That’s the problem the next tutorial tackles. And it’s the one Maya will actually use to make her case to Delgado.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three rules an Instrumental Variable must satisfy to be valid?

Understand In your own words, explain what the “Exclusion” rule means using the tutorial’s system_delay example—why would it be a problem if the system delay made people cancel for reasons other than the extra lead time it caused?

Apply The tutorial reports a naive regression effect of 0.003476 versus an IV estimate of 0.003135. Using these two numbers, calculate how many times larger the naive estimate is than the IV estimate, and explain why this gap is much smaller than the “off by 3x” gaps that sometimes show up in IV case studies.

Analyze The tutorial says IV estimates are “often more accurate but less precise” than standard regressions. Walk through why 2SLS’s Stage 1 (predicting lead_time using only system_delay) necessarily throws away information compared to using the full lead_time variable directly—and why that’s the source of the reduced precision (hint: look at the standard-error comparison in Section 5).

Evaluate The tutorial calls “Weak Instrument Bias” a catch: if the instrument’s nudge is too small, the estimate becomes wild and unreliable. Critique the process of finding an instrument in the first place: given how specific the three validity rules are, what’s the risk of a data scientist convincing themselves a weak or borderline instrument is valid just because it’s the only lever they could find? Use the tutorial’s F-statistic of 135.58 as a reference point for what “not weak” looks like.

Create Design a plausible instrumental variable for a different causal question: “does a longer customer-support hold time cause customers to churn?” Propose a candidate instrument and check it against all three rules (Relevance, Exclusion, Independence), noting any rule you’re unsure about.

---