How to Properly Handle Missing Data (Is Imputation Always the Right Answer?)

Imagine looking at a spreadsheet of customer orders. Ten percent of the “Shipping Address” column sits empty. Your first instinct might be to fill those gaps—a complete dataset feels more professional, right?

But how you fill them, or whether you fill them at all, changes the story your data tells. Missing data isn’t just an annoyance. It’s a decision point that shapes your whole analysis.

1. The Missing Data Problem: Why It Matters More Than You Think

When we see a NaN or a null in Python, we usually want to make it go away. Three main choices: drop the rows, fill them in (imputation), or build a model that doesn’t care. Most practitioners default to imputation without asking why the data is missing. So what happens when we handle it naively?

import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression

# Create a dataset where missingness is tied to the outcome
np.random.seed(42)
n = 1000
income = np.random.normal(50000, 10000, n)
spending = 0.5 * income + np.random.normal(0, 2000, n)

df = pd.DataFrame({'income': income, 'spending': spending})

# Scenario: High earners are less likely to report income (Missing Not At Random)
df.loc[df['income'] > 65000, 'income'] = np.nan

# 1. Naive Deletion
df_dropped = df.dropna()
model_dropped = LinearRegression().fit(df_dropped[['income']], df_dropped['spending'])

# 2. Naive Mean Imputation
df_imputed = df.copy()
df_imputed['income'] = df_imputed['income'].fillna(df_imputed['income'].mean())
model_imputed = LinearRegression().fit(df_imputed[['income']], df_imputed['spending'])

print(f"True Slope: 0.5")
print(f"Dropped Slope: {model_dropped.coef_[0]:.3f}")
print(f"Imputed Slope: {model_imputed.coef_[0]:.3f}")
print(f"Dropped R2: {model_dropped.score(df_dropped[['income']], df_dropped['spending']):.3f}")
print(f"Imputed R2: {model_imputed.score(df_imputed[['income']], df_imputed['spending']):.3f}")

In this example, the true relationship is 0.5. Both models print a nearly identical slope (~0.488). Mean imputation doesn’t distort the slope here because the imputed rows all sit at the same X value (the mean), contributing almost nothing to the slope calculation. The damage shows up elsewhere — check the R² for each model. The imputed model’s R² is dramatically lower, dropping from roughly 0.81 to 0.57. Mean imputation didn’t bias the coefficient in this setup, but it injected a pile of unexplained noise: 75 rows where a single fixed X value is paired with a wide range of high spending values the model can’t account for. We made our data look complete, but we made our model’s fit dishonest.

2. Three Types of Missingness: Why the Reason Matters More Than the Count

Picking the right fix starts with understanding the “Missingness Mechanism.” The reason for the gap matters more than the size of it.

1. MCAR (Missing Completely At Random): Data is missing by pure chance. A sensor battery died. A form got lost in the mail. None of it relates to the data itself.
2. MAR (Missing At Random): The name is confusing. What it means: the missingness depends on data we can see, not on the missing value itself. Men might be less likely to report their weight. But if we know their age and height, we can estimate it accurately.
3. MNAR (Missing Not At Random): This is the tricky one. The missingness depends on the value that’s missing. People with very high debt might simply skip the debt question entirely.

# Visualizing patterns
import seaborn as sns
import matplotlib.pyplot as plt

# Let's check if 'spending' predicts if 'income' is missing
df['is_missing'] = df['income'].isnull()
sns.boxplot(x='is_missing', y='spending', data=df)
plt.title("Is spending different when income is missing?")
plt.show()

If the boxplots look different, your data is likely not MCAR. If spending is higher when income is missing, you’ve got a pattern you need to account for.

3. When to Just Delete Rows (And When It’s a Trap)

Deleting rows (listwise deletion) is the most honest approach. You aren’t making anything up. But it’s only safe if your data is MCAR.

If you delete rows under MAR or MNAR, you introduce bias. You’ve systematically removed a specific group of people from your study. You also lose “power.” Smaller datasets mean wider confidence intervals, which makes it harder to prove your results are real.

Rule of thumb: If less than 5% of your data is missing and it looks like pure noise, deleting is usually fine.

4. Imputation: The Swiss Army Knife (And Its Limitations)

Imputation fills missing values so your models can actually train.

• Mean/Median: Fast. But it collapses variance, assuming every gap sits at the average.
• KNN Imputation: Looks at similar rows — the “neighbors” — to estimate a value. Smarter than the mean by a good margin.
• Multiple Imputation (MICE): Rather than one guess, it makes several — five or ten, typically. I’d call this the gold standard: it at least admits the true value is uncertain.

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
df_knn = pd.DataFrame(imputer.fit_transform(df[['income', 'spending']]), columns=['income', 'spending'])
print(f"KNN Imputed Mean: {df_knn['income'].mean():.2f}")

5. The Imputation Illusion: Why More Data Doesn’t Always Mean Better Data

Here’s the catch: once you impute, your software treats those filled-in values as fact. Fill 20% of your dataset with the mean and your standard errors shrink. The results look tighter. But that confidence isn’t earned — you’re adding points you never measured and treating the inflated sample as if it were real.

6. Alternatives: When to Use Indicators or Keep the Missingness

Sometimes missing data is the signal.

• Missing Indicator: Rather than only imputing, add a column is_income_missing. The model can then pick up that people who don’t report income behave differently.
• Native Handling: XGBoost and LightGBM handle NaN internally. At each split, they route missing values whichever direction reduces error. That often beats manual imputation.

7. A Decision Tree: How to Choose Your Strategy

How do you choose? Three questions usually settle it:

1. Is it MCAR? If yes, and missingness is under 5%, Delete.
2. Is it MAR? If yes, go with KNN Imputation or Multiple Imputation.
3. Is it MNAR? If yes, you’ll need Missing Indicators or domain knowledge. Plain imputation will likely fail you here.

8. Real Example: E-Commerce Order Data

Consider a real case. An e-commerce site is missing 5% of its shipping addresses, and most of those gaps trace to mobile users.

• Deletion: We lose 5% of the data — the mobile users — so the analysis over-represents desktop.
• Mean Imputation: You can’t mean-impute a street address, so this doesn’t apply.
• Indicator: We flag these as “Mobile_Missing.” The model picks up that mobile users have a different conversion rate, and that signal is what we want.

9. Implementation Checklist

1. Count: Start with df.isnull().sum() — it tells you the scale at a glance.
2. Visualize: The missingno library reveals whether gaps cluster or scatter randomly.
3. Diagnose: Check whether missingness tracks with other variables in the dataset.
4. Decide: Walk through the decision tree above.
5. Validate: Run your model both ways, with and without the missing rows. If results shift drastically, your imputation is probably biasing things.

10. What’s Next?

Missing data in static tables is manageable enough. But what about time series, or claims that X causes Y? Next we’ll look at how missing values can derail causal inference, and how DoWhy keeps the analysis sound.

Explore more: Run a KNN imputer on your current project and compare the variance against a simple mean fill. The difference in your data’s “shape” is often striking.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are MCAR, MAR, and MNAR, and which one describes the article’s high-earners example?

Understand In your own words, explain why the article’s mean-imputation experiment left the regression slope almost unchanged but tanked the R² score — what’s the difference between a coefficient being biased and a model’s fit being degraded?

Apply Using the article’s decision tree (MCAR + <5% missing → Delete; MAR → KNN/Multiple Imputation; MNAR → Indicators/domain knowledge), what would you do with a column that’s 3% missing, confirmed MCAR via a boxplot check?

Analyze The article’s e-commerce example finds that missing shipping addresses correlate with mobile users, and recommends a “Mobile_Missing” indicator over deletion or mean imputation. Walk through why deleting those rows specifically biases the analysis toward desktop users, in a way that a random 5% deletion wouldn’t.

Evaluate The article says “if you delete rows under MAR or MNAR, you introduce bias” and calls this the hidden cost of losing statistical power. Critique the reverse risk: can Multiple Imputation (MICE), the article’s “gold standard,” also introduce its own kind of bias if the imputation model itself is misspecified? What would that look like?

Create Design a missingness diagnosis and strategy for a new dataset: a churn-prediction dataset where last_login_date is missing for 15% of users. Propose a specific check (like the article’s boxplot-against-outcome test) to determine whether this looks like MCAR, MAR, or MNAR, and pick a strategy from the article’s toolkit based on what you’d expect to find.

---