Reference: Preprocessing

Preprocessing is everything that happens between “raw data arrived” and “the model saw a feature matrix.” Most of it is unglamorous, and most of it is where silent bugs live. This reference covers the three preprocessing stages that bite hardest: missing-data imputation, row/column hygiene (deduplication, type coercion, dtype alignment), and the leakage traps that appear the moment you run any of the above on the wrong side of a train/test split.

Roster

Method / step	One-line definition	Applies to	When to use	When to avoid	Used in
Mean imputation	Replace missing numeric with the column mean	Numeric, continuous	MCAR, small gap fraction, baseline sanity check	Skewed data, MAR with informative missingness	how-to-properly-handle-missing-data-is-imputation)
Median imputation	Replace missing numeric with the column median	Numeric, continuous	Skewed numeric, robust to outliers	Categorical, MNAR where the missing value itself carries signal	how-to-properly-handle-missing-data-is-imputation)
Mode imputation	Replace missing categorical with the most frequent category	Categorical, low-cardinality	Small fraction missing, one dominant category	High-cardinality categoricals; creates a spurious “popular” mode	how-to-properly-handle-missing-data-is-imputation)
KNN imputation	Fill each missing value from its k nearest rows (by observed features)	Numeric or encoded categorical	MAR, when rows cluster in feature space	Very large datasets, high dimensionality, MCAR with no structure	how-to-properly-handle-missing-data-is-imputation)
MICE	Iteratively regress each incomplete column on the others; draw from the posterior	Numeric + categorical	MAR, multivariate structure, downstream model cares about uncertainty	Single missing values, time-critical inference, MNAR	how-to-properly-handle-missing-data-is-imputation)
Indicator column	Add a binary was_missing column alongside the imputed value	Any	When the fact of missingness is itself predictive (MNAR or strong MAR)	MCAR (adds noise, no signal)	how-to-properly-handle-missing-data-is-imputation)
Deduplication	Remove exact or near-exact duplicate rows	Rows	After ingestion, before any aggregation	When duplicates are legitimate (repeated events)	—
Type coercion	Convert a column from object/string to int/float/category/datetime	Any column	After dtype audit, before modeling	When the conversion loses information (e.g., forcing IDs to int)	—
Dtype alignment	Ensure the same logical column has the same dtype across train/serve	Any pipeline	After schema is fixed, before fitting	When upstream schema is genuinely different	building-an-ml-pipeline-that-avoids-train-serving)
Fit-on-train-only preprocessing	Compute any learned statistic (mean, scaler, KNN graph) on train only	Any learned step	Always, for any learned transform	Never — see leakage section	data-leakage-why-your-perfect-model-is-probably-ly)

Which imputation for which missingness pattern

The single most useful question to ask before choosing an imputation strategy is: why is the value missing? That answer falls into one of three regimes (Rubin, 1976), and the choice of imputer is downstream of which regime you are in.

1. MCAR (Missing Completely At Random) — the probability that a value is missing does not depend on the value itself or on any other observed variable. Example: a sensor glitch that drops one row’s reading at random. Safe to use any imputation; mean/median/mode are fine because there is no selection bias to worry about.
2. MAR (Missing At Random) — the probability of missingness depends on observed values but not on the unobserved value itself. Example: men in a survey are less likely to report their weight; whether weight is missing depends on the observed sex. Use KNN or MICE — the observed features can recover the structure that the simple imputers throw away.
3. MNAR (Missing Not At Random) — the probability of missingness depends on the unobserved value. Example: people with very low income refuse to report it. No purely statistical imputer can fix this; you need a missingness indicator, domain knowledge, and often a sensitivity analysis. Mean/median/KNN/MICE will all be biased the same way.

Decision tree:

• Are >40% of the values in a column missing?
  • Yes → Drop the column unless the fact of missingness is predictive; in that case keep only the indicator and drop the imputed column.
  • No → continue.
• Is the missingness MCAR?
  • Yes, and the column is numeric → mean (if symmetric) or median (if skewed / outlier-prone).
  • Yes, and the column is categorical → mode (if low-cardinality) or “missing” as its own category.
• Is the missingness MAR?
  • Numeric, few features, ≤ ~10k rows → KNN imputation (scale first).
  • Numeric + categorical mixed, or > 10k rows, or you need multiple imputations for uncertainty → MICE.
  • Strong prior that the fact of missingness is itself predictive → also add a missingness indicator column.
• Is the missingness MNAR?
  • All statistical imputers are biased. Add a missingness indicator, document the assumption, run a sensitivity analysis (impute low vs high and see if predictions flip). Prefer a model class that tolerates missing values natively (e.g., gradient boosting with native NaN handling) so the model can learn the missingness signal directly.
• Is the column a leakage candidate (e.g., computed from the target)?
  • Re-examine the data dictionary. Drop or rewrite. See the data leakage article).

Let $Y$ be the variable of interest, $R \in \{0, 1\}$ the missingness indicator ($R = 1$ means $Y$ is observed), and $X$ the other observed variables.

$\text{MCAR: } P(R = 1 \mid Y, X) = P(R = 1)$

$\text{MAR: } P(R = 1 \mid Y, X) = P(R = 1 \mid X)$

$\text{MNAR: } P(R = 1 \mid Y, X) \text{ depends on } Y \text{ (even given } X\text{)}$

Under MCAR, the complete-case estimator $\hat\theta_{\text{CC}}$ is unbiased: $\mathbb{E}[\hat\theta_{\text{CC}}] = \theta$. Under MAR it is in general biased, but the imputed-data estimator $\hat\theta_{\text{IMP}}$ with a model $\mathbb{E}[Y \mid X]$ is unbiased if the imputation model is correctly specified. Under MNAR no purely statistical imputer suffices.

Plain English	Statistical symbol	Python equivalent
Missingness indicator	$R$	df['col'].isna().astype(int)
Complete-case estimator	$\hat\theta_{\text{CC}}$	df.dropna().mean()
Imputed value	$\hat y_i = \mathbb{E}[Y \mid X = x_i]$	imputer.transform(X)
Imputation model variance ignored by single imputation	$\mathrm{Var}(\hat\theta_{\text{IMP}}) - \mathrm{Var}_{\text{between}}$	MICE’s m draws

Worked example: comparing imputers on one messy dataset

We construct a small, deliberately messy dataset so the comparison is visible. In real life you would not know the ground truth — here we do, so we can measure imputation error directly.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import IterativeImputer
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(7)

# Ground truth: age ~ income (positively), with some noise
n = 400
age = rng.normal(45, 12, n).clip(18, 90)
income = 30000 + 900 * age + rng.normal(0, 8000, n)
sex = rng.choice(["M", "F"], n, p=[0.5, 0.5])

df = pd.DataFrame({"age": age, "income": income, "sex": sex})

# Inject MAR missingness on income: men report income less often
p_missing = np.where(df["sex"] == "M", 0.35, 0.05)
mask = rng.random(n) < p_missing
df["income_true"] = df["income"]
df.loc[mask, "income"] = np.nan

print(df.isna().mean().round(3))
# age            0.000
# income         0.192
# sex            0.000
# income_true    0.000

# Split FIRST — leakage-safe imputation happens after this line
df_train, df_test = train_test_split(df, test_size=0.3, random_state=7)

results = {}

# Strategy 1: median (robust to skew, ignores sex)
m = SimpleImputer(strategy="median")
m.fit(df_train[["income"]])  # train-only fit
pred = m.transform(df_test[["income"]])
results["median"] = mean_squared_error(df_test["income_true"], pred)

# Strategy 2: median per group (uses sex, a crude MAR correction)
# Only fill the rows that are actually missing, via .fillna() — mirroring how
# SimpleImputer/KNNImputer/IterativeImputer's .transform() leaves already-observed
# values untouched. A version that blanket-replaces every row (missing or not)
# throws away the ~80% of test rows where the observed value was already correct,
# and badly understates this strategy.
group_medians = df_train.groupby("sex")["income"].median()
pred_grouped = df_test["income"].fillna(df_test["sex"].map(group_medians)).values.reshape(-1, 1)
results["median_by_sex"] = mean_squared_error(df_test["income_true"], pred_grouped)

# Strategy 3: KNN imputer (uses age + sex one-hot + income)
X_train = pd.get_dummies(df_train[["age", "income", "sex"]], drop_first=True)
X_test  = pd.get_dummies(df_test[["age", "income", "sex"]],  drop_first=True)
X_test = X_test.reindex(columns=X_train.columns, fill_value=0)  # align columns
knn = KNNImputer(n_neighbors=5, weights="distance")
knn.fit(X_train)
X_test_imp = knn.transform(X_test)
income_idx = list(X_train.columns).index("income")
results["knn"] = mean_squared_error(df_test["income_true"], X_test_imp[:, income_idx])

# Strategy 4: MICE / IterativeImputer
mice = IterativeImputer(random_state=7, max_iter=20)
mice.fit(X_train)
X_test_mice = mice.transform(X_test)
results["mice"] = mean_squared_error(df_test["income_true"], X_test_mice[:, income_idx])

print(pd.Series(results).round(0).sort_values())
# mice             14629219.0
# knn              17989668.0
# median_by_sex    22326633.0
# median           22558732.0

The exact numbers will shift with your random seed and package versions, but the ordering is the lesson — and it’s a sharper one than “condition on whatever predicts missingness.” median_by_sex only narrowly beats plain median (22.3M vs. 22.6M) even though sex is, by construction, exactly the variable driving missingness here (this dataset is MAR in sex). That’s because sex predicts whether income is missing, not what income is — in this dataset income is a function of age, not sex, so grouping the imputer by sex buys almost nothing. KNN and MICE win by a wide margin (14.6M and 18.0M) precisely because they condition on age, the feature that actually correlates with the value being imputed. The takeaway: matching your imputer to the missingness mechanism (MAR → condition on something) is necessary but not sufficient — you also need to condition on variables that predict the value, not just the presence, of the data.

Lines 1–9: imports. enable_iterative_imputer is a feature-gate import — IterativeImputer is still “experimental” in scikit-learn and has to be explicitly unlocked before the sklearn.impute.IterativeImputer import will resolve.

Lines 12–19: synthetic data construction. age is clipped to a realistic 18–90 range; income is a linear function of age plus Gaussian noise. This gives us ground truth we can later score against.

Lines 22–25: MAR missingness injection. Men have a 35% chance of missing income, women 5%. This is exactly the MAR case: missingness depends on the observed variable sex, not on the unobserved income itself.

Lines 31–36: the single most important line in the script is train_test_split before any imputation. Everything that follows — the median, the group medians, the KNN graph, the MICE regressors — is fit on df_train only. If you fit on the whole dataframe and then split, you have leaked test information into the imputer (see the leakage section below).

Lines 39–42 (median): SimpleImputer(strategy="median") learns one number — the median of income on the training set — and broadcasts it to every missing row. It ignores age and sex, which is why it narrowly loses to the grouped approach below, and loses badly to KNN/MICE, which use age.

Lines 45–52 (median by sex): a hand-rolled grouped imputer. The pattern df_test["sex"].map(group_medians) is the simplest possible MAR-aware imputer: it conditions on one observed variable. Two details matter here: the group medians are computed on df_train only, and pred_grouped is built with .fillna(...) so it only overwrites rows that are actually missing in df_test["income"] — matching how SimpleImputer/KNNImputer/IterativeImputer behave (their .transform() leaves observed values untouched). Scoring a version that blanket-replaces every row regardless of missingness makes this strategy look far worse than it is.

Lines 55–62 (KNN): one-hot encoding, then KNNImputer. The .reindex(columns=X_train.columns, fill_value=0) step is critical — if a category only appears in train but not in test (or vice versa), pd.get_dummies will produce mismatched columns and the imputer will crash or silently misalign features. n_neighbors=5 and weights="distance" are reasonable defaults; on larger datasets you’d want to scale the numeric columns first, since KNN is distance-based and unscaled income (range ~30k–80k) would dominate age (range 18–90).

Lines 65–69 (MICE): IterativeImputer is scikit-learn’s MICE. It regresses each incomplete column on all the others, iterates until max_iter or convergence. random_state=7 makes the draws reproducible. For multiple imputation (drawing m imputed datasets and combining the estimates with Rubin’s rules), you would call IterativeImputer(sample_posterior=True) m times — see van Buuren for the full procedure.

Final print: the comparison. We score against income_true, which we preserved at the top. In a real pipeline you don’t have this — the metric here is a diagnostic to pick an imputer, not a production metric.

Deduplication, type coercion, dtype alignment

These three are the boring preprocessing steps that don’t have elegant statistical theory but cause most production bugs.

Deduplication

Run df.duplicated().sum() after every ingestion. Two failure modes:

• The upstream producer replays the same row with a new timestamp, so you have logical duplicates that exact-match deduplication will miss.
• Genuinely different events happen to share all observed fields.

Define a stable primary key (a tuple of columns that should be unique per logical event) and dedup on that, not on the whole row.

Type coercion

After pd.read_csv, check df.dtypes. Anything stored as object is a Python string masquerading as a column. Common fixes:

• pd.to_numeric(df['x'], errors='coerce') for numerics trapped in strings.
• pd.to_datetime(...) for timestamps.
• astype('category') for low-cardinality strings (this shrinks memory and unlocks faster groupby).

Coerce with errors-as-NaN so you can audit the rows that fail; do not silently cast.

Dtype alignment

This is the train-serving analog of leakage. If your training pipeline reads amount as float64 but the serving pipeline reads it as int64 (because the upstream schema changed or the column was cast somewhere mid-pipeline), the model will silently receive a different feature distribution at serve time. The fix is a schema object — pandera, pydantic, or even just an explicit dtype dict — that both pipelines validate against. See the train-serving skew article) for the full treatment.

Leakage: the canonical preprocessing-before-split mistake

This is the single most common reason a model that scores 0.97 on a notebook benchmark deploys at 0.62 in production.

The mistake. You receive a dataset, you run df.fillna(df.mean()) on the whole dataframe, then you split into train/test. The test-set means are now in the imputed values. The test set is no longer a held-out sample from the same distribution — it’s a sample whose missing values have been replaced with statistics that include themselves.

The canonical counter-example:

# WRONG — leakage
from sklearn.linear_model import LogisticRegression

# Toy classification dataset for this example (separate from the imputation
# walkthrough above — that one had no target column).
df = pd.DataFrame({
    "age": rng.normal(45, 12, 400),
    "income": rng.normal(60000, 15000, 400),
    "sex": rng.choice(["M", "F"], 400),
    "target": rng.integers(0, 2, 400),
})
df.loc[rng.random(400) < 0.15, "income"] = np.nan
model = LogisticRegression()

# numeric_only=True avoids a TypeError on the non-numeric "sex" column — pandas
# >= 2.0 no longer silently drops non-numeric columns from .mean(), it raises.
# The leak is still here: ALL rows, including the future test rows, are used
# to compute the fill values.
df_imputed = df.fillna(df.mean(numeric_only=True))
X = df_imputed.drop(columns=["target", "sex"])
y = df_imputed["target"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3)
model.fit(X_tr, y_tr)
# model.score(X_te, y_te)  # inflated — the test set was peeked at during imputation

# RIGHT — imputation is part of the fitted pipeline
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

X = df.drop(columns=["target", "sex"])
y = df["target"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=7)

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("model", LogisticRegression()),
])
pipe.fit(X_tr, y_tr)        # imputer fits on X_tr only
pipe.score(X_te, y_te)      # imputer transforms X_te using X_tr stats

The second version is correct and is what you deploy: the pipeline is a single object that carries its fitted state from training to serving. There is no opportunity to forget to refit the imputer, because the imputer lives inside the pipeline that gets serialized.

The same leakage pattern appears for other learned steps, and the fix is the same in every case:

• Scaling — StandardScaler.fit on all rows.
• Encoding — TargetEncoder.fit on all rows, which leaks the target and is worse than leaking a feature.
• Feature selection — SelectKBest.fit on all rows, which leaks both features and target.
• Outlier clipping — computing the 1st/99th percentiles on all rows.

The rule is mechanical: if the step learns anything from the data, it is fitted on the training set only, and the same fitted object is applied to the test set and to future serving data.

For the production-side version of the same failure (the serving pipeline computes the mean differently from the training pipeline, or the training pipeline used a df.mean() that included the test fold, or the schema drifted between fit and serve), see the data leakage article) and the train-serving skew article).

Edge cases and common mistakes

• Imputing categoricals with mean. SimpleImputer(strategy="mean") on a string column raises. strategy="median" on a high-cardinality categorical is meaningless. Use strategy="most_frequent" or treat missing as its own category "__missing__".
• KNN imputation without scaling. KNN is distance-based; a column with range 0–100,000 dominates a column with range 0–1. Always scale numerics before KNNImputer, ideally inside a Pipeline so the scaler is also train-only.
• MICE with sample_posterior=False. Single imputation understates uncertainty: every missing value gets one deterministic draw, so downstream confidence intervals are too narrow. For inference (vs. prediction) use sample_posterior=True and run m imputations, then combine with Rubin’s rules.
• Forgetting that pd.get_dummies is stateful in shape. Train produces N dummy columns; test produces M. If a category is missing in test, you get a shape mismatch at predict time. Always reindex(columns=..., fill_value=0) or, better, use sklearn.preprocessing.OneHotEncoder(handle_unknown="ignore") which is designed for this.
• Mean/median on a column with >50% missing. You’re now imputing more than half the column from a minority of real values. The column is mostly the imputer’s output, not data. Drop it, or keep only the missingness indicator.
• Imputing before computing derived features. If ratio = a / b and b is missing, you must impute b before computing ratio, not after. Better: compute the ratio in a way that propagates NaN and then impute the ratio column.
• Deduplicating by drop_duplicates() on the whole row. Two legitimate rows that happen to share all columns get dropped. Dedup on a primary key.
• astype('category') on a column with new categories at serve time. pd.CategoricalDtype(categories=[...]) is the fix; otherwise new categories become NaN and you’ve silently introduced missingness at serve time that wasn’t in training.
• Fitting the imputer on the whole dataset. See the leakage section. This is the #1 cause of inflated cross-validation scores.
• Imputing with a value that doesn’t exist in the domain. Age imputed to 0 when the minimum is 18; income imputed to -3 because of an outlier-pulled mean. Clip imputed values to the column’s plausible range.
• Assuming the missingness pattern is the same in production. A model trained on data where 5% of income was missing will behave strangely if at serve time 30% is missing because the upstream data feed broke. Monitor the missingness rate at serve time, not just the predictions.

Cross-references

• Data Leakage: Why Your Perfect Model Is Probably Lying) — the canonical article on the leakage pattern this reference’s leakage section points to.
• How To Properly Handle Missing Data: Is Imputation?) — the narrative tutorial on imputation that this reference supports.
• Building an ML Pipeline That Avoids Train-Serving Skew) — the production-side version of the preprocessing-leakage story: dtype drift, schema mismatch, and the same fit-on-all-then-split mistake resurfacing as a deploy-time bug.

Further reading

• Stef van Buuren, Flexible Imputation of Missing Data (2nd ed., 2018, CRC Press). The canonical reference for MICE and multiple imputation; the companion website has the full text free and R code for every example.
• Roderick J. A. Little & Donald B. Rubin, Statistical Analysis with Missing Data (3rd ed., 2019, Wiley). The original treatment of MCAR/MAR/MNAR and the framework every later method builds on. Dense, theory-heavy.
• scikit-learn user guide, section 6.4: Imputation of missing values — covers SimpleImputer, KNNImputer, IterativeImputer, and the experimental MissingIndicator.
• pandas user guide, Working with missing data — the API reference for isna, fillna, interpolate, and the dtype behavior of NaN across numeric vs. object columns.