Feature Engineering Techniques Reference A Checkli

You’ve tried XGBoost. You’ve tuned LightGBM until your laptop fan sounded like a jet engine. You even dusted off a neural network, hoping deep magic would save you. And yet — your validation score won’t budge. That stubborn metric sits there, mocking you, while you cycle through every algorithm in scikit-learn like a chef who keeps switching pans but never changes the ingredients.

Here’s the dirty secret the tutorials don’t always tell you: the raw data you’re feeding into your model is probably in a shape your model can’t digest. Kuhn and Johnson put it perfectly in their work on predictive modeling — relevant predictors are often represented in ways that models struggle with. Think of it this way: you’re handing a master chef a pile of unwashed, whole vegetables and expecting a Michelin-star meal. The information is all there, but it’s not in a form the chef can work with.

Feature engineering is translation. You’re not changing the information — you’re speaking the model’s language. And after you’ve done it right, even a simple linear model can suddenly see patterns that were invisible before.

Let me prove it. Here’s a tiny taste of what we’re talking about:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score

# Generate a toy dataset with mixed types and different scales
np.random.seed(42)
n_samples = 500

# Numerical feature on a huge scale (dollars)
income = np.random.lognormal(mean=10, sigma=1, size=n_samples)
# Categorical feature (education level)
education = np.random.choice(['high school', 'bachelors', 'masters', 'phd'], size=n_samples)
# Target: people with high income OR advanced degrees are more likely to buy
log_odds = (np.log(income) - 10) * 0.5
log_odds[education == 'masters'] += 1.0
log_odds[education == 'phd'] += 2.0
prob = 1 / (1 + np.exp(-log_odds))
y = (np.random.random(n_samples) < prob).astype(int)

# Put into a DataFrame
import pandas as pd
df = pd.DataFrame({'income': income, 'education': education})

# Split first — this rule will echo through the whole article
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.3, random_state=42)

# Model 1: Raw data, no preprocessing — the model sees income in raw dollars and education as strings
# (this will crash, so we'll just use income for the raw model)
model_raw = LogisticRegression(max_iter=1000)
model_raw.fit(X_train[['income']], y_train)
y_pred_raw = model_raw.predict(X_test[['income']])
score_raw = accuracy_score(y_test, y_pred_raw)

# Model 2: Scaled and one-hot encoded
preprocessor = ColumnTransformer([
    ('num', StandardScaler(), ['income']),
    ('cat', OneHotEncoder(drop='first'), ['education'])
])
pipeline = Pipeline([('preprocess', preprocessor), ('model', LogisticRegression(max_iter=1000))])
pipeline.fit(X_train, y_train)
y_pred_processed = pipeline.predict(X_test)
score_processed = accuracy_score(y_test, y_pred_processed)

print(f"Raw model accuracy (income only, unscaled): {score_raw:.3f}")
print(f"Processed model accuracy (scaled + encoded): {score_processed:.3f}")
print(f"Improvement: +{(score_processed - score_raw) * 100:.1f} percentage points")

That jump isn’t from a fancier algorithm. It’s the same logistic regression under the hood. The only difference is that we cleaned the vegetables before we handed them over.

This article is a checklist — seven gates your dataset should pass through before it ever touches model.fit(). We’ll walk through each one: why it matters, when to use it, what happens if you skip it, and exactly how to code it. By the end, you’ll have a mental (and literal, printable) checklist that catches the silent killers of model performance before they strike.

But before we even think about scaling or encoding, there’s a gate we have to pass through first. A gate that, if you miss it, makes everything else a lie.

Gate Zero: The Leakage Audit (Do This Before Anything Else)

Imagine you’re studying for a final exam. The night before, someone hands you the answer key. You memorize it, ace the test, and feel brilliant. But when you get to the real world and someone asks you to solve a problem that wasn’t on that answer key, you’re lost. You never actually learned anything — you just memorized the answers.

That’s data leakage. Your model saw information during training that won’t exist when it’s making real predictions in production. It memorized the answer key instead of learning the patterns. Your validation score looks amazing. Your production model is useless.

This isn’t a theoretical edge case. It’s the most common catastrophic failure in machine learning, and it’s heartbreakingly easy to do by accident. Let me show you exactly how it happens.

The Noise-Data Demo: Finding Patterns in Pure Randomness

We’ll generate a dataset that is, by construction, pure noise. Every feature is random. The target is random. There is absolutely no relationship to find. A model that’s not cheating should get an AUC around 0.5 — no better than a coin flip.

Then we’ll make one mistake: we’ll select features using the entire dataset before splitting into train and test. Watch what happens.

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

# Generate 100 samples, 1000 features — all pure random noise
np.random.seed(42)
n_samples, n_features = 100, 1000
X_noise = np.random.randn(n_samples, n_features)
y_noise = np.random.randint(0, 2, n_samples)

# ============================================
# THE WRONG WAY: Feature selection on the FULL dataset before splitting
# ============================================
# Select the 10 "best" features using the entire dataset
selector_wrong = SelectKBest(score_func=f_classif, k=10)
X_selected_wrong = selector_wrong.fit_transform(X_noise, y_noise)

# Now split — but the selector already saw the test data through the target!
X_train_wrong, X_test_wrong, y_train_wrong, y_test_wrong = train_test_split(
    X_selected_wrong, y_noise, test_size=0.3, random_state=42
)

model_wrong = RandomForestClassifier(random_state=42)
model_wrong.fit(X_train_wrong, y_train_wrong)
y_pred_wrong = model_wrong.predict_proba(X_test_wrong)[:, 1]
auc_wrong = roc_auc_score(y_test_wrong, y_pred_wrong)

# ============================================
# THE RIGHT WAY: Split first, then select features ONLY on training data
# ============================================
X_train_right, X_test_right, y_train_right, y_test_right = train_test_split(
    X_noise, y_noise, test_size=0.3, random_state=42
)

selector_right = SelectKBest(score_func=f_classif, k=10)
X_train_right_selected = selector_right.fit_transform(X_train_right, y_train_right)
X_test_right_selected = selector_right.transform(X_test_right)  # transform only, no fit!

model_right = RandomForestClassifier(random_state=42)
model_right.fit(X_train_right_selected, y_train_right)
y_pred_right = model_right.predict_proba(X_test_right_selected)[:, 1]
auc_right = roc_auc_score(y_test_right, y_pred_right)

print(f"WRONG way AUC (feature selection before split): {auc_wrong:.3f}")
print(f"RIGHT way AUC (split first):                  {auc_right:.3f}")
print(f"Expected AUC on pure noise:                  ~0.500")

Look at that wrong-way AUC. On data that is literally random numbers, the model is reporting a score far above 0.5. It found patterns that don’t exist. How? Because when we ran SelectKBest on the full dataset, it peeked at the relationship between every feature and the target across all rows — including the ones that would later become the test set. It selected the 10 features that, by pure random chance, happened to correlate slightly with the target in this particular sample. Then when we tested on those same rows, the model looked brilliant. But it was just memorizing noise.

The right-way AUC hovers around 0.5. The model correctly reports that it learned nothing, because there was nothing to learn.

The Cardinal Rule

This is the rule that echoes through every gate in this checklist:

Split first. Fit every transformer only on the training set. Transform the test set using the fitted transformer — never fit on test data.

Every fit() call is a potential leak. Every time you compute a statistic from data — a mean, a standard deviation, a target encoding, a feature importance score — you must ask: “Did I compute this using only the training data?” If the answer involves the test set in any way, you’ve leaked.

We’ll see specific, sharp-edged versions of this rule throughout the article. Target encoding without cross-fitting (Gate Three) is the most notorious offender. Feature selection before cross-validation (Gate Five) is another. And temporal leakage (Gate Six) is a whole different beast — it’s not about methodology, it’s about whether your feature even exists at prediction time.

But for now, internalize this: the leakage audit is Gate Zero. It comes before everything else. If you skip it, every score you see from here on is suspect.

Gate One: Missing Data — Impute, But Don’t Erase the Signal

Real-world data has holes in it. Sensors fail. Customers skip survey questions. Databases have nulls. Your first instinct might be to drop rows with missing values or fill them with the mean and move on. Both of those instincts are wrong — or at least, they’re throwing away information.

Here’s the counterintuitive truth: missingness is information. The fact that a customer didn’t fill in their income is itself a signal. Maybe they’re privacy-conscious. Maybe they’re unemployed and embarrassed. Maybe they’re in a hurry. Whatever the reason, customers with missing income often behave differently from customers who reported it — and if you blindly fill in the mean, you erase that signal.

Three Imputation Strategies (and When to Use Each)

scikit-learn gives us three main tools, and they solve different problems:

SimpleImputer is your baseline. It replaces missing values with a constant — the mean, median, most frequent value, or a fixed number. It’s fast, it’s simple, and it’s the right starting point. Use it when you need a quick, interpretable fix and you’re reasonably confident the data is missing at random.

IterativeImputer is more sophisticated. It treats each feature with missing values as a target and uses all the other features to predict what should be there. It does this in rounds, refining its estimates each time. Use it when your features have relationships with each other — for example, if income and education level are correlated, knowing someone’s education helps you make a smarter guess about their missing income.

KNNImputer uses the k-nearest neighbors approach. For each row with a missing value, it finds the k most similar rows (based on the features that are present) and fills in the gap with the average of those neighbors. Use it when local structure matters — when similar observations should have similar values.

The Missingness Indicator: Don’t Throw Away the Signal

Here’s the move that separates novices from practitioners: add_indicator=True. This parameter (available on SimpleImputer and IterativeImputer) stacks a new binary column next to each imputed feature. A 1 in that column means “this row’s value was missing and got filled in.” A 0 means “this value was real.”

Why does this matter? Because now your model can learn different patterns for imputed vs. real values. If customers with missing income really do behave differently, the model can use that indicator column to adjust its predictions. You’ve preserved the signal of missingness instead of drowning it in the mean.

Let’s see all of this in action:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer, IterativeImputer, KNNImputer
from sklearn.experimental import enable_iterative_imputer  # noqa
import matplotlib.pyplot as plt

# Create a dataset with a real relationship and then punch holes in it
np.random.seed(42)
n = 200
x = np.random.uniform(0, 10, n)
y_true = 2 * x + np.random.normal(0, 1.5, n)

# Punch ~30% holes in x, completely at random
mask = np.random.random(n) < 0.3
x_with_holes = x.copy()
x_with_holes[mask] = np.nan

# Put into a DataFrame
import pandas as pd
df = pd.DataFrame({'x': x_with_holes, 'y': y_true})

# Split first — the rule from Gate Zero applies here too
X_train, X_test, y_train, y_test = train_test_split(df[['x']], df['y'], test_size=0.3, random_state=42)

# --- Strategy 1: SimpleImputer (mean) with missingness indicator ---
simple_imp = SimpleImputer(strategy='mean', add_indicator=True)
X_train_simple = simple_imp.fit_transform(X_train)
X_test_simple = simple_imp.transform(X_test)

print("SimpleImputer with add_indicator=True:")
print(f"  Original columns: {X_train.shape[1]}")
print(f"  Output columns:   {X_train_simple.shape[1]} (one for imputed x, one for the indicator)")
print(f"  First 5 rows of training output:\n{X_train_simple[:5]}")
print()

# --- Strategy 2: IterativeImputer ---
iter_imp = IterativeImputer(random_state=42, add_indicator=True)
X_train_iter = iter_imp.fit_transform(X_train)
X_test_iter = iter_imp.transform(X_test)

print("IterativeImputer with add_indicator=True:")
print(f"  Output columns: {X_train_iter.shape[1]}")
print(f"  First 5 rows of training output:\n{X_train_iter[:5]}")
print()

# --- Strategy 3: KNNImputer ---
knn_imp = KNNImputer(n_neighbors=5)
X_train_knn = knn_imp.fit_transform(X_train)
X_test_knn = knn_imp.transform(X_test)

print("KNNImputer:")
print(f"  Output columns: {X_train_knn.shape[1]} (no indicator available for KNNImputer)")
print(f"  First 5 rows of training output:\n{X_train_knn[:5]}")
print()

# --- Visualize the distortion from mean imputation ---
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# Before imputation: plot the complete original data (before we punched holes)
axes[0].scatter(x, y_true, alpha=0.6, edgecolors='k', linewidth=0.5)
axes[0].set_title('Original Data (No Missing Values)')
axes[0].set_xlabel('x')
axes[0].set_ylabel('y')

# After mean imputation: reconstruct what the imputed dataset looks like
x_imputed = x_with_holes.copy()
x_imputed[mask] = np.nanmean(x_with_holes)  # fill holes with the mean
axes[1].scatter(x_imputed, y_true, alpha=0.6, edgecolors='k', linewidth=0.5)
# Highlight the imputed points
axes[1].scatter(x_imputed[mask], y_true[mask], color='red', alpha=0.8, 
                edgecolors='k', linewidth=0.5, label='Imputed (mean)')
axes[1].set_title('After Mean Imputation\n(Red points are imputed — notice the vertical stripe)')
axes[1].set_xlabel('x (imputed)')
axes[1].set_ylabel('y')
axes[1].legend()

plt.tight_layout()
plt.show()

Look at that second scatter plot. See the vertical stripe of red points? That’s the distortion. Every missing x value got replaced with the exact same number — the mean. So all those points now sit on the same vertical line, regardless of what their y value is. The real relationship between x and y was a diagonal smear; after mean imputation, you’ve got an artificial stripe cutting through it. You’ve changed the shape of your data.

This is the hardest part of imputation: there is no perfect fix. Every imputation method makes assumptions about why the data is missing and what the true values might be. The best you can do is (a) preserve the signal of missingness with an indicator column, (b) choose an imputation strategy that matches your data’s structure, and (c) always, always fit the imputer on the training set only.

Gate Two: Scaling — When Your Model Cares About Size (and When It Doesn’t)

Imagine you’re comparing two people. One is 180 centimeters tall. The other has an annual income of 180,000 dollars. If you just look at the raw numbers — 180 vs. 180,000 — the income number is a thousand times bigger. Does that mean income is a thousand times more important than height? Of course not. But that’s exactly how many machine learning models see the world.

Distance-based models — SVMs, k-nearest neighbors, PCA — work by measuring how far apart data points are in feature space. If one feature is measured in dollars (range 0–100,000) and another in years (range 0–100), the dollar feature dominates every distance calculation. The model isn’t making a judgment about what’s important; it’s just doing math, and the math is overwhelmed by the scale difference.

Gradient-based models — neural networks, regularized linear/logistic regression — have a related problem. When features are on different scales, the loss function forms steep, narrow valleys. The optimizer bounces around inefficiently, and regularization penalizes coefficients unevenly. scikit-learn’s preprocessing guide puts it plainly: models with RBF kernel SVMs and regularized linear models assume features are centered around zero with comparable variance.

Which Models Need Scaling (and Which Genuinely Don’t Care)

Need scaling: SVMs (especially with RBF kernel), k-NN, neural networks, PCA, regularized linear/logistic regression (Ridge, Lasso, ElasticNet).

Don’t need scaling: Tree-based models — decision trees, random forests, gradient boosting. Why? Because trees make splits based on order, not magnitude. A decision tree asks “is income > 50,000?” not “how far is this income from that one?” Multiplying all values by 1000 doesn’t change the order, so the tree makes the same splits. The tree genuinely doesn’t care.

StandardScaler vs. MinMaxScaler vs. RobustScaler

StandardScaler subtracts the mean and divides by the standard deviation. The result has mean 0 and variance 1. This is your default choice — it works well for most situations, especially when your data is roughly normally distributed.

MinMaxScaler squishes everything into a fixed range, usually [0, 1]. Use it when you need a bounded range — for example, neural networks with sigmoid output layers, or when you’re working with image pixel data that expects values between 0 and 1.

RobustScaler uses the median and interquartile range instead of the mean and standard deviation. Use it when you have outliers. The mean and standard deviation get pulled around by extreme values; the median and IQR don’t. If your data has a few enormous values, RobustScaler keeps them from warping the scaling for everything else.

Let’s see what happens when we skip scaling for a model that desperately needs it:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt

# Create a toy dataset with two features on wildly different scales
np.random.seed(42)
n_samples = 300

# Feature 1: measured in dollars, range ~0 to 100,000
income = np.random.lognormal(mean=10, sigma=1, size=n_samples)
# Feature 2: measured in years, range ~18 to 80
age = np.random.uniform(18, 80, n_samples)

# Target: people with high income OR middle age are class 1
# (The true boundary depends on both features roughly equally)
log_odds = (np.log(income) - 10) * 0.8 + (age - 45) * 0.1
prob = 1 / (1 + np.exp(-log_odds))
y = (np.random.random(n_samples) < prob).astype(int)

X = pd.DataFrame({'income': income, 'age': age})

# Split first
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# --- SVM WITHOUT scaling ---
svm_raw = SVC(kernel='rbf', random_state=42)
svm_raw.fit(X_train, y_train)
y_pred_raw = svm_raw.predict(X_test)
score_svm_raw = accuracy_score(y_test, y_pred_raw)

# --- SVM WITH scaling ---
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

svm_scaled = SVC(kernel='rbf', random_state=42)
svm_scaled.fit(X_train_scaled, y_train)
y_pred_scaled = svm_scaled.predict(X_test_scaled)
score_svm_scaled = accuracy_score(y_test, y_pred_scaled)

# --- Random Forest (shouldn't care about scaling) ---
rf_raw = RandomForestClassifier(random_state=42)
rf_raw.fit(X_train, y_train)
score_rf_raw = accuracy_score(y_test, rf_raw.predict(X_test))

rf_scaled = RandomForestClassifier(random_state=42)
rf_scaled.fit(X_train_scaled, y_train)
score_rf_scaled = accuracy_score(y_test, rf_scaled.predict(X_test_scaled))

print(f"SVM without scaling:    {score_svm_raw:.3f}")
print(f"SVM with scaling:       {score_svm_scaled:.3f}")
print(f"Random Forest (raw):    {score_rf_raw:.3f}")
print(f"Random Forest (scaled): {score_rf_scaled:.3f}")
print()
print("Notice: SVM jumps dramatically with scaling. Random Forest doesn't care.")

# Visualize the decision boundaries
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Create a mesh for plotting decision boundaries
# We'll use the scaled space for both plots for fair comparison
x_min, x_max = X_train_scaled[:, 0].min() - 1, X_train_scaled[:, 0].max() + 1
y_min, y_max = X_train_scaled[:, 1].min() - 1, X_train_scaled[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))

# SVM without scaling — but we need to predict in the original (unscaled) space
# To plot on the scaled axes, we'll inverse-transform the mesh, predict, and plot
# Actually, let's just plot both in scaled coordinates for visual comparison
# SVM raw: we'll scale the mesh, predict, then plot
mesh_scaled = np.c_[xx.ravel(), yy.ravel()]

# For the raw SVM, we need to inverse-transform the mesh to original space
# But we don't have an inverse for the raw model's implicit scaling
# So let's just train both on scaled data and show one with good C, one with bad C
# Actually, the cleanest demo: show the SVM decision boundary in the ORIGINAL space
# by plotting on the original axes

# Let's redo this more cleanly:
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Plot 1: SVM on raw (unscaled) data — decision boundary in original space
x_min_raw, x_max_raw = X_train['income'].min(), X_train['income'].max()
y_min_raw, y_max_raw = X_train['age'].min(), X_train['age'].max()
xx_raw, yy_raw = np.meshgrid(np.linspace(x_min_raw, x_max_raw, 200),
                              np.linspace(y_min_raw, y_max_raw, 200))
mesh_raw = np.c_[xx_raw.ravel(), yy_raw.ravel()]
Z_raw = svm_raw.predict(mesh_raw).reshape(xx_raw.shape)

axes[0].contourf(xx_raw, yy_raw, Z_raw, alpha=0.3, levels=[-0.5, 0.5, 1.5], colors=['blue', 'red'])
axes[0].scatter(X_train['income'], X_train['age'], c=y_train, cmap='bwr', alpha=0.6, edgecolors='k')
axes[0].set_title(f'SVM WITHOUT Scaling\nAccuracy: {score_svm_raw:.3f}')
axes[0].set_xlabel('Income ($)')
axes[0].set_ylabel('Age (years)')

# Plot 2: SVM on scaled data — but plot in original space by inverse-transforming
# We'll create mesh in scaled space, predict, then plot on the same original axes
# Actually, let's just plot in scaled space and label axes appropriately
x_min_s, x_max_s = X_train_scaled[:, 0].min() - 1, X_train_scaled[:, 0].max() + 1
y_min_s, y_max_s = X_train_scaled[:, 1].min() - 1, X_train_scaled[:, 1].max() + 1
xx_s, yy_s = np.meshgrid(np.linspace(x_min_s, x_max_s, 200),
                          np.linspace(y_min_s, y_max_s, 200))
mesh_s = np.c_[xx_s.ravel(), yy_s.ravel()]
Z_s = svm_scaled.predict(mesh_s).reshape(xx_s.shape)

axes[1].contourf(xx_s, yy_s, Z_s, alpha=0.3, levels=[-0.5, 0.5, 1.5], colors=['blue', 'red'])
axes[1].scatter(X_train_scaled[:, 0], X_train_scaled[:, 1], c=y_train, cmap='bwr', alpha=0.6, edgecolors='k')
axes[1].set_title(f'SVM WITH Scaling\nAccuracy: {score_svm_scaled:.3f}')
axes[1].set_xlabel('Income (standardized)')
axes[1].set_ylabel('Age (standardized)')

plt.tight_layout()
plt.show()

Look at the accuracy difference. The SVM without scaling is essentially guessing — it can’t find a meaningful boundary because the income feature completely dominates the distance calculations. The scaled SVM finds the pattern. And the random forest? Identical scores either way. Trees genuinely don’t care about feature scale.

The rule from Gate Zero applies here too: fit the scaler on the training set, then transform both train and test. If you fit on the full dataset, you’ve leaked information about the test set’s mean and variance into your training pipeline.

Gate Three: Categorical Encoding — Ordinal, One-Hot, and the Target-Encoding Trap

Models eat numbers, not strings. If you’ve got a column full of “red”, “blue”, “green”, you need to turn those into something mathematical. This sounds simple until you have a column with 10,000 unique values and you realize one-hot encoding would create 10,000 new columns.

There are three families of encoding you need in your toolkit, and one of them has a razor-sharp edge that cuts most practitioners at least once.

Ordinal Encoding: When Order Exists

If your categories have a natural order — “low” < “medium” < “high”, or “freshman” < “sophomore” < “junior” < “senior” — use OrdinalEncoder. It maps each category to an integer that preserves the order. The model can now understand that “high” is greater than “low” in a meaningful way.

One-Hot Encoding: When Order Doesn’t Exist

If your categories are just names with no inherent order — “red”, “blue”, “green” — use OneHotEncoder. It creates a new binary column for each category. A row with “red” gets a 1 in the red column and 0s everywhere else. This tells the model “these are distinct options, not ordered levels.”

But here’s the catch: if you have a column with 10,000 unique values (like product IDs or zip codes), one-hot encoding creates 10,000 new columns. Your dataset explodes in width, and most of those columns are sparse — almost entirely zeros. This is where target encoding comes in.

Target Encoding: Powerful, But the Sharpest Gotcha in Feature Engineering

Target encoding replaces each category with the mean target value for that category. If you’re predicting whether a customer will churn, and customers from California churn at a rate of 0.23, then every row with “California” gets the value 0.23. One column, no explosion, and it captures the relationship between the category and the target directly.

This is incredibly powerful for high-cardinality features. Daniele Micci-Barreca introduced it in 2001 specifically for this problem — when you have too many categories for one-hot encoding but you don’t want to throw away the information.

Now here’s the trap. And this is the single sharpest gotcha in all of feature engineering.

If you compute the target mean for “California” using all the rows — including the row you’re currently encoding — you’ve just handed the model the answer. For that row, the encoded value contains the target itself. It’s like a student who gets to see their own grade before taking the test.

Let’s see this in brutal clarity:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder, TargetEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score

# Create a dataset with a high-cardinality categorical feature
np.random.seed(42)
n_samples = 1000

# 50 unique categories
categories = [f'cat_{i}' for i in range(50)]
cat_column = np.random.choice(categories, size=n_samples)

# Target depends partly on the category and partly on noise
# Each category has a different base probability
category_means = {cat: np.random.uniform(0.2, 0.8) for cat in categories}
base_prob = np.array([category_means[c] for c in cat_column])
noise = np.random.normal(0, 0.1, n_samples)
prob = np.clip(base_prob + noise, 0.05, 0.95)
y = (np.random.random(n_samples) < prob).astype(int)

df = pd.DataFrame({'category': cat_column})

# Split first
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.3, random_state=42)

# ============================================
# Ordinal Encoding
# ============================================
ord_pipe = Pipeline([
    ('encode', OrdinalEncoder()),
    ('model', LogisticRegression(max_iter=1000))
])
ord_pipe.fit(X_train, y_train)
score_ord = accuracy_score(y_test, ord_pipe.predict(X_test))

# ============================================
# One-Hot Encoding
# ============================================
ohe_pipe = Pipeline([
    ('encode', OneHotEncoder(sparse_output=False, handle_unknown='ignore')),
    ('model', LogisticRegression(max_iter=1000))
])
ohe_pipe.fit(X_train, y_train)
score_ohe = accuracy_score(y_test, ohe_pipe.predict(X_test))

# ============================================
# Target Encoding — THE WRONG WAY (naive fit_transform on train)
# This is what happens if you don't use cross-fitting
# ============================================
# WARNING: This is intentionally wrong — it leaks the target
# We're showing it so you can see how seductive the fake score is
target_enc_wrong = TargetEncoder()
X_train_te_wrong = target_enc_wrong.fit_transform(X_train, y_train)
X_test_te_wrong = target_enc_wrong.transform(X_test)

model_wrong = LogisticRegression(max_iter=1000)
model_wrong.fit(X_train_te_wrong, y_train)
score_te_wrong = accuracy_score(y_test, model_wrong.predict(X_test_te_wrong))

# ============================================
# Target Encoding — THE RIGHT WAY (with cross-fitting)
# sklearn's TargetEncoder does cross-fitting internally by default
# when you use fit_transform, it computes the mean for each row
# using only the OTHER rows' data
# ============================================
target_enc_right = TargetEncoder(cv=5)  # 5-fold cross-fitting
X_train_te_right = target_enc_right.fit_transform(X_train, y_train)
X_test_te_right = target_enc_right.transform(X_test)

model_right = LogisticRegression(max_iter=1000)
model_right.fit(X_train_te_right, y_train)
score_te_right = accuracy_score(y_test, model_right.predict(X_test_te_right))

print("Encoding comparison:")
print(f"  Ordinal encoding:          {score_ord:.3f}")
print(f"  One-hot encoding:          {score_ohe:.3f}")
print(f"  Target encoding (WRONG):   {score_te_wrong:.3f}  <-- looks great, but it's cheating")
print(f"  Target encoding (correct): {score_te_right:.3f}")
print()
print("The wrong way looks better because it leaked the target into the features.")
print("In production, that model would fall apart because new categories won't have")
print("pre-computed target means to leak.")

See that gap? The naive target encoding looks better — sometimes dramatically better — because it cheated. The correct version, with cross-fitting, gives you an honest score. scikit-learn’s TargetEncoder with cv=5 computes the target mean for each row using only the other four folds’ data. That row never gets to see its own target.

If you’re not using scikit-learn’s built-in cross-fitting, you must do out-of-fold encoding manually. Otherwise your validation score is a lie, and your production model will fail in ways you won’t understand until it’s too late.

Gate Four: Interactions and Polynomial Terms — Teaching Your Model About Combinations

Linear models have a blind spot. They assume each feature acts independently — that the effect of square footage on house price is the same whether the house has 2 bedrooms or 5. But that’s not how the world works. A 3,000-square-foot house with 2 bedrooms is a weird, open-plan loft. A 3,000-square-foot house with 5 bedrooms is a large family home. The combination matters.

This is what interaction terms capture. When you multiply two features together, you’re saying “the effect of feature A depends on the value of feature B.” PolynomialFeatures does this systematically: given features [a, b] at degree 2, it produces [1, a, b, a², ab, b²]. The cross-term ab is the interaction — it captures how a and b work together.

When to Use Polynomial Features

Use them when you’re working with linear or logistic regression and you suspect relationships aren’t purely additive. Tree-based models and neural networks learn interactions automatically — a decision tree can split on square footage, then split again on bedrooms, effectively creating an interaction without you having to specify it. But linear models need you to spell it out.

The Cost: Combinatorial Explosion

Here’s the catch. With 100 features at degree 2, you get roughly 5,000 output features. At degree 3, it’s around 170,000. This is why Gate Five (feature selection) is essential — you’ll create a flood of features, and most of them are noise.

Use interaction_only=True if you only care about the cross-terms (ab) and not the squared terms (a², b²). This keeps the feature count manageable when you only want combinations.

And remember Gate Two: scale first, then create interactions. If you create polynomial features on unscaled data, your squared terms inherit all the scale problems we just fixed.

Let’s see what happens when the true relationship includes an interaction but we don’t tell the model:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error

# Create a dataset where the true relationship includes an interaction
np.random.seed(42)
n = 200

x1 = np.random.uniform(0, 10, n)
x2 = np.random.uniform(0, 10, n)

# True relationship: y = 3*x1 + 2*x2 + 5*x1*x2 + noise
# The interaction term (x1*x2) is actually the strongest component
y = 3 * x1 + 2 * x2 + 5 * x1 * x2 + np.random.normal(0, 5, n)

X = pd.DataFrame({'x1': x1, 'x2': x2})

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# --- Model 1: Linear regression WITHOUT interactions ---
# Scale first, then model
pipe_no_inter = Pipeline([
    ('scale', StandardScaler()),
    ('model', LinearRegression())
])
pipe_no_inter.fit(X_train, y_train)
y_pred_no_inter = pipe_no_inter.predict(X_test)
rmse_no_inter = np.sqrt(mean_squared_error(y_test, y_pred_no_inter))

# --- Model 2: Linear regression WITH polynomial features (degree 2) ---
pipe_with_inter = Pipeline([
    ('scale', StandardScaler()),
    ('poly', PolynomialFeatures(degree=2, include_bias=False)),
    ('model', LinearRegression())
])
pipe_with_inter.fit(X_train, y_train)
y_pred_with_inter = pipe_with_inter.predict(X_test)
rmse_with_inter = np.sqrt(mean_squared_error(y_test, y_pred_with_inter))

# Show what PolynomialFeatures actually produces
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
poly = PolynomialFeatures(degree=2, include_bias=False)
X_train_poly = poly.fit_transform(X_train_scaled)
feature_names = poly.get_feature_names_out(['x1', 'x2'])

print("PolynomialFeatures expands [x1, x2] at degree 2 to:")
print(f"  {list(feature_names)}")
print()
print(f"Original features: {X_train.shape[1]}")
print(f"After PolynomialFeatures (degree 2): {X_train_poly.shape[1]}")
print()
print(f"RMSE without interactions: {rmse_no_inter:.2f}")
print(f"RMSE with interactions:    {rmse_with_inter:.2f}")
print()
print("The model with interaction terms captures the x1*x2 relationship")
print("that the simple linear model completely misses.")

Look at the feature names: x1, x2, x1², x1·x2, x2². That cross-term x1·x2 is doing the heavy lifting — it’s capturing the fact that the true relationship multiplies the two features together. The linear model without it can only fit a flat plane through the data; the model with it can fit a curved surface that follows the actual pattern.

Gate Five: Feature Selection — Keeping What Matters, Ditching What Doesn’t

After Gate Four, you might have thousands of features. Many of them are noise — random fluctuations that happen to correlate with your target in the training data but won’t generalize. If you keep them, your model will memorize those spurious patterns. Your training score will look great. Your test score won’t.

Feature selection is about finding the signal and discarding the noise. It’s not just about speed — it’s about preventing overfitting.

Three Families of Feature Selection

Univariate filters (SelectKBest): The fastest, simplest approach. Each feature is scored independently against the target using a statistical test (f_classif for classification, f_regression for regression). You keep the top k. The downside: it ignores feature interactions. A feature that’s useless alone but powerful in combination gets discarded.

Recursive Feature Elimination (RFE/RFECV): Train a model, find the least important feature, remove it, repeat. This is model-aware — the feature importance comes from an actual model, so interactions are accounted for. RFECV goes further: it uses cross-validation to decide how many features to keep. The downside: it’s slow. Training a model hundreds of times adds up.

Model-based selection (SelectFromModel): Train a model once, look at the feature importances or coefficients, and keep only the ones above a threshold. With L1-regularized linear models (Lasso), this is particularly elegant — the L1 penalty naturally pushes coefficients to exactly zero. As scikit-learn’s feature selection guide notes: “Linear models penalized with the L1 norm have sparse solutions: many of their estimated coefficients are zero.” Increase the regularization strength, and more coefficients vanish. The model is its own feature selector.

The Leakage Trap (Again)

Feature selection must happen inside cross-validation, not before. If you select features on the full dataset and then cross-validate, you’ve peeked at the test folds — the feature selector saw which features correlate with the target across all the data, including the hold-out sets. That’s leakage. Your cross-validation score will be inflated.

The fix: use RFECV (which bakes cross-validation into the selection) or wrap your feature selection in a Pipeline inside cross_val_score. Or use SelectFromModel with a model that has built-in regularization.

Let’s see all three approaches on a dataset where half the features are pure noise:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.feature_selection import SelectKBest, f_regression, RFECV, SelectFromModel
from sklearn.linear_model import Lasso, LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

# Generate a dataset with 100 features — only 10 are actually predictive
np.random.seed(42)
n_samples, n_features = 300, 100
n_informative = 10

# Create the informative features
X_informative = np.random.randn(n_samples, n_informative)
true_coefficients = np.random.uniform(1, 5, n_informative)
y = X_informative @ true_coefficients + np.random.normal(0, 1, n_samples)

# Add 90 noise features
X_noise = np.random.randn(n_samples, n_features - n_informative)
X = np.hstack([X_informative, X_noise])

# Shuffle columns so the informative ones aren't all at the beginning
shuffle_idx = np.random.permutation(n_features)
X = X[:, shuffle_idx]
# Track which columns are informative (first n_informative after shuffling)
informative_mask = np.zeros(n_features, dtype=bool)
informative_mask[shuffle_idx[:n_informative]] = True

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# --- Method 1: SelectKBest (univariate filter) ---
selector_kbest = SelectKBest(score_func=f_regression, k=20)
X_train_kbest = selector_kbest.fit_transform(X_train, y_train)
X_test_kbest = selector_kbest.transform(X_test)

model_kbest = LinearRegression()
model_kbest.fit(X_train_kbest, y_train)
score_kbest = np.sqrt(mean_squared_error(y_test, model_kbest.predict(X_test_kbest)))

# --- Method 2: RFECV (recursive elimination with cross-validation) ---
# RFECV automatically determines the optimal number of features
estimator_rfecv = LinearRegression()
selector_rfecv = RFECV(estimator_rfecv, step=5, cv=5, scoring='neg_mean_squared_error')
selector_rfecv.fit(X_train, y_train)
X_train_rfecv = selector_rfecv.transform(X_train)
X_test_rfecv = selector_rfecv.transform(X_test)

model_rfecv = LinearRegression()
model_rfecv.fit(X_train_rfecv, y_train)
score_rfecv = np.sqrt(mean_squared_error(y_test, model_rfecv.predict(X_test_rfecv)))

# --- Method 3: SelectFromModel with Lasso (L1 regularization) ---
# Lasso naturally pushes coefficients to zero
lasso = Lasso(alpha=0.1, max_iter=5000, random_state=42)
selector_lasso = SelectFromModel(lasso, prefit=False)
selector_lasso.fit(X_train, y_train)
X_train_lasso = selector_lasso.transform(X_train)
X_test_lasso = selector_lasso.transform(X_test)

model_lasso_final = LinearRegression()
model_lasso_final.fit(X_train_lasso, y_train)
score_lasso = np.sqrt(mean_squared_error(y_test, model_lasso_final.predict(X_test_lasso)))

# --- Baseline: No feature selection ---
model_baseline = LinearRegression()
model_baseline.fit(X_train, y_train)
score_baseline = np.sqrt(mean_squared_error(y_test, model_baseline.predict(X_test)))

print(f"Number of training features: {X_train.shape[1]}")
print(f"Number of truly informative features: {n_informative}")
print()
print(f"Baseline (no selection):     RMSE = {score_baseline:.3f}, features = {X_train.shape[1]}")
print(f"SelectKBest (k=20):          RMSE = {score_kbest:.3f}, features = 20")
print(f"RFECV (auto):                RMSE = {score_rfecv:.3f}, features = {selector_rfecv.n_features_}")
print(f"SelectFromModel (Lasso):     RMSE = {score_lasso:.3f}, features = {selector_lasso.get_support().sum()}")
print()

# Show the L1 sparsity in action: as alpha increases, more coefficients become zero
alphas = [0.001, 0.01, 0.1, 0.5, 1.0, 5.0]
print("L1 sparsity path (Lasso with different alpha values):")
print(f"{'Alpha':<10} {'Non-zero coefs':<20} {'% Zero':<10}")
print("-" * 40)
for alpha in alphas:
    lasso_path = Lasso(alpha=alpha, max_iter=5000, random_state=42)
    lasso_path.fit(X_train, y_train)
    n_nonzero = np.sum(lasso_path.coef_ != 0)
    pct_zero = (n_features - n_nonzero) / n_features * 100
    print(f"{alpha:<10} {n_nonzero:<20} {pct_zero:<10.1f}%")

Watch the L1 sparsity path. At alpha=0.001, the model keeps most features. At alpha=5.0, almost everything is zero — the model is forced to keep only the features that really matter. This is both a feature selector and a regularizer in one elegant package.

Notice that the baseline model, with all 100 features, has the worst RMSE. It’s overfitting to the 90 noise features. Feature selection doesn’t just make your model faster — it makes it better.

Gate Six: The Production Sanity Check — Does This Feature Exist at Prediction Time?

You’ve engineered brilliant features. Your validation score is stellar. You deploy the model. And it crashes. Or worse, it doesn’t crash — it silently produces garbage predictions — because a feature you trained on won’t be available until next week.

This is temporal leakage, and it’s heartbreakingly common. It’s a different flavor from the leakage we covered in Gate Zero. Train/test leakage is about methodology — did you accidentally let your training pipeline see the test data? Temporal leakage is about the real-world meaning of your features — does this information actually exist at the moment you make the prediction?

The Question to Ask for Every Feature

“At the moment I make this prediction, does this value exist?”

If you’re predicting next month’s customer churn, you can’t use this month’s support tickets as a feature — they haven’t happened yet. If you’re predicting whether a loan will default, you can’t use “number of late payments in the next 6 months” — that’s the future. If you’re building a recommendation system, you can’t use “did the user buy this item?” to predict whether they’ll buy it.

Some features are clearly illegal once you think about them. But others are sneakier. Aggregate features computed on the full dataset — like “average purchase amount across all customers” — often include future rows. If you compute that average on data that includes dates after your prediction point, you’ve leaked the future into the past.

The Fix: Time-Based Splits

Random train/test splits assume all rows are independent and identically distributed. Time-series data violates that assumption. If you randomly split time-series data, rows from the future end up in your training set, and rows from the past end up in your test set. Your model learns patterns from the future and appears to predict the past — which looks amazing in validation and fails completely in production.

The fix is simple: split by time. All training data must come from before all test data. This mimics the real world, where you train on historical data and predict the future.

Let’s see how seductive the fake score from a random split can be:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Simulate a time-series churn prediction scenario
np.random.seed(42)
n_samples = 1000

# Create a time index (day 0 to day 999)
time = np.arange(n_samples)

# Feature: "support tickets this month" — this is a legitimate feature
# It's historical: at prediction time, we know how many tickets they filed
support_tickets = np.random.poisson(lam=2, size=n_samples)

# Feature: "purchases next month" — THIS IS THE FUTURE
# At prediction time, we don't know this yet. Using it is temporal leakage.
future_purchases = np.random.poisson(lam=3, size=n_samples)

# Target: churn (1 = churned). 
# Churn is partly predictable from support tickets (real signal)
# and we'll make it also correlate with future purchases (the leak)
log_odds = (support_tickets - 2) * 0.3 + (future_purchases - 3) * (-0.8)
prob = 1 / (1 + np.exp(-log_odds))
y = (np.random.random(n_samples) < prob).astype(int)

# Create DataFrame with a legitimate feature and a leaky feature
X = pd.DataFrame({
    'support_tickets': support_tickets,
    'future_purchases': future_purchases,  # This is illegal at prediction time
    'time': time
})

# ============================================
# RANDOM SPLIT (wrong for time-series)
# ============================================
X_train_rand, X_test_rand, y_train_rand, y_test_rand = train_test_split(
    X.drop('time', axis=1), y, test_size=0.3, random_state=42
)

# Model using BOTH features (including the illegal future_purchases)
model_rand = LogisticRegression(max_iter=1000)
model_rand.fit(X_train_rand, y_train_rand)
score_rand = accuracy_score(y_test_rand, model_rand.predict(X_test_rand))

# Model using ONLY the legitimate feature
model_rand_legit = LogisticRegression(max_iter=1000)
model_rand_legit.fit(X_train_rand[['support_tickets']], y_train_rand)
score_rand_legit = accuracy_score(y_test_rand, model_rand_legit.predict(X_test_rand[['support_tickets']]))

# ============================================
# TIME-BASED SPLIT (correct)
# ============================================
# Sort by time and split: first 70% for training, last 30% for testing
split_idx = int(0.7 * n_samples)
X_train_time = X.iloc[:split_idx].drop('time', axis=1)
X_test_time = X.iloc[split_idx:].drop('time', axis=1)
y_train_time = y[:split_idx]
y_test_time = y[split_idx:]

# Model using BOTH features (including the illegal future_purchases)
model_time = LogisticRegression(max_iter=1000)
model_time.fit(X_train_time, y_train_time)
score_time = accuracy_score(y_test_time, model_time.predict(X_test_time))

# Model using ONLY the legitimate feature
model_time_legit = LogisticRegression(max_iter=1000)
model_time_legit.fit(X_train_time[['support_tickets']], y_train_time)
score_time_legit = accuracy_score(y_test_time, model_time_legit.predict(X_test_time[['support_tickets']]))

print("=" * 60)
print("RANDOM SPLIT (wrong for time-series):")
print(f"  With leaky feature:     {score_rand:.3f}  <-- looks great!")
print(f"  Without leaky feature:  {score_rand_legit:.3f}")
print()
print("TIME-BASED SPLIT (correct):")
print(f"  With leaky feature:     {score_time:.3f}  <-- still looks good, but...")
print(f"  Without leaky feature:  {score_time_legit:.3f}")
print()
print("The random split hides the temporal leakage. The time-based split")
print("reveals that the leaky feature won't actually help in production")
print("because in a real deployment, you're always predicting the future.")

In a random split, the leaky feature looks like it helps — because the model sees future information scattered throughout both train and test sets. In a time-based split, the gap between the leaky and legitimate models narrows or disappears — because in the real world, you don’t have a time machine.

The Practical Fix

Timestamp your data. For every feature, document the point in time it becomes available. If that point is after your prediction point, the feature is illegal. This isn’t a coding fix — it’s a discipline. You have to think about what each column actually means in the real world, not just what it looks like in a CSV.

The Full Checklist (Printable Reference)

Here’s everything, consolidated into a single scannable checklist. Run through this before every modeling project.

---

Feature Engineering Checklist

Gate 0: Leakage Audit

• Split data into train/test before ANY transformation
• Fit every transformer (imputer, scaler, encoder, selector) on train only
• Transform test using the fitted transformers — never fit on test

Gate 1: Missing Data

• Choose imputation strategy: Simple (baseline), Iterative (features predict each other), or KNN (local structure)
• Add missingness indicators (add_indicator=True) to preserve the signal of missingness
• Fit imputer on train, transform on test

Gate 2: Scaling

• If using SVM, k-NN, neural net, PCA, or regularized linear model: scale your features
• If using tree-based models: skip scaling (they don’t care)
• Choose scaler: StandardScaler (default), MinMaxScaler (bounded range needed), RobustScaler (outliers present)
• Fit scaler on train, transform on test

Gate 3: Categorical Encoding

• Ordinal encoding for ordered categories (low < medium < high)
• One-hot encoding for unordered, low-cardinality categories
• Target encoding for high-cardinality categories — with cross-fitting (cv≥5) to prevent leakage
• Fit encoder on train, transform on test

Gate 4: Interactions and Polynomials

• Add polynomial/interaction features if using linear/logistic regression
• Use interaction_only=True if you only need cross-terms
• Scale before creating polynomial features
• Watch for feature explosion — you’ll need Gate 5 afterward

Gate 5: Feature Selection

• Choose method: SelectKBest (fast, univariate), RFECV (model-aware, slow), SelectFromModel with L1 (sparse solution)
• Do feature selection inside cross-validation, not before
• Tree-based models can use built-in feature importance with SelectFromModel

Gate 6: Production Sanity Check

• For every feature, ask: “Does this value exist at prediction time?”
• Use time-based splits for temporal data, not random splits
• Document the availability timestamp for every feature

---

Print this. Tape it to your monitor. Run through it before every model.fit(). The gates are in order for a reason — each one builds on the discipline of the one before it.

Recap

We started with a model that wouldn’t budge and a dataset it couldn’t digest. We end with a systematic process for translating raw data into model-ready features — without leaking, without distorting, and without shipping features that don’t exist yet.

Here’s what we learned:

• Leakage is the silent killer. Split first, fit transformers on train only, and never let your pipeline peek at the test data.
• Missingness is information. Impute thoughtfully, and always add an indicator column so your model can learn from the fact that data was missing.
• Scale matters for some models and not others. Distance-based and gradient-based models need it; trees don’t. Choose your scaler based on your data’s quirks.
• Target encoding is powerful and dangerous. Use cross-fitting or you’re handing the model the answer key.
• Linear models need help seeing interactions. PolynomialFeatures spells them out — at the cost of a much wider dataset.
• Feature selection prevents memorization. Use L1 regularization, recursive elimination, or univariate filters — but always inside cross-validation.
• Production is the ultimate test. If a feature doesn’t exist at prediction time, it doesn’t matter how predictive it looks in your notebook.

Check Your Understanding

Remember: List the seven gates in order, from Gate Zero through Gate Six.

Understand: Explain in plain English why target encoding without cross-fitting is leakage. What specifically does the model see that it shouldn’t?

Apply: You’re given a dataset with missing numerical values and a high-cardinality categorical column. Write a pipeline (in pseudocode or real code) that correctly imputes, encodes, and scales the data without leakage.

Analyze: A colleague shows you a model with 0.99 validation accuracy on a customer churn problem. What five specific questions would you ask to check for leakage?

Evaluate: Compare mean imputation versus iterative imputation. Under what conditions would you choose one over the other? What are the tradeoffs?

Create: Design a feature engineering plan for a loan default prediction problem where data arrives monthly. Which features would be illegal to include, and why? How would you structure your train/test split?

Apply What You Learned

Task: Take the Titanic dataset (available via seaborn.load_dataset('titanic') or from Kaggle) — it has missing values, categorical features, and numerical features on different scales. Run it through all seven gates and compare a baseline logistic regression’s accuracy before and after your feature engineering pipeline.

Deliverable: A single Python script or notebook that:

1. Loads the Titanic data
2. Splits into train/test before any transformation
3. Applies Gates 1–5 (imputation with indicators, scaling, categorical encoding, optional interactions, feature selection)
4. Trains a logistic regression on both the raw data (minimally processed) and the fully engineered data
5. Prints the before/after accuracy scores

Self-check rubric:

• Did you split before any transformation? (Gate Zero)
• Did you add missingness indicators? (Gate One)
• Did you scale numerical features? (Gate Two — logistic regression has regularization by default)
• Did you encode categorical features appropriately? (Gate Three)
• Did you consider interactions or polynomial features? (Gate Four — optional but worth trying)
• Did you use feature selection inside cross-validation or on the training set only? (Gate Five)
• Did your score improve without obvious signs of leakage? (Gate Six — the Titanic data isn’t temporal, but the discipline still applies)