Feature Engineering for Tabular Data: The Techniques That Actually Move the Needle

In our last lesson, Sam sorted out feature scaling so every feature got an equal vote. The sellability classifier and the price model share the same listing features, so Sam prototypes today’s fix on price prediction — the effect is easier to see and measure on a continuous target than on a plateaued accuracy number. The price model’s R-squared has plateaued, though, and no amount of hyperparameter tuning seems to help.

Why Your Model is Stuck (and Why Hyperparameters Won’t Save You)

Sam has spent an entire afternoon tweaking max_depth or learning_rate, only to watch R-squared shift by a measly 0.001. It’s one of the most frustrating plateaus in data science. You feel like you have a good engine, but the car won’t go any faster.

What’s actually going on? Most of the time, the problem isn’t the model. It’s the food you’re feeding it. A world-class chef can’t make a five-star meal from flour and water alone—they need yeast, salt, and spices to create something worth eating. In machine learning, your features are those ingredients. If the signal is buried deep in raw numbers, even the most complex XGBoost model won’t find it. I’d call this the ‘Tuning Trap’: we focus on the algorithm when we should focus on the data.

Here’s what happens when we try to predict house prices from a raw dataset. Sam will use a version of the ‘House Prices’ data from the scaling tutorial, but with more complexity.

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

# Let's create a more realistic dataset
np.random.seed(42)
n_samples = 1000
data = {
    'SqFt': np.random.normal(2000, 500, n_samples),
    'Bedrooms': np.random.randint(1, 6, n_samples),
    'YearBuilt': np.random.randint(1950, 2023, n_samples),
    'ZipCode': np.random.choice(['ZIP_A', 'ZIP_B', 'ZIP_C', 'ZIP_D'], n_samples),
    'LastSaleDate': pd.date_range(start='2020-01-01', periods=n_samples, freq='D')
}

# Create a target price with some hidden logic -- including a real
# per-zip-code effect the baseline features below can't see, since they
# drop ZipCode entirely.
df = pd.DataFrame(data)
zip_effect = {'ZIP_A': 0, 'ZIP_B': 60000, 'ZIP_C': -30000, 'ZIP_D': 15000}
df['Price'] = (df['SqFt'] * 150) + (df['Bedrooms'] * 10000) + \
               ((2024 - df['YearBuilt']) * -500) + df['ZipCode'].map(zip_effect) + \
               np.random.normal(0, 5000, n_samples)

# Baseline: Drop non-numeric for now and fit
X_raw = df[['SqFt', 'Bedrooms', 'YearBuilt']]
y = df['Price']

X_train, X_test, y_train, y_test = train_test_split(X_raw, y, test_size=0.2, random_state=42)

model = RandomForestRegressor(random_state=42)
model.fit(X_train, y_train)
print(f"Baseline R-squared: {model.score(X_test, y_test):.4f}")

• np.random.seed(42) fixes the random number generator so the synthetic dataset is reproducible — every run produces the same numbers, which matters when comparing baseline vs. engineered models.
• np.random.normal(2000, 500, n_samples) generates 1,000 square-footage values drawn from a normal distribution with mean 2,000 and standard deviation 500 — most houses land between 1,500 and 2,500 sqft.
• np.random.randint(1, 6, n_samples) generates 1,000 bedroom counts uniformly between 1 and 5 — randint excludes the upper bound, so the range is [1, 5].
• np.random.choice([...], n_samples) randomly assigns one of four zip code labels to each sample — these are categorical strings that can’t be fed directly into a scikit-learn model without preprocessing.
• pd.date_range(start='2020-01-01', periods=n_samples, freq='D') creates 1,000 consecutive daily dates starting January 1, 2020 — this simulates a stream of sale dates.
• df['Price'] = ... constructs the target variable using a hidden linear formula: $150 per square foot, $10,000 per bedroom, minus $500 per year of age, a per-zip-code offset (ZIP_B commands $60,000 more than the ZIP_A baseline, ZIP_D $15,000 more, ZIP_C $30,000 less), plus Gaussian noise — the model doesn’t know this formula and must discover the relationships from the data.
• X_raw = df[['SqFt', 'Bedrooms', 'YearBuilt']] selects only the three numeric columns — ZipCode (strings) and LastSaleDate (datetime) are dropped because RandomForestRegressor can’t handle them without preprocessing.
• train_test_split(..., test_size=0.2, random_state=42) splits the data into 80% training (800 samples) and 20% testing (200 samples) — random_state ensures the same split every run.
• model.score(X_test, y_test) returns the R² (coefficient of determination) on the held-out test set — 1.0 is perfect, 0.0 means the model predicts the mean, and negative values mean it’s worse than just guessing the average.

That prints Baseline R-squared: 0.7931 — decent, but leaving real money on the table: at an average error of about $31,000 per house, a homeowner refinancing off this model’s number would notice. The model only sees raw dimensions, and one of the price drivers baked into the data — ZipCode — isn’t in its feature list at all. It doesn’t understand context. To get to the next level, stop tuning the engine and start enriching the fuel.

The Power of Interaction: When 1 + 1 Equals 3

Features often only make sense when they are combined. A 1,000-square-foot house is small. A $500,000 price tag is high. But a 1,000-square-foot house that costs $500,000 tells you something specific about luxury or location.

This is the intuition behind Interaction Terms and Ratios. A raw number like ‘Total Rooms’ works fine, but ‘Average Room Size’ (Total Area divided by Total Rooms) might be the signal that separates a cramped apartment from a spacious villa. These ratios reveal patterns the model can’t see when it looks at columns individually.

Here’s what happens when we create a ‘Price per Square Foot’ logic or combine features manually:

# Creating new 'Intuition' features
df_eng = df.copy()

# 1. The Ratio: How big are the rooms actually?
df_eng['Avg_Room_Size'] = df_eng['SqFt'] / df_eng['Bedrooms']

# 2. The Age: Models handle 'Age' better than 'Year'
df_eng['House_Age'] = 2024 - df_eng['YearBuilt']

# Let's check the correlation of our new features with Price
correlations = df_eng[['Avg_Room_Size', 'House_Age', 'Price']].corr()['Price']
print("Correlation with Price:")
print(correlations)
# Avg_Room_Size    0.1992
# House_Age       -0.0758
# Price            1.0000

• df_eng = df.copy() creates a copy of the DataFrame — modifications to df_eng won’t affect the original df, preserving the baseline for later comparison.
• df_eng['Avg_Room_Size'] = df_eng['SqFt'] / df_eng['Bedrooms'] divides square footage by bedroom count to create a ratio feature — a 2,000 sqft house with 4 bedrooms (500 sqft/room) tells a different story than 2,000 sqft with 1 bedroom (2,000 sqft/room), even though both have the same raw SqFt.
• df_eng['House_Age'] = 2024 - df_eng['YearBuilt'] converts an absolute year into a relative age — most models find “age” more intuitive than “year built” because the relationship between age and price is more linear (newer = more expensive) than the relationship between raw year and price.
• .corr()['Price'] computes the Pearson correlation coefficient between each selected column and the Price column — values range from -1 (perfect negative correlation) to +1 (perfect positive), with 0 meaning no linear relationship.

We’ve simplified the math for the model. Instead of the Random Forest having to figure out that 2024 - YearBuilt = Age, we’ve given it the answer directly. The correlation shows Avg_Room_Size has the stronger simple relationship with price at 0.20; House_Age sits at a weak -0.08, even though age genuinely lowers price by $500/year in the formula that generated this data. That’s not a contradiction — age’s effect is real but small next to square footage’s, which alone swings price by tens of thousands of dollars. A weak Pearson correlation doesn’t mean a feature is causally unimportant; it means its effect is being outscaled by something with more variance, which a tree-based model can still exploit through splits even when the raw correlation looks unremarkable.

Handling the ‘Messy’ Categories: Beyond One-Hot Encoding

Here’s the catch with categorical data: sometimes you have too much of it. Our dataset has ZipCode. Four zip codes? One-Hot Encoding (a column for each) works fine. Five hundred? Not so much.

This is the Curse of Dimensionality. Create 500 new columns and your data goes “sparse” — mostly zeros. The model slows down, and pattern-finding gets harder across that many columns.

So we use Target Encoding instead. We replace the zip code name with the average house price for that zip code. Now the model sees a number representing the neighborhood’s typical value rather than a label.

This is the hardest part to get right. Use the average price from your whole dataset and you’re “cheating” — showing the model answers from the future. That’s Data Leakage. Calculate these averages using only the training data.

# We'll use a simple manual target encoding for intuition
# In a real project, use 'category_encoders.TargetEncoder'

zip_means = df_eng.groupby('ZipCode')['Price'].mean()
df_eng['Zip_Encoded'] = df_eng['ZipCode'].map(zip_means)

print("Sample of Target Encoded Zip Codes:")
print(df_eng[['ZipCode', 'Zip_Encoded']].head(3))
#   ZipCode  Zip_Encoded
# 0   ZIP_D    325318.25
# 1   ZIP_C    286383.33
# 2   ZIP_D    325318.25

• df_eng.groupby('ZipCode')['Price'].mean() groups all rows by their ZipCode value and computes the average Price for each group — the result is a Series where the index is the zip code label and the value is the mean price for that zip.
• df_eng['ZipCode'].map(zip_means) looks up each row’s zip code in the zip_means Series and replaces the string label with its corresponding average price — ‘ZIP_A’ becomes 308592.0, ‘ZIP_B’ becomes 376123.0.
• Warning: This computes the mean using the entire dataset, including test rows — in a real workflow this causes data leakage (the model sees test-set information during training). Nothing in the code above flags this; the two comments at the top only note that this is a simplified manual encoding and point to category_encoders.TargetEncoder for production use. The leakage warning is a separate concern this reference is raising for you, not something the code itself surfaces — which is exactly why it’s easy to ship by accident. A production version would compute zip_means on the training set only and then .map() those training-set means onto the test set.
• .head(3) shows the first three rows to verify the encoding worked — each zip code should now have a numeric value representing that neighborhood’s average price.

The result is a single numeric column where ‘ZIP_A’ becomes 308592.0 and ‘ZIP_B’ becomes 376123.0 — the printed averages above. The model now knows ‘ZIP_B’ is genuinely the most expensive zip in this data (ZIP_C is the cheapest, at 286383) — no 500 separate columns needed.

Time is a Circle: Engineering Temporal Features

Dates carry useful information, but not as raw strings. Show a model “2023-12-25” and it sees characters, not a calendar. We need to break it down. Is it a weekend or a holiday?

There’s a trickier issue: Cyclicality. To a model, Hour 23 (11 PM) and Hour 0 (Midnight) sit 23 units apart. In reality, they’re adjacent. The fix is Cyclical Encoding — transform time into sine and cosine waves. This represents time as a circle, so the model learns that December sits next to January.

# Extracting basic date features
df_eng['Month'] = df_eng['LastSaleDate'].dt.month
df_eng['DayOfWeek'] = df_eng['LastSaleDate'].dt.dayofweek
df_eng['IsWeekend'] = df_eng['DayOfWeek'].apply(lambda x: 1 if x >= 5 else 0)

# Cyclical encoding for Month (1-12)
df_eng['month_sin'] = np.sin(2 * np.pi * df_eng['Month']/12)
df_eng['month_cos'] = np.cos(2 * np.pi * df_eng['Month']/12)

dec_row = df_eng[df_eng['Month'] == 12].iloc[[0]]
jan_row = df_eng[df_eng['Month'] == 1].iloc[[0]]
print("Cyclical Month Features (December vs January):")
print(pd.concat([dec_row, jan_row])[['Month', 'month_sin', 'month_cos']])
#      Month     month_sin  month_cos
# 335     12 -2.449294e-16        1.0
# 0        1  5.000000e-01   0.866025

• .dt.month extracts the month (1–12) from each datetime — pandas’ .dt accessor exposes datetime properties like month, day, hour as a Series.
• .dt.dayofweek extracts the day of week as an integer (0=Monday, 6=Sunday) — this is different from .dt.day_name() which returns the string name.
• .apply(lambda x: 1 if x >= 5 else 0) creates a binary “is weekend” flag — days 5 (Saturday) and 6 (Sunday) become 1, all others become 0.
• np.sin(2 * np.pi * df_eng['Month']/12) maps each month to a point on a sine wave — dividing by 12 normalizes the month number so that one full cycle (2π radians) spans exactly one year.
• np.cos(...) does the same with cosine — the sin/cos pair together represent each month as a (x, y) coordinate on a unit circle, so December (month 12) and January (month 1) end up physically adjacent on the circle rather than 11 units apart.
• df_eng[df_eng['Month'] == 12].iloc[[0]] and the matching == 1 line select an actual December row and an actual January row by their calendar month, not by position in the DataFrame. The first and last rows by position (df_eng.iloc[[0, -1]]) would be January 2020 and September 2022 here, since LastSaleDate runs day by day for 1,000 days — nowhere near the December/January boundary this section is illustrating.

Interaction Ratio and Cyclical Encoding

The interaction ratio (average room size) is a simple division:

$\text{AvgRoomSize} = \frac{\text{SqFt}}{\text{Bedrooms}}$

Cyclical encoding maps a periodic value $t$ (e.g., month 1–12) onto a unit circle using sine and cosine:

$\sin_t = \sin\!\left(\frac{2\pi \, t}{T}\right), \qquad \cos_t = \cos\!\left(\frac{2\pi \, t}{T}\right)$

where $T$ is the period (12 for months, 24 for hours, 7 for days of week).

Plain English	Statistical symbol	Python equivalent
Square footage	$\text{SqFt}$	df_eng['SqFt']
Number of bedrooms	$\text{Bedrooms}$	df_eng['Bedrooms']
Average room size (ratio)	$\frac{\text{SqFt}}{\text{Bedrooms}}$	df_eng['SqFt'] / df_eng['Bedrooms']
Cyclic value (e.g. month)	$t$	df_eng['Month']
Period (e.g. 12 months)	$T$	12
Sine component	$\sin(2\pi t / T)$	np.sin(2 * np.pi * df_eng['Month'] / 12)
Cosine component	$\cos(2\pi t / T)$	np.cos(2 * np.pi * df_eng['Month'] / 12)

Now, the sine and cosine values for Month 12 and Month 1 sit close together — a Euclidean distance of about 0.52 on the unit circle, versus 2.0 (the circle’s full diameter) between December and June. We’ve taught the model how a calendar works.

The Results: Did We Actually Move the Needle?

So what did all that work get us? We can find out by comparing our engineered model to the original baseline.

# Prepare engineered data
features = ['SqFt', 'Bedrooms', 'House_Age', 'Avg_Room_Size', 'Zip_Encoded', 'month_sin', 'month_cos', 'IsWeekend']
X_eng = df_eng[features]

X_train_e, X_test_e, y_train_e, y_test_e = train_test_split(X_eng, y, test_size=0.2, random_state=42)

model_eng = RandomForestRegressor(random_state=42)
model_eng.fit(X_train_e, y_train_e)

print(f"New R-squared: {model_eng.score(X_test_e, y_test_e):.4f}")
# New R-squared: 0.9767

# Feature Importance
importances = pd.Series(model_eng.feature_importances_, index=features).sort_values(ascending=False)
print("\nMost Important Features:")
print(importances)
# SqFt             0.8306
# Zip_Encoded      0.1260
# Avg_Room_Size    0.0140
# House_Age        0.0139
# Bedrooms         0.0088
# month_sin        0.0032
# month_cos        0.0027
# IsWeekend        0.0008

• features = [...] lists the engineered feature set — note that YearBuilt is replaced by House_Age, the raw ZipCode string is replaced by the numeric Zip_Encoded, and three new temporal features (month_sin, month_cos, IsWeekend) plus one ratio (Avg_Room_Size) are added.
• train_test_split(X_eng, y, ...) uses the same random_state=42 as the baseline split — this ensures the same rows go to train and test, so the R² comparison is apples-to-apples (only the features changed, not the split).
• model_eng.feature_importances_ returns an array of how much each feature contributed to reducing impurity across all trees in the forest — higher values mean the feature was used more often and more effectively for splitting.
• pd.Series(..., index=features) wraps the importance array with the feature names as the index — this makes the output readable and lets .sort_values(ascending=False) rank features from most to least important.

The R-squared jumped from 0.7931 to 0.9767 — and the number that actually matters for a homeowner, average error per house, dropped from about $31,000 to about $8,900. That is real money: the model went from being wrong by nearly a full percentage point of a typical home’s value to being wrong by roughly a quarter of that.

Now look at the feature_importances_. Zip_Encoded ranks second, behind only SqFt and well above every raw feature we kept (Bedrooms included) — it earned that rank because ZipCode carries a real $90,000 swing (from ZIP_C at the bottom to ZIP_B at the top) in the data the baseline model never got to see. House_Age and Avg_Room_Size rank lower, just above Bedrooms: real, small effects, not headline features. The model found one of our “spices” — the zip encoding — far more useful than most of the raw “flour,” but not all of them equally.

Hand-crafted feature engineering vs. letting the model find interactions

Not every model benefits equally from manual feature engineering. Here’s when the extra effort pays off for Sam:

Approach	When it wins	When it falls short
Manual feature engineering (ratios, interactions, target encoding)	Linear/logistic regression, neural networks — these models can only learn linear combinations of the features you give them, so SqFt / Bedrooms must be created by hand	When the model can already discover the interaction internally — adding redundant features just increases dimensionality
Tree-based models (Random Forest, XGBoost, LightGBM)	Trees can naturally discover interactions by splitting on one feature and then another — a tree can learn “if SqFt > 2000 and Bedrooms <= 2” without you creating an explicit ratio	Tree models still benefit from some manual features (target encoding for high-cardinality categoricals, cyclical encoding for time) but don’t need explicit multiplication/division interactions
Target encoding vs. one-hot	Use target encoding when a categorical column has hundreds of unique values — one-hot would create hundreds of sparse columns	Don’t use target encoding on categoricals with very few values — one-hot is simpler and avoids leakage risk
Cyclical encoding vs. raw numbers	Always prefer cyclical encoding for time features (hours, months, days) — it preserves the “wrap-around” relationship that both linear and tree models miss	Rarely a downside, but if the period is irrelevant to the target (e.g., day-of-month when only season matters), skip it to reduce noise

Sam’s rule of thumb: If the model is linear or neural-network-based, hand-craft every interaction you can think of — the model literally cannot discover them on its own. If it’s tree-based, focus your engineering effort on encoding categoricals and time features; the trees will handle the numeric interactions themselves.

Recap of what we learned:

• Ratios and Interactions: Let the model see relationships (like room size) without computing the division itself.
• Target Encoding: Handles messy categories with hundreds of values without bloating the dataset.
• Cyclical Features: Uses Sine and Cosine to teach the model that time (months, hours) repeats in a circle.
• The Big Lesson: We didn’t change the algorithm; we changed what it saw.

With better features in hand, each of Sam’s individual models is stronger. But he wonders whether combining several different models could beat any single one. In the series finale, we’ll bring everything together with Ensemble Stacking—combining multiple models (including ones built on these engineered features) the right way.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is Target Encoding, and what problem does it solve that One-Hot Encoding doesn’t handle well?

Understand In your own words, explain why Hour 23 and Hour 0 need Cyclical (Sine/Cosine) Encoding instead of just being left as the numbers 23 and 0.

Apply Using the article’s Avg_Room_Size formula (SqFt / Bedrooms), calculate the average room size for a house with SqFt = 2400 and Bedrooms = 4, and for a second house with SqFt = 1200 and Bedrooms = 1. Which house likely feels more spacious per room, even if you only knew the ratio and not the raw numbers?

Analyze The article warns that Target Encoding “cheats” by showing the model future answers if you compute zip-code averages using the whole dataset instead of just the training data. Walk through exactly how a test-set house’s price would leak into its own Zip_Encoded feature if you didn’t separate train and test before computing zip_means.

Evaluate The article’s final comparison shows engineered features like House_Age and Zip_Encoded ranking above raw features in feature_importances_. Critique using feature importance alone as proof that engineering “worked”: what else would you want to check (beyond just seeing new features rank highly) before concluding the R-squared improvement is trustworthy and not a fluke of this one train/test split?

Create Design two new engineered features (an interaction/ratio and a temporal one) for a different tabular dataset: predicting whether a restaurant will fail within its first year, given seating_capacity, avg_meal_price, opening_date, and neighborhood. Explain what raw relationship each new feature would expose that the model couldn’t see from the raw columns alone.

---

Related articles

• P10: Feature Scaling — Why Your Model Might Be Ignoring Half Your Data) — the previous part where Sam sorted out feature scaling so square footage and bedroom count got equal billing before hitting the accuracy plateau that feature engineering addresses.
• P12: Ensemble Stacking — Combining Models the Right Way) — the series finale where Sam combines his individually tuned models into a stacked ensemble for the final sellability forecast.

References & Further reading

• Zheng, A. & Casari, A. (2018). Feature Engineering for Machine Learning. O’Reilly Media. — the canonical reference on feature engineering techniques for tabular data, covering text vectorization, target encoding, interaction terms, and feature selection strategies with practical Python examples.
• Kaggle: House Prices — Advanced Regression Techniques — the canonical regression competition where feature engineering (especially target encoding of neighborhoods and interaction terms) separates top leaderboard scores from middle-of-the-pack submissions on real Ames, Iowa housing data.
• pandas: Time series / date functionality — official pandas documentation for datetime handling, including the .dt accessor properties and date_range generation used in the cyclical encoding examples.

---

<Apply title=“✍️ Essay prompt — “Did feature engineering actually move the needle?” (Data Scientist)”>

Brief: You’re at a stakeholder review defending the claim that feature engineering lifted the House Prices model’s R². The article itself admits the target-encoding step (zip_means = df_eng.groupby('ZipCode')['Price'].mean()) was computed on the entire 1,000-row dataframe — including the 200 test rows — which is textbook data leakage. On top of that, the target Price was synthesized from a known linear formula ($150·\text{SqFt} + 10000·\text{Bedrooms} − 500·\text{Age} + \text{zip effect} + \text{noise}$), and the entire before/after comparison rests on a single train_test_split(..., random_state=42).

In 200–400 words, argue a clear position: is the article’s R² improvement trustworthy enough to report to stakeholders, or is it inflated? Pick a side (defend / attack / conditional) and defend it.

Deliverable: A stakeholder memo (200–400 words) with a one-line position up top, then the case.

Rubric:

• States an explicit position (defend / attack / conditional) in the first sentence.
• Names at least 3 distinct threats to validity beyond the leakage bug — e.g. (a) single train/test split with no cross-validation, (b) the target was synthesized from a known linear formula the model is essentially rediscovering, (c) feature_importances_ is impurity-based and biased toward high-cardinality numeric features like Zip_Encoded.
• References at least 2 concrete numbers from the article (n_samples=1000, test_size=0.2 → 200 test rows, the $150/sqft coefficient, the four zip codes, etc.).
• Proposes at least one concrete validation step the team should run before reporting the lift — e.g. k-fold CV with category_encoders.TargetEncoder fit inside each fold, a leakage-free baseline, or a permutation-importance cross-check.
• Stays under 400 words; memo tone, not lab-report tone.