Ensemble Stacking — Combining Models the Right Way

Why One Model Isn’t Enough (and Why You Can’t Just Average Them)

Even the best machine learning models have bad days. Sam has seen this at HomeMatch. One model might nail prices for luxury mansions but stumble on small apartments. Another might handle suburban homes well yet get confused by city condos.

Every algorithm has blind spots. Linear Regression assumes straight-line relationships. A Decision Tree builds rigid boxes. Rely on just one, and you inherit its systematic errors. The obvious fix is to train several models and average their predictions.

So let’s try that. We’ll use the house price dataset from our previous feature engineering tutorial, featuring SqFt, Bedrooms, and House_Age.

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

# Recreating our running house price dataset
np.random.seed(42)
n_samples = 1200
data = {
    'SqFt': np.random.normal(2000, 500, n_samples),
    'Bedrooms': np.random.randint(1, 6, n_samples),
    'House_Age': np.random.randint(0, 70, n_samples),
    'Quality': np.random.uniform(1, 10, n_samples)
}
df = pd.DataFrame(data)
df['Price'] = (df['SqFt'] * 150) + (df['Bedrooms'] * 10000) - (df['House_Age'] * 500) + (df['Quality'] * 5000) + np.random.normal(0, 5000, n_samples)

X = df.drop('Price', axis=1)
y = df['Price']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train 3 diverse base models
model_lr = LinearRegression().fit(X_train, y_train)
model_rf = RandomForestRegressor(random_state=42).fit(X_train, y_train)
model_gb = GradientBoostingRegressor(random_state=42).fit(X_train, y_train)

# Naive Averaging
pred_lr = model_lr.predict(X_test)
pred_rf = model_rf.predict(X_test)
pred_gb = model_gb.predict(X_test)
pred_avg = (pred_lr + pred_rf + pred_gb) / 3

print(f"Linear Regression R2: {r2_score(y_test, pred_lr):.4f}")
print(f"Random Forest R2: {r2_score(y_test, pred_rf):.4f}")
print(f"Naive Average R2: {r2_score(y_test, pred_avg):.4f}")

• np.random.seed(42) fixes the random number generator so the synthetic house-price dataset is reproducible across runs — important when comparing the naive average against full stacking later.
• np.random.normal(2000, 500, n_samples) generates 1,200 square-footage values from a normal distribution with mean 2,000 and standard deviation 500.
• np.random.randint(1, 6, n_samples) generates bedroom counts between 1 and 5 — randint excludes the upper bound, so the range is [1, 5].
• np.random.randint(0, 70, n_samples) generates house ages between 0 and 69.
• np.random.uniform(1, 10, n_samples) generates quality scores between 1 and 10 — a continuous scale representing build quality or amenities.
• df['Price'] = ... constructs the target using a hidden linear formula: $150 per square foot, $10,000 per bedroom, minus $500 per year of age, plus $5,000 per quality point, plus Gaussian noise — none of the models know this formula and must discover the relationships from the data.
• df.drop('Price', axis=1) drops the target column to create the feature matrix X — axis=1 means “drop a column,” not a row.
• train_test_split(..., test_size=0.2, random_state=42) splits the data 80% train (960 rows) / 20% test (240 rows) with a fixed seed so the same rows go to train and test every run.
• Three diverse base models are fit: LinearRegression (captures global linear trends), RandomForestRegressor (non-linear tree ensemble), and GradientBoostingRegressor (sequential error-correcting trees) — diversity is key to stacking.
• (pred_lr + pred_rf + pred_gb) / 3 is the naive average — equal weight to all three models regardless of which is better in which price range.
• r2_score(y_test, pred) computes the R² (coefficient of determination) on the held-out test set for each model and for the average — 1.0 is perfect, 0.0 means the model just predicts the mean.

The naive average usually beats the worst model, but it rarely outperforms the best one by much. Simple averaging treats every model as equally capable. A Linear Regression might be much better at handling large houses, while a Random Forest excels on old houses. Equal weighting wastes that specific expertise.

Stacking tackles this directly: What if we let a second model learn when to trust each base model?

The Stacking Architecture: Base Models, Meta-Learner, and Holdout Data

Stacking isn’t just a model; it’s a hierarchy. Think of it as a corporate structure.

1. Layer 1 (The Workers): These are your Base Models. You train several different models (like our LR and RF above) on your training data.
2. Layer 2 (The Manager): This is the Meta-Learner. Instead of looking at the original features (SqFt, Bedrooms), the Manager only looks at the predictions made by the Workers.
3. The Input: The Manager’s “features” are the outputs of Layer 1.

Here’s the tricky part—and the hardest to get right. You can’t train the Manager on the same data the Workers used. If you do, the Manager will see that the Workers “memorized” the training answers and trust them too much. That’s overfitting.

To fix this, we need a Holdout Set. We train the Workers on Part A of the data, let them predict on Part B, then train the Manager on those Part B predictions.

Data Leakage: The Silent Killer of Stacking

Using “in-sample” predictions—predictions on data the model already saw during training—commits Data Leakage.

Picture a student who glimpses the exam answers beforehand and scores 100%. A teacher (the Meta-Learner) watching this would think the student is brilliant and trust them with everything. But on the real final exam (the Test Set), that student fails. They didn’t learn—they memorized.

Here’s what that looks like in code:

# WRONG WAY: Training meta-learner on training predictions
train_preds_rf = model_rf.predict(X_train)
train_preds_gb = model_gb.predict(X_train)

# The meta-learner 'memorizes' how well RF did on data it already saw
meta_X_bad = np.column_stack([train_preds_rf, train_preds_gb])
meta_model_bad = LinearRegression().fit(meta_X_bad, y_train)

# Test performance will likely be worse than a single model
test_preds_meta = meta_model_bad.predict(np.column_stack([pred_rf, pred_gb]))
print(f"Leaky Stacking R2: {r2_score(y_test, test_preds_meta):.4f}")

• model_rf.predict(X_train) generates predictions on the training data — because the Random Forest has already seen these rows, its predictions will be unrealistically accurate (it effectively memorized the answers).
• np.column_stack([train_preds_rf, train_preds_gb]) stacks the two in-sample prediction arrays as columns to form the meta-learner’s feature matrix — each row is [RF_pred, GB_pred] for one house, but both columns are “contaminated” by memorization.
• LinearRegression().fit(meta_X_bad, y_train) trains the meta-learner to map base-model predictions to the true price — but because the base-model predictions are in-sample, the meta-learner learns to trust them far too much.
• meta_model_bad.predict(np.column_stack([pred_rf, pred_gb])) applies the leaky meta-learner to the test set — the test predictions themselves are fine (out-of-sample), but the meta-learner’s weights were calibrated on garbage, so the mapping will collapse on unseen data.

So your meta-learner is built on a lie. Performance looks great in the training logs, then collapses the moment it meets a new house.

Cross-Validation Stacking: The Right Way to Use All Your Data

To avoid leakage without wasting half our data on a holdout set, we use K-Fold Stacking.

We split the training data into, say, 5 chunks (folds). Train the base models on 4 of them, predict on the 5th. Repeat until every row in the training set has an “out-of-sample” prediction. Now the Meta-Learner can train on the entire dataset safely—each prediction it sees came from a model that hadn’t seen that specific row.

Choosing Your Base Models: Diversity Over Accuracy

A stack of five identical Random Forests is useless. You’d just get the same answer five times—like asking five people who went to the same school and read the same book.

Stacking thrives on Diversity. You want models that make different mistakes.

• Model A: A Linear Model (good at global trends).
• Model B: A Tree-based Model (good at non-linear interactions).
• Model C: A K-Nearest Neighbors model (good at local patterns).

If the correlation between Model A’s errors and Model B’s errors is low, the Meta-Learner has something to work with. It can say, “When Model B says the price is high but Model A says it’s low, Model A is usually right.”

Choosing Your Meta-Learner: Simple Often Wins

What should the “Manager” model be? A complex Neural Network is tempting, but Logistic Regression (for classification) or Ridge Regression (for regression) is usually the better call.

Why? The Meta-Learner only sees a handful of features — one per base model. A complex model overfits almost immediately at that scale. A simple linear model just finds the right weight for each worker.

The Meta-Learner’s Input/Output Relationship

The meta-learner receives the $K$ base-model predictions as its only features and outputs a single weighted combination:

$\hat{y}_{\text{meta}}(\mathbf{x}) = w_0 + \sum_{k=1}^{K} w_k \, \hat{y}_k(\mathbf{x})$

where $\hat{y}_k(\mathbf{x})$ is the prediction of base model $k$ for input $\mathbf{x}$, $w_k$ is the weight the meta-learner assigns to model $k$, and $w_0$ is the intercept. With Ridge regularization (the recommended meta-learner), the weights $w_k$ are shrinkage-constrained to avoid over-trusting any single base model.

Plain English	Statistical symbol	Python equivalent
Base model $k$‘s prediction for input $\mathbf{x}$	$\hat{y}_k(\mathbf{x})$	model_k.predict(x)
Meta-learner’s intercept	$w_0$	meta_learner.intercept_
Meta-learner’s weight for base model $k$	$w_k$	meta_learner.coef_[k]
Final stacked prediction	$\hat{y}_{\text{meta}}(\mathbf{x})$	stack_model.predict(x)
Number of base models	$K$	len(base_models)

Stacking in Practice: A Complete Example

Let’s build a proper, leak-free stack using scikit-learn. Our base models: a Random Forest, a Gradient Booster, and a Linear Regression. A Ridge Regressor manages the ensemble.

from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import Ridge

# Define the 'Workers'
base_models = [
    ('rf', RandomForestRegressor(n_estimators=50, random_state=42)),
    ('gb', GradientBoostingRegressor(random_state=42)),
    ('lr', LinearRegression())
]

# Define the 'Manager'
meta_learner = Ridge()

# The StackingRegressor handles the K-Fold logic for us automatically!
stack_model = StackingRegressor(
    estimators=base_models,
    final_estimator=meta_learner,
    cv=5
)

stack_model.fit(X_train, y_train)
stack_preds = stack_model.predict(X_test)

print(f"Final Stacking R2: {r2_score(y_test, stack_preds):.4f}")
print(f"Improvement over best single model: {r2_score(y_test, stack_preds) - r2_score(y_test, pred_gb):.4f}")

• base_models = [(...)] defines the list of base models as named tuples — each entry is (name, estimator), and StackingRegressor uses the name to track and fit each model internally.
• RandomForestRegressor(n_estimators=50, random_state=42) uses 50 trees (reduced from the default 100 for speed) — fewer trees are fine here because the stack combines multiple models, so each individual model needn’t be maximally tuned.
• Ridge() is the meta-learner — Ridge regression adds L2 regularization, which shrinks the learned weights $w_k$ toward zero and discourages the meta-learner from over-trusting any single base model (the same principle Sam learned in the regularization tutorial).
• StackingRegressor(estimators=..., final_estimator=..., cv=5) wires everything together — cv=5 tells scikit-learn to use 5-fold cross-validation to generate out-of-sample predictions for the meta-learner, automatically avoiding the leakage problem from the “WRONG WAY” block.
• stack_model.fit(X_train, y_train) fits all base models using internal K-fold cross-validation (each model trains on 4/5 of the data and predicts on the held-out 1/5), then fits the meta-learner on those out-of-fold predictions — all in one call.
• stack_model.predict(X_test) runs all base models on the test set, feeds their predictions to the meta-learner, and returns the meta-learner’s final output.
• r2_score(y_test, stack_preds) - r2_score(y_test, pred_gb) computes the R² gain of the stacked ensemble over the best single base model (Gradient Boosting) — a positive value means stacking found exploitable patterns in the base models’ errors.

A positive improvement (e.g., 0.015) means the stacking model found patterns in the individual models’ errors. Even a 1% gain can be worth millions in high-stakes industries like finance.

Common Pitfalls and When to Use Stacking

The Pitfalls:

1. Too much complexity: Similar base models mean stacking just burns compute for no real gain.
2. Inference Latency: Production means running all base models plus the meta-learner. Need a prediction in 10 milliseconds? Stacking might be too slow.

When to use it:

• When accuracy matters above all else (think Kaggle competitions).
• When you have diverse models that score similarly but disagree on specific cases.
• When you have enough data for proper cross-validation.

Naive averaging vs. weighted averaging vs. full stacking

Not every project needs a full meta-learner. Here’s the complexity/payoff ladder for a small team like Sam’s at HomeMatch:

Approach	Complexity	When it wins	When it falls short
Naive averaging (equal-weight mean)	Trivial — one line of code	Quick baseline when base models are similarly accurate and their errors are uncorrelated; great sanity check before investing in stacking	Treats every model as equally smart; can’t adapt to “trust LR on mansions, RF on starter homes”; often barely beats the best single model
Weighted averaging (manually tuned weights)	Low — tune a few weights on a holdout set	When you can characterize each model’s strengths (“RF is better on older houses”) but don’t have enough data or time for full K-fold stacking	Still a fixed linear combination — can’t learn conditional trust (“use model A when SqFt > 3000”); requires a clean holdout set to tune weights without leakage
Full stacking (meta-learner on K-fold CV predictions)	High — cross-validation machinery + extra model + careful pipeline	Accuracy is the top priority; diverse base models with low error correlation; enough data for reliable 5-fold CV; Kaggle-style competitions	Slow inference (all base models + meta-learner must run per request); added deployment complexity; overkill for a small team’s first production model

Sam’s rule of thumb: Start with naive averaging as a baseline — if it doesn’t beat your best single model, stacking probably won’t either (your models aren’t diverse enough). Move to weighted averaging if you can quantify each model’s regional strengths. Reserve full stacking for the final production model where every fraction of a percent matters and you can afford the inference cost.

Wrapping Up the Series

What we covered:

• Intuition: A meta-learner weighs the opinions of diverse base models.
• The danger: Data leakage creeps in if the meta-learner sees “in-sample” predictions.
• The fix: K-fold cross-validation generates out-of-sample predictions for the meta-learner.
• Diversity: Pick base models that make different types of mistakes.

Stacking is the “final boss” of classical machine learning — and it’s where Sam’s journey ends. He started by learning to balance bias and variance on a simple toy problem; now he ships a stacked ensemble for HomeMatch’s real sellability model. It’s built on gradient descent, kept honest by regularization, and chosen from among the boosting libraries. He worked through it one tree at a time, strengthened by forests and boosting, sharpened by SVMs, made robust against high dimensions. A cheap Naive Bayes side tool helps with the spam inbox; scaling made it fair. Feature engineering sharpened it further. And finally, a meta-learner combines it all — one that knows which model to trust when.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the two layers in a Stacking architecture, and what does each layer look at?

Understand In your own words, explain why training the Meta-Learner on “in-sample” predictions (predictions the base models made on their own training data) causes Data Leakage, using the article’s exam-answers analogy.

Apply Using the article’s improvement formula (stacking R2 - best single model R2), if the stacking model scores 0.87 and the best single base model (Gradient Boosting) scores 0.855, what is the improvement, and does the article’s example threshold (“even a 1% gain can be worth millions”) apply here?

Analyze The article says K-Fold Stacking trains base models “on 4 chunks and predicts on the 5th,” repeating until every row has an out-of-sample prediction. Walk through why this specifically avoids the leakage problem shown in the “WRONG WAY” code block, without requiring a separate holdout set that would waste data.

Evaluate The article recommends a simple model (Ridge or Logistic Regression) as the Meta-Learner, warning that a complex model “will quickly overfit to these few features.” Critique this reasoning: given the Meta-Learner only sees a handful of base-model predictions as its input features, why exactly does its input being narrow (not the underlying problem being simple) make a complex Meta-Learner risky here?

Create Design a 3-model stack for a different problem: predicting customer lifetime value from behavioral and demographic data. Choose three genuinely diverse base models (justify diversity using the article’s “different mistakes” criterion) and a meta-learner, and note one pitfall from the article’s list that would be most likely to bite this specific stack in production.

---

Related articles

• P11: Feature Engineering for Tabular Data — The Techniques That Actually Move the Needle — the previous part where Sam enriched the house-price features (ratios, target encoding, cyclical time) before combining models in this finale.
• P01: Can You Explain the Bias-Variance Tradeoff in Plain English? — where Sam’s journey began, balancing a too-simple model against a too-complex one; this finale closes the loop by combining many of those models into a single stack.

References & Further reading

• Wolpert, D. H. (1992). Stacked Generalization. Neural Networks, 5(2), 241–259. — the foundational paper that introduced stacking as a formal method for combining multiple learning algorithms via a meta-learner trained on out-of-sample cross-validated predictions.
• Kaggle: House Prices — Advanced Regression Techniques — the canonical regression competition where stacked ensembles (typically combining XGBoost, LightGBM, and a linear model) dominate the top of the leaderboard on real Ames, Iowa housing data.
• scikit-learn: StackingRegressor documentation — official API reference for the StackingRegressor and StackingClassifier used in this tutorial, including the cv parameter that handles K-fold out-of-sample prediction automatically.

---