In our last lesson, Sam saw how Gradient Descent helps a model walk down the mountain of error to find the right settings. There’s a catch. Sometimes the model gets too good at this descent. It learns a path so specific to the training data that it fails on a new mountain. This is overfitting. Today, we’ll learn to fix it using Regularization.

1. The ‘Over-Eager’ Learner: Why Models Try Too Hard

Think of a model as a student preparing for a math test. A good student learns the rules of addition and subtraction. An over-eager student just memorizes every practice question and answer in the textbook.

On test day, if the teacher asks something from the book, that student gets it 100% right. Change a single number, though, and they fail. They didn’t learn math. They learned the noise of the textbook.

Models are lazy by default. They want to drive error to zero as fast as they can. Give a complex model a small dataset. It will find a ‘wiggly’ line that passes through every data point perfectly, even when those points are just random coincidences. This is the Complexity Tax: we need a way to punish the model for being too ‘wiggly.’

Here’s what happens when a standard Linear Regression tries too hard on a tiny, noisy dataset of ice cream sales vs. temperature.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# Create a small, noisy dataset
np.random.seed(42)
X = np.sort(5 * np.random.rand(10, 1), axis=0)
y = 2 * X.ravel() + np.random.randn(10) * 2

# Fit a 'complex' model (Polynomial) to show overfitting
poly = PolynomialFeatures(degree=9)
X_poly = poly.fit_transform(X)
model = LinearRegression().fit(X_poly, y)

# Predict over a smooth range
X_test = np.linspace(0, 5, 100).reshape(-1, 1)
y_pred = model.predict(poly.transform(X_test))

print(f"Model Coefficients: {model.coef_[1:4]} ... (very large numbers)")
# The coefficients are huge (e.g., 500 or -1000). 
# This means the model is reacting violently to tiny changes in X.

• np.random.seed(42) fixes the random number generator so the same “random” data is produced every run — essential for reproducible tutorials.
• X = np.sort(5 * np.random.rand(10, 1), axis=0) generates 10 random values in [0, 5) and sorts them so the plot line connects left-to-right; the shape (10, 1) makes X a column vector.
• PolynomialFeatures(degree=9) expands a single feature x into 10 features: [1, x, x², …, x⁹] — a degree-9 polynomial with 10 coefficients can wiggle through all 10 points perfectly, which is exactly the overfitting we want to demonstrate.
• model.coef_[1:4] prints coefficients 1 through 3 (skipping the intercept term at index 0) to show they’re enormous — a sure sign the model is overreacting to noise.

What’s actually going on? The model is using massive weights to force the line through every noisy dot. To fix this, we tell the model: “You can minimize error, but you have to keep your weights small.”

2. L2 Regularization (Ridge): The Gentle Brake

Ridge Regression (also called L2) is like putting a speed limiter on a car. It doesn’t tell the car where to go. It just keeps it from hitting 200 mph.

Ridge adds a penalty to the error score — the square of the weights. A weight of 1 costs 1. A weight of 10 costs 100. Squaring makes big numbers expensive, so the model keeps weights small and spread out.

Every feature stays in the model, but none gets to dominate. That wild, wiggly line becomes a smooth, sensible curve.

from sklearn.linear_model import Ridge

# Alpha is the 'strength' of the penalty
ridge_model = Ridge(alpha=1.0)
ridge_model.fit(X_poly, y)
y_ridge_pred = ridge_model.predict(poly.transform(X_test))

print(f"Ridge Coefficients: {ridge_model.coef_[1:4]}")
# These numbers are much smaller (close to 1 or 2).
# The 'wiggle' is gone because the model couldn't afford the 'cost' of big weights.

• Ridge(alpha=1.0) creates a Ridge regression model; alpha is the regularization strength — higher means a heavier penalty on large weights.
• ridge_model.fit(X_poly, y) fits the model on the same polynomial-expanded features, but now the L2 penalty constrains the coefficients during fitting.
• ridge_model.coef_[1:4] shows the first few non-intercept coefficients are now small (near 1 or 2) instead of the hundreds or thousands seen in the unregularized model — the “speed limiter” is working.

3. L1 Regularization (Lasso): The Ruthless Editor

If Ridge is a gentle brake, Lasso (L1) is a pair of scissors.

Instead of squaring the weights, Lasso adds the absolute value of the weights to the error. The math difference sounds small. The result is stark: Lasso sets weights to exactly zero.

Say you have 100 variables but only 5 matter. Lasso zeroes out the other 95. That’s built-in feature selection, no extra step.

The Catch: two similar variables — like ‘Square Footage’ and ‘Number of Rooms’ — and Lasso might keep one at random and drop the other. A ruthless editor.

from sklearn.linear_model import Lasso

lasso_model = Lasso(alpha=0.1, max_iter=10000)
lasso_model.fit(X_poly, y)

# Count how many coefficients are exactly zero
zeros = np.sum(lasso_model.coef_ == 0)
print(f"Lasso set {zeros} features to exactly zero.")
# This means Lasso decided those 'wiggly' polynomial features weren't worth the cost.

• Lasso(alpha=0.1, max_iter=10000) creates a Lasso model; alpha=0.1 is a modest penalty, and max_iter=10000 raises the maximum iterations because Lasso’s coordinate-descent solver sometimes needs more steps to converge.
• np.sum(lasso_model.coef_ == 0) counts how many coefficients are exactly zero — Lasso’s absolute-value penalty creates a geometric “corner” at zero that the optimization can land on precisely, unlike Ridge which only approaches zero asymptotically.

The regularized loss functions for Ridge and Lasso add a penalty to the standard sum of squared errors:

Ridge (L2) loss:

$L_{\text{ridge}} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^{p} w_j^2$

Lasso (L1) loss:

$L_{\text{lasso}} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^{p} |w_j|$

Elastic Net loss (blend of both):

$L_{\text{elastic}} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2 + \alpha \left[ \rho \sum_{j=1}^{p} |w_j| + (1-\rho) \sum_{j=1}^{p} w_j^2 \right]$

Plain English	Statistical symbol	Python equivalent
Sum of squared errors (data fit)	$\sum (y_i - \hat{y}_i)^2$	np.sum((y - y_pred)**2)
L2 penalty — sum of squared weights	$\alpha \sum w_j^2$	alpha * np.sum(w**2)
L1 penalty — sum of absolute weights	$\alpha \sum \|w_j\|$	alpha * np.sum(np.abs(w))
Regularization strength	$\alpha$	alpha
L1/L2 mixing ratio (Elastic Net)	$\rho$ (rho)	l1_ratio

4. Elastic Net: The Best of Both Worlds

Here’s the thing: what if you want the grouping stability of Ridge but the editing power of Lasso?

Elastic Net is just a mix of the two. It uses a slider (l1_ratio) to set how much of each penalty applies.

• l1_ratio at 1.0 means pure Lasso.
• l1_ratio at 0.0 means pure Ridge.
• 0.5 is a 50/50 split.

Use Elastic Net on large datasets with many correlated variables. It keeps related variables grouped together while still trimming the junk.

from sklearn.linear_model import ElasticNet
import pandas as pd

en_model = ElasticNet(alpha=0.1, l1_ratio=0.5)
en_model.fit(X_poly, y)

# Compare the 'strength' of the first few coefficients
results = pd.DataFrame({
    'Ridge': ridge_model.coef_[:5],
    'Lasso': lasso_model.coef_[:5],
    'ElasticNet': en_model.coef_[:5]
})
print(results)
# You'll see ElasticNet is usually 'between' the other two.

• ElasticNet(alpha=0.1, l1_ratio=0.5) creates a model that blends L1 and L2 penalties; l1_ratio=0.5 means 50% Lasso and 50% Ridge — you can slide it from 0 (pure Ridge) to 1 (pure Lasso).
• The pd.DataFrame compares the first 5 coefficients across all three models side by side, making it easy to see that ElasticNet’s values typically fall between Ridge’s (larger) and Lasso’s (smaller or zero).

5. How to Pick Your ‘Alpha’: The Tuning Secret

How much regularization do you actually need? The alpha parameter controls the dose.

• Alpha = 0: Standard Linear Regression, no penalty. Overfitting is almost certain.
• Alpha = 1,000,000: The penalty is so high that every weight gets crushed to zero. The model learns nothing.

We’re after the Goldilocks Zone. The standard approach: try a range of alphas (say, 0.01, 0.1, 1, 10) and check which one performs best on validation data.

alphas = [0.001, 0.1, 10]
for a in alphas:
    m = Ridge(alpha=a).fit(X_poly, y)
    # As 'a' goes up, the line gets flatter and 'safer'.
    print(f"Alpha {a}: Sum of weights is {np.sum(np.abs(m.coef_)):.2f}")

• for a in alphas loops through three regularization strengths spanning four orders of magnitude (0.001 → 0.1 → 10) to show how the penalty constrains the model.
• np.sum(np.abs(m.coef_)) computes the L1 norm of the weight vector — as alpha increases, this sum shrinks because the model can’t “afford” large weights, making the fitted line progressively flatter and less prone to overfitting.

So, how do you choose between Ridge, Lasso, and Elastic Net? Here is the rule of thumb:

• Use Ridge (L2) when you want all features to stay in the model but none should dominate — it shrinks weights gently and spreads credit across correlated variables. The tradeoff: no feature selection, so you keep noisy variables that just have small weights.
• Use Lasso (L1) when you suspect many features are useless and want automatic feature selection — it zeroes out irrelevant weights entirely. The tradeoff: with highly correlated features (like ‘Square Footage’ and ‘Number of Rooms’), it picks one and drops the rest arbitrarily, which can be unstable.
• Use Elastic Net when you have both useful and useless features AND many correlated groups — it combines Lasso’s pruning with Ridge’s grouping stability. The tradeoff: you now tune two hyperparameters (alpha and l1_ratio) instead of one.

What we learned today:

• Overfitting happens when a model memorizes noise instead of patterns.
• Regularization adds a penalty to the model’s error based on weight size.
• Ridge (L2) shrinks all weights gently, keeping every feature in play.
• Lasso (L1) zeros out the useless weights and drops them entirely.
• Elastic Net blends both approaches, which tends to suit complex, real-world data well.

With overfitting under control, Sam is ready to move beyond straight lines and try a fundamentally different family of models — gradient-boosted trees — in the next part of our series.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the key mechanical difference between L1 (Lasso) and L2 (Ridge) regularization—squaring the weights versus taking their absolute value—and what practical effect does that difference have?

Understand In your own words, explain why Lasso can set a weight to exactly zero while Ridge only ever shrinks weights close to (but not exactly) zero.

Apply Using the article’s penalty formulas, if a model has two weights, w1 = 2 and w2 = 10, calculate the L2 penalty (sum of squared weights) and the L1 penalty (sum of absolute weights) for this pair, and compare how much more the L2 penalty “punishes” the larger weight relative to L1.

Analyze The article warns that Lasso “might randomly pick one to keep and delete the other” when two variables are highly correlated (like Square Footage and Number of Rooms). Walk through why this instability specifically happens with Lasso’s absolute-value penalty and not with Ridge’s squared penalty.

Evaluate The article frames alpha tuning as finding a “Goldilocks Zone” by testing a handful of values (0.01, 0.1, 1, 10) against validation data. Critique this approach: what could go wrong if you only tested those four specific values instead of a denser range, and how would you know if you’d missed the true optimum?

Create Design a regularization choice for a new scenario: a marketing dataset with 500 highly correlated ad-spend features (many nearly identical spend categories) and only 200 rows of data. Which of Ridge, Lasso, or Elastic Net would you pick, and justify your choice using the tradeoffs described in the article.

---

Related articles

• P02: How Gradient Descent Actually Works — where Sam learned how models “walk downhill” to find the best weights, the prerequisite for understanding what regularization constrains.
• P04: XGBoost vs LightGBM vs CatBoost — the next step for Sam: trading linear models for gradient-boosted trees, a fundamentally different family of learners.

References & Further reading

• Tibshirani, R. (1996). “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society: Series B, 58(1), 267–288. — the foundational paper that introduced Lasso (L1 regularization) and proved its feature-selection properties.
• Hoerl, A. E., & Kennard, R. W. (1970). “Ridge Regression: Biased Estimation for Nonorthogonal Problems.” Technometrics, 12(1), 55–67. — the seminal paper that introduced Ridge regression (L2 regularization) and showed that a small amount of bias can dramatically reduce variance.
• Kaggle: House Prices — Advanced Regression Techniques — a competition where Ridge, Lasso, and Elastic Net are standard tools for preventing overfitting on a tabular housing dataset.
• Scikit-learn documentation: Ridge & Lasso — production-grade implementations with full parameter references.

---