Reference: Regularization

Why models overfit at all

A model overfits when it learns patterns that exist in the training set but not in the underlying data-generating process. The symptom is familiar: training loss keeps falling, validation loss plateaus or rises, and the gap between the two grows. The cause is that any model with enough capacity will eventually start memorizing the noise in your particular training sample — and noise, by definition, does not generalize.

The classical framing is the bias-variance decomposition. For a target $y = f(x) + \epsilon$ with noise variance $\sigma^2$, the expected test squared error of an estimator $\hat f$ splits into three pieces: a bias term (how wrong the average prediction is), a variance term (how much the prediction wobbles across different training sets), and the irreducible noise $\sigma^2$. A simple model has high bias but low variance — it underfits. A complex model has low bias but high variance — it overfits. Regularization is the umbrella term for every trick that adds bias in exchange for a larger reduction in variance, sliding you along the U-shaped test-error curve toward its minimum.

The rest of this reference catalogs the most common regularizers, what each one costs, and when each is the right tool.

Roster of regularizers

Regularizer	One-line definition	Typical range	When to use	Used in corpus
L2 (Ridge)	Add $\lambda\lVert w\rVert_2^2$ to the loss	$\lambda \in [10^{-5}, 10]$	Default for linear models; basis of weight decay in NNs	classical-ml-p03, lora-and-qlora
L1 (Lasso)	Add $\lambda\lVert w\rVert_1$ to the loss	$\lambda \in [10^{-4}, 1]$	When you need sparse, interpretable features	classical-ml-p03
ElasticNet	$\lambda\big[\rho \cdot \text{L1} + (1-\rho) \cdot \text{L2}\big]$	penalty $\lambda \in [10^{-5}, 10]$; mixing weight $\rho$ (l1_ratio) $\in [0,1]$	Correlated features where L1 is too aggressive	classical-ml-p03
Dropout	Zero out a $p$ fraction of activations during training	$p \in [0.2, 0.5]$	Deep nets, especially wide fully connected layers	attention-p11
BatchNorm	Normalize each feature to mean 0, var 1 across the batch	momentum $\approx 0.9$	CNNs and deep nets with a comfortable batch size	attention-p11
LayerNorm	Normalize across features per example, not across batch	—	Transformers, RNNs, batch-size-1 settings	attention-p03
Early stopping	Stop when val loss has risen for $k$ epochs	$k \in [3, 20]$	Universal; essentially free; costs only a validation split	learning-curves
Weight decay	L2 penalty applied through the optimizer (AdamW)	$10^{-4}$ to $10^{-2}$	Deep nets, especially transformers and large fine-tunes	lora-and-qlora
Data augmentation	Transform inputs, keep labels, train on the union	varies by modality	Vision, audio, increasingly text; data-scarce regimes	—

Which regularizer for which architecture

Pick the regularizer by what your model is, not by what is currently fashionable:

• Linear / logistic regression
  • Need interpretable sparse features? → L1 (Lasso) — it will zero out the irrelevant ones.
  • Just want to shrink coefficients and stabilize? → L2 (Ridge) — shrinks coefficients smoothly without zeroing them; a safe choice when you don’t need sparsity.
  • Features are correlated and L1 alone is unstable? → ElasticNet at mixing weight $\rho \approx 0.5$.
• Tree-based models (XGBoost, LightGBM, RandomForest)
  • Don’t reach for L1/L2 — they don’t apply to trees.
  • Control depth: max_depth, min_samples_leaf.
  • Control per-tree randomness: subsample, colsample_bytree.
  • Slow the ensemble down: lower learning_rate, raise n_estimators proportionally.
• Deep neural networks
  • Overfitting concentrated in wide FC layers? → Dropout at $p = 0.3$–$0.5$.
  • Activations exploding or training is unstable? → BatchNorm (CNN) or LayerNorm (Transformer).
  • Still overfitting after dropout + normalization? → Early stopping + weight decay (via AdamW).
  • Very little training data? → Data augmentation.
• Every architecture, for free
  • Early stopping — set patience to ~10 epochs, log val loss every epoch, ship the checkpoint with the lowest val loss.

L1, L2, and ElasticNet — penalty-based regularizers

These three are the oldest and best-understood regularizers. The recipe is the same in every case: take the original training objective $\mathcal{L}(w)$ and add a penalty on the size of the weights.

L2 (Ridge). The penalty is $\lambda \lVert w \rVert_2^2 = \lambda \sum_i w_i^2$. Because the penalty is smooth, every weight gets pulled gently toward zero — but rarely all the way there. L2 produces small weights, not zero weights. It is the default regularizer for linear regression and the conceptual basis of weight decay in deep learning.

L1 (Lasso). The penalty is $\lambda \lVert w \rVert_1 = \lambda \sum_i |w_i|$. The kink at $w_i = 0$ in $|w_i|$ changes the optimization geometry enough that the optimal solution frequently has entire coefficients pinned exactly at zero. This is the sparsity property: L1 doubles as automatic feature selection. Use it when you have many features and only a few matter.

ElasticNet. A convex blend $\rho \cdot \text{L1} + (1-\rho) \cdot \text{L2}$, scaled by an overall penalty strength $\lambda$ — the same $\lambda$ knob as Ridge and Lasso (sklearn’s alpha). When features are correlated, L1 has a habit of picking one and zeroing the other arbitrarily; L2 keeps both shrunk but nonzero. ElasticNet splits the difference. Set the mixing weight $\rho$ (sklearn’s l1_ratio) near 1 for sparsity-dominant, near 0 for ridge-dominant — don’t confuse it with the penalty strength $\lambda$, which is a separate argument.

The Ridge objective and its gradient:

$\mathcal{L}_{\text{ridge}}(w) = \mathcal{L}(w) + \lambda \sum_i w_i^2, \qquad \nabla \mathcal{L}_{\text{ridge}} = \nabla \mathcal{L}(w) + 2\lambda w$

The $2\lambda w$ is the whole of what L2 adds to the gradient. Stepping through where it comes from, with the data-fit term $\mathcal{L}$ left unspecified because the penalty does not care what it is:

$$\frac{\partial}{\partial w} \left(\lambda w^{2} + L\right)$$

the derivative we want: how does this move when $w$ moves?

$$\term{focus}{\frac{\partial}{\partial w} \left(\lambda w^{2}\right) + \frac{\partial}{\partial w} L}$$

sum rule: differentiate each term separately

$$\term{focus}{\lambda \frac{\partial}{\partial w} w^{2}} + \frac{\partial}{\partial w} L$$

constant multiple rule: pull the constant out front

$$\lambda \term{focus}{2 w \frac{\partial}{\partial w} w} + \frac{\partial}{\partial w} L$$

power rule on the outside, chain rule on the inside

$$2 \lambda w \term{focus}{1} + \frac{\partial}{\partial w} L$$

the derivative of $w$ with respect to itself is $1$

$$2 \lambda w + \frac{\partial}{\partial w} L$$

so the derivative is

The unresolved $\frac{\partial}{\partial w} L$ in the final line is the point: L2 leaves your original gradient alone and adds a term proportional to the weight, which is why it shrinks large weights harder than small ones.

The Lasso objective and its subgradient (the absolute value is non-differentiable at zero, so we use a subgradient):

$\mathcal{L}_{\text{lasso}}(w) = \mathcal{L}(w) + \lambda \sum_i |w_i|, \qquad \partial \mathcal{L}_{\text{lasso}} = \nabla \mathcal{L}(w) + \lambda \, \text{sign}(w)$

Why L1 is sparse but L2 is not. Picture the unregularized loss as a set of concentric ellipses in weight space, centered at the unregularized solution. The L2 constraint $\lVert w \rVert_2^2 \le c$ is a sphere; the L1 constraint $\lVert w \rVert_1 \le c$ is a diamond with corners on the axes. The smallest ellipse that touches the sphere usually touches at a smooth, off-axis point — all coordinates nonzero. The smallest ellipse that touches the diamond almost always hits a corner — and a corner is a point where at least one coordinate is zero. That is the entire geometric argument for sparsity.

Bias-variance decomposition. For reference, the three-way split that motivates everything in this file:

$\mathbb{E}\!\left[(y - \hat f)^2\right] = \big(\mathbb{E}[\hat f] - f\big)^2 + \mathbb{E}\!\left[\big(\hat f - \mathbb{E}[\hat f]\big)^2\right] + \sigma^2$

Plain English	Statistical symbol	Python equivalent
Mean squared prediction error	$\mathbb{E}[(y - \hat f)^2]$	mean_squared_error(y, y_pred)
Squared L2 norm of weights	$\lVert w\rVert_2^2 = \sum_i w_i^2$	(w ** 2).sum()
L1 norm of weights	$\lVert w\rVert_1 = \sum_i \lvert w_i\rvert$	np.abs(w).sum()
Regularization strength	$\lambda$ (alpha in sklearn)	Ridge(alpha=1.0)

Dropout, BatchNorm, and LayerNorm — structural deep-net regularizers

Once the model has nonlinear hidden layers, weight penalties alone do less work than you would expect. The next family of regularizers changes the architecture of training rather than the loss.

Dropout. During training, sample a random subset of activations in a layer and set them to zero with probability $p$ (typically $0.2$ to $0.5$). At inference, all activations are used, scaled to compensate. The effect is that no single neuron can become a load-bearing feature, because in any given step it might be switched off. The network learns redundant, distributed representations — and those generalize better. Dropout is most useful in wide fully connected layers; on conv layers it is usually redundant once BatchNorm is in place, and in transformers it has largely been displaced by LayerNorm plus weight decay.

BatchNorm. Normalize each feature channel across the batch to mean 0 and variance 1, then learn a per-channel scale and shift. The immediate effect is to stabilize training — gradients flow more cleanly and you can use larger learning rates. The regularizing effect is a side benefit: the per-batch statistics inject noise, which itself suppresses overfitting. BatchNorm needs a large enough batch (say 32+) for the statistics to be meaningful; with batch size 1, or with variable-length sequences, it stops being a good idea.

LayerNorm. Same idea, different axis: normalize across the features of a single example, not across the batch. This makes it batch-size-agnostic and the standard choice for transformers and RNNs. LayerNorm’s regularizing effect is weaker than BatchNorm’s — it is mostly there for training stability — but it does not break on the workloads where BatchNorm does.

Early stopping, weight decay, and data augmentation — the non-penalty family

The last group of regularizers does not touch the loss at all. They work by changing when you stop, how you update, or what you feed in.

Early stopping. Monitor a validation metric. Stop training the first time it gets worse for $k$ consecutive epochs (the patience), and roll back to the best checkpoint. It is the cheapest regularizer you have — no retraining, no hyperparameter sweep beyond patience, and it composes cleanly with everything else. The deeper reason it works is that gradient descent explores low-norm solutions first; the longer you train, the larger the effective norm of the weights. Early stopping is approximate L2 with a data-dependent $\lambda$.

Weight decay. In modern optimizers (AdamW, Adam with weight_decay set), weight decay is the deep-learning version of L2 — but the implementation matters. The original Adam coupled the L2 penalty into the gradient, which interacts badly with Adam’s adaptive moment scaling. AdamW decouples them: it shrinks the weights first by a factor $(1 - \eta \lambda)$, then applies the gradient step. Always use AdamW rather than Adam-with-L2 when you want weight decay in a transformer.

Data augmentation. Generate new training examples by applying label-preserving transformations: flips and crops for images, pitch shifts and noise for audio, paraphrase or back-translation for text. Augmentation is the only regularizer that increases the effective size of your dataset rather than constraining the model, which is why it is the first thing to reach for when data is scarce. It composes with everything else.

Worked Python: an overfit model and what each regularizer does to it

The first example uses a deliberately hostile linear-regression setup: 140 samples, 60 features, only 5 of which carry signal, with the noise cranked high enough relative to the informative coefficients that an unregularized fit genuinely overfits.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
from sklearn.datasets import make_regression

# 140 samples, 60 features, only 5 carry signal -- the rest are noise.
# noise=60 keeps the signal-to-noise ratio low enough, relative to this
# sample size and feature count, that OLS actually overfits.
X, y, true_coef = make_regression(
    n_samples=140, n_features=60, n_informative=5,
    noise=60, coef=True, random_state=7,
)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)

def fit_and_score(model, name):
    model.fit(X_train, y_train)
    tr = r2_score(y_train, model.predict(X_train))
    te = r2_score(y_test,  model.predict(X_test))
    nz = int(np.sum(np.abs(model.coef_) > 1e-6))
    print(f"{name:18s}  train R²={tr:.3f}  val R²={te:.3f}  nonzero coefs={nz}")

fit_and_score(LinearRegression(),                         "OLS (no reg)")
fit_and_score(Ridge(alpha=20.0),                          "Ridge (L2)")
fit_and_score(Lasso(alpha=10.0),                          "Lasso (L1)")
fit_and_score(ElasticNet(alpha=2.0, l1_ratio=0.9),        "ElasticNet")

Verified output of this exact run (scikit-learn, random_state=7):

OLS (no reg)        train R²=0.941  val R²=0.288  nonzero coefs=60
Ridge (L2)          train R²=0.897  val R²=0.547  nonzero coefs=60
Lasso (L1)          train R²=0.813  val R²=0.699  nonzero coefs=6
ElasticNet          train R²=0.877  val R²=0.638  nonzero coefs=47

The story is in the last two columns. OLS uses all 60 coefficients and overfits by 65 R² points (0.941 train vs. 0.288 validation). Ridge keeps all 60 coefficients but shrinks them at $\lambda=20$ — train drops, validation nearly doubles. Lasso, with its penalty raised to $\lambda=10$, throws away 54 of the 60 coefficients — close to the 5 that actually matter — and gets the best validation score of the four. ElasticNet (penalty 2, mixing weight 0.9) lands in between: 47 nonzero coefficients and a validation R² between Ridge’s and Lasso’s.

The second example moves to a small deep net and shows what dropout, weight decay, and early stopping each do to the train-vs-validation loss curve. The target now includes a nonlinear interaction term the network has to work to fit, the training set is a genuinely small 60 rows, and — critically — the DataLoader is built only from those 60 training rows, not the full dataset, so the validation split never leaks into training.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(7)
X = torch.randn(160, 20)
true_w = torch.zeros(20); true_w[:5] = torch.tensor([1.5, -2.0, 0.8, 3.0, -1.2])
# linear signal + a nonlinear interaction term + noise -- makes 60 training
# rows genuinely hard to fit without memorizing
y = X @ true_w + 1.5 * torch.sin(X[:, 0] * X[:, 1]) + 1.5 * torch.randn(160)

X_tr, y_tr, X_val, y_val = X[:60], y[:60], X[60:], y[60:]   # split BEFORE building the loader
loader = DataLoader(TensorDataset(X_tr, y_tr), batch_size=16, shuffle=True)  # loader sees only training rows

def make_model(p_drop=0.0):
    return nn.Sequential(
        nn.Linear(20, 128), nn.ReLU(), nn.Dropout(p_drop),
        nn.Linear(128, 128), nn.ReLU(), nn.Dropout(p_drop),
        nn.Linear(128, 1),
    )

def train(p_drop=0.0, epochs=150, lr=1e-3, weight_decay=0.0, patience=None):
    torch.manual_seed(7)   # identical init + batch order across regularizers, for a fair comparison
    model = make_model(p_drop)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=weight_decay)
    loss_fn = nn.MSELoss()
    tr_hist, val_hist = [], []
    best, best_ep, wait = float("inf"), 0, 0
    for ep in range(epochs):
        model.train()
        for xb, yb in loader:
            opt.zero_grad()
            loss = loss_fn(model(xb).squeeze(-1), yb)
            loss.backward(); opt.step()
        model.eval()
        with torch.no_grad():
            tr_loss  = loss_fn(model(X_tr ).squeeze(-1), y_tr ).item()
            val_loss = loss_fn(model(X_val).squeeze(-1), y_val).item()
        tr_hist.append(tr_loss); val_hist.append(val_loss)
        if val_loss < best - 1e-4:
            best, best_ep, wait = val_loss, ep, 0
        else:
            wait += 1
            if patience is not None and wait >= patience:
                break
    return tr_hist, val_hist, best, best_ep

for label, kwargs in [
    ("baseline",            dict()),
    ("dropout=0.3",         dict(p_drop=0.3)),
    ("weight_decay=2.0",    dict(weight_decay=2.0)),
    ("early_stop=10",       dict(patience=10)),
]:
    tr, val, best, best_ep = train(epochs=150, **kwargs)
    print(f"{label:22s}  final train={tr[-1]:.3f}  final val={val[-1]:.3f}  "
          f"epochs_run={len(tr)}  best_val={best:.3f} @ epoch {best_ep}")

Verified output of this exact run (PyTorch, torch.manual_seed(7)):

baseline                final train=0.000  final val=5.727  epochs_run=150  best_val=5.117 @ epoch 21
dropout=0.3             final train=0.097  final val=5.811  epochs_run=150  best_val=5.105 @ epoch 41
weight_decay=2.0        final train=0.007  final val=5.775  epochs_run=150  best_val=5.254 @ epoch 21
early_stop=10           final train=0.215  final val=5.411  epochs_run=32   best_val=5.117 @ epoch 21

Every run shows the same shape: validation loss falls to a minimum somewhere in the first 20–40 epochs while training loss keeps falling toward zero, then validation loss climbs back up as the net starts memorizing the 60-row training set — genuine overfitting. None of the three regularizers here reliably beats an untuned baseline’s own best checkpoint by a wide margin: dropout’s best (5.105) edges out the baseline’s (5.117) only slightly, and weight decay’s best (5.254) is actually a little worse. What early stopping buys you isn’t a better minimum than training finds on its own — its best checkpoint (5.117 at epoch 21) is identical to the baseline’s, since both follow the same trajectory up to that point — it’s stopping 118 epochs sooner, once validation has drifted for 10 straight epochs, so you don’t have to babysit a full run and hand-pick the checkpoint. Which regularizer helps most is specific to this data and network, not a universal ranking — see the closing edge case below for how to choose one.

The linear part. make_regression constructs a regression problem where n_informative=5 of the 60 features actually drive the target; the other 55 are pure noise, and noise=60 keeps the signal weak enough relative to 140 samples and 60 features that OLS has room to fit noise. The fit_and_score helper prints train R², validation R², and the count of coefficients above $10^{-6}$ in absolute value — that last number is what makes the L1 sparsity story visible.

The interesting comparison is between Ridge and Lasso. Ridge (alpha=20) keeps all 60 features nonzero but shrinks them, narrowing the train/validation gap from 65 R² points to 35. Lasso (alpha=10) zeroes 54 of the 60 coefficients and lands at the highest validation R² of the four — the sparsity property from the math toggle, made visible in numbers. ElasticNet (alpha=2, l1_ratio=0.9) zeroes fewer coefficients than Lasso and lands between Ridge and Lasso on validation R², which is its whole point: splitting the difference between L1’s aggressive zeroing and L2’s uniform shrinkage.

The deep part. make_model builds a 20-128-128-1 MLP with ReLU and (optionally) dropout between layers. The train function is a minimal training loop with two things worth flagging: it re-seeds torch.manual_seed(7) at the top of every call so each regularizer variant starts from the same initial weights and sees the same batch order — without that, differences between runs would be confounded with different random initializations, not the regularizer itself — and it tracks the best validation loss and the epoch it occurred at, not just the value at whichever epoch training happens to stop.

Three things to notice in the output:

1. Dropout raises train loss (0.000 → 0.097) and finds a very slightly better checkpoint than baseline (5.105 vs. 5.117) — a real but modest effect in this run, not a dramatic win.
2. Weight decay (AdamW) also raises train loss slightly (0.000 → 0.007), but its best checkpoint here (5.254) is worse than baseline’s. Regularization strength and architecture interact — a fixed weight-decay value doesn’t universally help every problem, which is why you sweep it rather than assume a default is doing something.
3. Early stopping doesn’t find a better minimum than the unregularized run eventually finds on its own — both bottom out at val loss 5.117 at epoch 21, because they follow an identical trajectory until early stopping’s patience runs out. Its payoff is stopping at epoch 32 instead of running the full 150 epochs, and not needing a human watching the curve to know when to stop.

The function deliberately returns the full history, not just the final number — the train-vs-validation curve is the diagnostic you actually want to look at, not a single point. Plot tr_hist and val_hist against each other and the gap between them is the overfitting gap, made visible.

Edge cases and common mistakes

• Confusing weight decay with L2. With plain SGD they are equivalent. With Adam they are not: the L2 penalty gets rescaled by the gradient’s second moment, which is wrong. Use AdamW, which decouples them. The PyTorch and HuggingFace defaults do this correctly; old code using torch.optim.Adam(..., weight_decay=...) does not.
• Setting dropout too high. $p = 0.5$ is the textbook maximum and it is already a lot. Above $0.5$ you are usually underfitting on purpose. If you need that much regularization, your model is too big or your data is too small — fix one of those first.
• Using BatchNorm with batch size 1. The batch statistics become meaningless; the variance estimate is essentially zero and the normalization does nothing useful, or does something actively harmful. Use LayerNorm or GroupNorm in any setting where batches are tiny or variable — federated learning, online inference, recurrent sequence models.
• Putting LayerNorm where BatchNorm belongs. LayerNorm does not get the per-batch noise that gives BatchNorm its regularizing side effect; on CNNs with comfortable batch sizes it usually underperforms. Match the norm to the architecture: BatchNorm for CNNs, LayerNorm for transformers.
• Assuming L1 will give you interpretability. L1 gives you sparsity, which is necessary but not sufficient for interpretability. With correlated features L1 picks one arbitrarily; the nonzero set is not stable across resamples. If you actually need interpretable features, pair L1 with stability selection or Shapley-style attribution.
• Forgetting to retrain after early stopping. If you early-stop on a validation set and then ship the model, you have baked the validation set’s noise into the stopping decision. Either hold out a separate test set for the final metric, or — if you must retrain on all data — use the epoch count from the early-stopping run as a fixed schedule.
• Augmenting the test set. Augmentation applies to training only. The test set is the test set — transform it the same way you transform inference inputs (resize, normalize, etc.), but never flip, crop, or jitter it.
• Using L1 with a tiny $\lambda$. The sparsity kicks in over a narrow $\lambda$ range. Too small and you get no sparsity (you have just got OLS); too large and you have zeroed everything. Always sweep $\lambda$ on a log scale (e.g. [1e-4, 1e-3, 1e-2, 1e-1, 1]) rather than a linear one.
• Adding regularization before you have established a baseline. Regularization adds hyperparameters, and hyperparameters cost you tuning budget. Fit the unregularized model first, look at the train-vs-validation curve, and only add the regularizer that addresses the actual failure mode you see. If the curve shows no overfitting gap, no amount of dropout will help — it will just underfit.

Cross-references

• Bias-Variance Tradeoff in Plain English) — the conceptual underpinning for everything in this file.
• Regularization Explained: Why Your Models Overfit) — the full narrative version of the L1/L2 story.
• Why Deep Networks Die: Solving the Vanishing Gradient) — covers the gradient-flow problems that BatchNorm and LayerNorm partially address.
• Batch Normalization and Dropout: The Regularization Pair) — the deep-net regularizer family in detail.
• LoRA and QLoRA Explained: Fine-Tuning Big Models on a Single GPU) — uses weight decay and dropout in the context of parameter-efficient fine-tuning.
• Learning Curves: How to Read Your Model’s Mind to Fight Overfitting) — the diagnostic that tells you which regularizer you actually need.

Further reading

• Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1), 267–288. The original Lasso paper; still the cleanest exposition of the sparsity property.
• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56), 1929–1958.
• Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167. The original BatchNorm paper.
• Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. arXiv:1711.05101. The AdamW paper — read this before reaching for weight_decay in any optimizer.
• Library docs: scikit-learn linear models, PyTorch nn.Dropout, PyTorch nn.LayerNorm, PyTorch optim.AdamW.