Reference: Optimizers

This reference covers the gradient-based optimization family used to train neural networks — from vanilla gradient descent through AdamW — plus the learning-rate schedules that ride on top of them. It opens with a short recap of the chain-rule mechanic that all of these optimizers depend on, then walks each algorithm, a decision grid, runnable comparison code, and practical learning-rate guidance.

Backprop: the chain-rule mechanic optimization rides on

Every optimizer in this reference updates a parameter vector θ by moving it in the direction of the negative gradient of a loss L with respect to θ. The hard part is not the direction of the move — it is computing ∂L/∂θ for thousands or millions of parameters efficiently. That computation is backpropagation: an application of the chain rule from multivariable calculus, arranged so that each layer’s gradient is computed from the layer above it, reusing intermediate values.

For a layer that computes z = Wx + b and then a = φ(z), the gradient of the loss with respect to W is:

$$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial a}\;\frac{\partial a}{\partial z}\;\frac{\partial z}{\partial W}$$

The three factors are, in plain order: “how does the loss change if this layer’s activation changes,” “how does the activation change if the pre-activation changes” (φ′), and “how does the pre-activation change if the weights change” (which is just xᵀ, the input to this layer).

Plain English	Statistical symbol	Python equivalent
Loss with respect to parameters	∂L/∂θ	grad = torch.autograd.grad(loss, params)
Upstream gradient coming into a layer	∂L/∂a	grad_a (passed from next layer)
Local derivative of activation	φ′(z)	relu_grad = (z > 0).float()

Backprop is not an optimizer — it is the gradient supplier. Which optimizer consumes those gradients is the subject of the rest of this reference. For the full derivation, see the attention-from-scratch backprop article cross-referenced at the bottom.

Roster

Optimizer	One-line formula	Typical LR (η)	When to use	Used in corpus article
Gradient Descent (GD / “batch”)	θ ← θ - η·∇L(θ)	0.01–1.0	Toy problems, full datasets that fit in memory; theoretical baseline	How Gradient Descent Actually Works)
Stochastic GD (one-sample)	θ ← θ - η·∇L_i(θ)	0.01–0.1	Very large datasets where one sample is cheap; high-noise regime	How Gradient Descent Actually Works)
Mini-batch SGD	θ ← θ - (η/B)·Σ∇L_i(θ)	0.01–0.1	The practical default when paired with momentum; batch sizes 32–512	Why Isn’t My Model Learning)
Momentum	v ← βv + η·∇L; θ ← θ - v	0.01–0.1	Large-batch training; ill-conditioned (long narrow valleys) losses	Why Isn’t My Model Learning)
RMSprop	θ ← θ - η · g / √(E[g²]+ε)	1e-4–1e-3	RNNs; non-stationary objectives where gradients’ scale drifts	Building a Miniature Transformer)
Adam	RMSprop + momentum; θ ← θ - η·m̂/√v̂	1e-4–1e-3	Default for most deep nets; vision, NLP, tabular DL	Why Isn’t My Model Learning)
AdamW	Adam with decoupled weight decay; θ ← (1-ηλ)θ - η·m̂/√v̂	1e-5–5e-4	Transformer pretraining and fine-tuning; wherever weight decay matters	LoRA and QLoRA Explained)

Learning-rate schedules (Step, Cosine, Warmup) sit on top of any optimizer and modulate η over training:

Schedule	Formula (effective LR at step t)	When to use	Used in corpus article
Step	η · γ^(⌊t/s⌋) every s steps	Computer-vision CNNs; quick experiments	Building a Miniature Transformer)
Cosine	η_min + ½(η_max−η_min)(1+cos(πt/T))	Transformer training; long runs that benefit from graceful anneal	Building a Miniature Transformer)
Linear warmup + cosine	cosine after linear ramp from 0 to η_max over T_warm	Transformer pretraining and fine-tuning to avoid early-step instability	LoRA and QLoRA Explained)

Decision tree: which optimizer when

• Are you training a Transformer (GPT-style, BERT-style, ViT)?
  • Fine-tuning a pretrained model → AdamW with linear warmup + cosine decay, LR ~1e-5 to 5e-4.
  • Pretraining from scratch → AdamW with warmup ~1e-4 to 6e-4 depending on scale.
• Are you training a CNN for image classification?
  • Large-batch (>256) → SGD + momentum (0.9) with step decay, LR ~0.1.
  • Small-batch, quick experiment → Adam as a fast fallback; LR ~1e-3.
• Are you training an RNN / LSTM / GRU?
  • → RMSprop or Adam with LR ~1e-4 to 1e-3; gradient clipping strongly recommended.
• Are you training a tabular MLP or shallow net?
  • → Adam, LR ~1e-3, default β1=0.9, β2=0.999.
• Are you doing classical ML (logistic regression, shallow models) on data that fits in memory?
  • → Plain batch GD with a fixed LR or LBFGS; you don’t need the machinery below.
• Is your loss surface a long narrow valley (Hessian has very different singular values)?
  • → Momentum or Adam; the per-coordinate scaling helps most.
• Are you unsure and just want a default that usually works?
  • → Adam at LR 1e-3 with a cosine schedule and ~500 warmup steps. Move to AdamW if you add weight decay.

The optimizers, walked

Throughout, g denotes the gradient ∇L(θ) of the current mini-batch, θ the parameter vector, and η the learning rate. Subscripts t denote the time step.

1. Gradient Descent (GD)

The simplest possible update: take a step in the direction of steepest descent of the loss computed over the entire training set.

# Full-batch gradient descent. `epochs`, `lr`, and compute_grad_over_full_dataset
# stand in for your own loop count, learning rate, and loss gradient.
for epoch in range(epochs):
    grad = compute_grad_over_full_dataset(theta)
    theta = theta - lr * grad
# Shape of the result: on a convex loss with a small enough lr, this decreases
# monotonically every single epoch, with no noise -- every step sees the exact
# same full-dataset gradient. (A worked, fully-executed comparison of GD,
# momentum, RMSprop and Adam on a concrete loss surface follows in Section 9.)

GD is the theoretical reference point (it is what convergence proofs are built on) but is rarely used in deep learning because computing the full-batch gradient is too expensive per step. Its variance is zero, which is its main charm.

2. Stochastic Gradient Descent (SGD)

Use a gradient estimated from a single random sample per step. Noisy, but cheap — and the noise turns out to help escape saddle points and sharp minima.

# One-sample SGD. shuffle(dataset) and compute_grad_on_one_sample stand in
# for your own data loader and per-example loss gradient.
for x, y in shuffle(dataset):
    grad = compute_grad_on_one_sample(theta, x, y)
    theta = theta - lr * grad
# Shape of the result: the loss no longer decreases monotonically -- each step
# is a noisy one-sample estimate of the true gradient, so it bounces around a
# band whose width scales with the per-sample gradient variance, instead of
# settling smoothly the way full-batch GD does above.

3. Mini-batch SGD

The workhorse compromise: estimate the gradient using B samples (B = 32 to 512 is common). The mini-batch gradient is an unbiased estimator of the full-batch gradient with variance scaling as 1/B.

for X_batch, y_batch in dataloader:   # batch_size = 64
    grad = compute_grad(theta, X_batch, y_batch)
    theta = theta - lr * grad
# Shape of the result: averaging the gradient over 64 samples cuts the
# per-step noise roughly sqrt(64) ~= 8x relative to one-sample SGD above, so
# the loss curve sits between full-batch GD's smooth line and SGD's noisy one.

Why batch size matters for the optimizer choice: smaller batches give noisier gradients (better for escaping saddle points, worse for the final polish), and larger batches behave more like full-batch GD (smoother, but can converge to sharper, less-generalizable minima — the “large-batch training generalization gap” that motivates the SGD-with-momentum choice below).

4. Momentum

Add a velocity vector v that accumulates a discounted running average of past gradients. The update becomes θ ← θ - v, where v ← βv + η·g. The hyperparameter β (typically 0.9) controls how much history to retain.

Momentum is equivalent to optimizing a smoothed loss where the gradient is replaced by an exponential moving average of recent gradients. In a narrow valley, plain SGD oscillates across the valley while slowly creeping along it; momentum’s averaging damps the cross-valley oscillation while preserving the along-valley drift, so the effective step size along the good direction grows by a factor up to 1/(1 − β) ≈ 10 in the limit.

Plain English	Symbol	Python
Velocity / accumulated gradient direction	v	v = beta * v + lr * grad
Discount factor (history weight)	β	beta = 0.9
Update to parameters	θ ← θ − v	theta = theta - v

import numpy as np

def bowl_grad(p):
    # The elongated bowl used throughout the rest of this reference:
    # L = 0.5*(x^2 + 100*y^2) -- eigenvalues 1 (x, mild) and 100 (y, steep).
    return np.array([p[0], 100.0 * p[1]])

theta0 = np.array([5.0, 5.0])
lr, beta = 0.01, 0.9   # lr is capped near 1/100 by the steep y-axis's stability limit

# Plain gradient descent, no momentum
theta = theta0.copy()
for t in range(200):
    theta = theta - lr * bowl_grad(theta)
# theta = [0.6699, 0.0], loss = 0.224 -- still closing the mild x-axis after 200 steps

# The same 200 steps, with momentum added
theta = theta0.copy(); v = np.zeros(2)
for t in range(200):
    g = bowl_grad(theta)             # gradient of the loss at theta
    v = beta * v + lr * g
    theta = theta - v
# theta = [-2.15e-05, -8.66e-05], loss = 3.7e-7 -- momentum's accumulated
# velocity closes the mild x-axis about 600,000x faster than plain SGD at
# the SAME learning rate, without destabilizing the steep y-axis.

5. RMSprop

Momentum smooths gradients across time; RMSprop rescales them across coordinates. The idea: divide each gradient coordinate by a running estimate of its recent magnitude, so coordinates with consistently large gradients get smaller effective steps and vice versa. This is the right thing to do when the gradient scale changes a lot across parameters (common in RNNs).

theta = theta0.copy(); s = np.zeros(2)   # reusing bowl_grad and theta0 from Section 4
lr, rho, eps = 1e-3, 0.9, 1e-8
for t in range(200):
    g = bowl_grad(theta)
    s = rho * s + (1 - rho) * g**2
    theta = theta - lr * g / (np.sqrt(s) + eps)
# loss drops from 1262.5 to 1159.3 in 200 steps -- real, but unimpressive at
# this section's own default lr, because 1e-3 is calibrated for thousands of
# steps on a real network, not 200 steps on a toy quadratic. Raise lr to 1e-2
# (still within RMSprop's typical range) and the same 200 steps close the
# loss to 440.2; give it the 300 steps used in Section 9's full comparison
# and it reaches 199.5.

6. Adam (Adaptive Moment Estimation)

Adam combines momentum (a first-moment estimate m) and RMSprop (a second-moment estimate v) and adds bias correction for the early steps where both running averages start from zero and are therefore biased low.

The full Adam update is:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \\ \hat{m}_t = m_t / (1 - \beta_1^t), \quad \hat{v}_t = v_t / (1 - \beta_2^t) \\ \theta_{t+1} = \theta_t - \eta \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$$

The bias-corrected quantities m̂ and v̂ undo the initialization bias: at t = 1, m₁ = (1 − β₁)g₁ which is small if β₁ ≈ 1; dividing by (1 − β₁ᵗ) = (1 − β₁) gives back g₁ itself. The same trick applies to v.

Plain English	Symbol	Python
First-moment estimate (mean of grad)	m	m = b1*m + (1-b1)*g
Second-moment estimate (uncentered variance)	v	v = b2*v + (1-b2)*g**2
Bias correction (time t)	1 − βᵗ	1 - beta**t
Effective step per parameter	η·m̂/√v̂	lr * m_hat / (np.sqrt(v_hat) + eps)

theta = theta0.copy(); m = np.zeros(2); v = np.zeros(2)   # reusing bowl_grad and theta0
lr, b1, b2, eps = 1e-3, 0.9, 0.999, 1e-8
for t in range(1, 201):
    g = bowl_grad(theta)
    m = b1 * m + (1 - b1) * g
    v = b2 * v + (1 - b2) * g**2
    m_hat = m / (1 - b1**t)
    v_hat = v / (1 - b2**t)
    theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
# loss drops from 1262.5 to 1164.3 in 200 steps at this default lr -- the same
# lr-vs-step-budget mismatch as RMSprop above. At lr=1e-2 over the 300 steps
# used in Section 9's full comparison, Adam reaches 299.2.

Default hyperparameters (Kingma & Ba): η = 1e-3, β₁ = 0.9, β₂ = 0.999, ε = 1e-8. These work for most problems, which is why Adam is the “default and move on” optimizer.

7. AdamW

Standard Adam couples weight decay (L2 regularization) into the gradient: it adds λθ to g before the moment estimates. This couples the regularization strength to the adaptive scaling — so weight decay ends up being applied unevenly across parameters. AdamW (Loshchilov & Hutter 2017) decouples them: weight decay is applied directly to the parameters, and the gradient fed into the moment machinery is the unregularized gradient.

theta = theta0.copy(); m = np.zeros(2); v = np.zeros(2)   # reusing bowl_grad and theta0
lr, b1, b2, eps, wd = 1e-3, 0.9, 0.999, 1e-8, 1e-2
for t in range(1, 201):
    g = bowl_grad(theta)            # no L2 term added to g
    m = b1 * m + (1 - b1) * g
    v = b2 * v + (1 - b2) * g**2
    m_hat = m / (1 - b1**t)
    v_hat = v / (1 - b2**t)
    theta = theta - lr * (m_hat / (np.sqrt(v_hat) + eps) + wd * theta)
# loss = 1159.6 after 200 steps.
#
# Compare against coupling the same decay into the gradient instead
# (g = bowl_grad(theta) + wd * theta, fed through the identical Adam update):
# at wd=0.01 the coupled form gives loss = 1164.3 -- barely different from
# AdamW's 1159.6. Raise wd tenfold to 0.1 and the gap becomes stark: AdamW's
# loss drops further to 1118.1, responding to the stronger decay, while the
# coupled form is UNCHANGED at 1164.3. Adam's own per-coordinate
# normalization divides away most of the coupled penalty before it can act --
# that cancellation is exactly the failure Loshchilov & Hutter's decoupling fixes.

This is the optimizer essentially all modern Transformer fine-tuning recipes use (LoRA, QLoRA, full fine-tuning). The reason is that weight decay matters more in large nets and coupling it into Adam’s per-coordinate scaling messes it up; decoupling restores the simple L2 behavior on the weights themselves.

8. Learning-rate schedules

A schedule modulates η over training instead of holding it constant. Three schedules cover 95% of practical use.

import numpy as np

def step_lr(t, base_lr=0.1, step_size=30, gamma=0.1):
    return base_lr * (gamma ** (t // step_size))

def cosine_lr(t, T_total=200, lr_max=0.1, lr_min=0.0):
    return lr_min + 0.5 * (lr_max - lr_min) * (1 + np.cos(np.pi * t / T_total))

def warmup_cosine_lr(t, T_warm=10, T_total=200, lr_max=0.1, lr_min=0.0):
    if t < T_warm:
        return lr_max * (t / T_warm)
    return cosine_lr(t - T_warm, T_total - T_warm, lr_max, lr_min)

# At t = 5:    step 1.00e-01,  cosine 0.0998,  warmup-cosine 0.0500
# At t = 100:  step 1.00e-04,  cosine 0.0500,  warmup-cosine 0.0541
# At t = 199:  step 1.00e-07,  cosine 0.0000,  warmup-cosine 0.0000
#
# The step schedule decays faster than it looks: step_size=30 means 6 drops
# by t=199 (0.1 * 0.1**6 = 1e-7), so the back two-thirds of a 200-step run
# trains at a vanishingly small LR. Match step_size to total training length
# (something like T_total // 3) for a handful of clean drops spread across
# the whole run, rather than six crammed into the first third.

• step_lr drops the LR by a factor gamma every step_size steps. The result is a piecewise-constant staircase. This is the classic CV-CNN schedule and is forgiving: training proceeds, you observe the loss plateau, you drop the LR, repeat.
• cosine_lr smoothly anneals from lr_max to lr_min over T_total steps. Its smoothness tends to give slightly better final loss than step decay in long training runs because it spends more time near the end at low LR, doing the “final polish.”
• warmup_cosine_lr is cosine_lr with a linear ramp from 0 to lr_max during the first T_warm steps. The warmup is essential for Transformer training: the first few steps have huge noisy gradients (parameters are random,Adam’s moments are biased low, and per-layer activations can blow up). Starting at the full LR can spike the loss irrecoverably; ramping up gives Adam’s moment estimates time to fill in.

The t < T_warm branch returns lr_max * (t / T_warm), so the LR at step 0 is 0, and at step T_warm it is exactly lr_max. After T_warm we delegate to a cosine anneal of length T_total − T_warm, which means the cosine component of the schedule runs from the end of warmup to the end of training.

Worked comparison: convergence on a bowl vs. a saddle

To make the differences concrete, compare four optimizers (plain SGD, momentum, RMSprop, Adam) on two canonical 2D loss surfaces:

• Bowl: a quadratic L = ½(x² + 100y²) — an elongated bowl (Hessian eigenvalues 1 and 100).
• Saddle: L = ½(x² − 100y²) — a saddle at the origin with one upward and one downward direction.

import numpy as np

def bowl_grad(p):
    return np.array([p[0], 100.0 * p[1]])

def saddle_grad(p):
    return np.array([p[0], -100.0 * p[1]])

def sgd(grad_fn, theta0, lr=0.01, steps=300):
    theta = theta0.copy()
    for _ in range(steps):
        theta = theta - lr * grad_fn(theta)
    return theta

def momentum(grad_fn, theta0, lr=0.01, beta=0.9, steps=300):
    theta = theta0.copy(); v = np.zeros_like(theta0)
    for _ in range(steps):
        v = beta * v + lr * grad_fn(theta)
        theta = theta - v
    return theta

def rmsprop(grad_fn, theta0, lr=0.01, rho=0.9, eps=1e-8, steps=300):
    theta = theta0.copy(); s = np.zeros_like(theta0)
    for _ in range(steps):
        g = grad_fn(theta)
        s = rho * s + (1 - rho) * g**2
        theta = theta - lr * g / (np.sqrt(s) + eps)
    return theta

def adam(grad_fn, theta0, lr=0.01, b1=0.9, b2=0.999, eps=1e-8, steps=300):
    theta = theta0.copy(); m = np.zeros_like(theta0); v = np.zeros_like(theta0)
    for t in range(1, steps + 1):
        g = grad_fn(theta)
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g**2
        m_hat = m / (1 - b1**t)
        v_hat = v / (1 - b2**t)
        theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
    return theta

theta0 = np.array([5.0, 5.0])

for name, fn in [("SGD", sgd), ("Momentum", momentum), ("RMSprop", rmsprop), ("Adam", adam)]:
    bowl_dist = np.linalg.norm(fn(bowl_grad, theta0))
    saddle_y = fn(saddle_grad, theta0)[1]
    print(f"{name:>9}  bowl ||theta|| = {bowl_dist:.3e}   saddle y = {saddle_y:.3e}")

# Bowl loss: distance from optimum after 300 steps, lr = 0.01 for all four
#      SGD  bowl ||theta|| = 2.452e-01   (still closing the mild x-axis)
# Momentum  bowl ||theta|| = 9.377e-07   (momentum's acceleration wins decisively)
#  RMSprop  bowl ||theta|| = 2.811e+00   (per-coordinate normalization caps its
#                                         step below what this direction needs)
#     Adam  bowl ||theta|| = 3.442e+00   (same cap; no acceleration on the easy axis)

# Saddle: y-coordinate after 300 steps (the loss is unbounded along y, so no
# method converges here -- this measures how far each one has run away)
#      SGD  saddle y = 1.019e+91    (unconditionally unstable for any lr > 0)
# Momentum  saddle y = 2.301e+122   (velocity accumulates in the escaping
#                                    direction too, making the blow-up worse)
#  RMSprop  saddle y = 8.132e+00    (normalization caps the escape to roughly
#                                    a constant step per iteration)
#     Adam  saddle y = 8.330e+00    (same containment as RMSprop)

Bowl interpretation. The loss surface has one gently sloped direction (x, eigenvalue 1) and one steep direction (y, eigenvalue 100). All four optimizers share lr = 0.01 here — close to the largest lr plain gradient descent can use without diverging on the steep y-axis (stability needs lr·100 < 2). At that shared lr, momentum wins decisively (||theta|| ~= 9.4e-7): its accumulated velocity accelerates convergence along the mild x-direction far past what a single step size allows, without destabilizing y. Plain SGD is still visibly stuck (||theta|| ~= 0.245) — 300 steps at this lr isn’t enough to close the mild direction. RMSprop and Adam finish worse than either (2.81 and 3.44): their per-coordinate normalization caps the effective step at roughly the raw lr once gradients shrink, so they get none of momentum’s acceleration on the easy axis — and 0.01 is far smaller than the step they’d need to close a distance of 5 in 300 steps. The lesson this demo actually supports is narrower than “adaptive always wins”: Adam and RMSprop earn their keep by making a shared lr safe across coordinates of very different scale, not by out-accelerating a well-tuned momentum term on an easy direction.

Saddle interpretation. The saddle has no minimum along y — the loss decreases without bound as |y| grows, so no optimizer can make y settle near 0. Any positive step size makes plain gradient descent diverge geometrically along y (multiplying by 1 + 100·lr every step); at lr = 0.01 that’s ×2 per step, and after 300 steps SGD’s y has grown to ~1.0 × 10⁹¹. Momentum makes this worse, not better (~2.3 × 10¹²²): the same velocity term that won on the bowl’s easy direction accumulates just as eagerly in the escaping direction here. RMSprop and Adam don’t converge either — nothing can, given this loss — but their per-coordinate normalization caps the step in the escaping direction to roughly a constant lr-sized increment instead of a compounding one, so after 300 steps they’ve moved to about 8 rather than 10⁹¹ or 10¹²²: orders of magnitude better contained, without becoming stable in any sense. That containment, not convergence, is the property to take from this half of the demo.

Practical takeaway: in a network with parameters of wildly different curvature, a shared lr forces you to pick a value small enough for the worst-conditioned parameter, which stalls every well-conditioned one — RMSprop and Adam solve exactly that problem by rescaling each coordinate independently. What they do not do, as this demo shows, is guarantee faster convergence than a well-tuned momentum term on any single direction; their real advantages are not needing per-coordinate lr tuning, and staying bounded rather than exploding when a direction is genuinely unstable.

Practical learning-rate guidance

Default LRs to start with (always tune, but these are reasonable first guesses):

Architecture	Optimizer	Starting η	Schedule
Logistic regression / shallow MLP	Adam	1e-3	constant or cosine
CNN (ResNet-class)	SGD + momentum 0.9	0.1	step (×0.1 at 50%, 75% of training)
CNN, small / quick experiment	Adam	1e-3	cosine
RNN / LSTM / GRU	Adam (or RMSprop)	1e-3	cosine, with gradient clipping at 1.0
Transformer, pretraining	AdamW	6e-4 (small) to 3e-4 (medium)	linear warmup + cosine, warmup ~2% of total steps
Transformer, LoRA / QLoRA fine-tuning	AdamW	1e-4 to 5e-4	linear warmup 100–500 steps + cosine or constant
Diffusion model (U-Net)	AdamW	1e-4	cosine, warmup ~5k steps

Rules of thumb:

• If the loss explodes in the first 100 steps → LR is too high. Divide by 10.
• If the loss plateaus early and won’t go down further → LR might be too low, or you need a schedule that anneals it lower, or you’re stuck in a saddle (try more momentum or a different optimizer).
• If loss is noisy and bouncing → either reduce LR, increase batch size, or add gradient clipping.
• For Adam vs. SGD+momentum on CNNs: SGD+momentum often reaches slightly better final accuracy (the “generalization gap” literature), but Adam converges in fewer epochs. For a paper-grade ImageNet run, use SGD+momentum; for a quick prototype, use Adam.
• For Transformer fine-tuning, always use AdamW, not Adam — L2 regularization coupled into Adam’s gradient (which is what plain Adam + weight_decay does in most frameworks) is documented to underperform decoupled weight decay.

Edge cases and common mistakes

• Forgetting bias correction in Adam. If you implement Adam without the 1 - b**t correction, the first ~1000 steps will have artificially small effective LR because v starts at 0 and the second moment estimate is biased low. Always correct.
• Setting ε too large. With ε = 1e-8 (the default), the √v + ε denominator is dominated by √v once the second moment estimate has filled in. If you set ε = 1e-4 (which some old recipes did), the denominator never gets small enough, the effective LR is capped, and you get slow convergence on flat regions. Use the default unless you have a specific reason.
• Using Adam with weight_decay in the “old” way. PyTorch’s Adam takes a weight_decay argument that adds λθ to the gradient before the moment machinery. This is the coupled version; for Transformer training it underperforms AdamW. Don’t pass weight_decay to Adam; switch to AdamW.
• Picking LR without a schedule. A constant LR is fine for short runs but leaves performance on the table for anything longer than a few hundred steps. Cosine decay is essentially free and almost always helps the last few percent of accuracy.
• Warmup that’s too short on Transformers. If you skip warmup or warm up for only ~50 steps on a Transformer, the first forward pass with random weights can produce huge activations, and Adam takes a huge step before its moment estimates are accurate. Use 500–2000 warmup steps for small models; GPT-3-class models use thousands.
• Momentum too high on small batches. β = 0.99 (heavy momentum) on small batches accumulates very noisy gradients and can oscillate or diverge. Stick to β = 0.9 unless you have a specific reason (e.g., large-batch ImageNet training sometimes benefits from β = 0.98).
• Confusing β₁ and β₂. In Adam, β₁ is the first-moment (gradient mean) decay — usually 0.9 — and β₂ is the second-moment (gradient squared) decay — usually 0.999. The latter must be much closer to 1 because the squared gradient is much noisier per step.
• Not clipping gradients on RNNs and Transformers. None of the optimizers above handle exploding gradients gracefully; add torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) after loss.backward() and before optimizer.step().
• Using the full-batch LR for SGD. A LR that works for full-batch GD is too small for one-sample SGD by a factor roughly √B; if you switch from full-batch to mini-batch and the loss barely moves, multiply the LR by ~10.
• Restoring optimizer state incorrectly on resume. Adam, AdamW, and RMSprop have internal state (m, v, s) that must be checkpointed alongside the model. If you resume training with a fresh optimizer, you lose the accumulated moments and the schedule starts over — usually resulting in a loss spike.

Cross-references

• How Gradient Descent Actually Works and the Variance Story) — the foundational article; this reference is its deep-learning extension.
• Backpropagation Intuitively: How Networks Learn From Their Mistakes) — the full chain-rule derivation this reference only sketches.
• Building a Miniature Transformer From Scratch) — uses AdamW with cosine schedule end-to-end.
• Batch Normalization and Dropout: The Regularization Story) — interacts with optimizer choice via the effective gradient scale.
• Why Isn’t My Model Learning? A Friendly Guide to Diagnosing Training Issues) — the troubleshooting companion; this reference supplies the “what to switch to” answers.
• Borrowing Brains: A Beginner’s Guide to Transfer Learning) — sets up the fine-tuning context where AdamW and LoRA live.
• LoRA and QLoRA Explained: Fine-Tuning Big Models on a Single GPU) — the canonical AdamW + warmup-cosine fine-tuning recipe.

Further reading

• Rumelhart, Hinton & Williams (1986) — “Learning representations by back-propagating errors.” Nature 323. The original backprop paper; the chain-rule mechanic everything in this reference rides on.
• Kingma & Ba (2014) — “Adam: A Method for Stochastic Optimization.” arXiv:1412.6980. The Adam paper; read sections 2 and 3 for the moment-estimation framing.
• Loshchilov & Hutter (2017) — “Decoupled Weight Decay Regularization.” arXiv:1711.05101. The AdamW paper; section 2 has the precise argument for why decoupling matters.
• Reddi, Kale & Kumar (2018) — “On the Convergence of Adam and Beyond.” arXiv:1904.09237. Documents a non-convergence edge case in Adam and proposes AMSGrad; useful if you’re chasing the last percent of accuracy.
• Andrej Karpathy — “A Recipe for Training Neural Networks” and “The Bitter Lesson” blog posts. Practical, opinionated, and still current on which optimizer to reach for and why.
• Library docs: PyTorch torch.optim, Hugging Face Transformers Trainer optimizer/scheduler docs, TensorFlow tf.keras.optimizers.