Why Isn't My Model Learning? A Friendly Guide to Diagnosing Deep Learning Dead-Ends

Last time, Lina got her BookSight purchase-prediction model training stably with batch normalization and dropout. Now her newest model, built the same way, refuses to learn at all—and she’s about to play mechanic to figure out why.

You’ve spent hours setting up your environment. You coded your layers, picked a loss function, and hit Run. You wait for the magic to happen. Then you see it: the loss curve is flat. It isn’t going down. It isn’t even wiggling. That’s exactly where Lina finds herself with her newest BookSight model.

It feels like a personal failure. But even senior researchers at OpenAI and Google deal with models that refuse to learn. Think of your model like a car engine. If it won’t start, you don’t buy a new car—you check the spark and the fuel, one piece at a time. In this part of the series, we’ll be the mechanics for your neural network.

Before touching her full architecture, Lina builds the simplest possible stand-in for her stalled model: a single linear layer that should learn to multiply by 2. If this tiny proxy can’t learn, her real model doesn’t stand a chance.

import torch
import torch.nn as nn

# A simple model that should learn to multiply by 2
model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
criterion = nn.MSELoss()

# Dummy data: Input is 1.0, Target is 2.0
x = torch.tensor([[1.0]])
y = torch.tensor([[2.0]])

print("Starting training...")
for epoch in range(5):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

• import torch, import torch.nn as nn — The standard PyTorch imports used throughout this series: the core tensor library and the neural-network module API.
• nn.Linear(1, 1) — A single linear layer with one input feature and one output. This is the simplest possible model that should learn the mapping y = 2x — a stripped-down proxy for Lina’s stalled purchase-prediction model.
• torch.optim.SGD(model.parameters(), lr=0.001) — Stochastic gradient descent with a learning rate of 0.001. This is intentionally low — slow enough that the loss barely moves over 5 epochs, producing the “flat as a pancake” curve Lina is seeing.
• nn.MSELoss() — Mean squared error: penalizes the squared difference between the model’s prediction and the target. Simple and interpretable for a debugging toy.
• torch.tensor([[1.0]]) / torch.tensor([[2.0]]) — A single training example: input 1.0, target 2.0. In Lina’s real model, the input would be a vector of browsing-session features and the target a buy/no-buy label.
• The training loop: optimizer.zero_grad() clears stale gradients, model(x) does the forward pass, criterion(output, y) computes the loss, loss.backward() runs backpropagation to populate gradients, optimizer.step() applies the weight update, and the print statement shows the loss value after each epoch.
• With lr=0.001, the loss will barely budge — this is the “dead” training loop Lina is trying to diagnose.

If you run this and the loss stays at 1.2345 every epoch, something is broken. More data won’t help here. If the engine isn’t sparking, adding fuel just floods the system. We need to find the specific block in the plumbing.

Step 1: The ‘Overfit One Batch’ Trick

Your most useful diagnostic. Before training on a million images, try five. If your model can’t memorize 5 rows of data, the logic is fundamentally broken.

The hard part is admitting the code might have a bug, not just ‘bad data.’ What you’re really checking is whether the backpropagation from Part 2 is actually connected to the weights.

# The 'Overfit One Batch' Strategy
# Take a tiny slice of your data
tiny_x = x[:5]
tiny_y = y[:5]

# Crank the learning rate up and train for 100 iterations
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

for i in range(100):
    optimizer.zero_grad()
    pred = model(tiny_x)
    loss = criterion(pred, tiny_y)
    loss.backward()
    optimizer.step()

print(f"Final tiny-batch loss: {loss.item():.8f}")

• x[:5] / y[:5] — Slices the first 5 rows of the input and target tensors. With the toy single-example data above, this produces the same 1-row tensor; in Lina’s real model, this would be 5 browsing sessions pulled from her full dataset.
• torch.optim.SGD(model.parameters(), lr=0.1) — Recreates the optimizer with the learning rate cranked up 100x (from 0.001 to 0.1). The point is to force fast convergence on the tiny batch — if the model can learn, it should learn aggressively here.
• for i in range(100): — Runs 100 training iterations (not epochs) on the same 5 rows. This is brute-force memorization: if the loss doesn’t approach zero after seeing the same 5 examples 100 times, the training logic itself is broken.
• loss.item():.8f — Prints the final loss to 8 decimal places. A healthy result would be something like 0.00000001; a result like 2.34567890 means you have a “silent bug” — the plumbing is disconnected somewhere.
• The most common silent bugs this test catches: forgetting optimizer.step(), computing loss on the wrong variable, or having a disconnected computation graph (e.g., detaching a tensor with .detach() at the wrong point).

If the loss doesn’t hit near zero here, you’ve probably got a silent bug. Maybe you forgot optimizer.step(), or you’re calculating loss on the wrong variable. If it does hit zero, your plumbing works. The problem lies elsewhere.

Vanishing Act: When Gradients Go to Zero

Remember the ‘Telephone Game’ from Part 2? In deep networks, the ‘blame’ (gradient) has to travel from the last layer all the way back to the first. If the signal gets quieter at every step, the first layers never hear the instructions. This is the Vanishing Gradient.

To spot it, we look at the magnitude of your gradients. Tiny gradients mean the weights don’t move. Here’s what shows up when we print the ‘gradient norm’—a measure of the total signal strength.

# Let's check the 'pulse' of our layers
for name, param in model.named_parameters():
    if param.grad is not None:
        grad_norm = param.grad.norm().item()
        print(f"Layer: {name} | Gradient Strength: {grad_norm:.10f}")

• model.named_parameters() — Iterates over every learnable parameter in the model, yielding (name, tensor) tuples. For Lina’s single nn.Linear(1, 1), this returns two entries: weight and bias.
• param.grad — The gradient tensor for this parameter, populated by loss.backward(). If backward() hasn’t been called yet (or if the parameter isn’t part of the computation graph), grad will be None — the if param.grad is not None guard skips those.
• param.grad.norm().item() — Computes the L2 norm (Euclidean length) of the gradient vector and converts it to a Python float. This single number tells you the total “strength” of the learning signal reaching this parameter.
• A gradient norm of 0.0000000000 means the signal died before reaching this layer — the layer’s weights will never update, no matter how many epochs you train. This is the hallmark of a vanishing gradient.
• In Lina’s deeper purchase-prediction model, she’d check each layer’s gradient norm; if the first layers show 0.0000000000 while the last layers show healthy values, the signal is dying in transit.

Interpretation: If that number is 0.0000000000, your model is effectively deaf. The signal died before it reached that layer. This often happens with the wrong activation functions, or when the network is too deep without the ‘shortcuts’ — residual connections — we used back in Part 3 to fix this exact problem, and which reappear inside every Transformer block in Part 9.

The Goldilocks Zone: Learning Rates and Initialization

Sometimes the model is fine. The speed is just wrong. Learning rate is like a hiker’s pace — too fast and you overshoot the valley, too slow and you never leave the house.

Weight initialization matters too. Start every weight at exactly zero and every neuron does the same thing. Think of a choir where everyone sings the same note — no harmony possible. The model needs some random noise at the start so neurons can begin distinguishing features.

# Too high vs Too low learning rate
for lr in [10.0, 0.0000001]:
    # Reset model
    m = nn.Linear(1, 1)
    opt = torch.optim.SGD(m.parameters(), lr=lr)
    # ... training loop ...
    print(f"Testing LR {lr}: Resulting loss was either exploding or stagnant.")

• for lr in [10.0, 0.0000001]: — Tests two extreme learning rates: one absurdly high (10.0) and one absurdly low (0.0000001). The goal is to show the two failure modes at opposite ends of the spectrum.
• m = nn.Linear(1, 1) — Creates a fresh model for each learning rate test. If you reused the same model, the weights from the first test would contaminate the second.
• torch.optim.SGD(m.parameters(), lr=lr) — Applies the tested learning rate to the optimizer. With lr=10.0, the weight updates are so large that the loss overshoots the minimum and diverges to NaN; with lr=0.0000001, the updates are so tiny that the loss looks completely flat.
• The # ... training loop ... comment is illustrative — in practice you’d fill in the same zero_grad / forward / loss / backward / step sequence used throughout this article.
• Lina’s real diagnostic would test a range like [0.1, 0.01, 0.001] — one of those usually produces visible learning, which is how she’d know the right ballpark for her model.

Set the learning rate to 10.0 and your loss becomes NaN (Not a Number) — the weights fly off to infinity. Drop it to 0.0000001 and the loss looks flat. The weights are barely moving.

Is Your Data Lying to You?

If the model and the optimizer are fine, the problem is the ‘fuel.’ Neural networks don’t like big numbers. They want inputs centered near zero with a modest spread — not swinging between 0 and 3,600. Feed in 1000.0 with a target of 0.001 and the math gets unstable.

Your code can run perfectly and still learn garbage from messy data. Check your tensors before they go into the model. Lina’s time-on-page feature, for instance, could range from 0 to 3600 seconds if she forgets to normalize it.

def check_data(tensor):
    print(f"Mean: {tensor.mean():.2f}")
    print(f"Max: {tensor.max():.2f}")
    print(f"Min: {tensor.min():.2f}")

check_data(x)

• def check_data(tensor): — A simple diagnostic function that prints three summary statistics for any tensor. This is the kind of quick sanity check Lina runs on every feature before feeding it to her model.
• tensor.mean() — The average value of all elements. If this is 500.0 instead of near 0.0, the optimizer will struggle — the loss landscape becomes a long, narrow ravine that’s hard to navigate.
• tensor.max() / tensor.min() — The range of values in the tensor. If max is 3600 and min is 0 (un-normalized time-on-page in seconds), the gradients will be dominated by the large-magnitude features and the small ones will be ignored.
• check_data(x) — Called on the input tensor x. In Lina’s real model, she’d call this on each feature column of her browsing-session data — time-on-page, cart-items, scroll-depth, etc. — before concatenating them into the input vector.
• The rule of thumb: standardize each feature — subtract its mean and divide by its standard deviation — so it’s centered at 0 with roughly unit spread. That lands most values in about the -3 to 3 range, not inside a fixed [0, 1] or [-1, 1] box.

In practice: if your mean is 500.0, scale the data. Subtract the average and divide by the spread so the mean lands at 0.0. The loss landscape becomes smoother for the optimizer to navigate.

Your Diagnostic Checklist

When your model stops learning, don’t panic. Work through the checklist:

1. Overfit a tiny batch: Can it learn 5 rows? If not, look at your code logic.
2. Check your gradients: Are the numbers moving, or are they zeros?
3. Scale your data: Is each feature centered near 0 with roughly unit spread (most values within about -3 to 3), rather than a raw range like 0–3600?
4. Check the learning rate: Try 0.1, 0.01, and 0.001. One of them usually wiggles the needle.

Which Debugging Technique to Reach For First

The four techniques in this article each address a different failure mode. Here’s which to reach for depending on what you’re seeing:

Symptom	First technique	What it catches	When to reach for it	Tradeoff
Loss is completely flat from epoch 1	Overfit one batch	Broken training logic — missing optimizer.step(), disconnected computation graph, wrong loss variable — or a learning rate too small to move the needle in a handful of epochs.	Always start here. If the model can’t memorize 5 rows even with the learning rate cranked up, the logic is broken; if it can, the logic is fine and the flat curve was a hyperparameter problem all along.	A flat loss on a deep network can also be a vanishing gradient (Part 3) rather than a code bug or a slow learning rate — if overfit-one-batch passes, checking gradient norms is what tells those two apart.
Loss drops then plateaus abruptly on a deep network	Check gradient norms	Vanishing gradients — the “telephone game” signal dying before reaching early layers.	After overfit-one-batch passes but deep layers still won’t learn. Look for gradient norms near zero on the first layers specifically.	Only diagnoses gradient flow, not data quality or learning rate.
Loss is NaN or explodes to infinity	Learning rate sweep	Learning rate far too high — weights overshooting and diverging.	Immediately when you see NaN. Drop the learning rate by 10x and retry.	A too-low learning rate (the other extreme) produces the flat loss you started with — the sweep finds the sweet spot in between.
Training loss is low but validation loss is much higher (the classic overfitting gap)	Add regularization	The model has memorized the training set instead of learning patterns that generalize — see Batch Normalization and Dropout, Part 11.	After the model trains cleanly (low, stable training loss) but validation performance lags well behind it.	Regularization won’t fix a broken training loop, dead gradients, or unscaled data — it only helps once those are already ruled out.
Training is slow or jagged, or won’t drop below a plateau even after a learning-rate sweep	Data scaling check	Unscaled inputs stretching the loss landscape into a hard-to-navigate ravine — e.g., time-on-page in seconds (0–3600) sitting next to a feature already scaled to 0–1.	After a learning-rate sweep doesn’t fully fix a sluggish or jagged loss curve. Check mean, max, min of every input feature.	Scaling alone won’t fix a broken training loop, a vanishing gradient, or overfitting — it’s a fuel problem, not an engine problem.
Loss wiggles but never settles	Learning rate sweep (finer)	Learning rate slightly too high — bouncing around the minimum without converging.	After other checks pass. Try rates one order of magnitude apart: 0.1, 0.01, 0.001.	Time-consuming — each rate requires a full training run to evaluate.

Lina’s diagnostic journey: Her loss was flat from epoch 1, so she started with the overfit-one-batch test — and it passed almost immediately, hitting a loss of 0.00000000 once she cranked the learning rate up to 0.1. That ruled out a silent code bug: optimizer.step() was right there in the loop, and the plumbing was wired correctly. What she’d actually been looking at was a learning rate — 0.001 — too small to move the needle over just 5 epochs. Raising it, not rewriting her architecture or touching her data, was the actual fix. The checklist saved her from hours of guessing at the wrong layer of the stack.

And with that, Lina’s journey comes full circle. She started with a single neuron deciding whether a visitor would buy a book. Now she runs a full transformer-based model for BookSight. It’s built on backpropagation, kept trainable by solving vanishing gradients, extended to see images for cover verification, given memory for long sessions via RNNs then LSTMs, sped up and sharpened by attention and multi-head attention, assembled into a working mini-transformer, made practical by transfer learning, made stable by batch norm and dropout, and when it broke, debugged systematically instead of guessing. Every piece was built from scratch, one concept at a time.

---

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the “Overfit One Batch” trick, and what does it tell you if the loss doesn’t hit near zero?

Understand In your own words, explain why initializing all of a network’s weights to exactly zero prevents the model from learning, using the article’s “choir” analogy.

Apply The article tests learning rates of 10.0 and 0.0000001 and says one causes the loss to explode to NaN while the other looks “flat.” Given the article’s diagnostic checklist, if you saw a loss curve that was flat for 100 epochs, which two checklist items (of the four listed) would you try first, and in what order—and why that order?

Analyze The article says checking gradient norms can reveal a Vanishing Gradient, where a printed value of 0.0000000000 means “your model is effectively deaf.” Walk through why a gradient norm near zero on the first layer specifically (not the last layer) points to a vanishing-gradient problem rather than, say, a bad learning rate.

Evaluate The article’s diagnostic checklist is a fixed, ordered list: overfit a tiny batch, check gradients, scale data, check learning rate. Critique this ordering: is “overfit a tiny batch” really always the right first step, or can you think of a symptom (something in the loss curve or the code) where you’d want to check the learning rate or the data scale before attempting to overfit a tiny batch?

Create Design a “pre-flight checklist” a data scientist could run before starting a full training run (not after it fails) that would catch at least two of the four failure modes in this article ahead of time. Name the specific checks and what red flag each one would look for.

---

Related articles

• Batch Normalization and Dropout: The Regularization Tricks That Make Deep Learning Actually Work)
• Neural Networks Without the Calculus: What’s Actually Happening Inside a Single Neuron?)

References & Further reading

• Karpathy, A. (2019). “A Recipe for Training Neural Networks.” — The definitive practical guide to debugging and training neural networks, written by Andrej Karpathy (former Director of AI at Tesla). It codifies the same diagnostic philosophy Lina follows here: start with the simplest possible test, verify your data before blaming your model, overfit one batch before scaling up, and never assume your code is correct until proven otherwise. Available at karpathy.github.io.
• PyTorch Documentation: Troubleshooting and Performance Optimization Recipes — Official PyTorch recipes for common debugging scenarios, including gradient checking, learning rate scheduling, and data normalization — the same techniques Lina used to diagnose her stalled BookSight model.

---