Why Deep Networks Die: Solving the Vanishing Gradient Problem with ReLU, ResNets, and BatchNorm

In Part 2, Lina saw how a network learns by “walking backwards” from an error to assign blame to its weights. But a serious problem kept AI stuck for years. As networks get deeper, the training signal often disappears before it reaches the first layers.

If the first layers don’t learn, the whole network fails. A beautiful roof won’t help if the foundation is sand. Today we look at why networks “die”—and the three inventions that saved them.

1. The ‘Telephone Game’ of Deep Learning

Picture a neural network as a long chain of people playing Telephone. The first person gets a message, whispers it to the next. By person 50, it’s often garbled or gone.

In deep learning, this happens during backpropagation. To update the first layer, we multiply the “blame” (gradients) from every layer after it.

• Vanishing: If each person whispers slightly quieter than the last (multiplying by 0.1), the message fades to silence.
• Exploding: If each person shouts louder than the last (multiplying by 1.1), the message becomes deafening noise.

Here’s what a signal looks like after 50 layers in Python:

vanishing_signal = 1.0
exploding_signal = 1.0

for i in range(50):
    vanishing_signal *= 0.1
    exploding_signal *= 1.1

print(f"Vanishing signal after 50 layers: {vanishing_signal}")
print(f"Exploding signal after 50 layers: {exploding_signal:.2f}")

• vanishing_signal = 1.0 and exploding_signal = 1.0 — Both signals start at full strength. One will shrink layer by layer; the other will grow.
• vanishing_signal *= 0.1 — Each layer multiplies the signal by 0.1, simulating layers whose gradients are less than 1. After 50 layers, the signal is 1e-50 — effectively zero.
• exploding_signal *= 1.1 — Each layer multiplies by 1.1, simulating layers whose gradients exceed 1. After 50 layers, the signal is ≈117.39 — large enough to destabilize training.
• The for i in range(50) loop simulates 50 layers of backpropagation. The vanishing signal dies; the exploding signal becomes meaningless noise.

The vanishing signal drops to 1e-50 — basically zero. The exploding signal climbs to 117.39. In a real network, a gradient of zero means the weights never move, and a gradient of 117 multiplied across many weights breaks the math. That’s why your training logs might show a “Loss” that never changes — your gradients have died.

2. The Sigmoid Trap: Why Our Old Tools Failed

In the early days of AI, we used the Sigmoid activation function. It’s that classic ‘S’ curve — squashing any input into a range between 0 and 1. The appeal was biological: it roughly mimics how neurons fire.

The problem is what happens at the ends. The curve goes flat. Feed it 5.0 or -5.0, and the slope (gradient) drops to nearly zero.

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def sigmoid_gradient(x):
    # The math for the slope of a sigmoid
    s = sigmoid(x)
    return s * (1 - s)

print(f"Gradient at x=0 (the peak): {sigmoid_gradient(0):.4f}")
print(f"Gradient at x=5 (the flat part): {sigmoid_gradient(5):.4f}")

• def sigmoid(x): return 1 / (1 + math.exp(-x)) — The classic sigmoid function. Any input gets squashed to a value between 0 and 1. At x=0 the output is 0.5 (the midpoint); at x=5 the output is ≈0.993 (nearly saturated).
• s = sigmoid(x) — Compute the sigmoid once so we can reuse it in the derivative formula without recalculating it.
• return s * (1 - s) — The derivative of sigmoid: σ(x)(1 − σ(x)). At x=0 this is 0.25 (the maximum slope); at x=5 this is ≈0.0066 (nearly flat).
• The test prints show the gradient collapsing from 0.25 at the center to 0.0066 at x=5 — a 38× reduction from a single layer. Chain 20 such layers together and the gradient is effectively deleted.

Sigmoid Function

$\sigma(x) = \frac{1}{1 + e^{-x}}$

The sigmoid squashes any input $x$ into a value between 0 and 1. Large positive inputs approach 1; large negative inputs approach 0.

Sigmoid Derivative (Gradient)

$\sigma'(x) = \sigma(x)(1 - \sigma(x))$

The slope of the sigmoid at any point. The maximum value is 0.25 (at $x = 0$), and it shrinks toward 0 as the input moves away from the center. This shrinking derivative is what causes gradients to vanish when many sigmoid layers are stacked.

That identity is usually quoted rather than worked out. Here it is being worked out — one rule at a time:

$$\frac{\partial}{\partial x} \frac{1}{1 + e^{- x}}$$

the derivative we want: how does this move when $x$ moves?

$$\term{focus}{- \frac{\frac{\partial}{\partial x} \left(1 + e^{- x}\right)}{\left(1 + e^{- x}\right)^{2}}}$$

power rule on the outside, chain rule on the inside

$$- \frac{\term{focus}{\frac{\partial}{\partial x} 1 + \frac{\partial}{\partial x} e^{- x}}}{\left(1 + e^{- x}\right)^{2}}$$

sum rule: differentiate each term separately

$$- \frac{\term{focus}{0} + \frac{\partial}{\partial x} e^{- x}}{\left(1 + e^{- x}\right)^{2}}$$

constant rule: a term with no $x$ in it cannot change with $x$

$$- \frac{\term{focus}{e^{- x} \frac{\partial}{\partial x} \left(- x\right)}}{\left(1 + e^{- x}\right)^{2}}$$

the exponential is its own derivative, times the chain rule

$$- \frac{e^{- x} \term{focus}{- \frac{\partial}{\partial x} x}}{\left(1 + e^{- x}\right)^{2}}$$

constant multiple rule: pull the constant out front

$$\frac{e^{- x} \term{focus}{1}}{\left(1 + e^{- x}\right)^{2}}$$

the derivative of $x$ with respect to itself is $1$

$$\frac{e^{- x}}{\left(1 + e^{- x}\right)^{2}}$$

so the derivative is

$$\frac{e^{x}}{\left(1 + e^{x}\right)^{2}}$$

tidy up

The last line, $\frac{e^{x}}{(1 + e^{x})^{2}}$, is the same quantity as $\sigma(x)(1 - \sigma(x))$ — multiply that product out and the two agree. The second form is the one worth remembering, because it says the slope is largest when $\sigma(x) = 0.5$ and collapses as the output saturates toward 0 or 1.

Plain English	Statistical symbol	Python equivalent
Input to sigmoid	$x$	x
Sigmoid output (activation)	$\sigma(x)$	sigmoid(x)
Sigmoid derivative (gradient)	$\sigma'(x)$	sigmoid_gradient(x)
Euler’s number	$e$	math.exp(1)

Interpretation: At the center, the gradient is 0.25. Small. Multiply 0.25 by itself 20 times (for 20 layers), and you get about 0.00000000000091 (9.1 × 10⁻¹³). The “blame” signal is effectively gone. This is the Sigmoid Trap: the model is “thinking,” but the gradients are too small for it to learn from its mistakes.

3. ReLU: The Simple Switch That Saved AI

By 2010, researchers had realized we didn’t need fancy S-curves. A switch would do. They introduced ReLU (Rectified Linear Unit).

The rule is straightforward: if the number is positive, keep it. If it’s negative, zero it out.

def relu(x):
    return max(0, x)

def relu_gradient(x):
    return 1.0 if x > 0 else 0.0

# Let's compare 10 layers of Sigmoid vs 10 layers of ReLU
sigmoid_chain = 0.25 ** 10
relu_chain = 1.0 ** 10

print(f"Sigmoid signal after 10 layers: {sigmoid_chain:.10f}")
print(f"ReLU signal after 10 layers: {relu_chain:.10f}")

• def relu(x): return max(0, x) — ReLU in one line: if the input is positive, pass it through unchanged; if negative, zero it out.
• def relu_gradient(x): return 1.0 if x > 0 else 0.0 — The derivative of ReLU is 1 for positive inputs and 0 for negative inputs. For positive inputs, the gradient passes through at full strength — it never shrinks.
• sigmoid_chain = 0.25 ** 10 — Simulates 10 layers of sigmoid, each with the maximum gradient of 0.25. The result is ≈0.00000095 — nearly zero.
• relu_chain = 1.0 ** 10 — Simulates 10 layers of ReLU, each with a gradient of 1.0. The result is 1.0 — unchanged, no matter how many layers you add.

What this actually means: A gradient of 1.0 sends the “message” through at full volume. It doesn’t shrink, no matter how many layers you stack on top.

The catch: A neuron whose input stays negative gets a gradient of 0 every time. That’s a “Dying ReLU”—permanently off. The usual fix is Leaky ReLU, which lets a sliver of signal through even for negative numbers.

Which activation should you reach for?

Approach	What it fixes	Best for	Tradeoff
Sigmoid	(original baseline)	Shallow networks, output layer for binary classification	Gradients vanish in deep networks — the derivative maxes out at 0.25
ReLU	Shrinking gradients	Most hidden layers in deep networks	“Dying ReLU” — if inputs go negative, the neuron permanently outputs 0
Leaky ReLU	Dying ReLU	Networks prone to dead neurons	Slightly more complex; a small negative slope (e.g. 0.01) keeps dead neurons alive
ResNets	Long-range gradient flow	Very deep networks (50+ layers)	Extra memory for skip connections; more parameters
BatchNorm	Internal covariate shift	Stabilizing any deep network	Depends on batch size; adds two learnable parameters per layer

Rule of thumb: Start with ReLU in hidden layers. If you see dead neurons, switch to Leaky ReLU. Add BatchNorm if training is unstable. Add residual connections once your network exceeds ~20 layers. Use Sigmoid only for the final output layer when you need a probability.

4. ResNets: Building a ‘High-Speed Lane’ for Gradients

Even with ReLU, very deep networks (like 100+ layers) still struggled. The signal would get “confused” as it passed through so many transformations.

In 2015, Microsoft researchers introduced Residual Connections (or Skip Connections). Think of this as a “High-Speed Lane” on a highway. Instead of forcing the signal through every stop in a layer, we give it a shortcut. The signal can skip the layer entirely.

Here’s what a Residual Block looks like in pseudo-code:

def residual_block(input_data, weights):
    # 1. The 'Slow Lane': Process the data through the layer
    processed = some_math_layer(input_data, weights)
    
    # 2. The 'High-Speed Lane': Add the original input back to the result
    output = processed + input_data
    return output

• processed = some_math_layer(input_data, weights) — The “slow lane”: the layer does its normal transformation (weights multiply the input, activation squashes the result).
• output = processed + input_data — The “high-speed lane”: instead of returning just the processed output, we add the original input back. This creates a gradient shortcut — the gradient can flow backward through the + input_data term without being multiplied by any weights, so it never shrinks.

The tricky part: why does adding the input back help? It gives the gradient a direct path to flow backward without being touched by the weights. If a layer turns out useless, the network just sets the processed part to zero. The signal passes through the shortcut unchanged. That’s what let us train networks with thousands of layers.

5. Batch Normalization: Keeping the Signal Steady

Even with shortcuts and ReLU, layers can still go rogue. One layer starts outputting huge numbers, which forces the next layer to output even larger ones. This is called “Internal Covariate Shift.”

Batch Normalization (BatchNorm) works like a thermostat for your data. After every layer, it looks at a batch and recalibrates: re-centering so the average is 0 and the spread is 1.

import statistics

data = [10.5, 12.2, 9.8, 15.1, 11.0] # Data is 'drifting' high

def batch_norm(batch):
    mu = statistics.mean(batch)
    sigma = statistics.pstdev(batch)  # population stdev — this is what BatchNorm actually uses
    # Subtract the mean and divide by the spread
    return [(x - mu) / sigma for x in batch]

normalized = batch_norm(data)
print(f"Normalized data: {[round(n, 2) for n in normalized]}")
print(f"New Mean: {round(statistics.mean(normalized), 2)}")

• mu = statistics.mean(batch) — Compute the mean of the current batch of data. With the test data, this is ≈11.72.
• sigma = statistics.pstdev(batch) — Compute the population standard deviation (spread) of the batch — BatchNorm divides by N, not N - 1, since it’s normalizing the literal batch statistics rather than estimating a population parameter from a sample. With the test data, this is ≈1.86.
• return [(x - mu) / sigma for x in batch] — Subtract the mean and divide by the standard deviation. This re-centers the data to mean 0 and standard deviation 1, preventing any layer from drifting into extreme ranges where activation functions flatten out.
• The test call shows data drifting high (10.5–15.1); after normalization, the mean is 0.0 and the values are centered around zero.

Interpretation: The mean is now 0.0. By keeping the data centered, BatchNorm ensures the inputs to the next layer don’t get stuck in the flat zones of activation functions. It makes the network much less sensitive to how you initialize your weights.

6. Putting It All Together: A Stable Deep Network

So we have three tools working together to keep deep networks trainable:

1. ReLU keeps the gradient from shrinking at the activation step.
2. BatchNorm stops the data from drifting into extreme ranges.
3. ResNets give the gradient a shortcut across long distances.

Before training, you can run a quick “Stability Checklist” in your code to check gradient health:

def check_stability(gradient):
    if abs(gradient) < 1e-7:
        return "Vanishing! Layer won't learn."
    elif abs(gradient) > 100:
        return "Exploding! Math will break."
    else:
        return "Healthy."

print(f"Status: {check_stability(0.000000001)}")

• if abs(gradient) < 1e-7: — If the gradient is smaller than 0.0000001, the weight updates will be so tiny they have no effect — the layer is effectively frozen.
• elif abs(gradient) > 100: — If the gradient exceeds 100, the weight updates will be so large that the loss diverges — the math breaks.
• else: return "Healthy." — Between these extremes, the gradient is large enough to learn but small enough to be stable.
• The test call passes 0.000000001 (1e-9), which is well below the 1e-7 threshold, so it returns “Vanishing! Layer won’t learn.”

The gradient problem is settled here — for networks like this one, where the signal only has to survive a stack of distinct layers, each with its own weights. (Networks that reuse the same weights over and over, like the recurrent networks Lina meets in Part 5, run into a related but different version of this problem — one that ReLU alone can’t fix, since there’s no new activation function saving the signal at each step, just the same shrinking weight applied again and again.) Now Lina is curious about what happens when the input is a whole image rather than a handful of numbers—say, a listing photo of a book she needs to verify. Part 4 covers how Convolutional Neural Networks (CNNs) give computers a way to see and understand images.

Explore more: Try changing the 0.1 in the first code block to 0.99. How many more layers can the signal survive before it vanishes?

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What does ReLU do to a positive input, and what does it do to a negative input?

Understand In your own words, explain why the Sigmoid activation function causes gradients to vanish in deep networks—use the idea of the “flat” parts of the S-curve in your answer.

Apply Using the article’s exploding-signal formula (multiplying by 1.1 once per layer), what would the signal be after 30 layers instead of 50? Using the stability-checklist’s thresholds (vanishing below 1e-7, exploding above 100), would that count as vanishing, exploding, or healthy?

Analyze ReLU’s gradient chain stays at exactly 1.0 no matter how many layers you add (since 1.0 raised to any power is still 1.0), while Sigmoid’s gradient shrinks with every layer. Walk through why this specific mathematical property—not just “ReLU is different”—is what actually solves the vanishing gradient problem.

Evaluate The article calls Leaky ReLU the fix for “Dying ReLU.” Critique plain ReLU: under what real training circumstance would a neuron’s input become “always negative,” and why might that be more common than you’d expect in a poorly-initialized network?

Create Design your own stability-check function that adds a “borderline—watch closely” zone the article’s version doesn’t have (its version only recognizes “vanishing,” “exploding,” and “healthy,” with nothing in between the extremes and healthy). What threshold would you pick for the borderline zone, and why?

---

Related articles

• Backpropagation Intuitively: How Networks Learn From Their Mistakes)
• Why Computers See Better With CNNs: An Intuitive Guide to Image Recognition)

References & Further reading

• Hochreiter, S. (1991). “Untersuchungen zu dynamischen neuronalen Netzen.” Master’s thesis, Technische Universität München. — The original thesis that identified and analyzed the vanishing gradient problem in recurrent neural networks.
• Glorot, X., & Bengio, Y. (2010). “Understanding the difficulty of training deep feedforward neural networks.” In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 249–256. — The landmark analysis showing why sigmoid networks fail to train deep architectures and how proper initialization and ReLU activations help.

---