Why RNNs Forget: The Intuition Behind the Vanishing Gradient Problem

In Part 2, Lina saw how a neural network learns by “distributing blame.” She used backpropagation to walk backward from a mistake and nudge her weights until the model got smarter. Part 3 picked up that thread and appeared to close it: ReLU, ResNets, and BatchNorm tamed the vanishing gradient problem for deep networks. Part 4 then took a detour into a different kind of depth — Convolutional Neural Networks, for images. That detour is complete; we’re back on the sequence thread Part 2 opened, and heading straight into a data type where the backpropagation story breaks down again: sequences.

“Again” needs an asterisk, though. Part 3’s fixes all target depth: a signal traveling through many stacked layers, each with its own independent weights. RNNs create a different failure mode. A single RNN cell reuses the exact same recurrent weight at every time step — there’s no fresh ReLU, no fresh BatchNorm, no new set of weights to rescue the signal at step 30 that failed at step 29. The same multiplication just happens again, over and over, across time instead of depth. That’s why the vanishing gradient is back even though Lina already has three tools that supposedly solved it — those tools fix a shrinking chain of different weights, not a shrinking chain of the same weight repeated.

A sentence, a stock price, a heartbeat, a customer’s browsing session on BookSight — sequences have a beginning and an end. To understand the end, you usually need to remember the beginning. Standard Recurrent Neural Networks (RNNs) have a very short memory.

1. The Memory Problem: Why Can’t Your Model Remember the Start of the Session?

Say Lina is tracking a customer’s browsing session on BookSight, 50 pages deep. On page 1, the customer clicked into a rare $800 first-edition book. If you guessed their budget from page 50 alone—a $12 bookmark—you’d miss that early high-value signal entirely.

To reach page 50, that early context has to pass through 49 intervening pages of whatever else the customer browsed. The expensive first click fades because the information degrades at each step along the way.

Picture an RNN as someone reading a session log one page view at a time, holding a single sticky note. To make sense of page 50, the model checks its sticky note—a summary of pages 1 through 49. But by page 50, the ink from page 1 has been written over 49 times. The model loses the context it needs.

Let’s simulate that “leaky” memory in Python. We’ll track a running signal in a buffer that loses 20% of its old information every time a new page view arrives.

def leaky_memory_sim(sequence):
    memory = 0.0
    decay_rate = 0.8 # We keep 80%, lose 20%
    
    for i, val in enumerate(sequence):
        # Add new info, but old info fades
        memory = (memory * decay_rate) + val
        print(f"Step {i+1}: Current Memory Value = {memory:.4f}")
    return memory

# We start with a '10' (that $800 handbag view) and then add '1's (routine page views)
data = [10, 1, 1, 1, 1, 1]
final_mem = leaky_memory_sim(data)

• memory = 0.0 — The hidden state starts at zero: nothing has been seen yet.
• decay_rate = 0.8 — Each step, only 80% of the old memory survives. The other 20% is lost — this is the “leak.”
• memory = (memory * decay_rate) + val — The core update: shrink the old memory, then add the new page view’s value on top. The old signal fades with every step.
• data = [10, 1, 1, 1, 1, 1] — The first value (10) represents the rare, expensive book view; the 1s are routine browsing. By Step 6, the 10 has been multiplied by 0.8 five times, shrinking its contribution to roughly 3.3 — already a fraction of its original weight.

Here’s what that means: the value ‘10’ mattered at Step 1. By Step 6, five rounds of decay have diluted its influence. A model guessing “is this a big spender?” purely from memory at page 6 would already be losing track of that early high-value signal from page 1. By page 50, it’s essentially gone.

2. The Hidden State: A Single Suitcase for All Your Data

So how does an RNN actually work? Unlike the simple neurons from Part 1, an RNN loops. It doesn’t take in the whole session at once. Instead, it processes one page view alongside a Hidden State carried over from the past.

Picture the hidden state as a suitcase. At each step, the model looks at a new page view and decides how to pack it in. The suitcase has a fixed size, though. Put something new in, and something old falls out.

In math, that update is $h = \tanh(Wx + Uh)$. We multiply the new input ($x$) by some weights, multiply the old suitcase ($h$) by other weights, add them together, and squash the result so values don’t explode.

RNN Hidden State Update

$h_t = \tanh(W \cdot x_t + U \cdot h_{t-1})$

At each time step $t$, the new hidden state $h_t$ is formed by combining the current input $x_t$ (weighted by $W$) with the previous hidden state $h_{t-1}$ (weighted by $U$), then passing the result through $\tanh$ to keep values bounded between $-1$ and $1$.

Plain English	Statistical symbol	Python equivalent
New input at current step	$x_t$	new_word_vec
Previous hidden state	$h_{t-1}$	old_hidden_state
Weight matrix for input	$W$	W (0.5)
Weight matrix for hidden state	$U$	U (0.8)
Updated hidden state	$h_t$	new_hidden_state
Nonlinear squashing function	$\tanh$	np.tanh

import numpy as np

def rnn_step(new_word_vec, old_hidden_state):
    # Random weights for demonstration
    W = 0.5 
    U = 0.8
    
    # The 'Squashing' function (tanh) keeps values between -1 and 1
    combined = (new_word_vec * W) + (old_hidden_state * U)
    new_hidden_state = np.tanh(combined)
    
    return new_hidden_state

# Let's process 3 page views
hidden = 0.0
pageviews = [1.0, 0.5, -0.2]
for w in pageviews:
    hidden = rnn_step(w, hidden)
    print(f"New Suitcase State: {hidden:.4f}")

• W = 0.5 and U = 0.8 — Scalar weights standing in for full weight matrices. W controls how much the new input matters; U controls how much the old memory matters.
• combined = (new_word_vec * W) + (old_hidden_state * U) — The core RNN update: weight the incoming page view, weight the previous suitcase, and add them together.
• np.tanh(combined) — Squashes the combined value into the range $[-1, 1]$. Without this, values would grow without bound across many steps.
• hidden = rnn_step(w, hidden) — Each iteration feeds the updated hidden state back in as input to the next step — the “recurrent” loop.

Here’s what happens when we run rnn_step: the old_hidden_state gets multiplied by U (0.8) each time. Do that 100 times and the original information has been multiplied by 0.8 one hundred times over. Let’s see what that does to the signal.

3. The Vanishing Gradient: Why the Model Stops Learning

This is the hardest part: how does the model learn from its mistakes when the sequence runs long? As Lina saw in Part 2, the Chain Rule passes the “blame” backward through the network.

To correct an error at page 50, the signal has to travel all the way back to page 1. Picture a whisper that gets quieter with each pass. When the RNN weights are small—less than 1.0—multiplying them repeatedly shrinks the gradient until it vanishes.

That’s the Vanishing Gradient Problem. Take a gradient of 0.0000001. The weight update (Gradient × Learning Rate) becomes so tiny that the weight doesn’t budge. The model stops hearing the error signal from the start of the sequence.

gradient = 1.0
weight = 0.7 # A weight less than 1.0

print("Tracing the gradient back through time:")
for step in range(1, 21):
    gradient = gradient * weight
    if step % 5 == 0:
        print(f"Step {step} back: Gradient = {gradient:.6f}")

• gradient = 1.0 — We start with a full-strength error signal (gradient of 1.0) at the end of the sequence.
• weight = 0.7 — Each step back through time multiplies the gradient by the recurrent weight. Because 0.7 < 1.0, the signal shrinks with every multiplication.
• gradient = gradient * weight — The repeated multiplication that causes vanishing: $0.7^{20} \approx 0.0008$, essentially zero.
• if step % 5 == 0 — Prints every 5th step so we can watch the gradient decay: 0.7, 0.49, 0.343, 0.2401… by step 20 it’s 0.000798.

Here’s what’s happening: by step 20, the gradient sits at 0.000798—essentially zero. The model can’t see what happened at the start of the sequence. It never learns that the rare book view on page 1 caused a prediction error 50 pages later. The message got lost in the mail.

4. Seeing it in Action: The ‘Parity’ Task

Let’s prove this with code. The task is Parity. We hand the model a sequence of 0s and 1s, and it has to say whether the count of 1s is even or odd. To get this right, the model must remember every single number it has seen.

So we’ll compare a simple RNN on a short sequence (length 5) versus a long one (length 50).

import torch
import torch.nn as nn
import itertools

class SimpleRNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.rnn = nn.RNN(input_size=1, hidden_size=10, batch_first=True)
        self.fc = nn.Linear(10, 1)

    def forward(self, x):
        out, _ = self.rnn(x)
        # We only care about the last prediction
        return torch.sigmoid(self.fc(out[:, -1, :]))

def train_and_eval(x_train, y_train, epochs=400, seed=0):
    torch.manual_seed(seed)
    model = SimpleRNN()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
    loss_fn = nn.BCELoss()
    for _ in range(epochs):
        optimizer.zero_grad()
        prediction = model(x_train)
        loss_fn(prediction, y_train).backward()
        optimizer.step()
    with torch.no_grad():
        accuracy = ((model(x_train) > 0.5).float() == y_train).float().mean().item()
    return accuracy

# Length 5: only 32 possible sequences exist, so we train on every single one
combos = list(itertools.product([0, 1], repeat=5))
x_len5 = torch.tensor(combos, dtype=torch.float32).unsqueeze(-1)
y_len5 = x_len5.sum(dim=1) % 2
accuracy_len5 = train_and_eval(x_len5, y_len5)

# Length 50: too many sequences to enumerate, so we train on a large random sample
torch.manual_seed(0)
x_len50 = torch.randint(0, 2, (2000, 50, 1)).float()
y_len50 = x_len50.sum(dim=1) % 2
accuracy_len50 = train_and_eval(x_len50, y_len50)

print(f"Length 5 accuracy: {accuracy_len5:.2f}")
print(f"Length 50 accuracy: {accuracy_len50:.2f}")

• nn.RNN(input_size=1, hidden_size=10, batch_first=True) — A single-layer RNN with 10 hidden units. input_size=1 because each sequence element is a single number; batch_first=True means the input tensor shape is (batch, sequence, features).
• self.fc = nn.Linear(10, 1) — A final linear layer that maps the 10-dimensional hidden state to a single output (even/odd prediction).
• out, _ = self.rnn(x) — Runs the full sequence through the RNN. out contains the hidden state at every step; the discarded _ is the final hidden state (redundant with out[:, -1, :]).
• torch.sigmoid(self.fc(out[:, -1, :])) — Takes only the last time step’s hidden state, projects it to a single logit, and squashes it through sigmoid to get a probability. The [:, -1, :] slice selects the final step across all batch elements.
• train_and_eval(...) — Trains the model with Adam for a fixed number of epochs on whatever data it’s handed, then reports the fraction of predictions on the correct side of 0.5.
• combos = list(itertools.product([0, 1], repeat=5)) — Length-5 binary sequences have only 2**5 = 32 possible combinations, so instead of sampling, we train directly on every single one — nothing is left unseen.
• x_len50 = torch.randint(0, 2, (2000, 50, 1)).float() — Length-50 sequences can’t be enumerated (2**50 of them), so we train on 2,000 random samples instead.
• The two print calls report what actually happened when this code ran: Length 5 accuracy: 1.00 — every sequence learned. Length 50 accuracy: 0.54 — barely better than a coin flip.

Run it, and the numbers confirm the story:

• Length 5: 1.00 — perfect accuracy. Five bits of info fit in its suitcase without trouble.
• Length 50: 0.54 — barely above a coin flip.

A result that close to 50% on a binary Yes/No task means the model is close to guessing randomly. The vanishing gradient stops the error signal from reaching the first 40-plus numbers. The model only “looks” at the last few and guesses from that incomplete picture.

RNNs vs. Truncating or Summarizing Long Sessions

Approach	What it does	Best for	Tradeoff
Truncate (last-N pages)	Feed only the most recent N page views to a simple model	Short sessions, or when recent actions dominate the prediction	Throws away early signal entirely — that rare book viewed on page 1 is invisible.
Summarize (aggregate stats)	Collapse the whole session into features like total views, max price, category counts	Quick baselines, tabular models	Loses all order information — can’t tell “rare book first, then browsing” from the reverse.
RNN	Process the full sequence step by step, maintaining a hidden state	Sessions where order and long-range dependencies matter	Vanishing gradient on long sequences — struggles to connect page 1 to page 50 without architectural help.

Rule of thumb: Reach for a simple RNN when sessions are short and order matters. For longer sessions, truncation or summarizing gives a fast baseline — but if early actions (like a rare book view) strongly influence later predictions, the sequential model is worth the complexity, and you’ll need a better memory mechanism (LSTMs and GRUs, covered in Part 6).

5. Where Do We Go From Here?

We’ve hit a wall. Vanilla RNNs handle short bursts fine, but they’re blind to the distant past. They try to cram everything into one small suitcase, and backpropagation causes that memory to fade.

So what does that mean for Lina? She needs a memory system that doesn’t leak before she can trust her purchase-prediction model on full-length browsing sessions.

Next up: LSTMs (Long Short-Term Memory). These models add a “long-term” storage vault—a way to protect important information so it doesn’t get overwritten or multiplied into oblivion.

Recap of what we learned:

• RNNs process sequences one step at a time using a Hidden State (the suitcase).
• The Vanishing Gradient happens when we multiply small numbers over many steps, making the learning signal disappear.
• Long-range dependencies (like the start of a long session) are impossible for basic RNNs to learn.

Part 6 covers how we fix the memory leak: LSTMs and the Gated Cell.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the “Hidden State” in an RNN, and what analogy does the article use to describe it?

Understand In your own words, explain why an RNN with a weight less than 1.0 loses the ability to use information from early in a long sequence.

Apply Using the article’s leaky-memory formula (memory = memory * decay_rate + new_value), calculate the memory value after 3 steps if decay_rate = 0.5 and the incoming sequence is [8, 0, 0].

Analyze The parity task showed the RNN hits 100% accuracy on length-5 sequences but only about 50% on length-50 sequences. Walk through why 50% accuracy specifically indicates “random guessing” for this task, rather than just “somewhat worse performance.”

Evaluate The article’s leaky-memory simulation uses one fixed decay_rate for every step. Critique this: real customer sessions have page views of very different importance (a product view vs. a search-bar click). What’s the limitation of a single fixed decay rate that doesn’t account for that difference?

Create Design an alternative decay schedule (not a single fixed number) that would help an RNN remember an early high-value product view specifically, even across a long session. What would trigger a slower decay at that particular step?

---

Related articles

• Why Computers See Better with CNNs: An Intuitive Guide to Image Recognition)
• LSTMs and GRUs: Giving Networks a Memory)

References & Further reading

• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). “Learning representations by back-propagating errors.” Nature, 323(6088), 533–536. — The foundational paper on backpropagation that underpins training RNNs through backpropagation-through-time (BPTT).
• Bengio, Y., Simard, P., & Frasconi, P. (1994). “Learning Long-Term Dependencies with Gradient Descent is Difficult.” IEEE Transactions on Neural Networks, 5(2), 157–166. — The seminal paper formally analyzing why standard RNNs struggle with long-range dependencies, motivating gated architectures like LSTMs.

---