LSTMs and GRUs: Giving Networks a Memory

In Part 5, Lina saw why standard Recurrent Neural Networks (RNNs) are like goldfish. They have a short-term memory problem. Because of the Vanishing Gradient, the signal from the start of a long session fades before it reaches the end.

If you’re tracking a 500-page-view session, you need that rare $800 first-edition book view from page 1 to stick around — it’s what lets you flag a high-value customer by the end. A standard RNN usually forgets that signal within the first few dozen pages. So we’re building a better memory system.

1. The Goldfish Problem: Why standard networks forget

A standard RNN works like a customer session log. The first page view says “this customer clicked into a rare $800 first-edition book.” By the time the log passes through twenty more page views, the model has essentially lost track of it.

This is the vanishing gradient in mathematical terms. The network loses its train of thought. So let’s see what happens when a simple RNN tries to remember a signal from the very start of a long sequence.

import torch
import torch.nn as nn

# A simple sequence: the first number is a high-value signal (1.0 = viewed an $800 handbag)
# The rest are 'filler' page views (0s). Can the RNN remember that first signal at the end?
sequence_length = 20
input_data = torch.zeros(1, sequence_length, 1)
input_data[0, 0, 0] = 1.0  # The high-value view happens at the very first page

model = nn.RNN(input_size=1, hidden_size=1, batch_first=True)
# We initialize weights to 0.5 to simulate the 'fading' effect
for name, param in model.named_parameters():
    if 'weight' in name:
        nn.init.constant_(param, 0.5)

output, hidden = model(input_data)
print(f"Value at step 1: {output[0, 0, 0]:.4f}")
print(f"Value at step 20: {output[0, 19, 0]:.4f}")

• input_data = torch.zeros(1, sequence_length, 1) — Creates a batch of 1 sequence, 20 steps long, 1 feature per step, all zeros. The shape is (batch, seq, feature).
• input_data[0, 0, 0] = 1.0 — Sets the very first step to 1.0: the “high-value” signal at page 1. Everything else stays zero.
• model = nn.RNN(input_size=1, hidden_size=1, batch_first=True) — A minimal RNN with 1 input feature and 1 hidden unit. batch_first=True means the input tensor is shaped (batch, seq, feature).
• for name, param in model.named_parameters() — Iterates over all named parameters (weights and biases) in the model.
• if 'weight' in name: nn.init.constant_(param, 0.5) — Sets every weight to 0.5 to simulate the “fading” effect: each step multiplies the signal by 0.5, so after 20 steps the original signal has been multiplied by $0.5^{19}$.
• output, hidden = model(input_data) — output contains the hidden state at every step; hidden is just the final state. We use output[0, 0, 0] (step 1) and output[0, 19, 0] (step 20) to compare.

What this shows: The value at step 1 starts meaningfully above zero. By step 20, it has decayed to essentially 0.0000. The high-value signal is gone. If the model needs that information to decide whether this customer is worth a premium retargeting ad at the end of the session, it fails. It is guessing in the dark.

(Note: this model’s weights aren’t seeded, so the exact step-1 value will vary slightly each time you run it — the “decays to ~0 by step 20” pattern is what matters, not the precise first number.)

2. The LSTM: A filing cabinet for your data

Researchers built the LSTM (Long Short-Term Memory) to solve this. Instead of a single “suitcase” that gets messy, picture an LSTM as a person managing a filing cabinet.

That person follows three strict rules, called Gates:

1. The Forget Gate: Check the old files for anything stale. If a customer closes the tab on a product, we don’t need to keep tracking its price. Toss it.
2. The Input Gate: Check the new information and decide if it’s worth saving. If the customer views a premium product, write it down and file it.
3. The Output Gate: Based on what’s in the cabinet and what just came in, what do we say right now?

Here’s how that gating logic looks in code. We use a sigmoid function (which outputs a number between 0 and 1) as a “valve.” 0 means closed — forget everything — and 1 means open, keep everything.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def lstm_cell_logic(input_val, prev_cell_state):
    # Forget gate: Should we keep the old memory?
    # Let's say we decide to keep 90% of it
    forget_gate = 0.9 
    
    # Input gate: Should we add the new info?
    # Let's say this info is very important (1.0)
    input_gate = 1.0
    new_info = np.tanh(input_val)
    
    # The magic happens here: The Cell State update
    # We multiply old memory by the forget gate and ADD the new info
    updated_cell_state = (prev_cell_state * forget_gate) + (new_info * input_gate)
    
    return updated_cell_state

memory = 10.0 # Our high-value signal from earlier
for i in range(20):
    memory = lstm_cell_logic(0.0, memory) # Adding 'filler' info

print(f"LSTM memory after 20 steps: {memory:.4f}")

• def sigmoid(x): return 1 / (1 + np.exp(-x)) — Manual sigmoid: squashes any real number into the range $(0, 1)$. In a real LSTM, each gate learns its own sigmoid weights; here we hardcode the values for clarity.
• forget_gate = 0.9 — Hardcoded to “keep 90%.” A real LSTM computes this from the current input and previous hidden state.
• input_gate = 1.0 — Fully open: accept all new information. Again, normally learned.
• new_info = np.tanh(input_val) — The “candidate” value: squashes the incoming signal into $[-1, 1]$ so it can be cleanly added to the cell state.
• updated_cell_state = (prev_cell_state * forget_gate) + (new_info * input_gate) — The core LSTM update: shrink old memory by the forget gate, then add the gated new info. This addition — not just multiplication — is what keeps the signal alive across many steps.
• memory = 10.0 — The starting value representing the high-value signal from page 1.
• for i in range(20): memory = lstm_cell_logic(0.0, memory) — Feeds 20 steps of “filler” (input 0.0) to see whether the original signal of 10 survives. It does: after 20 steps of 0.9× decay, $10 \times 0.9^{20} \approx 1.22$, still non-zero.

Interpretation: The RNN hit zero, but the LSTM memory is still at 1.22. It protected the original value because the math uses addition rather than constant multiplication. Addition keeps the signal from shrinking into nothing.

LSTM Gate Equations

At each time step $t$, the LSTM computes three gates and a candidate value, then updates the Cell State and Hidden State:

$f_t = \sigma(W_f \cdot [h_{t-1},\; x_t] + b_f)$

$i_t = \sigma(W_i \cdot [h_{t-1},\; x_t] + b_i)$

$\tilde{C}_t = \tanh(W_C \cdot [h_{t-1},\; x_t] + b_C)$

$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$

$o_t = \sigma(W_o \cdot [h_{t-1},\; x_t] + b_o)$

$h_t = o_t \odot \tanh(C_t)$

The forget gate $f_t$ decides how much of the old cell state to keep. The input gate $i_t$ decides how much new candidate information $\tilde{C}_t$ to write. The output gate $o_t$ decides how much of the updated cell state to expose as the hidden state $h_t$. The $\odot$ symbol is the Hadamard (element-wise) product.

Plain English	Statistical symbol	Python equivalent
Forget gate (how much old memory to keep)	$f_t$	forget_gate
Input gate (how much new info to add)	$i_t$	input_gate
Candidate cell value (squashed new input)	$\tilde{C}_t$	new_info
Cell state (long-term memory)	$C_t$	updated_cell_state
Output gate (how much to reveal now)	$o_t$	(not shown in simplified code)
Hidden state (short-term output)	$h_t$	hn
Previous hidden state	$h_{t-1}$	prev_hidden_state
Current input	$x_t$	input_val
Sigmoid activation	$\sigma$	sigmoid
Hadamard (element-wise) product	$\odot$	*
Hyperbolic tangent	$\tanh$	np.tanh

3. This is the hardest part: The Cell State vs. Hidden State

LSTM code returns two things: h (Hidden State) and c (Cell State). This trips up a lot of beginners.

Here’s the mental model:

• The Cell State (c): Your notebook. The long-term record. Information flows through it with very little interference. That early high-value signal stays here for 50 steps.
• The Hidden State (h): What you’re currently thinking. A filtered version of the notebook, combined with the current word, used to make a prediction right now.

So how do they differ during a sequence?

lstm = nn.LSTM(input_size=1, hidden_size=1, batch_first=True)
input_seq = torch.randn(1, 5, 1)

output, (hn, cn) = lstm(input_seq)
print(f"Hidden State (Short-term): {hn.item():.4f}")
print(f"Cell State (Long-term): {cn.item():.4f}")

• lstm = nn.LSTM(input_size=1, hidden_size=1, batch_first=True) — A single-layer LSTM with 1 input feature and 1 hidden unit. batch_first=True means the input tensor shape is (batch, seq, feature).
• input_seq = torch.randn(1, 5, 1) — Random input: 1 batch, 5 time steps, 1 feature per step. Each value is drawn from a standard normal distribution.
• output, (hn, cn) = lstm(input_seq) — The LSTM returns a tuple: output has the hidden state at every step, and (hn, cn) is a tuple of the final hidden state and final cell state. hn is the short-term memory; cn is the long-term “notebook.”
• hn.item() / cn.item() — .item() extracts a Python float from a single-element tensor. Both are scalars here because hidden_size=1.

What this actually means: The Cell State is usually larger or more complex because it holds the raw history. The Hidden State is the refined version, ready for the next layer of the network.

4. The GRU: The ‘Lite’ version of memory

LSTMs are great, but they are heavy. All those parameters slow down training. In 2014, researchers introduced the GRU (Gated Recurrent Unit).

Think of the GRU as the efficient younger sibling. It skips the separate filing cabinet (the Cell State). Instead, it relies on a “sticky note” (a single Hidden State). It combines the Forget and Input gates into one Update Gate.

• LSTM: “Should I forget the old stuff? Okay. Now, should I add the new stuff?”
• GRU: “How much of the new stuff should replace the old stuff?”

So, how do they compare on a small task?

import time

def benchmark(model_type, data):
    model = model_type(1, 64, batch_first=True)
    start = time.time()
    for _ in range(100):
        _ = model(data)
    return time.time() - start

data = torch.randn(32, 100, 1)
lstm_time = benchmark(nn.LSTM, data)
gru_time = benchmark(nn.GRU, data)

print(f"LSTM time: {lstm_time:.4f}s")
print(f"GRU time: {gru_time:.4f}s")

• def benchmark(model_type, data) — A generic timing function that works with either nn.LSTM or nn.GRU passed as the model_type argument.
• model = model_type(1, 64, batch_first=True) — Creates either an LSTM or GRU with input_size=1 and hidden_size=64. Both classes accept the same constructor arguments.
• for _ in range(100): _ = model(data) — Runs 100 forward passes (no training) to get a stable wall-clock timing. The _ discards each output.
• data = torch.randn(32, 100, 1) — 32 sequences of length 100 with 1 feature each — a realistic batch size and sequence length for benchmarking.
• lstm_time = benchmark(nn.LSTM, data) / gru_time = benchmark(nn.GRU, data) — Times both architectures on identical data so the comparison is fair.

Interpretation: The GRU is almost always faster (usually 15-20% faster). It performs just as well as an LSTM on many tasks. If you are short on computing power, I’d lean toward starting with a GRU.

LSTM vs. GRU: Which Memory Architecture Should You Reach For?

Approach	What it does	Best for	Tradeoff
LSTM	Maintains a separate Cell State with three gates (forget, input, output). The Cell State acts as a protected “notebook” that carries long-range signal across many steps.	Long sequences where protecting an early signal is critical; tasks where you need fine-grained control over what to remember vs. what to forget.	More parameters (~33% more than a GRU for the same hidden size), slower to train.
GRU	Uses a single Hidden State with two gates (update, reset). No separate Cell State — the update gate merges “forget old” and “add new” into one decision.	Most tasks; short-to-medium sequences; compute-constrained settings; when you want a fast baseline.	Fewer parameters and faster, but no dedicated long-term storage. May struggle on very long sequences where the LSTM’s Cell State provides an advantage.

Rule of thumb: Start with a GRU for speed and simplicity — it trains faster and often matches LSTM accuracy. Reach for an LSTM when you need the Cell State to protect a specific long-range signal (like a rare-book view on page 1 that must survive 100+ steps to influence a prediction at the end). Empirically, the gap narrows on shorter sequences but widens on longer ones.

5. Putting it to work: Predicting purchase intent

Can a memory-capable network solve a simple prediction task? We’ll give it a session: the customer views a rare $800 first-edition book on step 1, then browses four unrelated, cheap items. We want it to predict a high purchase-value score from that first signal, five steps later.

class Predictor(nn.Module):
    def __init__(self, mode='LSTM'):
        super().__init__()
        if mode == 'LSTM':
            self.rnn = nn.LSTM(1, 10, batch_first=True)
        else:
            self.rnn = nn.GRU(1, 10, batch_first=True)
        self.fc = nn.Linear(10, 1)

    def forward(self, x):
        out, _ = self.rnn(x)
        return self.fc(out[:, -1, :]) # Predict based on last state

# Imagine 0.8 represents 'viewed the $800 handbag' and we want to predict 0.9 (high purchase-value likelihood)
model = Predictor(mode='LSTM')
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

x = torch.tensor([[[0.8], [0.1], [0.1], [0.1], [0.1]]]) # 'handbag view' followed by 4 filler page views
y = torch.tensor([[0.9]])

for epoch in range(100):
    pred = model(x)
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f"Final Prediction: {model(x).item():.4f} (Target: 0.9)")

• class Predictor(nn.Module) — A model that can be either LSTM or GRU based on the mode argument, making it easy to swap architectures.
• if mode == 'LSTM': self.rnn = nn.LSTM(1, 10, batch_first=True) — Creates an LSTM layer with 1 input feature and 10 hidden units. The 10-dimensional hidden state captures the session’s “memory.”
• else: self.rnn = nn.GRU(1, 10, batch_first=True) — Falls back to a GRU with the same dimensions (1 input, 10 hidden). Both nn.LSTM and nn.GRU share the same call signature, so swapping is trivial.
• self.fc = nn.Linear(10, 1) — A linear layer that maps the 10-dimensional final hidden state to a single scalar output (predicted purchase-value score).
• out, _ = self.rnn(x) — Runs the full 5-step sequence through the recurrent layer. out has shape (1, 5, 10) — the hidden state at every step. The discarded _ is the final (h_n, c_n) tuple (or just h_n for GRU).
• return self.fc(out[:, -1, :]) — Takes only the last time step’s hidden state (out[:, -1, :] has shape (1, 10)) and projects it through the linear layer to get a single prediction.
• criterion = nn.MSELoss() — Mean squared error: penalizes the squared difference between prediction and target. Good for regression-style outputs.
• optimizer = torch.optim.Adam(model.parameters(), lr=0.01) — Adam optimizer with learning rate 0.01. Adam adapts per-parameter learning rates and is a safe default for most small models.
• optimizer.zero_grad() — Clears accumulated gradients from the previous epoch. Without this, gradients would sum across iterations.
• loss.backward() — Computes gradients via backpropagation-through-time (BPTT), flowing the error from step 5 all the way back to step 1.
• optimizer.step() — Updates all weights using the gradients computed by backward().

Interpretation: The model hits the target almost perfectly. Because it’s an LSTM, the “0.8” signal from the first page view didn’t vanish. The loss (the “surprise” factor) dropped to near zero—the model carried the memory of that early rare-book view through the filler steps and made an accurate prediction at the end.

Recap:

• Standard RNNs forget because gradients vanish during multiplication.
• LSTMs use Gates as a filing cabinet, protecting important information.
• The Cell State is the long-term notebook; the Hidden State is the short-term thought.
• GRUs are faster, streamlined versions of LSTMs that work well for most tasks.

Even with LSTMs, there’s a limit. If a session runs 1,000 page views, even a filing cabinet gets full. To solve that, we need to stop trying to remember everything and start paying Attention.

With a working memory system, Lina’s model can now remember the whole session—but processing every page view one at a time is slow. There’s a faster way to zero in on what matters.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three Gates in an LSTM, and what question does each one answer?

Understand In your own words, explain why the LSTM’s cell-state update uses addition instead of just multiplication, and why that specifically prevents the signal from shrinking to zero.

Apply Using the article’s lstm_cell_logic formula (updated_cell_state = prev_cell_state * forget_gate + new_info * input_gate), calculate the cell state after 2 steps starting from memory = 5.0, with forget_gate = 0.8, input_gate = 1.0, and new_info = tanh(1.0) added at each step.

Analyze The article says the Cell State is a “notebook” (long-term) and the Hidden State is “what you’re currently thinking” (short-term). Walk through what would go wrong if a model only had a Hidden State and no separate Cell State—why isn’t the short-term version enough on its own?

Evaluate The article recommends GRUs when you’re “short on computing power” since they’re 15-20% faster with similar performance. Critique that advice: describe a scenario where the LSTM’s extra Cell State (which the GRU doesn’t have) would likely matter enough to be worth the slower training.

Create Design a scenario (a specific sequence of page-view signals) where you’d expect the LSTM’s Forget Gate to matter a lot—where forgetting old information at the right moment is just as important as remembering the beginning.

See you in Part 7, where we build the mechanism that changed AI forever: Attention.

---

Related articles

• Why RNNs Forget: The Intuition Behind the Vanishing Gradient Problem
• The Attention Mechanism, Finally Explained Without the Math

References & Further reading

• Hochreiter, S., & Schmidhuber, J. (1997). “Long Short-Term Memory.” Neural Computation, 9(8), 1735–1780. — The foundational paper introducing the LSTM architecture and the gated cell state that solved the vanishing gradient problem for recurrent networks.
• Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.” Proceedings of EMNLP 2014. — The paper that introduced the GRU, a simplified gated recurrent architecture with fewer parameters than the LSTM.

---