Backpropagation Intuitively: How Networks Learn From Their Mistakes

In Part 1, Lina built a single neuron that predicted whether a BookSight visitor would buy. It took inputs — time on page and items in cart — multiplied them by weights, added a bias, and squashed the result into a probability. But she chose those weights herself. A real neural network has to figure them out on its own.

How does it do that? It learns by making mistakes. The process is called Backpropagation. Sounds like scary calculus. It’s really just a way of assigning blame when predictions go wrong.

The ‘Oops’ Moment: Why we need to work backwards

Imagine you’re Lina, BookSight’s junior ML engineer, reviewing yesterday’s purchase predictions. Your neuron looked at one visitor’s behavior and predicted only a 40% chance they’d buy—not very confident. But they bought anyway.

To improve the model, you don’t throw out all the weights and start over. You work backwards. Which part of the calculation was too cautious? Maybe weight_cart wasn’t aggressive enough, or the bias pulled the score down too far. You’re looking for the specific inputs that caused the error.

The “Forward Pass” is the neuron making its prediction. The “Loss” is how wrong that prediction turned out to be. Backpropagation is walking back through that calculation to figure out which weight to adjust, and by how much.

Here’s how we measure that “Oops” moment in code. We call this the Loss Function. A common one is Mean Squared Error (MSE)—essentially, “square the distance between the guess and the truth.”

def calculate_error(prediction, actual_target):
    # How far off were we?
    distance = prediction - actual_target
    
    # Square it so the error is always positive 
    # (and to penalize big mistakes more than small ones)
    loss = distance ** 2
    return loss

# Let's say the customer WAS a buyer (target=1.0), but the neuron guessed 0.4
print(f"Loss: {calculate_error(0.4, 1.0):.4f}") # Output: 0.3600

• distance = prediction - actual_target — Computes the raw difference between what the neuron guessed (0.4) and what actually happened (1.0). A negative distance means the neuron under-predicted; a positive one means it over-predicted.
• loss = distance ** 2 — Squaring the distance does two things: it makes every error positive (so under- and over-predictions don’t cancel out), and it penalizes large mistakes disproportionately more than small ones.
• The test call passes 0.4 (the neuron’s prediction) and 1.0 (the actual outcome — the visitor did buy). The result is 0.36, telling us the neuron was meaningfully wrong on this example.

A loss of 0.36 tells us we’re off. If the loss were 0.0, the prediction would be perfect. The goal of learning is to drive this number as close to zero as possible.

The Chain Rule: It’s just a game of Telephone

This is the hardest part of backpropagation: how do we know which specific weight caused the error? If the prediction was off, was it weight_cart, or was it the bias?

Think of the network as a game of Telephone.

1. The Weight affects the Sum.
2. The Sum affects the Prediction.
3. The Prediction affects the Loss.

To see how the Weight affects the Loss, multiply the sensitivities along the chain. Say a 1-unit change in weight_cart changes the sum by 2 — this visitor had 2 items in their cart. A 1-unit change in the sum then changes the prediction by 0.8. So a 1-unit change in weight_cart changes the prediction by $2 \times 0.8 = 1.6$.

In math terms, this is the Chain Rule — just figuring out how much “pull” each lever has on the final result. We call that total “pull” the Gradient.

# Let's simulate two connected parts: weight_cart -> Prediction
weight_cart = 0.5
items_in_cart = 2.0

# Sensitivity 1: How much does weight_cart affect the result?
# If result = weight_cart * items_in_cart, the sensitivity is just items_in_cart.
derivative_weight_to_output = items_in_cart

# Sensitivity 2: How much does the output affect the loss?
# (Simplified for this example)
derivative_output_to_loss = 0.8 

# The Chain Rule: Multiply them to find the total 'blame' for the weight
gradient = derivative_weight_to_output * derivative_output_to_loss
print(f"weight_cart's gradient is: {gradient}") # Output: 1.6

• derivative_weight_to_output = items_in_cart — The sensitivity of the prediction to weight_cart is just items_in_cart (2.0), because prediction = weight_cart * items_in_cart. Changing the weight by 1 unit changes the prediction by 2 units.
• derivative_output_to_loss = 0.8 — A simplified stand-in for how sensitive the loss is to the prediction. In a full sigmoid network, this would be the derivative of the sigmoid function at the current prediction value.
• gradient = derivative_weight_to_output * derivative_output_to_loss — The chain rule in one line: multiply the two local sensitivities to get the total “blame” assigned to weight_cart. The result (1.6) says a small increase in weight_cart will increase the loss by 1.6× that amount.

That 1.6 tells us: nudge weight_cart up a little, and the error rises by 1.6 times that amount. To improve, we move the weight in the opposite direction.

Let’s See What Happens: A manual backprop in Python

Let’s put this into a real but tiny script. We’ll teach weight_cart using one example: a visitor with 2 items in their cart who, in hindsight, was a strong buyer. Before Sigmoid squashes anything into a probability, we want the raw weighted score for a case like this to land around 10—big enough that after Sigmoid, the model is essentially certain. We’ll use a Learning Rate, which is just a small number that keeps us from overreacting and changing the weights too fast.

weight_cart = 0.5
items_in_cart = 2.0
target_raw_score = 10.0
learning_rate = 0.1

for i in range(3):
    # 1. Forward Pass (Make a guess)
    prediction = weight_cart * items_in_cart
    loss = (prediction - target_raw_score) ** 2
    
    # 2. Calculate Gradient (The 'Blame')
    # The math: 2 * (prediction - target) * items_in_cart
    gradient = 2 * (prediction - target_raw_score) * items_in_cart
    
    # 3. Update Weight (Nudge it in the right direction)
    weight_cart = weight_cart - (learning_rate * gradient)
    
    print(f"Step {i+1}: Prediction={prediction:.2f}, Loss={loss:.2f}, New Weight={weight_cart:.2f}")

• target_raw_score = 10.0 — The desired pre-sigmoid score. A raw score of 10 would pass through sigmoid as ≈0.99995, making the model nearly certain this visitor is a buyer.
• learning_rate = 0.1 — A small step size that prevents the weight from lurching too far on any single update — the network “learns” in small increments rather than jumping to extremes.
• prediction = weight_cart * items_in_cart — The forward pass: just the weighted sum. No bias or activation in this simplified example, so the prediction is the raw product.
• loss = (prediction - target_raw_score) ** 2 — MSE loss: square the gap between the current prediction and the target. On the first iteration this is (1.0 − 10.0)² = 81.0.
• gradient = 2 * (prediction - target_raw_score) * items_in_cart — The full chain-rule derivative of MSE with respect to weight_cart. The factor of 2 comes from differentiating the square; (prediction - target) is how sensitive the loss is to the prediction; items_in_cart is how sensitive the prediction is to weight_cart. Multiply them together and you get the total gradient.
• weight_cart = weight_cart - (learning_rate * gradient) — Gradient descent: move the weight opposite to the gradient (subtract, because the gradient points toward increasing loss). The learning rate scales how big each step is.

Look at the output. The loss starts high at 81.00 and drops fast—3.24 after step 1, then 0.13 after step 2. By the third step, weight_cart has moved from 0.5 all the way to 4.96, and the prediction (9.64) is nearly at our target of 10. We’re watching the network learn.

From the Loss to the Weight Update

$$\frac{\partial}{\partial w} \left(\hat{y} - t\right)^{2}$$

the derivative we want: how does this move when $w$ moves?

$$\term{focus}{2 \left(\hat{y} - t\right) \frac{\partial}{\partial w} \left(\hat{y} - t\right)}$$

power rule on the outside, chain rule on the inside

$$2 \left(\hat{y} - t\right) \term{focus}{\frac{\partial}{\partial w} \left(- t\right) + \frac{\partial}{\partial w} \hat{y}}$$

sum rule: differentiate each term separately

$$2 \left(\hat{y} - t\right) \left(\term{focus}{0} + \frac{\partial}{\partial w} \hat{y}\right)$$

constant rule: a term with no $w$ in it cannot change with $w$

$$2 \left(\term{focus}{w x} - t\right) \frac{\partial}{\partial w} \left(\term{focus}{w x}\right)$$

substitute the model, $\hat{y} = w x$

$$2 \left(w x - t\right) \term{focus}{x \frac{\partial}{\partial w} w}$$

constant multiple rule: pull the constant out front

$$2 x \left(w x - t\right) \term{focus}{1}$$

the derivative of $w$ with respect to itself is $1$

$$2 x \left(w x - t\right)$$

so the derivative is

$$2 \cdot 2.0 \left(0.5 \cdot 2.0 - 10.0\right)$$

put the real numbers in

$$\term{focus}{-36.0}$$

and that is the gradient

Plain English	Statistical symbol	Python equivalent
Prediction (weighted sum)	$\hat{y}$	prediction
Target value	$t$	target_raw_score
Weight for cart items	$w$	weight_cart
Cart items (input)	$x$	items_in_cart
Loss (MSE)	$L = (\hat{y} - t)^2$	loss = (prediction - target_raw_score) ** 2
Gradient of loss w.r.t. weight	$\frac{\partial L}{\partial w}$	gradient = 2 * (prediction - target) * items_in_cart
Learning rate	$\eta$	learning_rate
Weight update step	$w \leftarrow w - \eta \cdot \frac{\partial L}{\partial w}$	weight_cart = weight_cart - (learning_rate * gradient)

The Vanishing Gradient: Why deep networks get ‘tired’

Here’s the catch. A deep network has many layers. To find the blame for the first layer, you multiply the sensitivities of every layer that comes after it.

If those sensitivities are small — say, 0.1 — and you multiply them over and over ($0.1 \times 0.1 \times 0.1...$), the signal shrinks to almost zero. That’s the Vanishing Gradient Problem. The early layers never learn, because the “blame” signal from the end never reaches them. They just sit there with their original, random weights.

gradient = 1.0
for layer in range(1, 11):
    # Imagine each layer squashes the signal by half
    gradient = gradient * 0.5
    print(f"Layer {layer} gradient: {gradient:.5f}")

• gradient = 1.0 — Start with a full-strength gradient signal at the output layer of the network.
• gradient = gradient * 0.5 — Each layer multiplies the incoming gradient by 0.5, simulating a layer whose sensitivity is less than 1. This is what happens when activation functions like sigmoid squash their inputs — their derivatives are small numbers.
• The loop runs 10 layers. By layer 10, the gradient has decayed to 0.00098 — less than 0.1% of its original strength. The weight update for the first layer would be so tiny that it effectively never changes, no matter how many training steps you run.

By layer 10, the gradient is 0.00098. The signal is so weak that the weight barely moves. Building very deep networks was impossible for decades — until researchers found ways to keep the signal alive. We’ll see those tricks when we get to Attention.

Recap: The 3-Step Dance of Learning

Learning in AI isn’t magic; it’s a cycle:

1. Predict (Forward): Make a guess using current weights.
2. Measure (Loss): See how far off the guess was.
3. Distribute Blame (Backprop): Use the chain rule to find which weights caused the error, then nudge them.

The network is just a self-correcting machine. So we now know how a single path learns—and Lina’s model can learn from its mistakes. Here’s the catch. As networks get deeper, that blame assignment starts to break down. Part 3 walks through the Vanishing Gradient Problem.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What does the Loss Function measure, and what would a loss of 0.0 tell us about a prediction?

Understand In your own words, explain what the Chain Rule lets us calculate about weight_cart, and why we need it instead of just guessing which direction to nudge each weight.

Apply Using the gradient formula from this article (gradient = 2 * (prediction - target) * items_in_cart), calculate the gradient for a visitor with items_in_cart = 3, where the current prediction is 2.0 and target_raw_score is 10.0. Would the weight update increase or decrease weight_cart?

Analyze In the training loop, the loss dropped from 81.00 to 3.24 to 0.13—a huge drop at first, then much smaller drops. Walk through why the gradient (and therefore the size of each weight update) naturally shrinks as weight_cart gets closer to the right value, using the gradient formula itself.

Evaluate The vanishing gradient demo assumes every layer squashes the signal by exactly half. Critique that simplification: what would have to be true about a real network’s layers for the “signal dies by layer 10” conclusion to actually hold, and when might it not?

Create Design a hypothetical layer sensitivity pattern (a sequence of multipliers, one per layer) for a 10-layer network where the gradient does not vanish by the final layer. What does your pattern require to be true about most layers’ sensitivities, and why does that avoid the problem this article describes?

---

Related articles

• Neural Networks, Without the Calculus: What’s Actually Happening Inside a Single Neuron
• Why Deep Networks Die: Solving the Vanishing Gradient

References & Further reading

• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). “Learning representations by back-propagating errors.” Nature, 323, 533–536. — The landmark paper that introduced backpropagation as a practical algorithm for training multi-layer neural networks.
• Backpropagation (Wikipedia) — A detailed overview of the backpropagation algorithm, including its mathematical derivation and historical context.

---

Apply What You Learned

Topic: How backpropagation traces prediction error backward through a chain of local sensitivities to assign blame to individual weights, and why that chain can break in deep networks.

Draw a mindmap (paper, Excalidraw, Miro — anything) with at least these nodes:

• Forward Pass
• Loss Function (MSE)
• Chain Rule
• Gradient
• Learning Rate
• Weight Update (Gradient Descent)
• Vanishing Gradient
• reference-backpropagation.md

and at least these edges:

• Forward Pass → Loss Function (MSE)
• Chain Rule → Gradient
• Gradient → Weight Update (Gradient Descent)

Rubric: all 8 named nodes present; the 3 required edges drawn; one extra edge of your own with a one-sentence justification of why you added it.

Lina shipped the BookSight training-loop service to production overnight. This morning the monitoring dashboard shows the loss is increasing with every training step — starting at 81.00, then jumping to 262.44 and climbing from there. The service is making the model worse, not better.

Deliverable: A short debug report (≤ 200 words) that (1) identifies the planted bug in the weight-update line, (2) explains the mechanism — why adding the gradient instead of subtracting it drives the weight toward higher loss, and (3) shows the corrected line of code. Run the buggy loop, confirm the loss diverges; then fix it and confirm the loss drops 81.00 → 3.24 → 0.13 as the article shows.

Rubric:

• Identifies the specific buggy line: weight_cart = weight_cart + (learning_rate * gradient) — the sign is wrong
• Explains the mechanism: the gradient points in the direction of increasing loss; adding it moves the weight the wrong way (prediction goes negative, loss explodes)
• Shows the corrected code: weight_cart = weight_cart - (learning_rate * gradient)
• Confirms the fixed loop reproduces the article’s loss sequence (81.00 → 3.24 → 0.13) and weight reaching ~4.96

Reproduce the core backpropagation mechanics from Rumelhart, Hinton, & Williams (1986) — as simplified in this article — then break the vanishing-gradient assumption.

Brief: Implement the four equations from the article (MSE loss, chain-rule gradient, weight update, vanishing-gradient propagation) on the BookSight synthetic dataset. Then run the ablation: replace the article’s uniform per-layer sensitivity of 0.5 with a sensitivity pattern of your own design where the gradient survives to layer 10. Compare both patterns side by side and explain why yours avoids vanishing.

Starter: projects/attention-from-scratch-a-deep-p02-backpropagation-intuitively-how-networks-learn-fro/backprop-impl.py

Deliverable: A completed Python script plus a 150–250 word write-up containing: (1) the reproduced loss sequence from the article’s 3-step training loop (81.00 → 3.24 → 0.13), (2) the final gradient value at layer 10 for the article’s 0.5-sensitivity pattern vs your alternative pattern, and (3) a one-paragraph explanation of what property of your pattern prevents vanishing.

Rubric:

• mse_loss returns 0.36 for prediction=0.4, target=1.0 (matches article)
• compute_gradient returns −36.0 for the article’s first-step values (prediction=1.0, target=10.0, items=2.0)
• Training loop reproduces loss 81.00 → 3.24 → 0.13 and weight reaching ~4.96
• Vanishing-gradient demo reaches ~0.00098 at layer 10 with sensitivity=0.5
• Ablation pattern’s final gradient at layer 10 is ≥ 0.01 (i.e., does not vanish)
• Write-up explains that the alternative pattern requires most layers’ sensitivities to be ≥ 1.0, preventing the repeated sub-unit multiplication that causes vanishing