Last time, Sam and the team walked through the Archer and the Target. A model has to find the ‘sweet spot’ between too simple (Bias) and too complex (Variance). But how does the archer adjust their aim? Not by guessing. They feel the wind, check where the last arrow landed, then nudge the next shot.

That process of making small corrections to reduce error is called Gradient Descent. It sits under the hood of nearly every AI model — from linear regressions to the neural networks behind ChatGPT.

1. The Blindfolded Hiker: How Models ‘Feel’ Their Way to the Truth

Picture yourself as a hiker on a mountain. A thick fog rolls in. You’re effectively blindfolded. You know there’s a cozy cabin at the bottom of the valley, but you can’t see where it is. How do you get there?

Even without sight, your feet still help. You can feel the slope of the ground. If it tilts up to the left and down to the right, you step right. You don’t know whether the cabin is a mile away or ten feet away, but as long as you keep moving downhill, you’ll reach the lowest point.

In data science, the ‘mountain’ is your model’s error (or ‘Loss’). Higher up means more mistakes. The ‘valley’ is where your model is as accurate as possible. Gradient Descent is just ‘feeling’ the slope of the error and stepping toward the bottom.

2. The ‘Gradient’ is Just a Fancy Word for a Slope

When mathematicians say “gradient,” they really just mean which way is up, and how steep.

Picture standing on a hill. The gradient is a vector pointing straight up the steepest part of the slope. Since we want to go down, we move in the opposite direction.

But how big should each step be? That’s the Learning Rate.

• Too big, and you leap right over the valley and land on the other side.
• Too small, and you’ll be hiking for years before you reach the cabin.

In practice, this means adjusting the weights — the settings — of your model based on the size of the mistake. Here’s what a single update looks like in plain Python:

# A tiny example of a single weight update
weight = 0.5
gradient = 0.8  # The 'slope' we calculated from our error
learning_rate = 0.1 

# We move the weight in the opposite direction of the gradient
weight = weight - (learning_rate * gradient)

print(f"New weight: {weight:.2f}")
# The weight decreased because the slope was positive (uphill).

• weight = 0.5 initializes the model’s single parameter to a starting value — in a real model, weights are usually initialized randomly.
• gradient = 0.8 represents the slope of the loss with respect to the weight; a positive gradient means increasing the weight would increase the error, so we need to push it the other way.
• weight = weight - (learning_rate * gradient) is the core update rule: move the weight opposite to the gradient direction, scaled by the learning rate. With a positive gradient, the weight decreases — stepping downhill.

The gradient descent weight-update rule adjusts each parameter by taking a step proportional to the negative gradient of the loss:

$w \leftarrow w - \eta \, \nabla L(w)$

• $w$ — the current value of the weight (parameter) being updated.
• $\eta$ (eta) — the learning rate, a scalar controlling step size.
• $\nabla L(w)$ — the gradient of the loss function $L$ with respect to $w$; it points in the direction of steepest increase in error, so we subtract it.

Plain English	Statistical symbol	Python equivalent
Current weight (parameter) value	$w$	weight
Step size — how far to move each update	$\eta$ (learning rate)	learning_rate / lr
Slope of the loss w.r.t. the weight	$\nabla L = \frac{\partial L}{\partial w}$	gradient / dw
Weight update rule	$w \leftarrow w - \eta \, \nabla L$	weight -= lr * dw

3. Batch Gradient Descent: The Perfectionist’s Approach

So how do we actually use this with data? The most basic version is Batch Gradient Descent.

Picture that blindfolded hiker again. Before taking a single step, you radio 1,000 other hikers. You ask each one to feel the slope under their feet and report back. Then you average all their answers and take one step.

In data terms, “batch” means the model looks at every row in your dataset before it updates its weights.

The Catch: It’s stable and moves in a straight line toward the goal. But it’s slow. A million rows of data means a million calculations just to take one tiny step. Think of a committee that won’t budge until everyone has spoken.

import numpy as np

# Let's create some simple data: y = 3x + 4 + noise
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

# Batch Gradient Descent
weight = np.random.randn(1, 1)
bias = np.random.randn(1, 1)
lr = 0.1

for epoch in range(10):
    # 1. Predict for ALL 100 points
    predictions = X * weight + bias
    
    # 2. Calculate the average error (gradient)
    dw = (2/100) * X.T.dot(predictions - y)
    db = (2/100) * np.sum(predictions - y)
    
    # 3. Update weights
    weight -= lr * dw
    bias -= lr * db
    
    print(f"Epoch {epoch+1}: Weight is {weight[0][0]:.2f}")

• X = 2 * np.random.rand(100, 1) generates 100 random x-values uniformly in [0, 2) with shape (100, 1) so it’s a column vector.
• y = 4 + 3 * X + np.random.randn(100, 1) creates the target as a linear function (slope 3, intercept 4) plus Gaussian noise — the model’s job is to recover those coefficients.
• weight = np.random.randn(1, 1) and bias = np.random.randn(1, 1) initialize both as 2-D (1, 1) arrays, matching the shape that X.T.dot(...) produces for dw below — initializing them as 1-D np.random.randn(1) instead would raise ValueError: non-broadcastable output operand the first time weight -= lr * dw runs, because a shape-(1,) array can’t be updated in place from a shape-(1,1) gradient.
• dw = (2/100) * X.T.dot(predictions - y) computes the gradient of MSE loss with respect to the weight: the 2/100 comes from differentiating the mean squared error, and X.T.dot(...) applies the chain rule by multiplying the residuals by the inputs.
• db = (2/100) * np.sum(predictions - y) is the bias gradient — since bias has no input multiplication, you just sum the residuals.
• weight -= lr * dw is the in-place version of the update rule — equivalent to weight = weight - lr * dw.

After 10 steps, the weight is already approaching 3.0 (2.87 in this run). Steady progress, but each step was ‘expensive’ since we used the whole dataset.

4. Stochastic Gradient Descent (SGD): The Impatient Hiker

‘Stochastic’ is just a math word for ‘random.’ If Batch Gradient Descent is a committee, SGD is a lone hiker in a massive hurry.

Instead of looking at all the data, SGD looks at just one random row, calculates the error, and takes a step. Then it picks another random row and takes another.

The Catch: It only looks at one point at a time, so the gradient is very ‘noisy.’ One row might tell it to go left, while the next says go right. The path to the valley zig-zags. But each step is much faster to compute, and that noise can actually help the model jump out of small ‘potholes’ (local minima) in the mountain.

# SGD: Updating after every single point
weight = np.random.randn(1, 1)
bias = np.random.randn(1, 1)

for i in range(10): # Just look at the first 10 points
    xi = X[i:i+1]
    yi = y[i:i+1]
    
    prediction = xi * weight + bias
    dw = 2 * xi.T.dot(prediction - yi)
    db = 2 * (prediction - yi)
    
    weight -= lr * dw
    bias -= lr * db
    print(f"Step {i+1}: Weight is {weight[0][0]:.2f}")

• weight = np.random.randn(1, 1) re-initializes as a 2-D array here too, for the same reason as the Batch block above: xi.T.dot(prediction - yi) produces a shape-(1,1) gradient, and updating a shape-(1,) weight in place from it raises ValueError.
• xi = X[i:i+1] slices a single row from X while keeping the 2-D shape (1, 1) — using i:i+1 instead of X[i] preserves the column-vector structure that .dot() expects.
• dw = 2 * xi.T.dot(prediction - yi) drops the 1/N averaging factor because there’s only one data point — the gradient is computed from a single residual.
• The loop runs only 10 times (not over the full dataset), showing that SGD takes a step after each row, not after seeing all rows — hence the “impatient” label.

Notice how the weight jumps around much more than the Batch version? That’s the noise of SGD.

5. Mini-Batch: The ‘Goldilocks’ Solution

In practice, almost no one uses pure Batch or pure SGD. We use Mini-Batch Gradient Descent.

We take a small group of rows—typically 32, 64, or 128—and update the model on that group. You get the upside of both extremes:

1. Faster than Batch, since you skip looking at the whole dataset.
2. Smoother than SGD, because averaging 32 points gives a more reliable signal than a single one.

So why 32 or 64? This trips up beginners, but the answer usually comes down to how computer memory works. Processors handle small blocks of data efficiently when those blocks fit cleanly into cache.

batch_size = 20
for i in range(0, 100, batch_size):
    Xi = X[i:i+batch_size]
    yi = y[i:i+batch_size]
    
    # Update based on this 'mini-batch'
    predictions = Xi * weight + bias
    dw = (2/batch_size) * Xi.T.dot(predictions - yi)
    weight -= lr * dw
    print(f"Batch starting at {i}: Weight updated")

• for i in range(0, 100, batch_size) steps through the dataset in chunks of 20 — range(0, 100, 20) produces [0, 20, 40, 60, 80], yielding 5 mini-batches.
• Xi = X[i:i+batch_size] slices 20 rows at a time — small enough to fit in CPU cache, large enough to average out individual noise.
• dw = (2/batch_size) * Xi.T.dot(predictions - yi) restores the averaging factor but divides by batch_size (20) instead of the full dataset size (100) — keeping the gradient magnitude consistent regardless of batch size.

6. Adam: The Smart Hiker with a Memory

Standard Gradient Descent treats every step as if it’s the first day on the job. It forgets where it just came from.

Adam (Adaptive Moment Estimation) is the most popular optimizer today because it has a ‘memory.’ Picture a heavy ball rolling down a hill. It builds momentum. Roll it south for a while, and it will keep heading south even over a small bump.

Adam also uses ‘Adaptive Learning Rates.’ If a specific weight is changing a lot, Adam shrinks its step size. If a weight is barely moving, Adam gives it a nudge.

It speeds up in flat areas and slows down when things get tricky. This is why Adam is the ‘default’ for almost everyone. I’d lean toward it as a starting point, since it does the work of picking a learning rate for you.

7. Wrapping Up: Which One Should You Use?

So, how do you choose? Here is the rule of thumb:

• Use Batch if your dataset is tiny (a few hundred rows) and you want the most precise answer possible. Every step uses all the data, so the gradient signal is perfectly stable — but on large datasets each step is prohibitively slow, and the smooth path can get stuck in local minima.
• Use SGD if you have a massive dataset and need fast, rough updates — the per-step noise can actually help the model escape shallow local minima that Batch would settle into. The tradeoff is that convergence is jittery and the weight bounces around the minimum rather than settling cleanly.
• Use Mini-Batch for almost everything else. It is the industry standard for deep learning — it balances Batch’s stability with SGD’s speed, and batch sizes (32, 64, 128) are chosen to fit neatly into GPU/CPU cache for maximum hardware efficiency.
• Use Adam as your optimizer within Mini-Batch if you don’t want to spend all day tuning your learning rate. It is ‘smart’ enough to handle most problems by adapting the step size per-parameter on the fly.

What we covered:

• Gradient Descent feels the slope of error and moves downhill.
• Learning Rate is your step size — too big is dangerous, too small is slow.
• Batch is stable but slow; SGD is fast but chaotic; Mini-Batch sits between them.
• Adam is the modern favorite because it uses momentum and adapts to the data.

Sam’s model now updates its weights via gradient descent, but on a small HomeMatch training set it’s already starting to overfit. That’s what the next part on Regularization addresses — keeping the model from wandering too far and memorizing noise instead of the real pattern.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What does the “Learning Rate” control, and what goes wrong if it’s set too big versus too small?

Understand In your own words, explain why SGD’s path to the minimum is “zig-zagging” while Batch Gradient Descent’s path is smooth, using the article’s hiker analogies.

Apply Using the article’s weight-update formula (weight = weight - learning_rate * gradient), calculate the new weight if the current weight is 1.2, the gradient is 0.5, and the learning rate is 0.2.

Analyze The article says Mini-Batch is “faster than Batch” and “smoother than SGD.” Walk through why averaging the gradient over 32 random points (Mini-Batch) reduces the noise you’d get from a single point (SGD), without requiring the full computational cost of averaging over all 100+ points (Batch).

Evaluate The article recommends Adam as the default “if you don’t want to spend all day tuning your learning rate.” Critique this advice: is there ever a good reason to use plain (non-adaptive) Mini-Batch Gradient Descent over Adam, even though it requires more manual tuning?

Create Design a small experiment (not necessarily code) that would let you compare Batch, SGD, and Mini-Batch on the same dataset and show a teammate the “noise vs. speed” tradeoff the article describes. What would you plot, and what would you expect to see for each method?

---

Related articles

• P01: The Archer and the Target — Bias and Variance) — where Sam first learned about the error landscape that gradient descent navigates.
• P03: Regularization Explained — Why Your Models Overfit) — the “leash” that keeps gradient descent from driving a model into overfitting territory.

References & Further reading

• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). “Learning representations by back-propagating errors.” Nature, 323, 533–536. — the foundational paper that introduced backpropagation, making gradient descent practical for multi-layer models.
• 3Blue1Brown — What is Gradient Descent? — an animated visual walkthrough of how gradient descent navigates a loss landscape.
• Scikit-learn documentation: SGD classifiers & regressors — production-grade stochastic gradient descent for classical ML tasks.

---