Learning Curves: How to Read Your Model's Mind to Fix Overfitting and Underfitting

1. The ‘More Data’ Myth

Ever spent weeks collecting new data, sure it would fix your model’s low accuracy—only to watch the needle not move? You add more rows, retrain, wait. The score stays flat.

Blindly tweaking parameters or piling on data is like fumbling for a light switch in a dark room. You might find the door by luck. You won’t know how you got there. Learning Curves offer something better: an X-ray view of your model. Instead of guessing why it underperforms, you see what’s actually happening inside as the model takes in more information.

Now we’ll try solving a complex problem with a model that simply can’t handle it—no matter how much data we feed it.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import learning_curve, KFold

# Create a curvy dataset (a sine wave)
np.random.seed(42)
X = np.linspace(0, 10, 100).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.normal(0, 0.2, 100)

# X is sorted (0 to 10 in order), so the default cv=5 in learning_curve uses
# plain, UNSHUFFLED KFold for a regressor -- each fold becomes a contiguous
# block of X. That turns every validation fold into an extrapolation test
# (predicting a range of x the model never trained on) instead of a genuine
# held-out sample. Shuffling the folds is what makes the rest of this article
# measure what it claims to measure.
cv = KFold(n_splits=5, shuffle=True, random_state=42)

# Try to fit a straight line to a curve
model = LinearRegression()
train_sizes, train_scores, test_scores = learning_curve(
    model, X, y, cv=cv, train_sizes=np.linspace(0.1, 1.0, 10))

print(f"Final Training Score: {train_scores[-1].mean():.2f}")
print(f"Final Validation Score: {test_scores[-1].mean():.2f}")

Here the scores come out to 0.01 and -0.22. Both are essentially guessing — a validation score of 0 means “as good as always predicting the average,” and -0.22 is only slightly worse than that. Adding more data in this case is like giving a calculus exam to a student who only knows addition. They could review 1,000 examples and still not understand the underlying rules.

2. The Two Scores: Training vs. Validation

Reading a learning curve comes down to two numbers: the Training Score and the Validation Score.

• The Training Score is a practice test with the answer key in hand. It measures how well the model memorized the data it already saw.
• The Validation Score is the real exam. It uses data the model has never seen, so it shows whether the model learned the rules well enough to solve new problems.

The gap between those lines tells the story. If training sits at 100% but validation only reaches 50%, the student memorized the textbook without grasping the concepts.

from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Splitting the data to see the 'Practice' vs 'Real Exam' scores
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model.fit(X_train, y_train)

train_pred = model.predict(X_train)
val_pred = model.predict(X_val)

print(f"Training MSE: {mean_squared_error(y_train, train_pred):.3f}")
print(f"Validation MSE: {mean_squared_error(y_val, val_pred):.3f}")

Here, both errors are high. The model is failing the practice exam and the real one.

3. Scenario A: The ‘Stubborn’ Model (High Bias)

Picture a student who thinks every math problem can be solved by adding 2. Show them a hundred counterexamples and they still won’t budge. That’s Underfitting (or High Bias).

On a learning curve, both the training and validation lines flatten early and sit close together, low. Here’s the part beginners struggle with: more data won’t help. The model is too simple, too rigid to grasp the pattern.

# Visualizing High Bias
train_sizes, train_scores, test_scores = learning_curve(
    LinearRegression(), X, y, cv=cv)

plt.plot(train_sizes, np.mean(train_scores, axis=1), label='Training Score')
plt.plot(train_sizes, np.mean(test_scores, axis=1), label='Validation Score')
plt.title("High Bias: The Model is Too Simple")
plt.xlabel("Training Examples")
plt.ylabel("Score")
plt.legend()
plt.show()

Both lines stay low the entire way across — training peaks at 0.76 and ends at 0.01; validation ends at -0.22, having spent most of the curve between -5 and 0. There’s real noise early on, when only 8-26 points are available to fit or score a single-feature line against, but the destination is what matters: neither curve ever approaches a good score, no matter how much data is added. That’s the diagnostic signature of high bias — the model hasn’t learned the training data because a straight line can’t fit a curve. What you need is a more flexible model, like a Polynomial regression or a Random Forest.

4. Scenario B: The ‘Memorizer’ (High Variance)

Picture a student with a photographic memory. They memorize every digit of every question in the textbook. On the practice test, they score 100%. But change a ‘5’ to a ‘6’ on the real exam and they fall apart — they learned the noise, not the logic.

This is Overfitting (or High Variance). The model is too complex, so it treats random patterns as universal rules.

from sklearn.tree import DecisionTreeRegressor

# A Decision Tree with no limits will try to memorize every point
overfit_model = DecisionTreeRegressor(max_depth=None, random_state=42)
train_sizes, train_scores, test_scores = learning_curve(
    overfit_model, X, y, cv=cv)

plt.plot(train_sizes, np.mean(train_scores, axis=1), label='Training Score')
plt.plot(train_sizes, np.mean(test_scores, axis=1), label='Validation Score')
plt.title("High Variance: The Model is Memorizing")
plt.legend()
plt.show()

Training sits at a perfect 1.0 the entire way across — an unlimited-depth tree memorizes whatever it’s shown, regardless of how much of it there is. Validation ends at 0.80, for a final gap of about 0.2: a real but moderate overfitting signature, not a dramatic one, since 0.80 is a fairly good score in absolute terms. Where more data genuinely earns its keep is earlier on the curve: validation starts at -1.16 with only a handful of training points and climbs to 0.80 by the end — more data does help here, because it forces the model to find patterns that hold across more examples, which eventually drowns out the noise it would otherwise memorize.

5. The ‘Goldilocks’ Zone

A healthy model shows both lines converging toward a high score. Add more data, and the training score may dip slightly — it’s harder to memorize 1,000 points than 10 — while the validation score rises to meet it.

from sklearn.ensemble import RandomForestRegressor

# A balanced model
goldilocks_model = RandomForestRegressor(n_estimators=50, max_depth=3, random_state=42)
train_sizes, train_scores, test_scores = learning_curve(
    goldilocks_model, X, y, cv=cv)

plt.plot(train_sizes, np.mean(train_scores, axis=1), label='Training Score')
plt.plot(train_sizes, np.mean(test_scores, axis=1), label='Validation Score')
plt.title("The Goldilocks Zone: Healthy Convergence")
plt.legend()
plt.show()

Training ends at 0.92, validation at 0.85 — a gap of about 0.06, roughly a third of the overfit tree’s gap above, and validation itself is slightly higher here than the overfit tree’s 0.80. The forest beats the unlimited tree on both counts: a smaller gap and a better held-out score. At this point, the model generalizes well.

6. How to Fix It: Your Action Plan

Stop guessing. Use this checklist based on your learning curve plots:

If the lines are close together but the score is low (High Bias):

• Don’t spend money on more data.
• Do use a more complex model (e.g., switch from Linear to Random Forest).
• Do engineer better features that give the model more clues.

If there is a big gap between the lines (High Variance):

• Do get more data if possible; it helps the model stop memorizing noise.
• Do simplify the model (Regularization). For trees, limit max_depth. For linear models, use Lasso or Ridge.
• Do remove irrelevant features that might confuse the model.

Here’s a side-by-side fix. We’ll take our overfit tree and regularize it by limiting its depth.

fig, ax = plt.subplots(1, 2, figsize=(12, 4))

# Plot 1: Overfit (No depth limit)
t_s1, tr_s1, te_s1 = learning_curve(DecisionTreeRegressor(max_depth=None, random_state=42), X, y, cv=cv)
ax[0].plot(t_s1, np.mean(tr_s1, axis=1), label='Train')
ax[0].plot(t_s1, np.mean(te_s1, axis=1), label='Val')
ax[0].set_title("Before: Overfit (Big Gap)")

# Plot 2: Fixed (Limited depth)
t_s2, tr_s2, te_s2 = learning_curve(DecisionTreeRegressor(max_depth=3, random_state=42), X, y, cv=cv)
ax[1].plot(t_s2, np.mean(tr_s2, axis=1), label='Train')
ax[1].plot(t_s2, np.mean(te_s2, axis=1), label='Val')
ax[1].set_title("After: Regularized (Converged)")

plt.legend()
plt.show()

Training drops from a perfect 1.0 to 0.87 — the depth limit stops the tree from memorizing individual points — while validation stays essentially flat (0.80 before, 0.79 after). The gap is what closes, from about 0.20 to 0.08, not the validation score itself. That distinction matters: regularizing targets the symptom of overfitting (a training score that doesn’t reflect real generalization), and it doesn’t automatically raise validation, especially when validation was already decent going in, as it was here. What you get instead is a training score you can actually trust — 0.87 is a number close to what the model will really do on new data, where the unregularized tree’s 1.0 was not.

Summary Checklist:

1. Plot your learning curves before tuning hyperparameters.
2. Identify whether you have a Bias problem (lines close and low) or a Variance problem (big gap).
3. Choose your fix: smarter models for bias, more data or simpler models for variance.

Now that you can interpret what your learning curves are telling you, you won’t waste time collecting data you don’t need. Next, we’ll explore how to use Validation Curves to find the right ‘knob setting’ for your parameters.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember On a learning curve, what pattern indicates High Bias (underfitting) versus High Variance (overfitting)?

Understand In your own words, explain why the article says “adding more data will not help” for a High Bias model, using the “student who only knows addition” analogy.

Apply Using the article’s diagnostic checklist, if you plot a learning curve and see the training score at 0.95 and the validation score at 0.55, with a large gap between them, which fix would you try first: switching to a more complex model, or adding regularization?

Analyze The article says a healthy (“Goldilocks”) model’s training score might dip slightly as more data is added, while the validation score rises to meet it. Walk through why the training score naturally drops as training set size grows, even for a well-fit model — what’s different about memorizing 10 points versus memorizing 1,000?

Evaluate The article’s High Variance fix list includes “get more data” and “simplify the model” as parallel options. Critique this as an either/or menu: for a team with a fixed, unchangeable dataset size (e.g., a rare medical condition with only 200 patient records), which of the two fixes is actually available to them, and does the article’s ordering (data first) reflect that constraint?

Create Design a learning-curve diagnostic experiment for a new scenario: a spam classifier that gets 99% training accuracy and 97% validation accuracy (a small but nonzero gap). Would you call this Bias, Variance, or the Goldilocks zone, and what additional check (beyond the plotted curve shape) would you run before deciding no action is needed?

---