Evaluating Forecast Accuracy: MAPE, RMSE, and Why Averages Lie

Last time, Nora replaced her teammate’s plain K-Fold with walk-forward validation and landed a trustworthy accuracy number for the bakery demand model. Her regional manager’s reply came fast: “How much can we trust this?” That’s the question this article exists to answer.

1. The Problem: You Built a Forecast, Now What?

You finally finished your forecasting model. You plot the predictions against the actual results, and at first glance, the lines look pretty close. A surge of pride. Then a stakeholder asks: “How much can we trust this?”

For Nora, that stakeholder is her regional manager — the person who decides whether to bet the bakery chain’s daily inventory and staffing on the number Nora just handed over. So “how much can we trust this?” isn’t abstract anymore. It’s the last loose end in the forecasting effort.

This is where most people get stuck. You can’t just eyeball a forecast. Our eyes are drawn to the parts where the lines overlap, ignoring the gaps that could cost your business thousands of dollars. Pick the wrong way to measure your mistakes, and you might trust a model that is systematically broken.

So let’s try judging a forecast by eye alone.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Let's create some fake sales data
np.random.seed(42)
days = np.arange(1, 31)
actuals = 100 + 10 * np.sin(days/2) + np.random.normal(0, 5, 30)

# A 'Naive' forecast (just use yesterday's value)
forecast = np.roll(actuals, 1)
forecast[0] = 100 

plt.figure(figsize=(10, 5))
plt.plot(days, actuals, label='Actual Sales', marker='o')
plt.plot(days, forecast, label='Naive Forecast', linestyle='--')
plt.title("Can you tell if this is a 'good' forecast?")
plt.legend()
plt.show()

• np.random.seed(42) — fixes the random number generator so the noise pattern is reproducible across runs.
• np.arange(1, 31) — creates day labels 1 through 30 for the x-axis.
• 100 + 10 * np.sin(days/2) — builds a sine wave of amplitude 10 around a baseline of 100, simulating the cyclical up-and-down of daily sales.
• np.random.normal(0, 5, 30) — adds Gaussian noise (mean 0, standard deviation 5) on top of the wave so the series looks like real, imperfect sales data.
• np.roll(actuals, 1) — shifts the entire actuals array forward by one position, so forecast[i] equals actuals[i-1]; this is the classic “naive” forecast that just reuses yesterday’s value as today’s prediction.
• forecast[0] = 100 — manually patches the first element because np.roll wrapped the last actual value into index 0, which would be a look-ahead leak; setting it to the baseline 100 avoids that.
• plt.plot(...) — overlays the two series so you can visually judge how close the forecast tracks the actuals.

It looks okay, right? It follows the trend. But “looks okay” isn’t a metric. To make real decisions, we need numbers. And different numbers tell very different stories.

2. Meet the Big Three: MAE, RMSE, and MAPE

Forecast error comes down to one question: “How far off were we?” There are three main ways to answer.

• MAE (Mean Absolute Error): This is the average size of your mistakes. If your MAE is 5, you were off by 5 units on an average day. It’s simple, and it stays in the same units as your data (like dollars or liters).
• RMSE (Root Mean Squared Error): This also measures error, but it squares the mistakes before averaging them. Big mistakes end up counting for much more than small ones.
• MAPE (Mean Absolute Percentage Error): This tells you the error as a percentage. “We were off by 5% on average.” It sounds great for presentations, but it has a dark side.

Let’s calculate these for our data.

from sklearn.metrics import mean_absolute_error, mean_squared_error

def calculate_mape(y_true, y_pred):
    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100

mae = mean_absolute_error(actuals, forecast)
rmse = np.sqrt(mean_squared_error(actuals, forecast))
mape = calculate_mape(actuals, forecast)

print(f"MAE: {mae:.2f} units")
print(f"RMSE: {rmse:.2f} units")
print(f"MAPE: {mape:.2f}%")

• from sklearn.metrics import mean_absolute_error, mean_squared_error — imports the two built-in error functions; scikit-learn does not ship a MAPE function, which is why it is defined manually.
• def calculate_mape(y_true, y_pred) — a hand-rolled MAPE; it divides each absolute error by the corresponding actual value, takes the mean, and multiplies by 100 to express it as a percentage.
• np.abs((y_true - y_pred) / y_true) — element-wise absolute relative error; if any y_true is zero this line will divide by zero and produce inf, which is exactly the MAPE trap Section 4 unpacks.
• mean_absolute_error(actuals, forecast) — returns the average of |actuals - forecast|, the plain-dollar-amount “typical miss.”
• np.sqrt(mean_squared_error(actuals, forecast)) — squares each error, averages them, then takes the square root; the squaring step is what makes RMSE overweight the large errors.
• The three printed numbers give you the same underlying errors through three different lenses — same data, three very different stories.

What this actually means: Running the block above prints MAE ≈ 5.98 and RMSE ≈ 6.69. That’s our “typical” miss versus the outlier-weighted view — RMSE sits about 12% higher than MAE, which tells us a handful of larger errors are pulling the squared average up.

Given $n$ observations with actual values $y_t$ and forecasts $\hat{y}_t$, the three headline accuracy metrics are:

$\text{MAE} = \frac{1}{n}\sum_{t=1}^{n}\left|y_t - \hat{y}_t\right|, \qquad \text{RMSE} = \sqrt{\frac{1}{n}\sum_{t=1}^{n}\left(y_t - \hat{y}_t\right)^2}, \qquad \text{MAPE} = \frac{100}{n}\sum_{t=1}^{n}\frac{\left|y_t - \hat{y}_t\right|}{\left|y_t\right|}$

and the bias (mean error) that should always be reported alongside them:

$\text{ME (bias)} = \frac{1}{n}\sum_{t=1}^{n}\left(\hat{y}_t - y_t\right)$

Plain English	Statistical symbol	Python equivalent
Mean Absolute Error — average size of a mistake	$\text{MAE} = \frac{1}{n}\sum\lvert y_t - \hat{y}_t\rvert$	mean_absolute_error(y_true, y_pred)
Root Mean Squared Error —penalizes big misses	$\text{RMSE} = \sqrt{\frac{1}{n}\sum(y_t - \hat{y}_t)^2}$	np.sqrt(mean_squared_error(y_true, y_pred))
Mean Absolute Percentage Error — error as a %	$\text{MAPE} = \frac{100}{n}\sum\frac{\lvert y_t - \hat{y}_t\rvert}{\lvert y_t\rvert}$	calculate_mape(y_true, y_pred)
Mean Error (Bias) — are we consistently high or low?	$\text{ME} = \frac{1}{n}\sum(\hat{y}_t - y_t)$	np.mean(y_pred - y_true)
Individual forecast error at time $t$	$e_t = \hat{y}_t - y_t$	y_pred - y_true

The squaring inside RMSE is what makes the gap between MAE and RMSE diagnostic: a large RMSE-to-MAE ratio signals a few outsized misses rather than many uniform small ones.

3. Why RMSE Punishes Big Mistakes (And Why That Matters)

Here’s the tricky part: RMSE and MAE can disagree on which model is better.

Say you’re forecasting electricity demand. A small miss is fine. A big miss causes a blackout. You want a metric that reacts sharply when a large error shows up. That’s RMSE. Because it squares the error, a mistake of 10 isn’t twice as bad as a mistake of 5 — it’s four times as bad.

# Scenario A: Consistent small errors
actuals_a = np.array([100, 100, 100, 100])
pred_a = np.array([105, 95, 105, 95]) # Off by 5 every time

# Scenario B: Mostly perfect, but one huge miss
actuals_b = np.array([100, 100, 100, 100])
pred_b = np.array([100, 100, 100, 120]) # Off by 20 once

print(f"Model A - MAE: {mean_absolute_error(actuals_a, pred_a)}, RMSE: {np.sqrt(mean_squared_error(actuals_a, pred_a))}")
print(f"Model B - MAE: {mean_absolute_error(actuals_b, pred_b)}, RMSE: {np.sqrt(mean_squared_error(actuals_b, pred_b))}")

• actuals_a = np.array([100, 100, 100, 100]) — four identical true values so the only thing that varies is the size of the mistakes.
• pred_a = np.array([105, 95, 105, 95]) — Model A is off by exactly 5 every time, alternating over and under; a steady, predictable noise pattern.
• pred_b = np.array([100, 100, 100, 120]) — Model B is perfect three times and then misses by 20 on the last point; one catastrophic miss, the “blackout” scenario.
• mean_absolute_error(actuals_a, pred_a) — for Model A this averages |5|+|5|+|5|+|5| = 20 over 4, giving MAE = 5.
• np.sqrt(mean_squared_error(actuals_b, pred_b)) — for Model B the squared errors are 0,0,0,400; their mean is 100; the square root is 10, so RMSE = 10.
• The point: both models share MAE = 5, but Model B’s RMSE doubles to 10 because the single 20-unit miss gets squared before averaging — RMSE “screams” where MAE stays calm.

In Model A, the MAE and RMSE are both 5. In Model B, the MAE stays at 5, but the RMSE jumps to 10. RMSE caught the “blackout” risk that MAE missed.

4. The MAPE Trap: Why Percentages Can Lie

MAPE is the most popular metric in business, but it’s dangerous. It divides by the actual value.

Forecast something that gets close to zero, like daily sales of a specific shoe size, and MAPE will explode. If the actual value is 1 and you forecast 2, that’s a 100% error. If the actual is 100 and you forecast 101, that’s a 1% error. Sound unfair? You were off by exactly 1 unit in both cases.

# The zero problem
low_actuals = np.array([0.5, 100])
low_preds = np.array([1.5, 101])

print(f"MAPE with low values: {calculate_mape(low_actuals, low_preds):.2f}%")

• low_actuals = np.array([0.5, 100]) — two observations: one tiny (0.5) and one normal (100), illustrating how the same absolute miss behaves very differently under MAPE.
• low_preds = np.array([1.5, 101]) — the forecast is off by exactly 1 unit in both cases: 1.5 vs 0.5, and 101 vs 100.
• calculate_mape(low_actuals, low_preds) — for the first point the relative error is |1.5 − 0.5| / 0.5 = 1 / 0.5 = 2.0, i.e. 200%; for the second it is |101 − 100| / 100 = 0.01, i.e. 1%. Averaged, (2.0 + 0.01) / 2 = 1.005, times 100 gives exactly 100.5% — there’s no randomness in this block, so that’s the number every run prints, not a range.
• The lesson: MAPE treats a 1-unit miss on a $0.50 item as 200× worse than the same 1-unit miss on a $100 item, which is why any series with near-zero values makes MAPE meaningless.

One tiny number made our MAPE look like a disaster (100.5%). Our absolute errors were tiny. This is why practitioners are moving toward MASE (Mean Absolute Scaled Error). It compares your model to a simple baseline instead of dividing by zero-prone actuals.

5. A Concrete Example: Forecasting Store Revenue

Consider two models for store revenue.

• Model A is noisy. It’s off by about $500 every single day.
• Model B is accurate most of the time, but it missed that Friday was a holiday. That one day, it was off by $5,000.

np.random.seed(7)  # local seed so this cell is reproducible standalone

revenue_actuals = np.full(7, 10000)
model_a_preds = revenue_actuals + np.random.choice([-500, 500], 7)
model_b_preds = revenue_actuals.copy()
model_b_preds[6] = 5000 # The holiday miss

print(f"Model A (Consistent) MAE: {mean_absolute_error(revenue_actuals, model_a_preds):.0f}")
print(f"Model B (Holiday Miss) MAE: {mean_absolute_error(revenue_actuals, model_b_preds):.0f}")
print(f"--- SHIFT TO RMSE ---")
print(f"Model A RMSE: {np.sqrt(mean_squared_error(revenue_actuals, model_a_preds)):.0f}")
print(f"Model B RMSE: {np.sqrt(mean_squared_error(revenue_actuals, model_b_preds)):.0f}")

• np.random.seed(7) — a local seed for this cell, so it prints the same numbers whether you run the whole article top to bottom or just this block on its own.
• np.full(7, 10000) — seven days of identical $10,000 revenue, so the errors are the only thing that varies.
• np.random.choice([-500, 500], 7) — Model A’s predictions are scattered ±$500 around the true value every day, simulating consistent but uniform noise. Since every draw has the same magnitude, Model A’s MAE and RMSE come out to exactly $500 regardless of which days land high or low.
• model_b_preds[6] = 5000 — Model B is perfect for six days then drops to $5,000 on day 7 (the holiday it failed to recognize), a single $5,000 miss.
• mean_absolute_error(revenue_actuals, model_b_preds) — Model B’s MAE is $5,000/7 ≈ $714, because the one huge miss is diluted across seven near-perfect days.
• np.sqrt(mean_squared_error(revenue_actuals, model_b_preds)) — Model B’s RMSE is much higher (≈ $1,890) because the $5,000 miss gets squared (25,000,000) before averaging, so the single catastrophe dominates the metric.
• The contrast is the whole point: MAE says “these models are similar,” RMSE says “Model B is far riskier.”

MAE treats them as roughly equal — Model A sits at $500 and Model B at $714, both in the same ballpark. RMSE tells a different story: Model A stays at $500 but Model B jumps to about $1,890. That single $5,000 miss makes Model B far riskier. If you’re the manager who needs to keep enough cash in the drawer, Model B might get you fired, even with a low average error.

6. Choosing Your Metric: A Decision Framework

So, which one do you use? A quick checklist:

1. Use MAE if you want the average impact on your wallet and your data is stable.
2. Use RMSE if large errors hurt more than small ones (e.g., running out of stock).
3. Avoid MAPE if your data has zeros or values near zero.
4. Always use a Baseline. If your fancy AI model isn’t better than just “predicting the same as yesterday,” it’s not a good model — no matter how low the RMSE gets.

MAE vs. RMSE vs. MAPE — which accuracy metric should you reach for?

Metric	What it measures	When to use it	When it breaks
MAE (Mean Absolute Error)	Average absolute miss, in the same units as your data	When you want a plain-language “we’re off by about X units on a typical day” and the cost of errors is roughly linear in their size	When a few catastrophic misses matter far more than many small ones — MAE treats a $5,000 holiday miss the same as five $1,000 misses
RMSE (Root Mean Squared Error)	Square-rooted mean of squared errors; the squaring overweights large misses	When big errors are disproportionately costly (blackouts, stockouts, safety-critical forecasts) and you want the metric to scream at outliers	When you want a number that’s easy to interpret in business terms — RMSE is not “dollars off on an average day,” it’s a penalty-weighted blend that resists simple translation
MAPE (Mean Absolute Percentage Error)	Error as a percentage of actuals	When stakeholders think in percentages and your actual values are comfortably bounded away from zero (e.g., store revenue in the thousands)	Anywhere actuals can be zero or near-zero — a $1 miss on a $0.50 item reads as 100%+ error, blowing up the average; also asymmetric: over- and under-forecasting by the same absolute amount produce different percentages
MASE (Mean Absolute Scaled Error)	Your model’s MAE divided by a naive baseline’s MAE	When you want a scale-free, zero-safe metric that works across series of different magnitudes and directly answers “is this better than just predicting yesterday’s value?”	When the naive baseline itself is unstable (very short or highly seasonal series), which makes the denominator noisy and the ratio hard to interpret
Bias (Mean Error)	Average signed error — are you consistently high or low?	Always, as a companion to any of the above; it catches the systematic over- or under-forecasting that absolute-error metrics hide	On its own it’s not an accuracy metric at all — two forecasts with identical bias can have wildly different MAE/RMSE, so it must be paired with one

The key decision: always report at least one accuracy metric (MAE or RMSE depending on how much big misses hurt) and the bias. MAPE is fine for boardroom slides only when your actuals never approach zero. For Nora’s bakery, where daily sales of a slow-moving pastry can easily hit single digits, MAPE is a trap — MAE for the “typical miss” and RMSE for the “worst-day risk” are the two she can actually defend to her regional manager.

7. The Averaging Lie: Why a ‘Good’ Average Can Hide a Bad Forecast

Here’s the key lesson: Accuracy is not the same as Bias.

A forecast can post a perfect MAE and still be completely biased. Bias means you are consistently too high or too low. Over-forecast by $100 every time and you’ll sit on too much inventory. Come in $100 too high half the time and $100 too low the other half? Your inventory levels might balance out.

# Two forecasts with the same MAE
actuals_bias = np.array([100, 100, 100, 100])
forecast_high = np.array([110, 110, 110, 110]) # Always high
forecast_mixed = np.array([110, 90, 110, 90])  # High and Low

print(f"High Forecast MAE: {mean_absolute_error(actuals_bias, forecast_high)}")
print(f"Mixed Forecast MAE: {mean_absolute_error(actuals_bias, forecast_mixed)}")

# Calculate Bias (Mean Error)
print(f"High Forecast Bias: {np.mean(forecast_high - actuals_bias)}")
print(f"Mixed Forecast Bias: {np.mean(forecast_mixed - actuals_bias)}")

• forecast_high = np.array([110, 110, 110, 110]) — a forecast that is always 10 units too high; the errors are +10, +10, +10, +10.
• forecast_mixed = np.array([110, 90, 110, 90]) — a forecast that alternates 10 high and 10 low; the errors are +10, −10, +10, −10.
• mean_absolute_error(actuals_bias, forecast_high) and the mixed version — both return 10, because absolute-value averaging throws away the sign of each error.
• np.mean(forecast_high - actuals_bias) — the signed mean error for the high-only forecast is +10; it never cancels out, so the bias survives.
• np.mean(forecast_mixed - actuals_bias) — the signed mean error for the mixed forecast is 0, because the +10s and −10s cancel.
• The contrast proves the section’s point: two forecasts with identical MAE can have completely different bias, and the one with bias +10 will silently inflate your inventory every single day.

Both have an MAE of 10. The first has a bias of +10; the second has a bias of 0. Always report your Mean Error (Bias) alongside your accuracy metrics.

8. Putting It Together: A Forecast Evaluation Checklist

Run this function whenever you evaluate a model. It gives you the full picture.

def evaluate_forecast(y_true, y_pred):
    mae = mean_absolute_error(y_true, y_pred)
    rmse = np.sqrt(mean_squared_error(y_true, y_pred))
    bias = np.mean(y_pred - y_true)
    
    print(f"--- Forecast Report ---")
    print(f"MAE:  {mae:.2f} (Average size of mistake)")
    print(f"RMSE: {rmse:.2f} (Impact of outliers)")
    print(f"Bias: {bias:.2f} (Are we consistently high or low?)")
    
    plt.figure(figsize=(8, 4))
    plt.scatter(y_true, y_pred - y_true, alpha=0.5)
    plt.axhline(0, color='red', linestyle='--')
    plt.title("Residual Plot: Errors should be scattered randomly around 0")
    plt.xlabel("Actual Value")
    plt.ylabel("Error")
    plt.show()

evaluate_forecast(actuals, forecast)

• mae = mean_absolute_error(y_true, y_pred) — the “typical miss” in the data’s own units.
• rmse = np.sqrt(mean_squared_error(y_true, y_pred)) — the outlier-penalized miss; compare it to MAE to gauge how skewed the error distribution is.
• bias = np.mean(y_pred - y_true) — the signed mean error; a value far from 0 means the model is systematically leaning one way.
• plt.scatter(y_true, y_pred - y_true, alpha=0.5) — plots each residual (predicted minus actual) against the actual value; alpha=0.5 makes overlapping points visible.
• plt.axhline(0, color='red', linestyle='--') — draws the zero-error reference line; a healthy residual plot clusters randomly around this line with no funneling or curve.
• plt.xlabel("Actual Value") / plt.ylabel("Error") — labels the axes so the reader can see whether errors grow with the magnitude of the actuals (a classic heteroscedasticity warning).

9. Closing the Loop: Nora’s Full Forecasting Toolkit

We’ve spent this article — and this series — on one hard question from Nora’s regional manager: “How much can we trust this forecast?” A single number won’t earn that trust. You build it by reporting MAE for the typical miss, RMSE for the catastrophic ones, and bias for the direction the model leans. Don’t let an average hide a systematic flaw.

Nora started where a lot of teams start: an LSTM that flopped on a few months of sourdough demand data. From there she built a toolkit — ARIMA and SARIMA for the small, stable series where every parameter counts; gradient boosting and Prophet for the messier ones where holidays and structural breaks live; lag and rolling-window features that capture time instead of ignoring it; walk-forward cross-validation that doesn’t quietly cheat by reading tomorrow to predict yesterday; and the metric layer she can defend in plain language to whoever signs off on the inventory order. That is the toolkit. The forecast is just the last line of its output.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the key mathematical difference between MAE and RMSE that makes RMSE punish large errors more heavily?

Understand In your own words, explain why MAPE “explodes” when the actual value is close to zero, using the article’s shoe-size sales example.

Apply Using the article’s Scenario A/B setup (actuals = [100,100,100,100]), calculate the MAE and RMSE for a new prediction [100, 100, 100, 140] (one large miss of 40), and compare how much more dramatically RMSE reacts to this bigger single miss than MAE does.

Analyze The article’s Section 7 shows two forecasts with identical MAE (10) but different Bias (+10 vs 0). Walk through why a business that only tracks MAE could keep running an inventory-damaging forecast for months without ever noticing the problem, while tracking Bias would catch it immediately.

Evaluate The article recommends RMSE “if large errors hurt more than small ones (e.g., running out of stock).” Critique this recommendation for the electricity-demand blackout scenario mentioned earlier: is RMSE’s quadratic penalty actually the right shape for that risk, or would a metric with an even steeper penalty (or a hard threshold) better match how catastrophic a blackout actually is compared to a merely large miss?

Create Design a forecast evaluation report (following the article’s evaluate_forecast pattern) for a new scenario: a hospital forecasting ICU bed demand, where under-forecasting (running out of beds) is far more costly than over-forecasting (empty beds). Which metrics from the article would you prioritize, and would you treat positive and negative bias differently in your report?

---

Related articles

• Time-Series Cross-Validation: Why K-Fold Breaks on Temporal Data) — the immediately preceding article: Nora replaced plain K-Fold with walk-forward validation so the accuracy number she hands to her regional manager is actually trustworthy.
• ARIMA and SARIMA Intuitively: When Classical Forecasting Beats the Hype — article 1, where the whole journey began: Nora’s team tried an LSTM on sourdough demand, it flopped, and ARIMA/SARIMA won on the limited data they actually had.

References & Further reading

• Hyndman, R. J., & Koehler, A. B. (2006). “Another look at measures of forecast accuracy.” International Journal of Forecasting, 22(4), 679–688. doi.org/10.1016/j.ijforecast.2006.03.001 — the foundational paper that systematized the MAE/RMSE/MAPE comparison, introduced MASE as a scale-free, zero-safe alternative, and is the reason practitioners now treat percentage error with suspicion on low-magnitude series.
• Kaggle “Store Sales — Time Series Forecasting” competition: kaggle.com/competitions/store-sales-time-series-forecasting — a real-world retail benchmark where the choice of accuracy metric (and the honesty of your cross-validation) directly determines your leaderboard position; the daily store-level sales are structurally similar to Nora’s bakery data, making it an ideal sandbox for trying out the MAE/RMSE/bias report before trusting it in production.
• scikit-learn metrics documentation: scikit-learn.org/stable/modules/model_evaluation.html — official reference for mean_absolute_error, mean_squared_error, and the other regression metrics used throughout this article.

---