Classical Forecasting vs. Machine Learning: When Does ARIMA Beat an LSTM?

In the last article, Nora’s bakery team watched their LSTM faceplant on two years of sourdough sales data. A classical ARIMA/SARIMA model won comfortably. Was that a one-off, or a general pattern? She’s running the same head-to-head on a public benchmark — the classic airline-passengers dataset, an independent test case that isn’t the bakery’s own data.

1. The Puzzle: Why Does the Simpler Model Sometimes Win?

Deep Learning has been called the king of data science for years now. Got a complex problem? Throw a neural network at it. But in time series forecasting, the story flips. A math model from the 1970s—ARIMA—often outperforms a cutting-edge LSTM (Long Short-Term Memory) network.

Consider a door. If you want to predict whether it will open, you don’t need a supercomputer simulating every atom. You just need to understand the hinge. ARIMA works like that hinge model. It captures the basic mechanics of how a number moves over time.

So let’s put them head-to-head on a classic dataset: the monthly number of airline passengers.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.arima.model import ARIMA
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
from sklearn.metrics import mean_absolute_error

# Load data
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv'
df = pd.read_csv(url, index_col='Month', parse_dates=True)
data = df['Passengers'].values.astype('float32')

# Split: 80% train, 20% test
train_size = int(len(data) * 0.8)
train, test = data[:train_size], data[train_size:]

# 1. ARIMA Model
# We use a simple (5,1,0) configuration
arima_model = ARIMA(train, order=(5,1,0))
arima_result = arima_model.fit()
arima_pred = arima_result.forecast(steps=len(test))

# 2. LSTM Model
scaler = MinMaxScaler(feature_range=(0, 1))
train_scaled = scaler.fit_transform(train.reshape(-1, 1))

def create_dataset(dataset, look_back=1):
    X, Y = [], []
    for i in range(len(dataset)-look_back-1):
        X.append(dataset[i:(i+look_back), 0])
        Y.append(dataset[i + look_back, 0])
    return np.array(X), np.array(Y)

look_back = 12
X_train, y_train = create_dataset(train_scaled, look_back)
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))

lstm_model = Sequential([
    LSTM(4, input_shape=(1, look_back)),
    Dense(1)
])
lstm_model.compile(loss='mean_squared_error', optimizer='adam')
lstm_model.fit(X_train, y_train, epochs=50, batch_size=1, verbose=0)

# Prepare test data for LSTM
inputs = data[len(data) - len(test) - look_back:]
inputs = scaler.transform(inputs.reshape(-1, 1))
X_test, _ = create_dataset(inputs, look_back)
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))
lstm_pred = scaler.inverse_transform(lstm_model.predict(X_test))

print(f"ARIMA MAE: {mean_absolute_error(test, arima_pred):.2f}")
print(f"LSTM MAE: {mean_absolute_error(test, lstm_pred):.2f}")

• pd.read_csv(url, index_col='Month', parse_dates=True) — loads the airline CSV with the Month column parsed as datetime and set as the index, so pandas treats the data as a proper time series.
• data = df['Passengers'].values.astype('float32') — extracts the passenger counts as a NumPy array of 32-bit floats, the dtype Keras/TensorFlow prefers.
• train_size = int(len(data) * 0.8) — splits 80% of the series into training and 20% into test; data[:train_size] and data[train_size:] slice accordingly.
• ARIMA(train, order=(5,1,0)) — fits an ARIMA model with $p=5$ autoregressive lags, $d=1$ differencing, and $q=0$ moving-average terms.
• arima_result.forecast(steps=len(test)) — produces a forecast for the same number of steps as the test set, so we can compare directly.
• MinMaxScaler(feature_range=(0, 1)) — squashes the training data into the [0, 1] interval, which helps the LSTM’s gradient-based optimization converge.
• create_dataset — a helper that slides a window of look_back past values across the series, building input/output pairs for the LSTM’s supervised format.
• look_back = 12 — uses 12 months of history as the input window for each prediction, matching the yearly cycle.
• Sequential([LSTM(4, input_shape=(1, look_back)), Dense(1)]) — a minimal LSTM with 4 hidden units feeding a single dense output neuron.
• lstm_model.fit(..., epochs=50, batch_size=1, verbose=0) — trains for 50 passes over the data, one sample at a time, with training logs suppressed.
• scaler.inverse_transform(lstm_model.predict(X_test)) — converts the LSTM’s scaled predictions back into raw passenger counts for a fair comparison against the test set.

In this run, ARIMA’s MAE usually comes out lower than the LSTM’s. ARIMA might miss by 45 passengers, while the LSTM misses by 60. The pattern is clear: for small, seasonal datasets, the ‘fancy’ model overfits to noise. The simple model stays focused on the trend.

2. What ARIMA Actually Does (In One Sentence)

ARIMA stands for AutoRegressive Integrated Moving Average. That’s a mouthful, but it’s just three ideas combined.

1. AutoRegression (AR): The future looks like the past. If it rained yesterday, good chance it rains today.
2. Integrated (I): If the data trends upward, we don’t use raw values. We use the difference between today and yesterday. That makes the data ‘stationary’—its average doesn’t drift over time.
3. Moving Average (MA): We learn from past errors. If yesterday’s forecast was too high, nudge today’s down a bit.

So how does differencing (the ‘I’ part) clean up a trending dataset?

# Visualizing stationarity
plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.plot(data)
plt.title("Original (Non-Stationary)")

plt.subplot(1, 2, 2)
plt.plot(np.diff(data))
plt.title("Differenced (Stationary)")
plt.tight_layout()
plt.show()

• plt.subplot(1, 2, 1) — creates the left panel of a 1×2 grid for the original series.
• plt.plot(data) — plots the raw passenger counts, which trend upward and are clearly non-stationary.
• np.diff(data) — first-order differencing: subtracts each value from its predecessor, producing the series of changes that fluctuates around zero.
• plt.subplot(1, 2, 2) — creates the right panel for the differenced series.
• plt.tight_layout() — adjusts subplot spacing so the two panels don’t overlap or get clipped.

The second plot hovers around zero. That’s what ARIMA wants. Predicting a value that stays in a range is far easier than predicting one that flies off to infinity.

3. What Neural Networks Do Differently

Neural networks like LSTMs don’t assume the world is linear. Think of them as chefs who learn a recipe by watching someone cook a thousand times. They don’t need to be told “the future looks like the past.” They figure out the hidden patterns themselves.

Here’s the catch: LSTMs are data-hungry. A neural net has thousands of tiny ‘knobs’ (weights) to tune. Give it only 100 data points, and it memorizes them perfectly — then fails when it sees something new. That’s overfitting.

4. The Data Regime Matters Most

So, when do you reach for which? It comes down to the bias-variance tradeoff.

• ARIMA has high bias: It assumes the world is simple. If the world is actually complex, ARIMA will never be perfect.
• Neural nets have high variance: They are flexible enough to fit any shape. But they might “hallucinate” patterns in random noise.

Try simulating a regime shift—where the rules of the data suddenly change—and see how each model handles it.

# Synthetic data: 500 points of a simple wave, then a sudden jump
t = np.linspace(0, 100, 1000)
y = np.sin(t) + np.random.normal(0, 0.1, 1000)
y[500:] += 5 # The Regime Shift

# ARIMA usually struggles with sudden structural breaks
# LSTM can sometimes learn these if it has seen them before
plt.plot(y)
plt.axvline(x=500, color='r', linestyle='--', label='Regime Shift')
plt.legend()
plt.show()

• np.linspace(0, 100, 1000) — generates 1000 evenly spaced points from 0 to 100 as the time axis.
• np.sin(t) + np.random.normal(0, 0.1, 1000) — builds a sine wave with light Gaussian noise to simulate a stable oscillating pattern.
• y[500:] += 5 — adds a vertical jump of 5 units starting at the 500th point, simulating a structural break or “regime shift.”
• plt.axvline(x=500, color='r', linestyle='--', label='Regime Shift') — draws a red dashed vertical line at the break point for visual reference.

With 50,000 rows of data and weird jumps like this, the LSTM eventually learns that jumps happen. ARIMA just gets confused and produces a huge error.

5. Feature Engineering: ARIMA vs. Neural Nets

ARIMA handles extra information — weather, say — through “ARIMAX.” You have to be explicit about it. A neural net just takes another column in the input matrix.

The trade-off is interpretability. If ARIMA predicts a price increase, you can check the coefficients: “It’s the trend from last week.” A neural net’s answer is closer to “Because these 5,000 weights said so.” In plenty of businesses, why matters more than what.

6. The Real Trade-Off: Accuracy vs. Everything Else

Feature	ARIMA	LSTM (Neural Net)
Training Speed	Seconds	Minutes/Hours
Data Needed	Very Little (50+)	Lots (1000+)
Interpretability	High	Low
Hardware	Your Laptop	Often needs a GPU

If you are forecasting 10,000 different products for a small grocery store, you can’t wait 10 hours for a neural net to train for each one. ARIMA wins on scale and speed every time.

7. A Decision Tree: Which Should You Use?

Here’s a rule of thumb that can save you weeks of work:

1. Fewer than 500 data points? Use ARIMA.
2. Pattern is a simple up-and-down seasonal wave? Use ARIMA.
3. 10,000+ points with weird or nonlinear patterns? Try an LSTM.
4. Need to explain the result to a boss? Use ARIMA.

One more move: try an ensemble. Average both models. ARIMA and LSTM errors tend to cancel each other out.

8. Common Pitfalls: Why People Get This Wrong

The biggest mistake? Tuning Bias. A researcher spends three days tuning a neural network’s learning rate and layers, then runs a default ARIMA model for 30 seconds. The neural net wins. For a fair comparison, use tools like auto_arima to give the classical model its best chance.

9. A Real-World Example: Energy Demand Forecasting

Electricity demand is the middle ground. It has strong daily cycles — people wake up, turn on lights — and temperature adds its own influence.

A Hybrid approach usually wins here. ARIMA handles the basic daily rhythm. A small Neural Network catches the shocks from weather changes.

10. When to Revisit Your Choice

Data changes. This is called Concept Drift. If your company grows from 10 customers to 10,000, your data might become less noisy and more patterned. That’s the moment to switch from ARIMA to a Neural Net. So monitor your error rates. If ARIMA’s error starts creeping up month after month, the ‘simple hinge’ might have broken — and you need a more complex model.

11. Key Takeaways & What’s Next

• Complexity is not a feature; it’s a cost. Only pay it if you get a massive boost in accuracy.
• ARIMA is the baseline. Never trust a neural net forecast unless it can beat a simple ARIMA model first.
• Data size dictates the tool. Small data = Classical. Big data = Machine Learning.

The pattern holds. With limited data, simpler wins. Nora heads back to build out the bakery’s forecasting features properly, and we pick up there.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember According to the article’s decision tree, what two factors matter most when choosing between ARIMA and an LSTM?

Understand In your own words, explain why the article calls ARIMA “High Bias” and Neural Nets “High Variance,” connecting it back to the Bias-Variance Tradeoff.

Apply Using the article’s decision tree, would you reach for ARIMA or an LSTM to forecast daily sales for a brand-new store that has only 200 days of history with a clear weekly pattern? Walk through which rule(s) apply.

Analyze The article says the biggest mistake is “Tuning Bias”—spending days tuning a neural net but only seconds on a default ARIMA. Walk through why this comparison is unfair even before looking at accuracy numbers, and what a fair comparison would require from both sides.

Evaluate The article’s “Pro Tip” suggests ensembling ARIMA and LSTM because “the errors…cancel each other out.” Critique this as a default recommendation: under what circumstance would averaging a good model (ARIMA) with a bad, overfit model (LSTM on too little data) make the combined forecast worse than ARIMA alone?

Create Design a monitoring plan for the “Concept Drift” scenario the article describes — a company growing from 10 to 10,000 customers. What specific signal in your forecast error would tell you it’s time to reconsider ARIMA in favor of a neural net, and how often would you check it?

---

Related articles

• ARIMA and SARIMA, Intuitively: When Classical Forecasting Wins — the previous article: Nora’s bakery team tries an LSTM for sourdough demand and watches it flop against ARIMA.
• Feature Engineering for Time Series: Lags, Rolling Windows, and More — the next article: Nora goes back to the bakery’s own data to build lag and rolling-window features for daily sales.

References & Further reading

• Makridakis, E., Spiliotis, E., & Assimakopoulos, V. (2018). “Statistical and Machine Learning forecasting methods: Concerns and ways forward.” PLOS ONE, 13(3), e0194889. — the M4 competition paper that systematically benchmarked classical vs. ML forecasting methods.
• Kaggle “M5 Forecasting — Accuracy” competition: https://www.kaggle.com/competitions/m5-forecasting-accuracy — Walmart retail sales forecasting; a real-world benchmark where both classical and ML methods were stress-tested.
• statsmodels ARIMA documentation: https://www.statsmodels.org/stable/generated/statsmodels.tsa.arima.model.ARIMA.html

---