Reference: Evaluation Metrics

A consolidated roster of the evaluation metrics used across the stats, MLOps, explainability, time-series, and ML articles. Linked from each article’s 📚 References toggle.

Classification metrics

Name	Formula	Range	When to use	Blind spot	Used in
Accuracy	$\frac{TP+TN}{N}$	[0, 1]	Balanced classes	Lying on imbalanced data (99% on rare disease)	the-confusion-matrix…, which-score…
Precision	$\frac{TP}{TP+FP}$	[0, 1]	Cost of false positives high	Ignores false negatives	which-score…, is-your-model-actually-better…
Recall / Sensitivity	$\frac{TP}{TP+FN}$	[0, 1]	Cost of false negatives high (rare disease)	Ignores false positives	which-score…, the-confusion-matrix…
Specificity	$\frac{TN}{TN+FP}$	[0, 1]	Cost of FP high (complement of recall)	—	the-confusion-matrix…
F1	$\frac{2 \cdot P \cdot R}{P+R}$	[0, 1]	Balance precision vs recall equally	Doesn’t weight costs; not interpretable unlike P or R	which-score…
ROC-AUC	area under TPR vs FPR curve	[0, 1]	Ranking quality; separability of classes	Misleading under severe class imbalance	which-score…
PR-AUC	area under P vs R curve	[0, 1]	Severe class imbalance	—	which-score…
Log-loss / Cross-entropy	$-\frac{1}{n}\sum[y\log\hat p + (1-y)\log(1-\hat p)]$	[0, ∞)	Probabilistic classification; rewards well-calibrated	Sensitive to over-confident wrong predictions	calibration-curves…

Regression / forecasting metrics

Name	Formula	When to use	Blind spot	Used in
MAE	$\frac{1}{n}\sum	y-\hat y	$	Robust to outliers; interpretable units
MSE	$\frac{1}{n}\sum(y-\hat y)^2$	Big errors hurt a lot; smooth gradient	Square units; outlier-sensitive	classical-ml-p02 (loss), attention-p02 (backprop)
RMSE	$\sqrt{\frac{1}{n}\sum(y-\hat y)^2}$	Same units as y; big errors matter	Same as MSE	evaluating-forecast-accuracy…
MAPE	$\frac{100}{n}\sum\frac{	y-\hat y	}{	y
MASE	$\frac{MAE}{\text{naive baseline MAE}}$	Compare forecasts across series of different scales	Needs the naive baseline (last value)	evaluating-forecast-accuracy…
R²	$1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}$	Share of variance explained	Misleading for non-linear data; never falls on out-of-sample if negative	baseline-models…, classical-ml-p12 (stacking)
SMAPE	$\frac{100}{n}\sum\frac{	y-\hat y	}{(	y

Probability-calibration metrics

Name	Formula	When to use	Used in
Brier score	$\frac{1}{n}\sum(\hat p - y)^2$	Probability accuracy (predicted vs actual 0/1)	calibration-curves…
Calibration / reliability diagram	bin predicted prob → plot against empirical accuracy	When the probability itself must be trustworthy (decisions hinge on it)	calibration-curves…
Expected Calibration Error (ECE)	$\sum_b \frac{n_b}{n}\|\text{acc}_b - \text{conf}_b\|$	One-number calibration summary	mention

Honest-mirror metrics (don’t get fooled)

Name	Tests for	Used in
Confusion matrix	All four cells at once (TP/FP/TN/FN)	the-confusion-matrix…
Learning curve	Train vs val score as N grows — capacity diagnosis	learning-curves…
Calibration curve	Predicted-vs-actual probability	calibration-curves…
Uplift curve	Targeting lift vs random	causal-inference-p07 (CATE)

Decision trees (which metric goes with which problem)

Is the outcome yes/no?

• Imbalanced? → don’t use accuracy; PR-AUC + recall(precision at a threshold)
• Two error types have different cost? → threshold-based P/R with explicit cost matrix
• Probabilities matter for a downstream decision? → log-loss + calibration curve

Is the output a number?

• Outliers matter → use MAE or MAPE (robust), never just MSE
• Big errors must hurt → RMSE or MSE
• Compare forecasts across series of different scales → MASE
• Want a ”% variance explained” soundbite → R², but read the caveat

Information-theoretic quantities

Name	Formula	Range	When to use	Used in
Entropy	$H(p) = -\sum_i p_i \log p_i$	[0, ∞)	Uncertainty in a single distribution — how “spread out” a set of probabilities is	decision-trees-from-scratch-how-splits-actually-get-made.md (Entropy split criterion)
Cross-entropy	$H(p,q) = -\sum_i p_i \log q_i$	[0, ∞)	Same quantity as log-loss above — how well a predicted distribution $q$ matches the true one $p$	See Log-loss / Cross-entropy row above
KL divergence	$D_{KL}(p\|q) = \sum_i p_i \log\frac{p_i}{q_i} = H(p,q) - H(p)$	[0, ∞), 0 iff $p=q$	The “extra bits” wasted using $q$ instead of the true $p$; asymmetric ($D_{KL}(p\|q) \neq D_{KL}(q\|p)$)	Model-calibration diagnostics, variational inference
Mutual information	$I(X;Y) = \sum_{x,y} p(x,y)\log\frac{p(x,y)}{p(x)p(y)}$	[0, ∞)	How much knowing $X$ reduces uncertainty about $Y$ — used for feature selection and information gain	decision-trees-from-scratch-how-splits-actually-get-made.md (Information Gain = mutual information between a split and the label)
Perplexity	$\text{PPL} = e^{H(p,q)} = 2^{H(p,q) \text{ (base 2)}}$	[1, ∞)	LLM evaluation — “effective number of equally-likely next-token choices” the model is confused among; lower is better	building-a-miniature-transformer-from-scratch-the-lego-approach-to-deep-learning.md (softmax entropy over the mini-transformer’s vocabulary), building-your-first-rag-pipeline-chunking-embedding-and-retrieval.md (LLM output evaluation)

The relationships in one line: cross-entropy is what you actually compute (it’s log-loss); entropy is cross-entropy’s floor (the best any model could do, since $q=p$); KL divergence is the gap between them ($H(p,q) - H(p)$) — the wasted bits from an imperfect model; perplexity is just cross-entropy exponentiated back into “number of choices” units so it’s easier to reason about intuitively (a perplexity of 8 means the model is, on average, as uncertain as if it were choosing uniformly among 8 options).

import numpy as np

# A 4-way classification: true label is class 0 (one-hot), model predicts a distribution
p_true = np.array([1.0, 0.0, 0.0, 0.0])       # true distribution (one-hot)
q_pred = np.array([0.7, 0.1, 0.1, 0.1])       # model's predicted distribution

cross_entropy = -np.sum(p_true * np.log(q_pred + 1e-12))
entropy_true = -np.sum(p_true[p_true > 0] * np.log(p_true[p_true > 0]))  # 0 for one-hot
kl_div = cross_entropy - entropy_true
perplexity = np.exp(cross_entropy)

print(f"Cross-entropy: {cross_entropy:.4f}")   # ~0.357
print(f"KL divergence: {kl_div:.4f}")          # ~0.357 (equals cross-entropy since entropy_true=0)
print(f"Perplexity: {perplexity:.4f}")         # ~1.43 — model is about as uncertain as choosing among ~1.4 options

Cross-references

• Pick the score that matters: which-score-actually-matters-a-plain-english-guide-to-precision-recall-and-the-rest.md
• Why accuracy lies: the-confusion-matrix-why-it-s-the-honest-mirror-fo.md
• Forecast accuracy in practice: evaluating-forecast-accuracy-mape-rmse-and-why-averages-lie.md
• When probabilities lie: calibration-curves-when-your-model-s-probabilities-are-lying-to-you.md
• Honest model comparison (is A actually better than B?): is-your-model-actually-better-a-plain-english-guide-to-statistical-significance.md
• Algorithm family map (which metric suits which algorithm): algorithm-subcategory-map.md

Further reading

• Powers, D. (2011). Evaluation: from Precision, Recall and F-Factor.
• Hyndman, R. & Athanasopoulos, G. (2018). Forecasting: principles and practice — chapters on forecast accuracy.
• Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores. (Brier / CRPS)
• Kaggle — Store Sales – Time Series (use of MAPE/MASE on a real forecast).
• Kaggle — Home Credit Default Risk (precision/recall tradeoff on imbalanced credit).