Reference: Loss Functions (the Training Objective)

This reference covers loss functions from the training perspective — the quantity an optimizer actually minimizes during gradient descent. That is a distinct question from evaluation metrics (what you report on held-out data to compare models). The same formula, say Mean Squared Error, appears in both roles, but the conversation is different: a loss is something you differentiate and backpropagate through; a metric is something you score, threshold, and rank with. If you arrived here looking for the scoring side of the story (RMSE, MAE, accuracy, F1, AUC, Brier score), see evaluation-metrics.

A mental anchor for the whole document:

Loss = “what the optimizer pulls the weights toward.” Metric = “what the human reads off the dashboard.”

They are often mathematically related but rarely operationally identical — more on that in the “loss ≠ metric” section near the end.

---

Roster

Loss	Formula (one line)	Range	When to use	Probabilistic interpretation
MSE (L2 / squared error)	mean((ŷ − y)²)	[0, ∞)	Regression with near-Gaussian noise; smooth optimization landscape	Negative log-likelihood of a Gaussian
MAE (L1 / absolute error)	mean(|ŷ − y|)	[0, ∞)	Regression with heavy-tailed noise or outliers you do not want to chase	Negative log-likelihood of a Laplace
Huber (smooth L1)	½(ŷ−y)² if |ŷ−y| ≤ δ, else δ·(|ŷ−y| − δ/2)	[0, ∞)	Regression where most noise is small but a few wild outliers exist	Negative log-likelihood of a Huber density
Binary cross-entropy (log loss)	−[y log ŷ + (1−y) log(1−ŷ)]	[0, ∞)	Binary classification, calibrated probabilities	Negative log-likelihood of a Bernoulli
Categorical cross-entropy	−Σ y_k log softmax(z)_k	[0, ∞)	Multi-class classification with soft or one-hot targets	Negative log-likelihood of a categorical
Hinge	max(0, 1 − y·f(x)) with y ∈ {−1, +1}	[0, ∞)	Max-margin classifiers (SVMs); when you care about the decision, not probabilities	None — geometric margin
Contrastive (Siamese)	y · d² + (1−y) · max(0, m − d)	[0, ∞)	Metric learning from pairs (same / different labels)	None — geometric embedding
Triplet	max(0, d(a, p) − d(a, n) + m)	[0, ∞)	Metric learning from anchored (anchor, positive, negative) triples	None — geometric embedding

Throughout, y is the target, ŷ is the model’s prediction (often f(x) or a parameterized output like z before a softmax).

---

Which loss for which problem

• Is your target a continuous number on the real line?
  • Are there wild outliers in y that you suspect are noise, not signal?
    • Mostly small errors with a few big ones? → Huber (best of both worlds; quadratic near zero, linear in the tails).
    • Outliers are everywhere and the median is more meaningful than the mean? → MAE.
  • Noise is roughly Gaussian and you want a smooth, well-behaved gradient everywhere? → MSE.
• Is your target a single label from K classes?
  • Do you need calibrated probabilities at the end (e.g., for risk pricing or thresholding)? → Categorical cross-entropy (with a softmax output).
  • Do you only need the decision and want a max-margin style objective? → Hinge (multi-class variants like Crammer–Singer if K > 2).
• Is the target a single yes/no?
  • You want interpretable probabilities and the standard scoring rule for them. → Binary cross-entropy (log loss).
  • You want max-margin classification without probability semantics. → Hinge.
• Are you learning embeddings where similar items should be close in space?
  • You have explicit (similar, dissimilar) pairs. → Contrastive.
  • You can mine triplets (anchor, positive, negative) and are willing to maintain a mining strategy. → Triplet; falls back to easy, zero-gradient triplets without it.

---

Why MSE and cross-entropy are negative log-likelihoods

Almost every standard loss can be derived as a maximum-likelihood objective under an assumed noise model. This is not a footnote — it is the reason certain losses are the natural choice for certain data-generating processes, and it explains why, for example, switching from MSE to MAE is equivalent to switching your noise model from Gaussian to Laplacian.

Maximum likelihood says: pick the parameters θ that maximize the probability of the observed data under the model. Equivalently, minimize the negative log-likelihood (NLL):

$$\mathcal{L}_{\text{NLL}}(\theta) = -\log p_{\theta}(y \mid x)$$

Now plug in noise models:

1. Gaussian noise on the target. Assume y = f_θ(x) + ε, with ε ~ 𝒩(0, σ²). Then

$$-\log p(y \mid x) = \frac{(y - f_θ(x))^2}{2\sigma^2} + \text{const}$$

Drop the constants and the factor of 1/2 (absorbed into the learning rate) and you get MSE. So training with MSE is precisely maximum-likelihood under Gaussian-residual assumptions.

2. Laplacian noise. Assume ε ~ Laplace(0, b). Then -log p(y|x) ∝ |y - f_θ(x)|, which is MAE. The Laplace distribution has heavier tails than the Gaussian, which is why MAE is robust to outliers: extreme residuals are less surprising under the Laplacian and so contribute a smaller gradient.

3. Bernoulli target. y ∈ {0, 1}, p(y=1|x) = σ(z) where σ is the logistic sigmoid. The log-likelihood of a single observation is y log σ(z) + (1-y) log(1-σ(z)); negate it and you have binary cross-entropy.

4. Categorical target. y ∈ {1, ..., K} one-hot encoded, p(y=k|x) = softmax(z)_k. The NLL is -Σ_k y_k log softmax(z)_k, which is categorical cross-entropy.

Plain English	Statistical symbol	Python (NumPy / torch)
Model prediction (raw output / logit)	z	z = model(x)
Predicted probability of class k	p_k = softmax(z)_k	F.softmax(z, dim=-1)
Squared residual	(y - ŷ)²	(y - yhat)**2
Negative log-likelihood under Gaussian	(y - ŷ)² / (2σ²)	0.5 * (y - yhat)**2 / sigma**2
Negative log-likelihood under Bernoulli	-y log ŷ - (1-y) log(1-ŷ)	F.binary_cross_entropy(yhat, y)

The deep consequence: when you pick a loss, you are also picking a noise model. If your data does not match the noise model the loss implies, the optimizer will dutifully minimize the wrong objective.

---

Worked Python example per loss

Each block below is illustrative (not executed in this document) but the code and outputs are honest and realistic for what you would see on a small toy problem.

1. MSE

import numpy as np

y_true  = np.array([3.0, -1.0, 2.0, 5.0])
y_pred  = np.array([2.5, -0.5, 2.0, 4.8])

mse = np.mean((y_pred - y_true) ** 2)
# mse = 0.135

y_pred - y_true is the residual vector [−0.5, 0.5, 0.0, −0.2]. Squaring elementwise gives [0.25, 0.25, 0.0, 0.04], sum 0.54, mean 0.54 / 4 = 0.135. Note that the two largest residuals (−0.5 and 0.5) each contribute 0.25 — together 93% of the total, even though neither alone is “more than half.” This is the “outlier sensitivity” of MSE in microcosm: the squared term means big errors dominate the gradient.

2. MAE

mae = np.mean(np.abs(y_pred - y_true))
# mae = 0.3

The same residuals now contribute 0.5 + 0.5 + 0.0 + 0.2 = 1.2, divided by 4 = 0.3. No single error dominates — MAE treats all residuals linearly.

3. Huber

def huber(y_pred, y_true, delta=1.0):
    r = np.abs(y_pred - y_true)
    quad = 0.5 * r ** 2
    lin  = delta * (r - 0.5 * delta)
    return np.mean(np.where(r <= delta, quad, lin))

huber(y_pred, y_true, delta=1.0)
# 0.0675   # all residuals are <= 1.0, so every point uses the quadratic
#            branch -- which makes Huber exactly half of MSE (0.135), not
#            equal to it. The formula's quadratic branch is (1/2)r^2, so
#            whenever delta is large enough to keep every point inside it,
#            Huber = 0.5 * MSE by construction.
huber(y_pred, y_true, delta=0.3)
# 0.0575   # the big residuals now go into the linear branch

With δ = 1.0 every residual is inside the quadratic zone, so Huber lands at exactly half of MSE (0.0675 vs 0.135) — the 1/2 in the formula, not a coincidence. With δ = 0.3 the 0.5-magnitude residuals fall outside and the loss flattens, capping the gradient at δ — that is the whole point of Huber.

4. Binary cross-entropy (log loss)

import torch
import torch.nn.functional as F

logits = torch.tensor([0.9, -0.4, 2.1, -3.0])
labels = torch.tensor([1.0, 0.0, 1.0, 0.0])

probs  = torch.sigmoid(logits)
# probs  = tensor([0.7109, 0.4013, 0.8909, 0.0474])

bce    = F.binary_cross_entropy(probs, labels)
# bce = tensor(0.2546)

# Equivalently, the more numerically stable form:
bce_logits = F.binary_cross_entropy_with_logits(logits, labels)
# bce_logits = tensor(0.2546)

Always prefer binary_cross_entropy_with_logits: it uses the log-sum-exp trick internally and avoids the catastrophic cancellation you get from log(sigmoid(z)) when z is large negative.

5. Categorical cross-entropy

logits = torch.tensor([[2.0, 1.0, 0.1],   # row 0: true class is 0
                       [0.2, 3.0, 0.4]])   # row 1: true class is 1
labels = torch.tensor([0, 1])

ce = F.cross_entropy(logits, labels)
# ce = tensor(0.2719)

F.cross_entropy internally applies log_softmax and then indexes out the -log p_true_class term — numerically stable and the standard choice for multi-class problems.

6. Hinge

# Hinge expects labels in {-1, +1}, not {0, 1}
f      = np.array([0.6, -0.3, 2.0, -0.8])   # raw scores
y_pm   = np.array([ 1,  -1,   1,   -1])      # ±1 labels

hinge  = np.maximum(0, 1 - y_pm * f)
# hinge = array([0.4, 0.7, 0.0, 0.2])
loss   = np.mean(hinge)
# loss = 0.325

The third example (f = 2.0, y = 1) contributes 0 because the margin 1 − y·f = 1 − 2 = −1 is already satisfied — the model is on the correct side and far enough from the boundary that hinge does not push it.

7. Contrastive (Siamese)

def contrastive_loss(d, y, margin=1.0):
    # y = 1 if "same class", 0 if "different class"
    return torch.mean(y * d ** 2 + (1 - y) * torch.clamp(margin - d, min=0) ** 2)

d   = torch.tensor([0.2, 1.5, 0.8, 0.9])    # pairwise distances
y   = torch.tensor([1.0, 0.0, 1.0, 0.0])    # 1 = same, 0 = different

loss = contrastive_loss(d, y, margin=1.0)
# loss = tensor(0.1725)

Pair 0 (same class, distance 0.2) contributes 0.04. Pair 1 (different, distance 1.5 > margin) contributes 0 — already far enough apart. Pair 2 (same class, distance 0.8) contributes 0.8² = 0.64 — same-class pairs are penalized by squared distance with no cap, so this is the pair actually driving the loss, at 93% of the 0.69 total before averaging. Pair 3 (different, distance 0.9) contributes (1 − 0.9)² = 0.01, pushing them further apart.

8. Triplet

def triplet_loss(d_ap, d_an, margin=1.0):
    return torch.mean(torch.clamp(d_ap - d_an + margin, min=0))

d_ap = torch.tensor([0.4, 0.3, 0.5])   # anchor-positive distances
d_an = torch.tensor([1.5, 0.6, 0.4])   # anchor-negative distances

loss = triplet_loss(d_ap, d_an, margin=1.0)
# loss = tensor(0.6000)

The third triplet contributes 0.5 − 0.4 + 1.0 = 1.1 — the negative is closer than the positive, so the loss is large and the gradient pulls the negative away hard. The second triplet contributes 0.3 − 0.6 + 1.0 = 0.7 — a smaller but still active penalty. The first triplet contributes 0 because the margin is already satisfied. Mean of [0, 0.7, 1.1] is 0.6.

---

The gradient each loss produces

The “shape” of the gradient is what actually flows through your network, so it is the most important practical property of a loss.

Let z be the model’s pre-output (logit) for a single example. Below, ŷ denotes the post-activation prediction (e.g., ŷ = z for regression, ŷ = σ(z) for binary classification, p = softmax(z) for multi-class).

Loss	Gradient w.r.t. z	Behaviour
MSE	∂L/∂z = 2(ŷ − y)·(∂ŷ/∂z). If ŷ = z: 2(z − y)	Grows linearly with the error — large errors get big but well-behaved gradients
MAE	∂L/∂z = sign(ŷ − y)·(∂ŷ/∂z). If ŷ = z: sign(z − y)	Constant magnitude ±1 — never explodes, but sign(0) is undefined (subgradient) and progress is slow near the optimum
Huber	2(ŷ − y)·(∂ŷ/∂z) inside the quadratic zone, δ·sign(ŷ − y)·(∂ŷ/∂z) outside	Smooth transition; bounded gradient magnitude in the tail
BCE (with logits)	∂L/∂z = σ(z) − y	Beautiful: the residual in probability space. Vanishes when σ(z) = y
Categorical CE (with logits)	∂L/∂z_k = softmax(z)_k − y_k	Same story, vector form: “predicted minus one-hot”
Hinge	∂L/∂z = −y · x if y·f(x) < 1, else 0	Zero gradient when the margin is satisfied — frozen once you are on the right side
Contrastive	Pull/push terms: 2y·d (same), −2(1−y)(m − d) (diff), both × Jacobian of embedding	Symmetric “spring” with a slack cut-off in the different-class branch
Triplet	∂L/∂d_ap = +1, ∂L/∂d_an = −1 (when active)	Anchor-positive pulled in, anchor-negative pushed out — both scaled by embedding Jacobian

The BCE and categorical CE gradients are the reason these losses are the workhorses of classification: the gradient is exactly the calibration error ŷ − y, which means the model only stops learning when its predicted probabilities match the empirical labels. The same is not true of, say, MSE applied to a sigmoid output — there the gradient also carries a σ'(z) = σ(z)(1−σ(z)) factor that saturates to zero when the logit is large, famously causing the “saturation” problem that cross-entropy avoids.

A compact NumPy illustration of the three “workhorse” gradients on the same toy problem:

z = np.array([0.9, -0.4, 2.1, -3.0])
y = np.array([1.0, 0.0,  1.0,  0.0])

# BCE: gradient w.r.t. logit is (sigmoid(z) - y)
grad_bce = 1 / (1 + np.exp(-z)) - y
# grad_bce = array([-0.2891,  0.4013, -0.1091,  0.0474])

# Categorical CE: gradient w.r.t. logits is (softmax(z) - one_hot(y))
z_row = np.array([2.0, 1.0, 0.1])
p    = np.exp(z_row - z_row.max()); p = p / p.sum()
y_oh = np.array([1, 0, 0])
grad_ce = p - y_oh
# grad_ce = array([-0.3410,  0.2424,  0.0986])
# (negative on the true class — descent pulls it up)

# Hinge: gradient w.r.t. f is -y * x * 1[y*f < 1]
f = np.array([0.6, -0.3, 2.0, -0.8])
y_pm = np.array([1, -1, 1, -1])
grad_hinge = np.where(y_pm * f < 1, -y_pm, 0.0)
# grad_hinge = array([-1.,  1.,  0.,  1.])
# third entry: margin already satisfied -> zero gradient

---

Huber for robust regression

MSE is mathematically tidy and easy to differentiate, but it has one well-known failure mode: a single y value of, say, 100 when the rest of the data sits around 0 will pull the entire fit toward 100 because the squared residual 10000 dominates the loss. Huber is the standard remedy.

The Huber loss is defined piecewise:

$$L_\delta(r) = \begin{cases} \tfrac{1}{2} r^2 & \text{if } |r| \le \delta \\ \delta \left(|r| - \tfrac{1}{2}\delta\right) & \text{if } |r| > \delta \end{cases}$$

where r = y − ŷ is the residual and δ is a tunable threshold. The derivative is

$$L_\delta'(r) = \begin{cases} r & \text{if } |r| \le \delta \\ \delta \cdot \text{sign}(r) & \text{if } |r| > \delta \end{cases}$$

which is continuous at |r| = δ (both branches give δ). So Huber is C¹ — once differentiable everywhere — but not C² at the kink. That is enough for most first-order optimizers (Adam, SGD with momentum) and PyTorch’s SmoothL1Loss.

Practical recipe:

• Estimate δ from the inlier noise scale (e.g., the median absolute deviation of the residuals on a clean validation set). δ ≈ 1.35·σ is a common rule that makes Huber 95% as efficient as MSE on Gaussian noise.
• If δ is too large, Huber degenerates to MSE; if too small, it degenerates to a scaled MAE with a thin quadratic “softness” at the origin.
• Standard library: torch.nn.HuberLoss(reduction='mean', delta=...), or torch.nn.SmoothL1Loss(beta=δ) scaled by δ — the two are related by HuberLoss(delta=δ) = δ · SmoothL1Loss(beta=δ), not identical, so swapping one for the other silently rescales the loss (and the effective learning rate) by δ.

---

Contrastive and triplet losses for embedding models

For the last two losses the goal is not to predict a label but to learn a geometry — a vector space in which semantically similar items are close and dissimilar items are far. The downstream evaluation is usually retrieval (Recall@k, mean average precision), not classification accuracy, so the loss has to push the geometry rather than the per-example prediction.

Contrastive (Siamese) learns from explicit pairs (x_i, x_j, y_ij) where y_ij = 1 if same class, 0 if different. The loss is

$$L_{\text{con}} = y \cdot d^2 + (1 - y) \cdot \max(0, m - d)^2$$

where d = ‖f(x_i) − f(x_j)‖₂ is the embedding distance and m is a margin that says “dissimilar pairs must be at least m apart; closer than that incurs loss.”

Triplet learns from (anchor, positive, negative) triples. The loss is

$$L_{\text{tri}} = \max\!\left(0, \, d(a, p) - d(a, n) + m\right)$$

which says “the negative must be at least m further from the anchor than the positive is.” The triplet formulation learns relational structure — is this pair closer than that one? — rather than the absolute distances contrastive learns, which helps when your supervision is naturally ordinal (rankings, relative similarity judgments) rather than binary same/different. That relational framing is why FaceNet (Schroff et al., 2015) popularized it, though it comes at a real cost contrastive doesn’t carry: most triplets are easy and contribute zero gradient, so triplet training needs a mining strategy (see below) that pairwise contrastive can skip entirely.

Mining strategy matters: most triplets are easy (the margin is already satisfied) and contribute zero gradient. Semi-hard mining — keeping only triplets where d(a, p) < d(a, n) < d(a, p) + m — keeps the gradient signal meaningful. See attention-from-scratch-a-deep-p02-backpropagation-intuitively-how-networks-learn-fro.md) for the gradient-flow mechanics that make such losses trainable end-to-end.

---

“Loss ≠ metric” — the distinction, made explicit

Beginners often assume that if you train with MSE you should evaluate with MSE, or if you train with cross-entropy you should report accuracy. That is not a rule, just a coincidence in some cases. The two questions are independent:

• The loss answers: “What surface am I descending?”
• The metric answers: “How good is the resulting model for the human?”

Sometimes the two align (training a regressor with MSE and reporting RMSE on held-out data is the same equation with a square root tacked on). More often they do not, because:

1. Losses must be differentiable; metrics don’t have to be. Accuracy, F1, Recall@k, AUC — none of these are smooth, which is exactly why we use cross-entropy as a surrogate for accuracy during training.
2. Losses are often averaged per-example; metrics are often aggregated across a population. Ranking quality matters for metrics but is invisible to per-example cross-entropy.
3. A loss can match a metric probabilistically without being identical. Cross-entropy is a strictly proper scoring rule: its unique minimizer is the true conditional distribution p(y|x). So minimizing cross-entropy also optimizes Brier score and log loss (the metric) — but it does not directly optimize accuracy.
4. You may want to optimize for fairness, calibration, or ranking, but report the metric anyway. A model trained with focal loss (a cross-entropy variant) might be evaluated with plain accuracy — the loss reshapes the gradient to emphasize hard examples; the metric still reads off the model’s top-1 decision.

A pairing table for the most common combinations:

Loss (training)	Matching metric (evaluation)	Why the pairing
MSE	RMSE / R²	Same units as y; R² normalizes against the variance baseline
MAE	MAE	Identical formula, just no gradient needed at evaluation time
Huber	MAE (or RMSE if outliers are real signal)	Huber handles the training; the metric should reflect what you actually care about at deployment
Binary cross-entropy	Log loss, Brier score, AUC, calibration error	All four are proper scoring rules or ranking statistics computed from probabilities — see calibration-curves-when-your-model-s-probabilities.md)
Categorical cross-entropy	Top-1 / Top-k accuracy, log loss	Accuracy is the thresholded version of the probabilities the loss was learning
Hinge	Accuracy (or zero-one loss)	Hinge optimizes the decision; accuracy measures the decision
Contrastive / Triplet	Recall@1, Recall@k, MAP, NMI	Retrieval metrics on the embedding space — see baseline-models-why-you-should-always-build-the-du.md) for the random-baseline sanity check that should accompany any of these

For a deeper, scoring-side treatment of the right-hand column of this table, see evaluation-metrics.

---

Edge cases and common mistakes

1. MSE on a sigmoid output for classification. This is a classic mistake. The gradient carries a σ(z)(1 − σ(z)) factor that vanishes as |z| grows, so badly-wrong predictions get smaller gradients — the opposite of what you want. Use BCE-with-logits instead. The symptom: training stalls on examples the model is very confidently wrong about. See attention-from-scratch-a-deep-p12-why-isn-t-my-model-learning-a-friendly-guide-to-di.md) for the diagnostic path.

2. Using BCE without with_logits. F.binary_cross_entropy(torch.sigmoid(z), y) will overflow or underflow when z is large because log(1 − σ(z)) ≈ −z for large positive z, and the subtraction log(σ) − log(1 − σ) is numerically unstable. The with_logits variant uses logsumexp internally and is the only correct way.

3. Forgetting that F.cross_entropy averages over the batch. If you need a sum reduction (e.g., to weight examples per-batch) you must pass reduction='sum' or reduction='none' and sum yourself. The default is mean, which silently changes the effective learning rate when batch sizes vary.

4. Class imbalance + plain cross-entropy. Cross-entropy weights each example equally, so a 99%-negative dataset will train the model to predict “negative” almost always. Use class-weighted CE, focal loss, or resampling. The confusion matrix in the-confusion-matrix-why-it-s-the-honest-mirror-fo.md is the honest way to detect this failure.

5. Hinge with labels in {0, 1} instead of {-1, +1}. The Hinge formula max(0, 1 − y·f) assumes y ∈ {−1, +1}. Using 0 instead of −1 for the negative class produces a different (and meaningless) objective: every “negative” example contributes max(0, 1 − 0) = 1 regardless of how the model scores it.

6. Huber with δ set by gut feel. Pick δ from the inlier noise scale; otherwise Huber is either secretly MSE (when δ is too large) or secretly MAE (when δ is too small), and you have gained nothing.

7. Triplet loss without mining. Naive random triplets are mostly “easy” (margin satisfied, zero gradient). Either do semi-hard mining or use a modern variant (circle loss, multi-similarity loss) — otherwise training plateaus almost immediately.

8. Contrastive / triplet with un-normalized embeddings and no collapse guard. Without L2 normalization, the embedding space can collapse to a tiny region with very large norms (the loss is still “satisfied” because distances are all small). Guard against it one of two ways: normalize the embedding to the unit sphere, which bounds every distance and is the simpler default, or leave the embedding unnormalized and add a norm penalty instead, which costs a hyperparameter but preserves magnitude information the loss can use. Either is fine; shipping neither is what causes the collapse.

9. Confusing “loss went down” with “the model improved”. A loss can decrease because the model is becoming over-confident on a subset of the data, not because it is generalizing. Always pair the training loss with a held-out metric; if the loss drops but the metric plateaus, you are likely overfitting the loss surface rather than the data distribution. The baseline-models sanity check in baseline-models-why-you-should-always-build-the-du.md) is the right first move when this happens.

10. Using gradient descent on a non-differentiable loss without thinking. MAE and Huber are only sub-differentiable at the kink; most autograd libraries pick one subgradient (e.g., 0 at r = 0), which is usually fine but can cause dead gradients in pathological cases. If you see training stall at a flat plateau with MAE, this is why — switching to Huber or perturbing the targets slightly unstalls it.

---

Why this matters for the rest of the corpus

Loss functions sit at the intersection of three threads in this corpus:

• The gradient-descent mechanics in classical-ml-foundations-p02-how-gradient-descent-actually-works-and-the-varian.md) — the loss is the function L(θ) whose gradient ∇_θ L the optimizer steps along. Picking a different loss literally changes the geometry of that descent.
• The backpropagation intuition in attention-from-scratch-a-deep-p02-backpropagation-intuitively-how-networks-learn-fro.md) — the loss is the starting point of the chain rule. The shape of ∂L/∂z (last column of the gradient table above) is what propagates all the way back through the network.
• The debugging perspective in attention-from-scratch-a-deep-p12-why-isn-t-my-model-learning-a-friendly-guide-to-di.md) — most “my model isn’t learning” tickets trace back to a loss / target / output-layer mismatch (the MSE on a sigmoid mistake above being the canonical example).

---

Cross-references

• classical-ml-foundations-p02-how-gradient-descent-actually-works-and-the-varian.md) — the optimizer’s view of the loss as a surface to descend.
• attention-from-scratch-a-deep-p02-backpropagation-intuitively-how-networks-learn-fro.md) — how the loss gradient flows back through the network.
• attention-from-scratch-a-deep-p12-why-isn-t-my-model-learning-a-friendly-guide-to-di.md) — diagnosing the loss / output / target mismatches that stall training.
• the-confusion-matrix-why-it-s-the-honest-mirror-fo.md — what the metric side of the story looks like once training is done.
• calibration-curves-when-your-model-s-probabilities.md) — why a cross-entropy-trained model can still be mis-calibrated, and how to check.
• baseline-models-why-you-should-always-build-the-du.md) — the random / dumb baseline that a chosen loss should always beat before you call the model “good.”

For the matching evaluation-side reference, see evaluation-metrics.

---

Further reading

• Bishop, Pattern Recognition and Machine Learning, chapter 5 — the canonical derivation of loss functions as maximum-likelihood objectives under Gaussian, Laplacian, and Bernoulli noise models; the source of the probabilistic-interpretation table in this file.
• Goodfellow, Bengio, Courville, Deep Learning, chapters 6 (deep feedforward networks: the cross-entropy / saturation discussion) and 8 (optimization for training: the gradient-shape analysis) — the standard graduate-level reference for why cross-entropy is preferred to MSE on sigmoid/softmax outputs.
• Schroff, Kalenichenko, Philbin, FaceNet: A Unified Embedding for Face Recognition and Clustering, CVPR 2015 — the original modern treatment of triplet loss with semi-hard negative mining; the conceptual ancestor of every contrastive / triplet loss used in self-supervised representation learning today.
• Library docs — torch.nn loss classes (MSELoss, L1Loss, SmoothL1Loss, HuberLoss, BCEWithLogitsLoss, CrossEntropyLoss, MultiMarginLoss, TripletMarginLoss); sklearn.metrics for the evaluation-side counterparts in evaluation-metrics.