Reference: Activation Functions

A consolidated roster of the activation functions used across the Attention From Scratch series and the causal/ML articles. Linked from each article’s 📚 References toggle.

Roster

Name	Formula	Output range	When to use	Gradient property	Used in
Sigmoid / Logistic	$\sigma(x) = \frac{1}{1+e^{-x}}$	$(0, 1)$	Binary output probability; gate inside LSTM	Saturates → vanishing gradient for	x
Tanh	$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$	$(-1, 1)$	RNN hidden states when zero-centered output helps	Saturates like sigmoid (both tails)	attention-p05 (RNN hidden state)
ReLU	$\max(0, x)$	$[0, \infty)$	Default hidden layer in deep nets; fixes vanishing gradient	Non-saturating for x>0; “dead neuron” risk for x<0	attention-p03 (the ReLU fix), attention-p09 (mini-Transformer)
Leaky ReLU	$\max(0.01x,\; x)$	$(-\infty, \infty)$	When dead ReLU neurons are a problem (many units stuck at 0)	Small positive gradient for x<0 keeps units alive	mentioned in attention-p03 as a ReLU variant
ELU	$x$ if $x>0$ else $\alpha(e^x-1)$	$(-\alpha, \infty)$	Smoother than ReLU near zero; can speed learning	Negative saturation is smoother than Leaky	(cross-referenced from attention-p03)
GELU	$x \cdot \Phi(x)$ where $\Phi$ is the normal CDF	$(-0.17, \infty)$	Transformers (BERT, GPT, modern LLMs)	Smooth, non-monotonic region near zero	attention-p09 (mini-Transformer Feed-Forward)
Softmax	$\frac{e^{x_i}}{\sum_j e^{x_j}}$	$(0, 1)$, sums to 1	Multi-class output; attention weights	Produces a probability distribution over K classes/tokens	attention-p07 (attention scores → weights)

The one paragraph you actually need

The reason activation functions exist at all is nonlinearity. Without one, a stack of linear layers collapses to a single linear layer (linear of linear is still linear). The choice is almost always about the gradient:

• Saturating activations (sigmoid, tanh) flatten out at both tails, so their gradient → 0 there. Backprop multiplies those zeros down the stack and the gradient vanishes — that’s the whole story of attention-p03.
• ReLU keeps a positive gradient for x>0, so deep nets stop vanishing. But x<0 gives 0 gradient → a “dead neuron” that never recovers. Leaky ReLU / ELU / GELU all patch this differently.
• Softmax is the odd one out: it’s not a hidden-layer activation, it’s an output that turns scores into a probability distribution. It’s what attention uses to convert raw scores into weights that sum to 1.

Which one for which architecture (decision rule)

Architectural spot	Default	Why
Hidden layer, deep net	ReLU	non-saturating, cheap, won’t vanish
Hidden layer, if many dead units	Leaky ReLU or GELU	keeps a tiny gradient on the left
RNN hidden state	Tanh	zero-centered, bounded (RNNs need bounded memory)
LSTM / GRU gate	Sigmoid	must output a number in [0,1] (a “keep fraction”)
Multi-class output	Softmax	turn logits into a probability distribution
Transformer FFN	GELU	what the paper used; smooth, well-tested
Binary classifier output	Sigmoid	one probability in [0,1]

Cross-references

• Vanishing / exploding gradient deep-dive: attention-from-scratch-p03-why-deep-networks-die-solving-the-vanishing-gradie.md
• The single neuron (where sigmoid first appears): neural-networks-without-the-calculus-what-s-actually-happening-inside-a-single-neuron.md
• Attention (where softmax first appears): the-attention-mechanism-finally-explained-without-the-matrix-algebra.md

Further reading

• Ramachandran, P. et al. (2017). Searching for Activation Functions. (GELU)
• Maas, A. et al. (2013). Rectifier Nonlinearities Improve Neural Networks. (Leaky ReLU)
• Clevert, D. et al. (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs).
• He, K. et al. (2015). Delving Deep into Rectifiers… (PReLU / init for ReLU)