Activation Functions Reference Relu Sigmoid Tanh G

You’ve built a neural network. You’ve stacked a few layers. But inside every hidden layer, there’s a moment of truth — a single neuron receives a number. That number is the weighted sum of all its inputs plus a bias. It could be anything: -4.7, 0.0, 12.3, 800. The neuron has to make a decision. Should it fire? How strongly? Should it stay completely quiet?

That decision is made by an activation function. It’s the neuron’s rule for transforming that raw incoming number into an output. Without it, stacking layers would be pointless — a chain of linear transformations is still just one linear transformation. You’d never learn a curve, a boundary, or anything interesting.

In this article, we’re going deep on four activation functions that shaped deep learning: Sigmoid, Tanh, ReLU, and GELU. We won’t just stare at formulas. We’ll trace the historical arc — why researchers abandoned sigmoid for tanh, then abandoned tanh for ReLU, and why modern transformers use GELU. Each switch fixed a real, painful training problem. By the end, you’ll have a mental model for choosing the right one, not just a memorized list.

Let’s start by watching a single neuron make its decision.

import torch
import torch.nn as nn

# --- A single neuron's raw materials ---
# Pretend these are the weights and bias our neuron has learned so far
weights = torch.tensor([0.5, -0.8, 1.2])
bias = torch.tensor(0.1)

# A single input vector (3 features)
inputs = torch.tensor([2.0, -1.0, 3.0])

# Step 1: The weighted sum (the "raw logit")
z = torch.dot(weights, inputs) + bias

# Step 2: Apply three different activation functions to the SAME raw logit
sigmoid = nn.Sigmoid()
tanh = nn.Tanh()
relu = nn.ReLU()
gelu = nn.GELU()

print(f"Raw logit (z): {z:.4f}\n")
print(f"After Sigmoid: {sigmoid(z):.4f}")
print(f"After Tanh:    {tanh(z):.4f}")
print(f"After ReLU:    {relu(z):.4f}")
print(f"After GELU:    {gelu(z):.4f}")

Look at that output. The same raw number — the same evidence — gets interpreted completely differently depending on the activation function. Sigmoid squashes it between 0 and 1, like a probability. Tanh squashes it between -1 and 1, centered at zero. ReLU passes it through unchanged if it’s positive, or slams the door with a zero if it’s negative. GELU lets it through, but with a little hesitation — a soft, probabilistic gate.

That’s the whole game. Every activation function answers the same question — “what should I do with this number?” — with a different philosophy. Now let’s meet the classical answers, and understand why they eventually weren’t good enough.

The Classical Era: Sigmoid and Tanh

For years, if you built a neural network, you used sigmoid or tanh. They were the defaults. They’re smooth, elegant, and have nice mathematical properties. Sigmoid outputs a number between 0 and 1, which feels like a probability. Tanh outputs between -1 and 1, which centers your data at zero. What’s not to love?

Here’s the catch: they both have a fatal flaw in deep networks. It’s called saturation.

Think of an S-curve. When the input is a large positive number, the sigmoid output is pinned near 1.0. When the input is a large negative number, it’s pinned near 0.0. In those flat regions, the curve barely budges. The derivative — the slope, the “how much should I adjust?” signal — is nearly zero. The neuron is confident, but it’s also stuck. It can’t learn because the gradient has vanished.

In a network with 5, 10, or 50 layers, this is a disaster. The vanishing gradient multiplies layer after layer during backpropagation. The early layers receive a gradient so tiny they might as well be frozen. This is why deep networks were so brutally hard to train before 2010.

Let’s see this in action.

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# --- Generate input range ---
x = torch.linspace(-8, 8, 1000)

# --- Compute activations and their derivatives ---
sigmoid = nn.Sigmoid()
tanh = nn.Tanh()

# Sigmoid and its derivative
sigmoid_y = sigmoid(x)
sigmoid_deriv = sigmoid_y * (1 - sigmoid_y)  # derivative of sigmoid

# Tanh and its derivative
tanh_y = tanh(x)
tanh_deriv = 1 - tanh_y**2  # derivative of tanh

# --- Plotting ---
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

# Activation functions
ax1.plot(x.numpy(), sigmoid_y.numpy(), label='Sigmoid', linewidth=2)
ax1.plot(x.numpy(), tanh_y.numpy(), label='Tanh', linewidth=2)
ax1.set_title('Sigmoid and Tanh: The S-Curves')
ax1.set_xlabel('Input (x)')
ax1.set_ylabel('Output')
ax1.legend()
ax1.grid(True, alpha=0.3)

# Derivatives
ax2.plot(x.numpy(), sigmoid_deriv.numpy(), label='Sigmoid derivative', linewidth=2)
ax2.plot(x.numpy(), tanh_deriv.numpy(), label='Tanh derivative', linewidth=2)
ax2.set_title('Derivatives: Where the Gradient Vanishes')
ax2.set_xlabel('Input (x)')
ax2.set_ylabel('Derivative (gradient)')
ax2.legend()
ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# --- Concrete saturation example ---
x_big = torch.tensor(5.0)
sig_out = sigmoid(x_big)
sig_deriv_out = sig_out * (1 - sig_out)

print(f"Input: {x_big.item()}")
print(f"Sigmoid output: {sig_out.item():.4f}  (neuron is ~99.3% 'on')")
print(f"Sigmoid derivative: {sig_deriv_out.item():.6f}  (gradient is tiny — learning is frozen)")

Look at the right-hand plot. The derivatives of both sigmoid and tanh are a narrow bump centered at zero. Once your input wanders outside roughly [-3, 3], the gradient is practically zero. The neuron is in the flatlands. It’s shouting “I’m 99.3% sure!” but whispering “don’t bother teaching me anything new.”

Tanh is slightly better than sigmoid because it’s zero-centered — its outputs average around 0 instead of 0.5. This helps keep the gradients from all having the same sign, which reduces zig-zagging during optimization. But it still saturates. LeCun et al. (1998) were so aware of this that they recommended a specific scaled tanh — f(x) = 1.7159 * tanh(2/3 * x) — designed to keep activations in the linear-ish, non-saturating sweet spot. That was an early, clever patch. But it was still a patch.

In plain English: sigmoid and tanh work fine for shallow networks. But in deep networks, they choke. The signal dies out as it backpropagates. Researchers needed something that didn’t flatline.

The ReLU Revolution: Simple, Fast, and Deep

The breakthrough was almost absurdly simple. Instead of a smooth S-curve, what if the neuron just… turned off for negative inputs and passed positive inputs straight through?

That’s ReLU: the Rectified Linear Unit. ReLU(x) = max(0, x). If the input is positive, output it unchanged. If it’s negative, output zero. No exponentials. No division. No saturation for positive numbers. The gradient is a clean, crisp 1 for any positive input, no matter how large. The signal flows backward through deep networks without decaying.

This one idea made 10-layer, 50-layer, and eventually 100+-layer networks trainable. Nair and Hinton (2010) introduced it for restricted Boltzmann machines, showing it preserved information about relative intensities better than binary units. But its real impact was in plain feedforward and convolutional networks. ReLU became the default, and deep learning took off.

But there’s a catch. A big one. What happens when a neuron receives a negative input? It outputs zero. Its gradient is zero. If that neuron always receives negative inputs — maybe its weights drifted during training — it never activates again. It’s a dead neuron. It contributes nothing, and it never recovers because its gradient is always zero.

Let’s see ReLU’s beautiful simplicity and then manufacture a dead neuron to understand the risk.

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# --- Plot ReLU ---
x = torch.linspace(-5, 5, 1000)
relu = nn.ReLU()
relu_y = relu(x)

plt.figure(figsize=(8, 4))
plt.plot(x.numpy(), relu_y.numpy(), linewidth=2, color='#2c7bb6')
plt.title('ReLU: max(0, x)')
plt.xlabel('Input (x)')
plt.ylabel('Output')
plt.axvline(0, color='gray', linestyle='--', alpha=0.5)
plt.axhline(0, color='gray', linestyle='--', alpha=0.5)
plt.grid(True, alpha=0.3)
plt.show()

# --- Demonstrate a dying neuron ---
# We'll create a small linear layer and deliberately push it into negative territory
fc = nn.Linear(3, 1)
# Set weights and bias so the output is always negative for our input
fc.weight.data = torch.tensor([[-1.0, -1.0, -1.0]])
fc.bias.data = torch.tensor([-2.0])

relu_activation = nn.ReLU()

# Our input vector
x_input = torch.tensor([1.0, 2.0, 3.0])

# Forward pass
raw_output = fc(x_input)
activated_output = relu_activation(raw_output)

print(f"Raw weighted sum: {raw_output.item():.4f}")
print(f"After ReLU: {activated_output.item():.4f}")
print()

# Now check the gradient
# We need a loss to backprop from
loss = activated_output.sum()  # dummy loss
loss.backward()

print(f"Gradient of weight: {fc.weight.grad}")
print(f"Gradient of bias: {fc.bias.grad}")
print()
print("The gradients are all zero. This neuron is dead. It will never update.")

See that? The raw weighted sum is negative. ReLU outputs zero. When we backpropagate, the gradients are all zero. This neuron is a zombie. It’s in the network, taking up space, but it will never learn again.

There are fixes — Leaky ReLU gives a tiny slope (like 0.01) for negative inputs, so the gradient never completely dies. Parametric ReLU learns that slope. But plain ReLU, despite this flaw, is still widely used and often works perfectly well. Why? Because with good weight initialization and reasonable learning rates, most neurons stay alive. Enough of them are in the “on” state that the network learns just fine. ReLU is like a light switch — on or off. When it’s on, the signal flows perfectly. When it’s off, nothing gets through. Most of the time, enough lights are on.

GELU: The Probabilistic Upgrade

ReLU is a hard gate. If x > 0, let it through completely. If x ≤ 0, block it entirely. There’s no nuance. What if, instead of a strict bouncer, you had one who let people in proportion to how likely they were to belong?

That’s GELU: the Gaussian Error Linear Unit. Introduced by Hendrycks and Gimpel in 2016, GELU weights the input by the probability that it’s positive — using the cumulative distribution function (CDF) of a standard Gaussian. In math: GELU(x) = x · Φ(x), where Φ(x) is the probability that a random value from a normal distribution is less than x.

For a large positive x, Φ(x) is nearly 1, so GELU(x) ≈ x — just like ReLU. For a large negative x, Φ(x) is nearly 0, so GELU(x) ≈ 0 — again like ReLU. But near zero, it’s beautifully curved. A slightly negative input like -0.5 doesn’t get brutally zeroed out. It gets multiplied by maybe 0.3, producing a small negative output. That small negative signal carries gradient information that ReLU throws away. The smoothness helps optimization, especially in very deep or attention-based architectures.

This is the hardest part to grasp intuitively, so let’s say it in plain English: ReLU is a bouncer with a strict “no entry if under 0” policy. GELU is a bouncer who looks at you, decides how likely you are to belong, and lets you in with that probability. If you’re very clearly supposed to be in (x >> 0), you walk right through. If you’re very clearly not (x << 0), you’re turned away. But if you’re borderline (x near 0), you get partial entry. The network gets to learn from the borderline cases instead of having them zeroed out.

In practice, computing the exact Gaussian CDF is expensive, so frameworks use a fast tanh approximation. This is what BERT, GPT, and Vision Transformers actually run:

GELU(x) ≈ 0.5 * x * (1 + tanh(√(2/π) * (x + 0.044715 * x³)))

Let’s see GELU and ReLU side by side, and confirm the approximation is nearly perfect.

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# --- Generate input range ---
x = torch.linspace(-4, 4, 1000)

# --- Compute activations ---
relu = nn.ReLU()
gelu_exact = nn.GELU(approximate='none')  # exact Gaussian CDF
gelu_tanh = nn.GELU(approximate='tanh')   # the fast approximation

relu_y = relu(x)
gelu_exact_y = gelu_exact(x)
gelu_tanh_y = gelu_tanh(x)

# --- Plot ---
plt.figure(figsize=(10, 5))
plt.plot(x.numpy(), relu_y.numpy(), label='ReLU', linewidth=2, linestyle='--', color='#d7191c')
plt.plot(x.numpy(), gelu_exact_y.numpy(), label='GELU (exact)', linewidth=2, color='#2c7bb6')
plt.plot(x.numpy(), gelu_tanh_y.numpy(), label='GELU (tanh approx)', linewidth=2, linestyle=':', color='#fdae61')
plt.title('ReLU vs. GELU: Hard Gate vs. Soft Gate')
plt.xlabel('Input (x)')
plt.ylabel('Output')
plt.axvline(0, color='gray', linestyle='--', alpha=0.3)
plt.axhline(0, color='gray', linestyle='--', alpha=0.3)
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

# --- Concrete borderline example ---
x_borderline = torch.tensor(-0.5)
relu_out = relu(x_borderline)
gelu_out = gelu_exact(x_borderline)

print(f"Input: {x_borderline.item()}")
print(f"ReLU output: {relu_out.item():.4f}   (completely silenced)")
print(f"GELU output: {gelu_out.item():.4f}   (small negative signal preserved)")
print()
print("That small negative output carries gradient information.")
print("ReLU throws it away; GELU lets it flow.")

Look at the curve near zero. ReLU has a sharp corner — a sudden, violent cutoff. GELU is smooth. That smoothness means the gradient is never exactly zero anywhere. There are no flat spots, no sharp corners. For optimization algorithms like Adam or SGD, smooth landscapes are easier to navigate. This is why GELU consistently outperforms ReLU in transformers — the architecture is so deep and the attention patterns so complex that the harsh zeroing of ReLU becomes a real bottleneck.

Head-to-Head: When to Use Which

We’ve met all four. Now the practical question: when should you actually use each one?

Let’s run a small, honest experiment. We’ll train identical networks on a synthetic binary classification task, varying only the hidden-layer activation. The goal isn’t to prove GELU is “best” — it’s to see the character of each function’s learning curve.

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
import numpy as np
import matplotlib.pyplot as plt

# --- Generate synthetic data ---
X_np, y_np = make_moons(n_samples=1000, noise=0.2, random_state=42)
X = torch.tensor(X_np, dtype=torch.float32)
y = torch.tensor(y_np, dtype=torch.float32).unsqueeze(1)

# Split into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X_np, y_np, test_size=0.2, random_state=42
)
X_train = torch.tensor(X_train, dtype=torch.float32)
X_test = torch.tensor(X_test, dtype=torch.float32)
y_train = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
y_test = torch.tensor(y_test, dtype=torch.float32).unsqueeze(1)

# --- Build a configurable 3-layer network ---
class SimpleNet(nn.Module):
    def __init__(self, activation):
        super().__init__()
        self.fc1 = nn.Linear(2, 16)
        self.fc2 = nn.Linear(16, 16)
        self.fc3 = nn.Linear(16, 1)
        self.activation = activation
        
    def forward(self, x):
        x = self.activation(self.fc1(x))
        x = self.activation(self.fc2(x))
        x = torch.sigmoid(self.fc3(x))  # sigmoid on output for binary classification
        return x

# --- Train with each activation ---
activations = {
    'Sigmoid': nn.Sigmoid(),
    'Tanh': nn.Tanh(),
    'ReLU': nn.ReLU(),
    'GELU': nn.GELU()
}

histories = {}
accuracies = {}

for name, act in activations.items():
    torch.manual_seed(42)  # same starting point for fair comparison
    model = SimpleNet(act)
    criterion = nn.BCELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    
    loss_history = []
    for epoch in range(500):
        # Forward
        y_pred = model(X_train)
        loss = criterion(y_pred, y_train)
        
        # Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        loss_history.append(loss.item())
    
    histories[name] = loss_history
    
    # Evaluate
    with torch.no_grad():
        y_test_pred = model(X_test)
        y_test_class = (y_test_pred > 0.5).float()
        acc = (y_test_class == y_test).float().mean().item()
        accuracies[name] = acc

# --- Plot training curves ---
plt.figure(figsize=(10, 6))
colors = {'Sigmoid': '#d7191c', 'Tanh': '#fdae61', 'ReLU': '#2c7bb6', 'GELU': '#1a9641'}
for name, history in histories.items():
    plt.plot(history, label=name, color=colors[name], linewidth=2, alpha=0.8)
plt.title('Training Loss by Activation Function')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

# --- Print final accuracies ---
print("Final Test Accuracies:")
for name, acc in accuracies.items():
    print(f"  {name}: {acc:.3f}")

Look at those curves. Sigmoid and tanh struggle — their loss drops slowly, and they often plateau at a higher value. ReLU drops fast and converges to a low loss. GELU drops similarly fast, sometimes slightly smoother. The final accuracies tell the same story: ReLU and GELU both hit high 90s, while sigmoid and tanh lag behind.

This isn’t a universal law — on a different task, the gaps might be smaller. But the pattern is real and well-documented. Here’s a practical decision framework:

• Sigmoid: Still the standard for the output layer of binary classification. It squashes the final logit into a [0,1] probability. In hidden layers? Almost never — the saturation kills deep networks.
• Tanh: Occasionally used in RNNs and LSTMs, where the zero-centered output helps with the gating mechanisms inside the cell. Rare in modern feedforward or transformer architectures.
• ReLU: The safe default for CNNs and most feedforward networks. Fast, simple, works. Watch for dead neurons if your learning rate is too high, but with good initialization (He initialization), you’ll rarely have a problem.
• GELU: The standard for transformers — BERT, GPT, ViT. If you’re building an attention-based model, just use GELU. The smoothness genuinely helps gradient flow through all those layers and attention heads.

Think of it this way: if you’re building a CNN, start with ReLU. If you’re building a transformer, use GELU. If you’re doing binary classification, sigmoid on the output layer. If you’re building an LSTM, tanh is still baked into the standard cell formulation. These aren’t arbitrary choices — they directly control gradient flow, and gradient flow is everything in deep learning.

Reference Table: Formulas, Derivatives, and Properties

Here’s the at-a-glance reference you can bookmark. Every formula is in consistent notation. Every property is explained in plain English alongside the math.

Function	Formula	Derivative	Output Range	Saturation Risk	Computational Cost	Best Used For
Sigmoid	σ(x) = 1 / (1 + e⁻ˣ)	σ(x) · (1 - σ(x))	(0, 1)	High — saturates for |x| > 3	Moderate (exponential)	Binary classification output layer
Tanh	tanh(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)	1 - tanh²(x)	(-1, 1)	High — saturates for |x| > 3	Moderate (exponential)	RNNs, LSTMs (zero-centered helps)
ReLU	max(0, x)	1 if x > 0, else 0	[0, ∞)	None for x > 0; dead for x ≤ 0	Very low (comparison only)	Default for CNNs and feedforward nets
GELU	x · Φ(x)	Φ(x) + x · φ(x) (see note)	(-0.17, ∞) approx	None — smooth everywhere	Moderate (tanh approximation)	Transformers (BERT, GPT, ViT)

Notes on the table:

• Φ(x) is the standard Gaussian CDF — the probability a random value from N(0,1) is less than x.
• φ(x) is the standard Gaussian PDF — the bell curve itself.
• In practice, GELU uses the fast tanh approximation: 0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715 · x³))).
• ReLU’s derivative at exactly x=0 is technically undefined; frameworks set it to 0 by convention.
• “Saturation risk” means the gradient becomes near-zero for large-magnitude inputs, freezing learning.

Recap and What’s Next

We’ve traced a 30-year journey through the evolution of activation functions. Each step solved a real, painful training problem:

• Sigmoid and tanh gave us smooth, differentiable nonlinearities — but their saturation caused vanishing gradients that made deep networks nearly impossible to train.
• ReLU threw out smoothness for raw speed and non-saturating gradients. That one change — max(0, x) — unlocked 100-layer networks. The cost was dead neurons, but good initialization mostly solved that.
• GELU brought smoothness back, but in a smarter way — probabilistic gating instead of hard thresholding. It’s the modern standard for transformers, combining ReLU’s speed with the gradient-friendly curves of the classical functions.

The big lesson: activation functions aren’t arbitrary choices. They directly control gradient flow, and gradient flow is everything in deep learning. A dead gradient means a dead neuron. A vanishing gradient means a frozen layer. Choose your activation like you’d choose the fuel for an engine — the wrong one will leave you stranded.

In the next article, we’ll explore activation functions designed specifically for the output layer — Softmax, and how it turns raw logits into a probability distribution over multiple classes. That’s the final piece of the puzzle: once your hidden layers have done their work, how do you interpret the network’s answer?

Check Your Understanding

Remember: What is the output range of the tanh activation function, and why is being zero-centered helpful for training?

Understand: Explain the vanishing gradient problem in your own words. Why does sigmoid suffer from it more than ReLU?

Apply: You’re building a 3-layer CNN for image classification. Which hidden-layer activation do you reach for first, and why? What would you use on the output layer?

Analyze: A colleague tells you they replaced ReLU with sigmoid in their 20-layer network and training completely stalled. What’s the most likely explanation? Point to the specific mechanism.

Evaluate: Compare the dead-neuron problem in ReLU with the saturation problem in sigmoid. Which is easier to diagnose and fix? Which is more damaging in practice?

Create: Design a tiny experiment (in pseudocode or real code) to test whether GELU or ReLU trains faster on a spiral dataset with 5 hidden layers. What would you measure? What result would make you choose one over the other?

Apply What You Learned

Take the 3-layer network from the Head-to-Head section. Modify it to have 5 hidden layers instead of 2. Train it with ReLU and GELU on the same moons dataset. Measure: (1) final test accuracy, (2) number of epochs to reach 90% accuracy, (3) the percentage of dead neurons (neurons whose output is exactly 0 for the entire test set). Write a short paragraph interpreting your results — which activation would you choose for a deep version of this task and why?

Self-check rubric: Did you successfully train both networks? Did you correctly identify dead neurons by checking for all-zero outputs? Does your interpretation reference gradient flow and the smoothness near zero?