Activation Functions: Why Neural Networks Need Nonlinearity

An activation function is the small nonlinear step applied to each neuron's weighted sum. Without it, a neural network would be a stack of linear operations, and stacking linear operations never produces anything more powerful than a single linear operation. This article explains why nonlinearity is essential, compares the common activation functions, and shows how their shapes affect training.

Why linearity is not enough

Suppose a layer computes y = W x + b and the next layer computes z = V y + c. Substituting the first into the second gives z = (V W) x + (V b + c). The result is still a single linear map of x. Adding one hundred such layers would still collapse to one matrix multiplication plus a bias.

Linear models can only draw straight decision boundaries and fit straight trends. Real patterns, such as the boundary between cat and dog images or the relationship between a word and its meaning in context, are highly curved. Nonlinear activations let each layer bend the space, so that stacked layers can represent complicated functions.

The main candidates

Several activation functions have dominated practice at different times. Each has a different range and a different derivative, which determines how well gradients flow during backpropagation.

Function	Formula	Output range	Maximum derivative	Typical use
Sigmoid	1 / (1 + e^-x)	(0, 1)	0.25	Output probabilities, gates
Tanh	(e^x - e^-x) / (e^x + e^-x)	(-1, 1)	1.0	Older recurrent networks
ReLU	max(0, x)	[0, infinity)	1.0	Most convolutional and feed-forward networks
GELU	x times the Gaussian CDF of x	About (-0.17, infinity)	About 1.13	Transformer models

The maximum derivative column matters most for training. A derivative below 1 shrinks gradients at each layer, while a derivative near 1 preserves them.

Sigmoid: squashing into probabilities

The sigmoid maps any real number into the interval between 0 and 1, which makes it natural for expressing probabilities. Its value at zero is exactly 0.5, and it saturates at both extremes. For large positive or negative inputs, the curve becomes nearly flat, so the derivative approaches zero.

That saturation is the core problem. When a sigmoid unit receives a very large input, almost no gradient passes back through it, so the weights feeding it barely change. Stacking many sigmoid hidden layers multiplies together factors that are each at most 0.25, so the gradient reaching early layers can become negligibly small. This is the vanishing gradient problem.

Tanh: centered but still saturating

Tanh is a rescaled sigmoid that outputs values between negative one and positive one. Because its output is centered around zero, it avoids some of the drift that affects sigmoid-based layers. Its maximum derivative is 1 at zero, which is better than sigmoid, but it still saturates for large inputs and suffers from vanishing gradients in deep stacks.

ReLU: the simple workhorse

The rectified linear unit outputs zero for negative inputs and passes positive inputs through unchanged. It is cheap to compute and its derivative is either 0 or 1. For positive inputs the gradient passes through without shrinking, which is why deep convolutional networks trained far more easily once ReLU became common.

ReLU has a known failure mode called dying units. If a neuron's weights push its input permanently negative, its output and gradient stay at zero, and the neuron stops learning. Variants such as Leaky ReLU give negative inputs a small slope, such as 0.01, so the gradient never disappears completely.

GELU: smooth gating in transformers

The Gaussian error linear unit weights its input by the probability that a standard normal variable is less than that input. Unlike ReLU, it has a smooth curve with a small negative dip, and it does not switch abruptly at zero. Transformer models adopted GELU widely because the smooth shape seemed to train well at large scale. Exact and approximate forms are both used in practice.

Choosing an activation in practice

For hidden layers in most modern feed-forward and convolutional networks, ReLU or a close variant is the default. For transformer blocks, GELU or related smooth functions are common. For output layers, the choice depends on the task. Sigmoid produces independent probabilities for multi-label problems, and softmax produces a probability distribution across mutually exclusive classes.

Softmax is not used in hidden layers in typical designs. It takes a vector of scores and exponentiates and normalizes them so that the outputs sum to one. Its derivative couples all outputs together, which is why it pairs naturally with cross-entropy loss.

Initialization and activations work together

Activation choice interacts with weight initialization. If weights start too large, sigmoid and tanh units saturate immediately. If they start too small, signals shrink layer by layer. Initialization schemes such as Xavier and He were designed to keep the variance of activations roughly stable across layers, and they are chosen to match the activation function. Using He initialization with sigmoid, or Xavier with ReLU, undermines that balance.

Normalization layers, which rescale activations to a stable range, also reduce sensitivity to these choices. They are not a replacement for picking a sensible activation, but they make networks more forgiving.

Gradient flow through a deep stack

Consider a 20-layer network where each layer multiplies the gradient by 0.25, the largest derivative a sigmoid can produce. After 20 layers the surviving gradient is scaled by 0.25 raised to the 20th power, which is about 9.1 times 10 to the minus 13. A layer with derivative 1.0 keeps the same scale, and a layer with derivative 1.1 grows it. Real networks are not that uniform, but the arithmetic shows why gradient scale matters. Residual connections address the problem differently: the skip path carries gradient across layers unchanged, so early layers still receive a signal even when the transformed path saturates.

Monitoring activations during training

Activation statistics are a useful diagnostic. Recording the fraction of ReLU outputs that are exactly zero shows how many units are inactive on a given batch. A layer where nearly every unit is zero is a warning sign of dying units or a learning rate that is too high. For sigmoid or tanh layers, a histogram of pre-activation values that piles up at the saturated ends shows that gradients there are close to zero. Checking these numbers early costs little and often explains a stalled run faster than changing the optimizer.

Key takeaways

Activation functions supply the nonlinearity that lets stacked layers represent complex patterns. Without them, depth adds no expressive power.

Sigmoid and tanh saturate for large inputs, which shrinks gradients and slows learning in deep networks. ReLU avoids most of that problem for positive inputs, at the cost of possibly dead units.

GELU and related smooth functions are common in transformers. Match the activation to the layer role and to the initialization scheme, and check for dead or saturated units when training stalls.