How Backpropagation Trains Neural Networks, Step by Step

Backpropagation is the algorithm that makes deep neural networks trainable. It answers one question for every weight in a network: if this weight changed slightly, how would the overall error change? With that answer, training can nudge each weight in the direction that reduces error. This article walks through the idea using a single neuron and real arithmetic, then shows how the same logic extends to many layers.

The forward pass

Training begins with a forward pass. An input flows through the network, each layer multiplies its inputs by weights, adds a bias, and applies an activation function. The final output is compared with the target using a loss function. Every intermediate value is stored, because the backward pass will need them.

For a single neuron with input x, weight w, and bias b, the steps are z = w * x + b, then a = sigma(z), where sigma is the sigmoid function. The loss for one example with target t might be L = (a - t) squared.

The chain rule carries the blame

The chain rule from calculus says that if a quantity depends on another through a sequence of steps, its rate of change is the product of the rates along each step. Backpropagation applies this rule backward from the loss to every weight.

For our neuron, the path from weight to loss has three links: weight affects z, z affects a, and a affects L. So the derivative of the loss with respect to the weight is dL/da times da/dz times dz/dw. Each factor is simple on its own. Backpropagation is the bookkeeping that multiplies them efficiently across an entire network.

A worked example with real numbers

Set x = 1, w = 0.5, b = 0, target t = 1, and learning rate 0.5. The steps below use the sigmoid function, whose derivative is sigma(z) times (1 minus sigma(z)).

Step	Quantity	Value
1	z = 0.5 * 1 + 0	0.5
2	a = sigmoid(0.5)	0.62246
3	Loss = (0.62246 - 1) squared	0.14253
4	dL/da = 2 * (a - t)	-0.75508
5	da/dz = a * (1 - a)	0.23500
6	dz/dw = x	1.0
7	dL/dw = -0.75508 * 0.23500 * 1.0	-0.17745
8	New w = 0.5 - 0.5 * (-0.17745)	0.58873

After one update, the weight rose from 0.5 to about 0.58873. Running the forward pass again gives a = sigmoid(0.58873), about 0.64314, and a loss of about 0.12735. The loss dropped from 0.14253 to 0.12735 after a single step. Repeating this over many examples and many steps is the whole of training.

The negative gradient in step 7 is the signal. It says that increasing the weight would decrease the loss, so the update pushes the weight upward.

From one neuron to many layers

A real network has millions of weights arranged in layers. The same chain rule applies, but now the derivative for an early-layer weight passes backward through every later layer. The backward pass computes the gradient for the last layer first, then reuses those intermediate results to compute the gradient for the layer before it, and so on toward the input.

This reuse is why backpropagation is efficient. Computing each weight's gradient independently would repeat a huge amount of work. By storing the values from the forward pass and passing error signals backward layer by layer, the cost of computing all gradients is only a small constant multiple of the forward pass cost, commonly estimated at about two times.

Batches and averaging

In practice the loss is computed over a mini-batch of examples. The gradients from each example are averaged, and the averaged gradient drives one update. Averaging smooths out the noise from any single example while keeping the computation fast on parallel hardware such as GPUs.

The role of the activation derivative

Notice that step 5 depends on the activation function. The sigmoid derivative is at most 0.25, so each layer that uses sigmoid can shrink the gradient passing through it. Across many layers those small factors multiply, and gradients can become tiny. This problem is called vanishing gradients, and it is one reason modern networks rarely use sigmoid in hidden layers.

The choice of activation function therefore affects not only what the network can represent but how well backpropagation can train it. The next article in this section examines common activation functions and their gradient behavior.

Automatic differentiation in practice

Nobody writes these derivatives by hand for modern models. Frameworks such as PyTorch and JAX record the operations performed during the forward pass in a graph, then apply the chain rule automatically during the backward pass. This technique is called automatic differentiation. It computes exact derivatives of the code as written, rather than approximations from finite differences.

Understanding the manual version still matters. When training diverges, produces NaN values, or stalls, the cause usually traces back to a gradient that is too large, too small, or wrongly scaled. Knowing what the chain rule is doing makes those failures diagnosable.

Historical note

The idea of applying the chain rule to train layered networks was explored in several fields before it became standard. It was popularized for neural networks in a 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams. Their demonstration showed that hidden layers could learn useful internal representations, which helped revive interest in multi-layer networks.

Memory cost of the backward pass

The forward pass stores intermediate values so the backward pass can use them. For a deep network with large batches, those stored activations can use more memory than the weights themselves. Gradient checkpointing reduces this cost by storing only some layers' activations and recomputing the rest during the backward pass. It trades extra computation for lower memory, which is often the right trade when the model would otherwise not fit on the hardware.

Mixed-precision training also cuts memory by keeping many stored values in 16-bit floating point instead of 32-bit. These choices are engineering decisions about how the same gradients are computed and stored. They do not change the underlying mathematics of the chain rule.

Key takeaways

Backpropagation computes how the loss changes with respect to every weight by applying the chain rule backward through the network. Each weight gets a gradient that says which direction reduces error.

The worked example shows the full cycle: forward pass, loss, gradient, update. One step lowered the loss, and training is many repetitions of the same loop.

Backpropagation reuses intermediate results, which makes gradients for huge networks affordable. Frameworks now automate the process, but the underlying chain rule still explains both successful training and failures.

Activation functions shape the gradients that flow backward. Their derivatives determine whether signals survive across many layers, which is the topic of the next article.