Derivatives, Gradients & the Chain Rule
How backprop is just the chain rule, applied carefully.
The situation
Your model has billions of weights, and nudging each one to measure its effect would take billions of passes for a single step. The interviewer asks:
How does backpropagation use the chain rule?
Running example: a one-neuron model predicting whether a delivery will be late. Input x (distance, scaled), weight w, bias b; the neuron computes z = wx + b, then the probability a = σ(z) = 1/(1 + e−z), and the loss is the squared error L = (a − y)² against the true label y. Training needs ∂L/∂w and ∂L/∂b.
Which way should each weight move?
A model has thousands to billions of weights and one number, the loss, that says how wrong it is. To improve, each weight must move in the direction that reduces the loss, by an amount that matches how sensitive the loss is to it. Derivatives give exactly that sensitivity, and the chain rule makes it computable for deep, nested models.
The idea
Think of a row of gears. Turn the first one, and the last one moves by the product of every gear ratio in between. The chain rule is that product. Backprop walks the gears backwards from the loss to find how much each weight matters.
Play
Try to break it. Here z is the neuron's score before the sigmoid. Raise w and x until z is above about 5, for example w = 3 and x = 2.
Chain rule vs finite differences
Push |z| above about 5 (e.g. w = 3, x = 2): the sigmoid saturates, ∂a/∂z collapses towards 0 and the gradient vanishes even though the prediction may be badly wrong. That's the vanishing-gradient problem in miniature.
Now say why: why does the gradient vanish even when the prediction is badly wrong?
Maths
In one breath
A derivative answers "if I nudge this input a little, how much does the output change?". With many inputs (every weight in a model), each has a partial derivative, and together they form the gradient: an arrow pointing in the direction where the loss rises fastest. Training walks the other way. Models are long chains of simple steps (multiply, add, squash, compare to the target), and the chain rule says the sensitivity through a chain is the product of the local sensitivities. Backpropagation computes all of them in one backward pass by reusing shared products, so a gradient costs about as much as one forward pass.What you'll be able to do after this: differentiate a loss with respect to a weight by hand, explain what the gradient vector means, apply the chain rule on a computation graph, and check a gradient numerically.Derivative, partial derivative, gradient
MML §5.1–5.2. Useful rules: d(x²)/dx = 2x; dσ(z)/dz = σ(z)(1 − σ(z)); d(log x)/dx = 1/x.
The chain rule
The multivariate chain rule, MML §5.2.2. Backpropagation applies it systematically from the loss backwards (Rumelhart, Hinton & Williams, Nature 1986).
Forward: compute values. Backward: multiply local slopes.
- The neuron as a chain of simple steps: a computation graph.
- Forward pass: plug in numbers. With w = 1, x = 1, b = −0.2 and a true label y = 0 (on time), the model predicts 0.69 probability of late and the loss is 0.48.
- Backward pass: at each step compute the local slope. ∂L/∂a = 2(a − y) = 1.38; ∂a/∂z = a(1 − a) = 0.21; ∂z/∂w = x = 1 and ∂z/∂b = 1.
- Multiply along the path: ∂L/∂w ≈ 0.30. Positive, so increasing w would increase the loss: training will decrease w. In a big network, the shared products (here ∂L/∂z) are computed once and reused for every weight feeding into z.
In practice
Gradients in practice
- Autodiff: PyTorch, JAX and TensorFlow record the computation graph in the forward pass and apply the chain rule backwards automatically (reverse-mode automatic differentiation).
- Cost: reverse mode computes the gradient of one scalar loss with respect to all parameters for a small constant multiple of the forward cost (Goodfellow et al. §6.5), which is why it beats finite differences by a factor of n.
- Loss and activation choices: cross-entropy with a sigmoid gives ∂L/∂z = a − y, cancelling the saturating σ′ term; squared error with a sigmoid doesn't. That's one reason classifiers use cross-entropy (see Entropy & KL).
- Gradient checking: compare backprop to finite differences on a tiny model when you write custom layers.
The answer that sounds right
"Estimate each partial derivative by nudging each weight"
Finite differences, (L(w + h) − L(w))/h, work for one weight and are a great way to check a gradient. But a model with n weights needs n + 1 forward passes per step: for a 7-billion-parameter model, 7 billion passes to take one step. They're also numerically fragile: too large an h is inaccurate, too small loses precision to rounding. Backpropagation gets all n partial derivatives exactly for the price of roughly one forward and one backward pass.
Say it
How does backpropagation use the chain rule?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
Derive ∂L/∂z for sigmoid + cross-entropy.
Answer it out loud, then check.
L = −[y log a + (1 − y) log(1 − a)], a = σ(z): ∂L/∂a = (a − y)/(a(1 − a)), ∂a/∂z = a(1 − a), so ∂L/∂z = a − y.
Follow-up 2 of 3
Why reverse mode, not forward mode?
Answer it out loud, then check.
Reverse mode costs about one backward pass per scalar output, whatever the number of inputs: ideal for one loss and millions of parameters. Forward mode costs one pass per input.
Follow-up 3 of 3
What causes vanishing and exploding gradients?
Answer it out loud, then check.
Multiplying many local derivatives: values below 1 shrink exponentially (saturated sigmoids/tanh), above 1 grow. Fixes: ReLU-type activations, careful initialisation, normalisation, residual connections, gradient clipping.
The 30-second answer
A derivative measures how much an output changes for a small change in an input; with many parameters, the gradient is the vector of partial derivatives and points in the direction of steepest increase of the loss, so we step against it. Models are compositions of simple functions, and the chain rule says the derivative through a composition is the product of local derivatives. Backpropagation applies the chain rule from the loss backwards over the computation graph, reusing intermediate products, so all parameter gradients cost about one extra pass. Saturating activations make local derivatives tiny, which causes vanishing gradients.
Go further
What not to say
- "Backprop is an optimisation algorithm." It computes gradients; the optimiser (SGD, Adam) uses them.
- "The gradient points to the minimum." It points to the steepest local increase.
Key terms and sources
Derivatives as sensitivity, partial derivatives, the gradient vector, the chain rule, computational graphs, and backpropagation on a single neuron · built from Deisenroth et al., Mathematics for Machine Learning (ch. 5), Goodfellow et al., Deep Learning (ch. 6.5) and Rumelhart, Hinton & Williams (1986) · checked 28 Oct 2026
Key terms
| Term | Meaning |
|---|---|
| Derivative | Local rate of change of a function. |
| Partial derivative | Rate of change in one variable, others fixed. |
| Gradient | Vector of partials; points uphill. |
| Chain rule | Derivative of a composition = product of local derivatives. |
| Backpropagation | Reverse-mode chain rule over a computation graph. |
| Vanishing gradient | Products of small local slopes shrink the gradient towards 0. |
Sources
- M. P. Deisenroth, A. A. Faisal & C. S. Ong, Mathematics for Machine Learning, CUP 2020, mml-book.github.io (ch. 5).
- I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, MIT Press 2016, §6.5, deeplearningbook.org.
- D. Rumelhart, G. Hinton & R. Williams, "Learning representations by back-propagating errors", Nature 323, 1986.