Overview
Automatic differentiation makes neural-network training convenient, but it does not eliminate the need to understand backpropagation. This tutorial removes PyTorch's automatic backward pass from a two-layer multilayer perceptron containing embeddings, linear layers, tanh, batch normalization, and cross-entropy loss. It first decomposes the forward computation into atomic tensor operations and manually propagates gradients through indexing, reshaping, matrix multiplication, broadcasting, reductions, exponentiation, logarithms, and nonlinearities. PyTorch gradients serve as a reference, with exact or approximate comparisons used to validate every derivation. The tutorial then shows why mechanically differentiating every primitive is educational but inefficient: analytically differentiating fused cross-entropy and batch normalization produces much shorter backward expressions because many intermediate terms cancel. Several general rules emerge, including gradient accumulation at reused variables, the duality between broadcasting and summation, and the use of tensor shapes to reconstruct matrix-gradient formulas. The completed manual backward pass replaces autograd while achieving essentially the same training loss and samples. The practical conclusion is not that production systems should abandon autograd, but that developers who understand its mechanics can diagnose saturation, dead neurons, exploding or vanishing gradients, incorrect clipping, and shape-related bugs with far greater confidence.
Sections
Higher-Level Insights
Broader implications derived from the manual backward-pass exercise.
- The hardest parts of tensor backpropagation are often dependency accounting and shape semantics rather than differentiation itself. Local derivatives are usually elementary, while reused variables, broadcasting, reductions, and transposes create most implementation risk.
- Fused operations are valuable for more than runtime optimization: their analytical gradients expose the essential learning behavior that is obscured by a long chain of atomic operations.
- Autograd proficiency and manual differentiation are complementary. Autograd supplies scalable execution, while manual derivation supplies the conceptual model needed to validate objectives and interpret failures.
- Tiny random parameter initialization is useful for gradient checking because exact zeros can simplify expressions and conceal an incorrect derivative.
Technical Details
Specific derivative rules, tensor behaviors, and validation techniques used in the implementation.
- For mean negative log-likelihood over a batch of size n, initialize the log-probability gradient to zeros and place -1/n only at each example's target-class position.
- The local derivative of log(x) is 1/x; the derivative of exp(x) is exp(x); and the derivative of tanh expressed through its output h is 1-h².
- For D = A @ B + C, the gradients are dA = dD @ Bᵀ, dB = Aᵀ @ dD, and dC equals dD summed over the dimensions where C was broadcast.
- The compact batch cross-entropy gradient is softmax(logits), minus one at each target-class index, then divided by the batch size.
- The embedding-table gradient is a scatter-add: route each selected embedding's gradient back to its source row and add contributions when an index appears multiple times.
- Numerical-stability subtraction of the row maximum should theoretically have no effect on softmax or loss; its gradient is therefore expected to be zero, apart from very small floating-point residuals.
Recommended Practice
Concrete ways to apply the tutorial's method.
- Attempt each derivative independently in the provided notebook before consulting the demonstrated solution.
- Check every intermediate manual gradient against autograd using exact equality, approximate equality, and maximum absolute difference.
- Inspect tensor shapes before deriving an operation's backward pass, especially when broadcasting, reductions, or matrix multiplication are involved.
- Use small nonzero random values during gradient-checking exercises so degenerate zero-valued expressions do not mask mistakes.
- After validating the educational implementation, retain framework autograd for normal production training unless a custom fused operation genuinely requires a handwritten derivative.
Risks and Pitfalls
Common failure modes exposed by tensor-level differentiation.
- Clipping the loss instead of clipping gradients can zero the gradients of outliers and cause difficult examples to be ignored.
- Overlooking a second computational branch yields an incomplete gradient for a reused tensor.
- Ignoring broadcasting can produce gradients with the wrong shape or omit the reduction required across reused dimensions.
- Exact equality checks alone can reject correct implementations because mathematically equivalent floating-point calculations may differ slightly.
- Using different variance estimators between training and inference may introduce a batch-normalization mismatch, particularly for small minibatches.
- Handwritten backward passes are tedious and easy to misimplement despite their educational value.