Backpropagation

Method to train neural nets by adjusting its weights. Backpropagation is reverse-mode automatic differentiation applied to the computation graph of a network: it computes the gradient of the loss with respect to every parameter in a single backward sweep.

Steps

1. Forward Pass: Compute activations $a^l$ for all layers
2. Loss Calculation: Compute the Loss $L(\hat{y},y)$
3. Backward Pass: Use the Chain rule to compute ${\delta }^l$, the error term for each layer starting from the output and propagating backwards
4. Update weights and biases using the computed gradients by Gradient Descent.

$z^l=W^l{a}^{l-1}+b^l,a^l=f(z^l)$

$\frac{\partial L}{\partial w_{ij}}={\sum }_k\frac{\partial L}{\partial z_k}\frac{\partial z_k}{\partial w_{ij}}$

Reverse-Mode Autodiff

A network is a directed acyclic graph of primitive operations. Forward mode propagates derivatives input to output; reverse mode propagates them output to input. Because a loss has one scalar output and many parameters, reverse mode computes all parameter gradients at the cost of roughly one extra forward pass, which is why it is the method of choice. This is exactly what Autograd engines automate.

Chain Rule on the Graph

Define the error at layer $l$ as ${\delta }^l=\partial L/\partial z^l$. The recurrence that flows backward is:

$${\delta }^L={\nabla }_{a}L\odot f^{\prime }(z^L),{\delta }^l=((W^{l+1}{)}^{\top }{\delta }^{l+1})\odot f^{\prime }(z^l).$$

From the error terms the parameter gradients are:

$$\frac{\partial L}{\partial W^l}={\delta }^l(a^{l-1}{)}^{\top },\frac{\partial L}{\partial b^l}={\delta }^l.$$

Worked 2-Layer MLP

For $z^1=W^{1}x+b^1$, $a^1=\sigma (z^1)$, $z^2=W^2{a}^1+b^2$, $\hat{y}=\text{softmax}(z^2)$, with cross-entropy loss:

$${\delta }^2=\hat{y}-y,\frac{\partial L}{\partial W^2}={\delta }^2(a^1{)}^{\top },{\delta }^1=((W^2{)}^{\top }{\delta }^2)\odot {\sigma }^{\prime }(z^1),\frac{\partial L}{\partial W^1}={\delta }^1{x}^{\top }.$$

From-Scratch numpy (vectorized, batch N)

import numpy as np
 
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1
    A1 = 1 / (1 + np.exp(-Z1))              # sigmoid
    Z2 = A1 @ W2 + b2
    A2 = np.exp(Z2 - Z2.max(1, keepdims=True))
    A2 /= A2.sum(1, keepdims=True)          # softmax
    return Z1, A1, Z2, A2
 
def backward(X, Y, W2, Z1, A1, A2):
    N = X.shape[0]
    dZ2 = (A2 - Y) / N                      # softmax + cross-entropy
    dW2 = A1.T @ dZ2
    db2 = dZ2.sum(0)
    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * A1 * (1 - A1)               # sigmoid'
    dW1 = X.T @ dZ1
    db1 = dZ1.sum(0)
    return dW1, db1, dW2, db2

Torch Autograd Check

import torch
 
X = torch.randn(8, 4)
Y = torch.eye(3)[torch.randint(0, 3, (8,))]
W1 = torch.randn(4, 5, requires_grad=True)
W2 = torch.randn(5, 3, requires_grad=True)
 
A1 = torch.sigmoid(torch.einsum('nd,dh->nh', X, W1))
logits = torch.einsum('nh,hc->nc', A1, W2)
loss = -(Y * logits.log_softmax(1)).sum(1).mean()
loss.backward()                             # dW1, dW2 match the manual grads
print(W1.grad.shape, W2.grad.shape)

Pitfalls

• Vanishing/exploding gradients: repeated multiplication by $f^{\prime }$ and weights can shrink or blow up ${\delta }^l$ in deep nets; use ReLU, residual connections, or normalization.
• Cache the forward activations; recomputing them in the backward pass wastes work (or use checkpointing to trade compute for memory).
• A subtle sign or transpose error is best caught with numerical gradient checking or a torch autograd comparison as above.

Related

• Autograd generalizes this to arbitrary graphs.
• Gradient Descent consumes the gradients to update parameters.