ResNet

ResNet (Residual Network) makes very deep networks trainable by having each block learn a residual function added to its input, so the layers only need to model the difference from identity rather than a full transformation.

The Degradation Problem

Naively stacking more layers does not always help: past a point, deeper plain networks show higher training error, not just higher test error. This is degradation, and it is not overfitting (training error itself rises). The issue is optimization: solvers struggle to make many stacked nonlinear layers approximate an identity mapping when that is what is needed.

Key insight

If a shallower network already works, a deeper one should do at least as well by setting the extra layers to identity. Residual blocks make identity the easy default, so adding depth cannot hurt in principle.

Residual Learning

Instead of asking a block to learn a target mapping $H(x)$ directly, we let it learn the residual $F(x)=H(x)-x$ and reconstruct:

$y=F(x,\{W_i\})+x$

The $+x$ term is an identity shortcut (skip connection). Driving $F(x)\to 0$ is easy (push weights toward zero), which recovers the identity mapping and directly addresses degradation.

Why Residuals Help Gradient Flow

During backprop the gradient of the loss $L$ with respect to a block input is:

$\frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(1+\frac{\partial F}{\partial x}\right)$

The constant $1$ provides a direct highway for the gradient that bypasses $\partial F/\partial x$, so signals do not vanish through many layers. See Backpropagation for the chain-rule mechanics and Residual Connections for the general pattern.

Basic vs Bottleneck Blocks

• BasicBlock: two $3\times 3$ convolutions, used in ResNet-18/34.
• Bottleneck: $1\times 1$ (reduce) then $3\times 3$ then $1\times 1$ (expand), used in ResNet-50/101/152. The $1\times 1$ layers cut and restore channels so the expensive $3\times 3$ runs on fewer channels.

When spatial size or channel count changes, the shortcut uses a $1\times 1$ conv downsample so the added tensors align. BatchNorm is applied after each conv and before the ReLU.

Code

import torch.nn as nn
 
def conv3x3(cin, cout, stride=1):
    return nn.Conv2d(cin, cout, 3, stride=stride, padding=1, bias=False)
 
class BasicBlock(nn.Module):
    expansion = 1
    def __init__(self, cin, cout, stride=1, downsample=None):
        super().__init__()
        self.conv1 = conv3x3(cin, cout, stride)
        self.bn1 = nn.BatchNorm2d(cout)
        self.conv2 = conv3x3(cout, cout)
        self.bn2 = nn.BatchNorm2d(cout)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample
 
    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        if self.downsample is not None:
            identity = self.downsample(x)
        return self.relu(out + identity)
 
class Bottleneck(nn.Module):
    expansion = 4
    def __init__(self, cin, cmid, stride=1, downsample=None):
        super().__init__()
        cout = cmid * self.expansion
        self.conv1 = nn.Conv2d(cin, cmid, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(cmid)
        self.conv2 = nn.Conv2d(cmid, cmid, 3, stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(cmid)
        self.conv3 = nn.Conv2d(cmid, cout, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(cout)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample
 
    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        if self.downsample is not None:
            identity = self.downsample(x)
        return self.relu(out + identity)

When is downsample needed?

Pass a downsample = nn.Sequential(nn.Conv2d(cin, cout, 1, stride=stride, bias=False), nn.BatchNorm2d(cout)) whenever stride != 1 or cin != cout, so the identity branch matches the residual branch shape.

Related

ResNet is the archetypal deep CNN. It popularized Residual Connections now ubiquitous in Transformers, relies on BatchNorm for stable training, and its skip paths chiefly matter because of how they shape gradients during Backpropagation.