I’ve been thinking a lot about ResNet architectures lately, especially how they transformed deep learning by solving the vanishing gradient problem. It’s fascinating how such a simple idea—adding skip connections—could enable training of networks hundreds of layers deep. Let me share what I’ve learned about building and training these powerful models in PyTorch.

Have you ever wondered why very deep networks were so difficult to train before ResNets? The answer lies in how gradients propagate through layers. As networks get deeper, gradients can become extremely small during backpropagation, making weight updates almost negligible. This vanishing gradient problem limited how deep we could effectively train neural networks.

ResNets introduced an elegant solution: residual connections. These connections allow the network to learn identity functions, essentially letting information skip layers when needed. This simple addition made it possible to train networks with hundreds of layers while maintaining stable gradients.

Let me show you how a basic residual block works in code:

class BasicBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        
        self.downsample = None
        if stride != 1 or in_channels != out_channels:
            self.downsample = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels)
            )

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        
        if self.downsample is not None:
            identity = self.downsample(x)
            
        out += identity
        return self.relu(out)

Notice how the identity connection preserves the original input and adds it to the transformed output? This small change makes all the difference in training deep networks effectively.

What happens when we need even deeper networks? That’s where bottleneck blocks come in. They use 1x1 convolutions to reduce computational complexity while maintaining representational power:

class BottleneckBlock(nn.Module):
    expansion = 4
    
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        width = out_channels
        self.conv1 = nn.Conv2d(in_channels, width, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(width)
        self.conv2 = nn.Conv2d(width, width, 3, stride, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(width)
        self.conv3 = nn.Conv2d(width, width * self.expansion, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(width * self.expansion)
        self.relu = nn.ReLU(inplace=True)
        
        self.downsample = None
        if stride != 1 or in_channels != width * self.expansion:
            self.downsample = nn.Sequential(
                nn.Conv2d(in_channels, width * self.expansion, 1, stride, bias=False),
                nn.BatchNorm2d(width * self.expansion)
            )

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        
        if self.downsample is not None:
            identity = self.downsample(x)
            
        out += identity
        return self.relu(out)

When building custom ResNet architectures, I often start with a flexible base class that can accommodate different block types and configurations. This approach lets me experiment with various depths and widths without rewriting the entire architecture each time.

Training these models requires some special considerations. I’ve found that proper weight initialization is crucial, especially for the final layers in each residual block. Using He initialization and sometimes zero-initializing the last batch normalization layer in each block can help the network start training more effectively.

Did you know that the learning rate schedule can significantly impact ResNet training? I typically use a cosine annealing schedule with warm restarts, which helps the model escape local minima and continue improving throughout training.

Here’s a practical training snippet I often use:

def train_resnet(model, train_loader, val_loader, epochs=100):
    optimizer = torch.optim.SGD(model.parameters(), lr=0.1, 
                               momentum=0.9, weight_decay=1e-4)
    scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
        optimizer, T_0=10, T_mult=2
    )
    criterion = nn.CrossEntropyLoss()
    
    for epoch in range(epochs):
        model.train()
        for inputs, targets in train_loader:
            optimizer.zero_grad()
            outputs = model(inputs)
            loss = criterion(outputs, targets)
            loss.backward()
            optimizer.step()
        
        scheduler.step()
        
        # Validation phase
        model.eval()
        with torch.no_grad():
            # Calculate validation metrics
            pass

One question I often get is: how deep should my custom ResNet be? The answer depends on your specific problem and dataset. For most tasks, ResNet-50 provides an excellent balance between performance and computational requirements. However, for simpler problems, ResNet-18 might be sufficient, while extremely complex tasks might benefit from ResNet-152 or even deeper custom architectures.

Remember that deeper isn’t always better. The key is finding the right architecture for your specific use case through careful experimentation and validation.

I’d love to hear about your experiences with custom ResNet architectures! What challenges have you faced when building deep networks? Share your thoughts in the comments below, and don’t forget to like and share this article if you found it helpful.