Skip to content

Convolutional Neural Networks

What This Is

Convolutional neural networks detect spatial patterns — edges, textures, shapes — by sliding small learned filters across the input. Unlike fully connected layers, convolutions share weights across positions, which makes them efficient and translation-equivariant.

The core insight is that local patterns matter more than global position. A vertical edge is a vertical edge whether it appears on the left or the right of the image. That single insight drives every architecture choice on this page: stacked small filters, hierarchical pooling, residual connections, depthwise separables.

When You Use It

  • classifying or analyzing images
  • extracting features from grid-structured data (spectrograms, dense 2D maps)
  • building a backbone for detection, segmentation, or generation
  • when the input has spatial structure that a fully connected layer would ignore
  • when you need a model that can run under 10 ms on a CPU / mobile device — see the depthwise-separable section below

Do Not Use It When

  • the input is truly one-dimensional and long-range (use attention instead)
  • you need a model that is permutation-invariant on the input (a graph, a set) — see Graph Neural Networks
  • a pretrained transformer already beats your hand-rolled CNN on the benchmark — use the pretrained model

Architecture Building Blocks

Layer What It Does Key Parameters
nn.Conv2d applies learned filters to extract spatial features in_channels, out_channels, kernel_size, stride, padding, groups
nn.MaxPool2d downsamples by taking the maximum in each window kernel_size, stride
nn.AvgPool2d downsamples by averaging same
nn.AdaptiveAvgPool2d pools to a fixed output size regardless of input size output_size
nn.BatchNorm2d normalizes per channel for stable training num_features
nn.Dropout2d drops entire channels to regularize p
nn.Flatten reshapes spatial output into a vector for the classifier head —

How Convolution Works

A 3×3 filter slides across the input, computing a dot product at each position:

Input (5×5)         Filter (3×3)        Output (3×3)
┌─────────────┐     ┌─────────┐         ┌─────────┐
│ . . . . .   │     │ w w w   │         │ o o o   │
│ . . . . .   │  ×  │ w w w   │    =    │ o o o   │
│ . . . . .   │     │ w w w   │         │ o o o   │
│ . . . . .   │     └─────────┘         └─────────┘
│ . . . . .   │
└─────────────┘
  • stride controls how far the filter moves each step (stride=2 halves the spatial size)
  • padding adds zeros around the input to control the output size; padding="same" keeps the dimensions unchanged
  • groups enables grouped or depthwise convolutions (see below)

Minimal Architecture

import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(3, 32, kernel_size=3, padding=1),    # 3 channels in, 32 out
    nn.BatchNorm2d(32),
    nn.ReLU(),
    nn.MaxPool2d(2),                                 # halve spatial size

    nn.Conv2d(32, 64, kernel_size=3, padding=1),
    nn.BatchNorm2d(64),
    nn.ReLU(),
    nn.MaxPool2d(2),

    nn.AdaptiveAvgPool2d(1),                         # global average pool
    nn.Flatten(),
    nn.Linear(64, 10),                               # 10-class output
)

Receptive Field — The Single Most Useful Inspection

The receptive field of a unit is the region of input pixels that contributes to its value. Each layer expands it. For a stack of 3×3 convolutions (stride 1, padding 1), the receptive field grows additively by 2 per layer; with stride-2 steps or pooling, it grows multiplicatively.

Quick recipe for rectangular, non-dilated convs:

def receptive_field(layers):
    """layers: list of (kernel_size, stride) tuples from input toward output"""
    rf = 1
    jump = 1
    for k, s in layers:
        rf = rf + (k - 1) * jump
        jump = jump * s
    return rf

# example: three 3×3 convs + a stride-2 pool + two more 3×3 convs
layers = [(3,1), (3,1), (3,1), (2,2), (3,1), (3,1)]
print(receptive_field(layers))   # -> 16 pixels on a side

Why this matters: if your network's receptive field is smaller than the object you are trying to recognize, no amount of training will make it work. If you are classifying objects that can span 100 pixels, you need a receptive field that comfortably exceeds 100. The fix is either more layers, a larger kernel, dilation, or stride.

Small Filters Stacked Deep — Why

Stacking 3×3 convolutions grows the receptive field:

  • 1 layer: 3×3
  • 2 layers: 5×5
  • 3 layers: 7×7

With fewer parameters than a single 7×7 filter (3·(9 C²) vs. 49 C²) and more nonlinearities between them. This is the design rule behind VGG and everything after: prefer small, stacked filters over single large ones.

Depthwise-Separable Convolutions

A standard conv learns out_channels · in_channels · k · k parameters. A depthwise-separable conv factors that into two cheap steps:

  1. depthwise — one filter per input channel (groups=in_channels)
  2. pointwise (1×1 conv) — combine channels

Parameter cost drops from C_out · C_in · k² to C_in · k² + C_in · C_out. For a 3×3 conv with 256 in / 256 out, that is 589k params → 2.3k + 65.5k ≈ 68k — nearly a 9× reduction.

class DepthwiseSeparable(nn.Module):
    def __init__(self, c_in, c_out, k=3, stride=1):
        super().__init__()
        self.dw = nn.Conv2d(c_in, c_in, k, stride=stride,
                            padding=k//2, groups=c_in, bias=False)
        self.pw = nn.Conv2d(c_in, c_out, 1, bias=False)
        self.bn = nn.BatchNorm2d(c_out)
    def forward(self, x):
        return F.relu(self.bn(self.pw(self.dw(x))))

Depthwise-separables are the workhorse of MobileNet, EfficientNet, and most mobile/edge CNNs. The quality gap to full conv is small; the speed gap is large.

Residual Connections — Why Depth Works

Naïve deep CNNs (VGG-19 and deeper) degrade on training set as they grow deeper — not overfitting, optimization difficulty. Residual connections fix this by adding the input to the output of a block:

y = F(x) + x

At initialization, F(x) ≈ 0, so y ≈ x — gradients flow through the identity. The network only has to learn the residual correction. This is the ResNet insight, and it made >100-layer networks trainable.

class ResidualBlock(nn.Module):
    def __init__(self, c, k=3):
        super().__init__()
        self.conv1 = nn.Conv2d(c, c, k, padding=k//2, bias=False)
        self.bn1   = nn.BatchNorm2d(c)
        self.conv2 = nn.Conv2d(c, c, k, padding=k//2, bias=False)
        self.bn2   = nn.BatchNorm2d(c)
    def forward(self, x):
        h = F.relu(self.bn1(self.conv1(x)))
        h = self.bn2(self.conv2(h))
        return F.relu(h + x)

When channels or spatial size change (stride>1), the shortcut becomes a 1×1 projection so the shapes match. Modern variants (pre-activation, bottleneck) tweak the block structure but keep the identity skip.

Modern Block Trade-offs

Block When it wins Trade
plain conv + BN + ReLU (VGG-style) shallow networks, maximum interpretability does not scale past ~20 layers
residual block (ResNet) depth up to 152, strong general baseline slightly more compute per parameter
bottleneck residual (ResNet-50+) very deep networks, fewer params 1×1 → 3×3 → 1×1 complexity
depthwise-separable (MobileNet) mobile, edge, tight latency small quality drop vs. full conv
inverted residual (MobileNetV2) mobile detection/classification relies on narrow bottlenecks
SE block (Squeeze-and-Excitation) adds channel attention to any backbone ~1% param cost, small but real win
ConvNeXt block modern CNN competitive with ViT depthwise + LN + GELU pattern

Design Ladder

  1. Start small — 2–3 conv blocks with batch norm and pooling
  2. Overfit one batch — if the network cannot drive training loss to near-zero on 8 images, the architecture is broken
  3. Add depth gradually — more layers, more filters
  4. Switch to a pretrained backbone (ResNet, EfficientNet, ConvNeXt) before designing from scratch
  5. Fine-tune selectively — see Transfer and Fine-Tuning

Common Architectures

Architecture Depth Key Idea
LeNet shallow the original: conv → pool → conv → pool → FC
AlexNet 8 layers GPU training at scale, ReLU, dropout
VGG 16–19 layers stack 3×3 convs, double channels
ResNet 18–152 layers skip connections
MobileNet / EfficientNet scalable depthwise-separables, compound scaling
ConvNeXt 22–51 modernized CNN with transformer-era patterns

What To Inspect

  • receptive field vs. object size — the calculator above
  • training loss on a single batch — must drop to near-zero on a 4-class, 8-image overfit test
  • validation loss trajectory — should mirror training after a short lag; a flat validation loss signals the model is stuck, not slow
  • feature maps at each stage — are the early layers detecting edges? Are the later layers detecting class-relevant structure?
  • per-class confusion matrix — where does the CNN actually err?
  • gradient norms per layer — very deep networks without residuals show shrinking gradients in early layers

Failure Pattern

Building a CNN with no pooling or stride, keeping the spatial size constant throughout. The model becomes extremely slow and memory-hungry without learning hierarchical features.

A second failure: flattening too early. If you flatten a 14×14×256 feature map directly into a linear head, the classifier has 50k weights per output class and overfits immediately. Use AdaptiveAvgPool2d(1) before flattening.

A third failure: stacking more than ~20 plain conv layers without residuals. Training loss stops falling; the fix is residual connections, not more tuning.

Common Mistakes

  • forgetting padding=1 with 3×3 kernels — the feature map shrinks by 2 per layer
  • using several fully connected layers after convolutions instead of global average pooling
  • skipping batch normalization — deep CNNs train much harder without it
  • applying the model to inputs of the wrong spatial size without AdaptiveAvgPool2d
  • designing from scratch when a pretrained backbone would beat it in a day
  • ignoring the receptive field and wondering why small objects are missed
  • using large kernels (7×7) when stacked 3×3s would do the same job with fewer parameters

Decision: Which Backbone

Need Start with Why
classification from scratch, small data ResNet-18 pretrained the universal strong baseline
classification from scratch, plenty of data + compute ConvNeXt / EfficientNet-B3+ better accuracy per FLOP
mobile / edge inference MobileNetV3 / EfficientNet-Lite depthwise-separables pay off
very long range spatial patterns ViT or ConvNeXt attention-like global mixing
dense prediction (segmentation, detection) ResNet + FPN or Swin feature-pyramid-friendly

Practice

  1. Build a 3-layer CNN and overfit one batch of 8 CIFAR images to confirm the architecture is wired correctly.
  2. Compute the receptive field of your network. Compare it to the smallest object you must recognize. Adjust if the field is smaller.
  3. Replace MaxPool2d with a strided 3×3 convolution. Compare accuracy and training speed.
  4. Remove batch normalization. Re-train and measure convergence speed; explain the gap.
  5. Swap one plain conv block for a depthwise-separable block. Compare parameter count and accuracy.
  6. Stack 40 plain conv layers and train on CIFAR-10. Observe the training loss. Convert the same network to residual blocks and re-run.
  7. Fine-tune a pretrained ResNet-18 on the same dataset. Compare to your hand-built CNN; explain the result.

Runnable Example

This example stays local-only for now because the browser runner does not yet include PyTorch. A small CNN on CIFAR-10 trains in a few minutes on a laptop CPU and in under a minute on any modern GPU.

Longer Connection

CNNs sit next to:

A CNN is a tool for hierarchical spatial pattern extraction. The decisions — kernel size, stride, depth, residuals, separables — each have a specific job. The moment the student cannot name the job a block is doing, the architecture has become folklore and should be simplified.