AIML Rounds RoadmapMocksJobs
Concept20 minBefore this:ResNets and Skip Connections

CNN Architectures: AlexNet to ConvNeXt

How did CNN architectures evolve, and what idea did each one add?

The situation

You're choosing a backbone for an image model. One colleague wants the newest large model, and another says it must run on a cheap phone. A third asks why anyone still uses 1×1 convolutions. The interviewer asks:

How did CNN architectures evolve, and what idea did each one add?

Why it matters

Interviewers often ask how you'd choose a backbone, or what a 1×1 convolution is for. These ideas also explain trade-offs you'll meet in production: accuracy against memory, latency and battery. This builds on convolutions and ResNets.

The idea

Think of how buildings developed. Each new design solved one problem: taller frames, lifts, lighter materials. You don't need to memorise every building, just what each one made possible. CNNs are similar: each famous model added one idea that later ones kept.

ResNetSkip connectionsallow depthTrains deeperMobileNetSeparable filterscut costRuns on phonesEfficientNetScale everythingtogetherBalanced growth
Each architecture is one idea, plus a trade-off.

Play

Try to break it. Raise the channels out, N. Then change the feature map size.

What does separating cost?

Same input, same output shape
standard multiply-adds
–
separable multiply-adds
–
times cheaper
–
weights: standard, separable
–

Exact calculation, using the MobileNet formulas. Weights leave out biases.

Raise N, and the saving approaches K². Raise K to 7, and it's up to about 49 times, which is why large depthwise filters are affordable in ConvNeXt. Change F: both costs move together, so the saving stays the same.

Now say why: why doesn't the saving depend on the size of the feature map?

Maths

In one breath

Each famous CNN added one idea. VGG: stack small 3×3 filters. Inception: run filter sizes in parallel, and shrink channels with 1×1 convolutions. ResNet: add skip connections. DenseNet: reuse all earlier features. MobileNet: split a convolution in two, for about 8 to 9 times less compute. EfficientNet: scale depth, width and resolution together. ConvNeXt: borrow Transformer design choices, and stay a pure CNN.

One idea per architecture

ModelYearThe ideaA reported result
AlexNet2012A deep CNN on GPUs, with ReLU and dropout.15.3% top-5 error, winning ImageNet 2012.
VGG2014Only 3×3 filters, up to 19 layers.7.0% top-5 for a single net, with 138 to 144 million parameters.
Inception2014Parallel filter sizes, and 1×1 convolutions to cut cost.22 layers, 12 times fewer parameters than AlexNet. 6.67% with an ensemble.
ResNet2015Skip connections.152 layers, 11.3 billion FLOPs against VGG-19's 19.6. 3.57% with an ensemble.
DenseNet2016Each layer receives all earlier feature maps.L(L + 1)/2 direct connections for L layers.
MobileNet2017Depthwise separable convolutions.70.6% top-1 at 569 million multiply-adds, against 71.7% at 4,866 million.
EfficientNet2019Scale depth, width and resolution together.84.3% top-1 for B7, with 66 million parameters.
ConvNeXt2022A ResNet modernised using Transformer design choices.82.1% top-1 for its smallest version, on ImageNet-1K.

Figures come from each paper, and they use different test setups, so don't compare them directly.

Why a 1×1 convolution helps

It mixes channels at each pixel, and can reduce their number. Inception uses this before its 3×3 and 5×5 filters, so those filters have fewer channels to process.

Depthwise separable convolutions

A normal convolution does two jobs at once: it looks at neighbouring pixels, and it mixes channels. MobileNet separates them:

  • Depthwise: one K×K filter per input channel, with no mixing between channels.
  • Pointwise: a 1×1 convolution that mixes the channels.

It has less capacity for the same channel count, and that's why accuracy falls slightly.

Compound scaling

Scaling only one dimension soon gives diminishing returns. A larger image needs more layers to see it, and more channels for its finer patterns. EfficientNet scales all three using one number φ, keeping compute growth at about 2φ.

What ConvNeXt changed

Its authors began with ResNet-50, and applied Transformer choices step by step: a different stage ratio, a patch-like 4×4 stem, 7×7 depthwise filters, GELU and a single LayerNorm. Top-1 accuracy rose from 78.8% at the stage-ratio step to 81.5% at the LayerNorm step. Much of the gap between CNNs and Transformers came from these design and training choices.

The maths

    From the MobileNet paper, Equations 2 and 5 and the ratio after them.

    In practice

    In practice

    • Start from a pretrained backbone: a well-known model is usually better than a new design.
    • Choose by constraint: measure latency, memory and accuracy on your real device. FLOPs alone can mislead, because memory access matters too.
    • Phones and edge devices: MobileNet-style models are the usual start. Width and resolution multipliers let you trade accuracy for speed.
    • Training recipes matter: a better recipe can change results as much as a new architecture, so compare models under the same recipe.
    • CNN or Transformer: with limited data, CNNs are a safe start. See Vision Transformers.

    The answer that sounds right

    "The newest model is always the best choice"

    It sounds right, as results keep improving. But published scores assume particular data, training recipes and hardware. With limited data or a cheap device, an older, well-understood backbone can do better in your setting. Measure accuracy, latency and memory on your own problem.

    Say it

    How did CNN architectures evolve, and what idea did each one add?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why is a depthwise separable convolution cheaper?

    Answer it out loud, then check.

    It splits the work. Depthwise filters handle space, one per channel, and a 1×1 convolution mixes channels. The cost ratio is 1/N + 1/K². With 3×3 filters, that's about 8 to 9 times cheaper. The trade-off is less capacity, and a small loss in accuracy.

    Follow-up 2 of 3

    Why stack two 3×3 filters instead of using one 5×5?

    Answer it out loud, then check.

    They see the same 5×5 area with 18 weights rather than 25, per channel pair, and they add an extra non-linearity. That was the main insight in VGG.

    Follow-up 3 of 3

    You need image classification on a low-end phone. What would you do?

    Answer it out loud, then check.

    Start with a pretrained MobileNet-style model. Tune the width and resolution multipliers against measured latency on the actual device. Then consider quantisation. I'd also measure memory, as FLOPs alone won't predict real speed.

    Each architecture added one idea. VGG stacked small 3×3 filters. Inception ran filter sizes in parallel, using 1×1 convolutions to cut cost. ResNet added skip connections, so very deep networks could train. MobileNet split a convolution into depthwise and pointwise parts, which is about 8 to 9 times cheaper with 3×3 filters. EfficientNet scaled depth, width and resolution together. ConvNeXt showed that a CNN with Transformer-style design choices competes with Transformers. In practice, I'd start from a pretrained backbone, and choose by accuracy, latency and memory on the real device.

    Key terms and sources

    AlexNet, VGG, Inception, ResNet, DenseNet, MobileNet, EfficientNet and ConvNeXt: the one idea each added, and how depthwise separable convolutions cut cost.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative, and worked examples were checked by calculation.