AIML Rounds RoadmapMocksJobs

Vector and Matrix Norms

What's the difference between L1 and L2 norms, and why does L1 give sparse weights?

The situation

Your regression has 500 features, and you want most of the weights to go to zero. A teammate switches between L1 and L2 penalties and gets very different results. You also want to stop exploding gradients. The interviewer asks:

What's the difference between L1 and L2 norms, and why does L1 give sparse weights?

Why it matters

The norm you choose decides what "small" means. That choice sets which weights regularisation removes, when gradient clipping activates and how far a layer can amplify its input. This builds on vectors and length.

The idea

Think of getting from one place to another in a city. L2 is the straight-line distance, as a bird flies. L1 is the distance by road on a grid, adding every block. L∞ only cares about your longest stretch. It's the same trip, measured by different rules.

L1Add absolutevalues7L2Straight-linelength5L∞The largestvalue4
The same vector, (3, −4), measured three ways.

Play

Try to break it. Set the vector to (3, 0), then (3, 3). Then lower the clip limit.

Compare the unit balls

Same vector, different sizes
L1
–
L2
–
L∞
–
after L2 clipping
–

Diamond is L1, circle is L2 and square is L∞, each drawn at radius 2. The dot is your vector. Clipping scales the whole vector, so the direction is unchanged.

Set the vector to (3, 0): all three norms agree, because only one value is non-zero. Set it to (3, 3), and they differ. L1 counts every value, while L∞ only sees the biggest.

Now say why: why do the three norms agree when only one value isn't zero?

Maths

In one breath

A norm measures size. L2 is straight-line length, L1 adds absolute values, and L∞ takes the largest entry. Each has a different "unit ball". L1's is a diamond with corners on the axes, so penalised solutions tend to land where some weights are exactly zero. For matrices, Frobenius treats all entries like one long vector, and the spectral norm is the most a matrix can stretch any vector.

Measuring size

NormHowWhere it's used
L2Square root of the sum of squares.Ridge and weight decay, gradient clipping, Euclidean distance.
L1Sum of absolute values.Lasso, for sparse weights. Less sensitive to single large values.
L∞The largest absolute value.Limits on any single value, like adversarial attacks.
FrobeniusL2 of all a matrix's entries.Low-rank error, matrix factorisation.
SpectralThe largest singular value.How much a layer can amplify. Spectral normalisation in GANs.

Why L1 gives zeros

Regularisation asks for the lowest loss inside a norm's ball. The loss's contours usually first touch L1's diamond at a corner, and corners sit on an axis, where other weights are exactly zero. L2's circle has no corners, so its weights shrink but rarely reach zero.

The gradient tells the same story. L2's pull weakens as a weight approaches zero. L1's pull stays constant, so it can push a small weight all the way to zero.

A rule every norm follows

  • It's zero only for the zero vector.
  • Scaling a vector scales its norm.
  • The triangle inequality holds: going directly is never longer than going via another point.

The maths

    Worked example: for x = (3, −4), L1 = 7, L2 = 5 and L∞ = 4. See the SVD lesson for singular values.

    In practice

    In practice

    • Gradient clipping: if the gradient's L2 norm exceeds a limit, scale the whole gradient down. Direction stays the same, and only the step size changes.
    • Scale your features: norms add values across features, so a feature in large units can dominate. Standardise before using a norm penalty.
    • Normalise embeddings: dividing by the L2 norm makes the dot product equal cosine similarity.
    • Spectral normalisation: dividing a layer's weights by its spectral norm limits how much it can amplify.
    • Check for divergence: tracking the norm of weights and gradients is a quick health check during training.

    The answer that sounds right

    "L2 is better because it's smooth"

    It sounds right, since smooth functions are easier to optimise. But they do different jobs. L2 shrinks all weights and keeps them, while L1 removes some entirely. If you want a small set of features, L2 won't give you one. Choose by the outcome you want, and consider elastic net for both.

    Say it

    What's the difference between L1 and L2 norms, and why does L1 give sparse weights?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why does gradient clipping use the norm rather than clipping each value?

    Answer it out loud, then check.

    Clipping each value separately changes the gradient's direction, whereas scaling by the norm keeps it. Per-value clipping is simpler, and some use it, but norm clipping keeps pointing where the gradient said to go, with only a smaller step.

    Follow-up 2 of 3

    Lasso picks one of two highly correlated features. Why?

    Answer it out loud, then check.

    The L1 penalty doesn't care how weight is split, in total, between two identical features, so it can land on either, often zeroing the other. Which one it picks can change between samples. Elastic net adds some L2, which spreads weight more evenly.

    Follow-up 3 of 3

    When would you prefer the spectral norm to Frobenius?

    Answer it out loud, then check.

    When you care about the worst case: the most a matrix can stretch any input. That matters for stability and Lipschitz limits. Frobenius is about total energy across all directions, which is what matters for low-rank approximation error.

    A norm measures size. L2 is straight-line length, and L1 is the sum of absolute values. L1 gives sparse weights because its unit ball is a diamond with corners on the axes, so the loss's contours tend to touch it where some weights are exactly zero. In terms of gradients, L1 keeps a constant pull towards zero, while L2's pull fades. For matrices, Frobenius treats all entries as one vector, and the spectral norm is the largest singular value, which limits how much a layer can amplify.

    Key terms and sources

    L1, L2 and max norms for vectors, Frobenius and spectral norms for matrices, and how they drive regularisation, gradient clipping and stability.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative, and worked examples were checked by calculation.