SVD & Low-Rank Approximation
How LoRA and compression keep most of a matrix with a fraction of the numbers.
The situation
You need to fine-tune a huge language model, and a teammate suggests training only a low-rank update to its weights. The interviewer asks:
What is SVD, and what do you get by keeping only the top k singular values?
Running example: a 32 × 32 grayscale icon of a delivery scooter (1,024 pixel values), with a little random noise added, as if from a cheap camera. We compress it with the SVD and ask how many components we need, and whether dropping components can even remove noise.
Big matrices with little real information
A ratings matrix of 10 million users × 100,000 restaurants, a 12,288 × 12,288 weight matrix in a language model, an image: these are huge, but their content is often driven by a few underlying factors (taste profiles, dominant directions, shapes). We want to find those factors, store less, and throw away the noise. The eigendecomposition only works for square (and nicely behaved) matrices; the SVD works for any matrix.
The idea
Think of a picture as a stack of transparent layers, sorted from most to least important. The first few layers give you almost the whole picture. Later layers add fine detail, and eventually mostly noise. SVD finds those layers for any matrix.
Play
Try to break it. Slide the rank up from 1 and watch the error against the clean icon. Keep going all the way to 32.
Keep k singular values
Left: noisy image and its rank-k reconstruction. Right: the singular values, kept in teal.
The first singular value alone carries 74% of the energy; five carry 96%. Error against the noisy image falls steadily to zero at k = 32 (you reproduce the noise exactly). Error against the clean icon is lowest around k = 8 (13%, below the 18.7% error of the noisy image itself) and then rises again as later components add noise back. Truncation is a denoiser: the signal lives in a few strong directions, the noise is spread thinly over all of them.
Now say why: why does the error against the clean icon stop falling and then rise, even as you keep more of the image?
Maths
In one breath
Every matrix, any shape, can be written as A = UΣVᵀ: a rotation, a stretch along perpendicular axes, and another rotation. The stretches, the singular values σ₁ ≥ σ₂ ≥ …, say how important each direction is. Keep only the top k and you get the best possible rank-k approximation of A (Eckart–Young), stored in k(m + n + 1) numbers instead of m × n. Real data is often nearly low-rank, so a small k captures most of it and drops noise. That idea is PCA (SVD of centred data), matrix-factorisation recommenders (users × items ≈ low-rank), and LoRA (fine-tune a huge weight matrix by learning a rank-r update BA).What you'll be able to do after this: describe the three factors of the SVD, explain rank-k truncation and its error, compute the storage saving, and connect it to PCA, recommenders and LoRA.The decomposition
MML §4.5. The singular values are the square roots of the eigenvalues of AᵀA, and V's columns are its eigenvectors (Eigenvalues).
The best rank-k approximation
C. Eckart & G. Young, "The approximation of one matrix by another of lower rank", Psychometrika, 1936; MML Theorem 4.25.
LoRA: fine-tune with a rank-r update
The authors hypothesise that the change in weights during fine-tuning has a low "intrinsic rank". On GPT-3 175B, LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times" compared with full fine-tuning with Adam abstract, and "a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288" p. 2. Because BA has the same shape as W₀, it can be merged after training: no extra inference latency.
In practice
SVD across ML
- PCA: the SVD of the centred data X = UΣVᵀ gives the principal components (rows of Vᵀ) and variances σ²/(n − 1) (PCA).
- Recommenders: approximate the users × items ratings matrix by user factors × item factors of rank k (matrix factorisation, popularised by the Netflix Prize). Missing ratings need special solvers (alternating least squares, SGD), not plain SVD.
- Latent semantic analysis: truncated SVD of a term–document matrix finds topics.
- Pseudo-inverse and least squares: the SVD gives the stable solution to linear regression even when XᵀX is singular.
- Model compression: replace a big weight matrix by its rank-k factors to save memory and compute.
The answer that sounds right
"Compress by keeping a subset of rows or pixels"
Dropping rows, columns or pixels throws away specific parts of the data. The SVD instead keeps the most important patterns spread across all pixels. And there's a guarantee no ad-hoc scheme has: among all rank-k matrices, the truncated SVD is closest to A (Eckart–Young, 1936), with error equal to the root sum of squares of the dropped singular values.
Caveats
- Full SVD costs O(min(mn², m²n)); for big matrices use randomised or truncated SVD (
TruncatedSVD,svds). - Low rank works only if the data really is close to low rank; a full-rank weight change (e.g. a new language) needs larger r, as the LoRA authors note.
- Singular vectors are directions of variance, not guaranteed to be interpretable features.
Say it
What is SVD, and what do you get by keeping only the top k singular values?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
SVD vs eigendecomposition?
Answer it out loud, then check.
Eigendecomposition needs a square matrix and may not exist or be orthogonal; SVD exists for every matrix and always has orthonormal factors. Singular values of A are square roots of eigenvalues of AᵀA.
Follow-up 2 of 3
How does PCA use SVD?
Answer it out loud, then check.
Centre X, compute its SVD; the right singular vectors are the principal directions and σ²/(n − 1) the variances. It avoids forming the covariance matrix and is more stable.
Follow-up 3 of 3
Why does LoRA work?
Answer it out loud, then check.
Fine-tuning updates appear to have low intrinsic rank; a rank-r update BA with r ≪ d captures most of it, cutting trainable parameters and optimizer memory, and it can be merged into W₀ at inference.
The 30-second answer
The singular value decomposition writes any matrix as UΣVᵀ: orthonormal input directions, non-negative singular values, and orthonormal output directions, or equivalently a sum of rank-one layers ordered by importance. Keeping the top k gives the best rank-k approximation in Frobenius norm, by the Eckart–Young theorem, with error equal to the root sum of squares of the dropped singular values, and storage of k(m + n + 1) numbers. It underlies PCA, matrix-factorisation recommenders, latent semantic analysis and LoRA, which fine-tunes a large model by learning a low-rank update BA.
Go further
What not to say
- "SVD only works on square matrices." It works on any shape.
- "Keeping more components is always better." Against noisy data, too many components reproduce the noise.
Key terms and sources
The singular value decomposition, rank-k truncation and the Eckart–Young theorem, compression and denoising, and how low rank powers PCA, recommenders and LoRA · built from Deisenroth et al., Mathematics for Machine Learning (ch. 4), Eckart & Young (1936) and Hu et al., "LoRA" (2021) · checked 28 Oct 2026
Key terms
| Term | Meaning |
|---|---|
| SVD | A = UΣVᵀ for any matrix. |
| Singular value | Non-negative stretch σᵢ; importance of a direction. |
| Rank | Number of non-zero singular values. |
| Truncated SVD | Keep the top k terms. |
| Eckart–Young | Truncated SVD is the best rank-k approximation. |
Sources
- M. P. Deisenroth, A. A. Faisal & C. S. Ong, Mathematics for Machine Learning, CUP 2020, mml-book.github.io (ch. 4.5–4.6).
- C. Eckart & G. Young, "The approximation of one matrix by another of lower rank", Psychometrika 1, 1936.
- E. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", ICLR 2022, arXiv:2106.09685.
The icon is generated procedurally with fixed-seed noise; the SVD is computed live in your browser (Jacobi eigen-decomposition of AᵀA).