Concept30 minBefore this:Eigenvalues & Eigenvectors

Principal Component Analysis

Ten correlated columns, two real ideas: how does PCA find them, and why scale first?

The situation

You have ten customer columns, and most of them move together. Feeding them all into a model counts the same idea several times. The interviewer asks:

What does PCA do, and what are the principal components?

Running example: the food-delivery app again, now with ten numbers per customer: orders per month, app sessions, active days, push-notification opens (all "engagement"); average basket, share of premium items, tip, delivery fee paid (all "spend"); coupon use (a bit of both); and the rating they give (mostly unrelated). The 300 simulated customers are generated from just two hidden factors, engagement and spend, plus noise. Can PCA discover that ten columns are really about two things? A 2-D version (orders vs app sessions) comes first.

Ten correlated columns, two real ideas

Customers who order more also open the app more and are active more days: those columns move together. Feeding all ten into a model wastes capacity on duplicates, makes distances (k-means, k-NN) count the same idea several times, and makes plots impossible. We want a few new features that summarise the ten with as little loss as possible, discovered from the data, with no labels.

The idea

Think of photographing a long, thin object. From most angles it looks small, but one angle shows it at full length. PCA finds that angle for your data: the direction with the most spread, then the next-best one at a right angle to it.

Illustrative. Keeping only PC1 here would keep most of the spread.

Play

Try to break it. Keep just one component and read the variance kept. Then add components one at a time until the gain gets small.

Choosing k

The explained-variance (scree) plot

Components kept

variance kept

–

reconstruction error (share of total)

–

columns stored per customer

–

Ten standardised customer features. Bars: variance per component. Line: cumulative share.

The first component explains 55.0%, the second 24.4% (79% together), the third 8.2%, and every other one about 2% or less, roughly what a pure-noise direction would explain. The bend after two components matches the two hidden factors, plus a third for the "rating" column, which is mostly its own independent noise. Common rules: keep enough components for 90–95% of the variance (scikit-learn accepts a fraction, e.g. n_components=0.95), or cut at the bend, or choose k by the downstream model's validation score.

Now say why: why do the first two components capture so much here, and where would you stop?

Maths

In one breath

PCA finds new axes for your data, the principal components: the first points along the direction where the data varies most, the second along the most-varying direction at right angles to the first, and so on. Keep the first few and you have a smaller set of features that holds most of the information (variance), with the least possible reconstruction error for a linear projection. Mathematically the components are the eigenvectors of the covariance matrix (computed via SVD), and each eigenvalue is the variance it captures. Choose how many to keep from the explained-variance curve. Always standardise first when features have different units, and remember PCA only finds linear structure, and high variance isn't the same as useful for prediction.What you'll be able to do after this: explain PCA as variance maximisation and as best reconstruction, read a scree plot, interpret loadings, show why scaling matters, and know when to use something else.

data pointsPC1 / chosen directionPC2variance numberslost information / failure

Variance along a direction

Centre the data (subtract each column's mean). Pick a direction, a unit vector w. Project every customer onto it: their score is w·x, a single number. The variance of those scores says how much of the data's spread that direction captures. If the covariance matrix of the data is Σ, that variance has a compact form:

Direction angle

variance captured by this direction

–

left over (squared distance to the line)

–

150 customers: orders per month vs app sessions (centred, correlation ≈ 0.85). Short red lines show each customer's distance to the line. Turn the line: captured + left over always add up to the total variance.

Rotate the line and watch the two numbers trade off. The direction that captures the most variance is also the one with the smallest squared distances to the line. Those are the same problem, which is the key fact about PCA. Here PC1 captures 93.7% of the variance; PC2, at right angles, gets the remaining 6.3%.

One principle

Components are eigenvectors of the covariance matrix

scikit-learn: PCA decomposes data "in a set of successive orthogonal components that explain a maximum amount of the variance" decomposition guide.

Because Σ is symmetric, its eigenvectors are orthogonal: each component captures new variance not captured by earlier ones, and the component scores are uncorrelated. In practice libraries don't form Σ; they take the singular value decomposition of the centred data matrix, X_c = U S V^T. The rows of V^T are the components and the squared singular values divided by (n − 1) are the eigenvalues. It's more numerically stable, and randomised SVD makes it fast when you only want a few components.

Go deeper: the two views give the same answer

For a unit vector w, each centred point splits into its projection (w·x)w and a residual. By Pythagoras, ‖x‖² = (w·x)² + ‖residual‖². Summed over points, the left side is fixed, so maximising the projected variance Σ(w·x)² is the same as minimising the total squared residual. With k components the same argument shows the top-k eigenvectors give the best rank-k linear reconstruction (the Eckart–Young theorem for the SVD).

What a component means

Reading the loadings

Each bar is the weight of one original feature in the component (the components are unit vectors, so weights are comparable).

PC1 has positive weights on everything (0.19–0.38): it's "overall activity", customers who do more of everything. PC2 is a contrast: negative on the engagement features, positive on the spend features, so it separates "frequent, cheap" from "occasional, big-basket" customers. PCA didn't recover the hidden factors "engagement" and "spend" one per axis; it found their sum and their difference. Components are directions of variance, not guaranteed to be meaningful concepts. (Factor analysis with a rotation such as varimax aims at more interpretable axes.) my take

In practice

The dimensionality-reduction toolbox

Method	Keeps	Use for
PCA (Pearson 1901, Hotelling 1933)	Global variance, linear	Compression, de-correlation, preprocessing, quick plots
Truncated SVD	Same without centring	Sparse data (text TF-IDF, "LSA")
Kernel PCA	Non-linear variance via kernels	Curved structure, small n
t-SNE / UMAP	Local neighbourhoods, non-linear	2-D visualisation of clusters (next lesson)
Autoencoders	Whatever reconstructs best, non-linear	Large data, images, learned embeddings

PCA, truncated SVD and kernel PCA are in scikit-learn's decomposition module decomposition guide.

The whole recipe

Standardise (unless all features share a meaningful scale).
Fit PCA on training data only, then transform train and test.
Choose k from cumulative explained variance (e.g. 95%) or downstream validation.
Inspect loadings to describe the top components.
Big data: svd_solver="randomized", IncrementalPCA for data that doesn't fit in memory.

Cost: exact PCA is O(n_max² · n_min); randomised PCA is O(n_max² · n_components) decomposition guide. IncrementalPCA processes mini-batches for out-of-core data.

When to use it

Use PCA when

Many correlated numeric features: compress before clustering, k-NN or linear models
You need a fast first look at the data's structure
Denoising, or reducing to ~50 dimensions before t-SNE/UMAP

Reach for something else when

Visualising clusters: t-SNE/UMAP
Non-linear structure: kernel PCA, autoencoders
Sparse text: truncated SVD (no centring)
You need interpretable factors: factor analysis, NMF

The answer that sounds right

The trap

"Just drop the columns that look redundant"

Keeping "orders per month" and dropping "app sessions" throws away whatever sessions knew that orders didn't. Averaging hand-picked groups needs you to already know the structure. And when features are on different scales, "most variance" just means "biggest units": measure basket size in rupees instead of standardised units and PCA's first component becomes basket size alone, explaining 99.5% of the variance, which says nothing about customers (see the scaling lab). PCA has to be done on the right scale.

Scale first

Units decide the answer if you let them

variance explained by PC1

–

Multiply just the basket column by 30 (rupees instead of standardised units) and PC1 explains 99.5% of the variance with a weight of 1.00 on that single column: PCA has found "the column with the biggest numbers". scikit-learn: PCA "centers but does not scale the input data for each feature before applying the SVD" decomposition guide, so put a StandardScaler before it whenever units differ. (If all features share a unit and their scale is meaningful, such as pixel intensities, don't standardise.)

What PCA can't do

Linear only, and variance isn't relevance

Linear: a curved structure (a spiral, a ring) can't be unrolled by a rotation. Kernel PCA, UMAP or autoencoders handle that.
Variance ≠ usefulness: the feature that predicts churn might be a low-variance direction PCA throws away. PCA is unsupervised; if you're reducing dimensions for a predictor, validate k on the prediction task. my take
Outliers: squared distances let a few extreme points tilt the components (robust PCA exists).
Interpretability: each component mixes all original features.
For visualising clusters, PCA often overlaps them: in the UMAP paper's benchmark, a k-NN classifier on a 2-D PCA embedding of MNIST scored about 0.46–0.47 accuracy against about 0.96–0.97 for t-SNE or UMAP UMAP paper, Table 2. PCA is still the standard first step before t-SNE.

Caveats

Strengths

Fast, deterministic, no hyperparameters except k
Optimal linear compression (least squared reconstruction error)
Decorrelates features; denoises by dropping small components
Applies to new data with one matrix multiply

Weaknesses

Linear only
Scale-sensitive: standardise first
High variance ≠ predictive
Components are mixtures, often hard to name

Say it

What does PCA do, and what are the principal components?

Say your answer out loud

Aim for 30 seconds. It feels different from reading it, and that's the point.

Tick what you covered

PCA finds orthogonal directions that capture the most variance.
Each component maximises the remaining variance, orthogonal to the earlier ones.
They're the eigenvectors of the covariance matrix, and eigenvalues are the variance captured.
It's computed in practice by SVD of the centred data.
You pick k from cumulative variance, and standardise features first.
It's linear and unsupervised, so it can miss curves and useful low-variance directions.