Linear Representations and Universality
What does it mean that concepts are linear inside a model, and when does that break?
The situation
A teammate found a "sentiment" direction with a probe and wants to add it during generation to make replies more positive. Another says directions inside a neural network don't mean anything. The interviewer asks:
What does it mean that concepts are linear inside a model, and when does that break?
The idea
Think of a mixing desk. Each slider is a direction: its position tells you how much of that sound is in the mix, and moving it changes the mix. Most concepts behave like sliders, but a few are dials that wrap around.
Play
Try it. Start at "king" and push along the gender direction. Then push along royalty.
Subspace, measurement, intervention
Park et al. (2024) point out that “linear representation” is used in three different senses, and prove how they connect Park et al. 2024, §1–2:
| Meaning | Claim | Tool it justifies |
|---|---|---|
| Subspace | All pairs that differ only in the concept differ along one shared direction: queen − king ∥ woman − man. | word-vector arithmetic |
| Measurement | The probability of the concept is logit-linear in the activation. | linear probes |
| Intervention | Adding a vector changes the concept and leaves other concepts alone. | steering vectors |
Their result: the subspace direction in the output (unembedding) space acts as an ideal probe, and the matching direction in the input space acts as a clean steering vector. In LLaMA-2, adding the male⇒female direction to “The prince matured and eventually became the” moves the top prediction from “king” to “queen”, without changing whether the next word is capitalised Park et al. Fig. 5, Table 5.
Try it: steer along one direction (3D)
A toy activation space with three concept directions: male ⇒ female, royalty, and capitalisation. The point starts at the context for “king”. Slide α to add the male ⇒ female direction and watch which word is closest. Drag the plot to rotate it.
Illustrative geometry with orthogonal concept directions, not a real model. Park et al. show the real directions are orthogonal only under the right inner product (section 3).
Notice: the same direction works for several words. That's the linear bet: one direction per concept, usable for both reading and editing.
How it works
In one breath
Many concepts behave like directions: project onto one to measure the concept, add along it to change it. That's the bet behind probes, steering and sparse autoencoders. It doesn't hold everywhere: some concepts are circles or need several directions, and which directions count as independent depends on the geometry you choose. Different models often learn similar structures, but similarity scores are fragile.A concept is a direction you can read and push
The survey’s statement: a concept can be represented by a linear direction φ in activation space. Adding or subtracting (a multiple of) φ changes the related behaviour, and the projection φ⊤h(x) approximately measures how strongly the concept is present p. 27.
- Strong version: every concept is best represented this way, so activations are approximately a sum of features: h(x) ≈ Σ αi φi.
- Weak version: only some concepts are linear p. 27.
Why the field cares: if concepts are directions, the toolkit is just linear algebra. You can find them, measure them, scale them and add them back. Probes, steering vectors and sparse autoencoders all assume this p. 27.
“Orthogonal” depends on which inner product you use
Cosine similarity and projection need an inner product, and nothing in training picks one out. Any invertible linear change to the representation (with the inverse applied on the other side) leaves the model’s outputs unchanged, so the plain Euclidean dot product has no special claim to be meaningful Park et al. §3.
Park et al. propose a causal inner product under which concepts that can vary independently (language and gender, say) come out orthogonal. In practice it is estimated from the covariance of the unembedding vectors. In LLaMA-2 the ordinary Euclidean product happened to work reasonably well; in Gemma-2B it did not, while the causal one still did Park et al. App. D.2. my take When a paper reports a cosine between two feature directions, ask which inner product it used.
Only an element-wise operation makes the axes special
Inside an MLP, the nonlinearity (ReLU, GELU) acts on each neuron separately, so the coordinate axes are computationally meaningful: a privileged basis. In the residual stream, nothing acts on coordinates one at a time. Rotate the axes, apply the opposite rotation to the reading and writing weights, and the model computes exactly the same thing: a non-privileged basis p. 27–28, Fig. 4.
Consequence: in a privileged basis a single neuron can be a meaningful feature. In a non-privileged one, features can point anywhere and need methods that look for directions p. 27.
Try it: rotate the basis and see what changes
An activation vector passes through a layer. We rotate the coordinate system by θ, apply the layer, and rotate back. If the axes don’t matter, the output shouldn’t change.
Same algorithm, or same representation?
The survey: models trained independently on similar tasks tend to develop similar representations or algorithms, often approximating a canonical form set by the task. How much depends on size, architecture and data p. 31.
| Weak universality | Strong universality | |
|---|---|---|
| Claim | Models converge on the same algorithms and computational patterns, possibly encoded differently | The representations themselves converge to a shared canonical form |
| Evidence | Recurring attention patterns, features and circuits across models and scales (induction heads, curve detectors, arithmetic circuits) | Representational similarity metrics; recent studies find these results fragile |
| Status | reasonably supported | less supported |
Survey §5.4 p. 31. Divergence can come from differences in architecture, data or the optimisation path.
Compare geometry, not neurons
You cannot just match neuron 7 in model A with neuron 7 in model B: there is no reason their axes line up, and in a non-privileged space (this lesson) the axes mean nothing anyway. So similarity metrics compare geometry, meaning how each model arranges the same inputs relative to each other. Examples are CKA (centred kernel alignment), SVCCA, and mutual nearest neighbours Huh et al. §2.
Try it: one model, rotated and noisy
Model B is model A’s representation of the same 30 inputs, rotated, with some noise added. Three ways of measuring similarity are shown. Watch them disagree.
30 points in 3 clusters (fixed seed), shown in their first two coordinates. Neuron matching = average correlation of neuron i in A with neuron i in B. CKA = linear centred kernel alignment. Mutual k-NN = share of each point’s 3 nearest neighbours that are the same in both models. All computed in your browser.
That is the fragility the survey points to: “are these models the same?” depends on which metric you use p. 31. Even the authors of the Platonic representation paper report that CKA showed only a very weak trend in their data, which is partly why they used nearest neighbours instead Huh et al. App. A.
In practice
Evidence for the weak version, not the strong one
Support
Linear probes work across many models and concepts; steering along single directions changes behaviour; sparse autoencoders recover linearly accessible, steerable features at scale p. 27.
Counterexamples
Nonlinear feature interactions; algorithms that rely on nonlinear transformations (the clock-like modular-addition circuits); features spread across several interacting subspaces; circles such as days of the week (Features, Polysemanticity and Superposition) p. 27 Engels et al.. Some argue features may not exist as explicit objects at all, but emerge from circuit dynamics p. 27–28.
The survey’s verdict: the hypothesis is “best viewed as a useful approximation that holds in some settings, rather than as a universal law” p. 28.
Does what we learn about one model transfer?
If strong universality holds, fine-grained mechanistic findings and the tools built on them transfer between models. If only weak universality holds, findings may stay tied to a model family or task. Then even a complete mechanistic understanding of one model might not carry over to the next p. 31. my take That makes universality a quiet assumption behind every research orientation in Why Look Inside a Model?: it decides whether interpretability can keep up with new models or has to start again for each one.
Single directions that control behaviour
The strongest recent evidence for the linear picture is behavioural. Arditi et al. (NeurIPS 2024) found that refusal is mediated by one direction in the residual stream across 13 open chat models up to 72B parameters: remove it and the model stops refusing harmful requests, add it and the model refuses harmless ones. Anthropic's persona vectors (August 2025) find directions for traits such as sycophancy and use them to monitor and steer models.
The counter-evidence is also real: some features are multi-dimensional, like the circles for days and months (Engels et al., 2024). So a direction is a good first hypothesis, and an intervention is how you check it.
The answer that sounds right
"The probe found a direction, so adding it will steer"
It sounds like the same thing, but a probe finds whatever direction best separates the labels, which can be a correlated feature the model never uses. Steering needs the direction the model reads. Test it: if adding the direction doesn't change behaviour, it's a readout, not a control.
Say it
What does it mean that concepts are linear inside a model, and when does that break?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
What is a privileged basis?
Answer it out loud, then check.
Where an elementwise operation like ReLU acts, the neuron axes become special, so single neurons can mean something. The residual stream has no such operation, so its axes are arbitrary.
Follow-up 2 of 3
What's weak versus strong universality?
Answer it out loud, then check.
Weak: different models learn similar algorithms. Strong: they learn the same representations. Recurring circuits support the weak version; the strong one rests on similarity metrics that depend on how you measure.
Follow-up 3 of 3
Why does the inner product matter?
Answer it out loud, then check.
Two directions can look orthogonal under the plain dot product but not under the geometry the model uses for its outputs, so whether two concepts are independent depends on that choice.
The linear representation hypothesis says many concepts are directions in activation space. Projecting onto the direction measures the concept, and adding along it changes it, which is what probes, steering and sparse autoencoders rely on. There's good evidence, for example a single refusal direction in many chat models. But it isn't universal: days of the week sit on a circle, some concepts need a subspace, and a direction that separates labels isn't always the one the model uses. So I'd validate any direction with an intervention.
Key terms and sources
Built from Barez et al., Understanding AI from the Inside (arXiv 2611.08438) and 5 more sources · checked 13 Nov 2026
Sources
- Barez et al., Understanding AI from the Inside (survey, 2026)
- Park et al., The Linear Representation Hypothesis and the Geometry of LLMs (2023)
- Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (NeurIPS 2024)
- Chen et al., Persona Vectors: Monitoring and Controlling Character Traits (Anthropic, 2025)
- Engels et al., Not All Language Model Features Are One-Dimensionally Linear (2024)
- Huh et al., The Platonic Representation Hypothesis (2024)
Explanations are ours, built from these sources. Chapters marked with page numbers come from the survey. Lab numbers are illustrative unless a lab says otherwise. Many recent papers are preprints.