In early neuroscience, researchers proposed a whimsical theory called the “Grandmother Cell Hypothesis”: the idea that somewhere in your brain exists a single, dedicated neuron that fires exclusively when you look at your grandmother. If that neuron died, you would recognize everyone else on Earth, but your grandmother would look like a total stranger. For decades, artificial intelligence researchers hoped artificial neural networks would work the same way, with one neuron for cats, one for semicolons, and one for German verbs.
When researchers actually inspected the feed-forward neurons of modern transformers, they were met with baffling chaos. A single neuron (e.g., Neuron 412 in Layer 7) would fire strongly when it saw an academic paper on quantum mechanics, a photograph of a golden retriever, and a poem by William Shakespeare. The neuron was not clean and monosemantic; it was stubbornly, maddeningly polysemantic. Why would a billion-dollar neural network multitask like a frantic amateur?
The Superposition Hypothesis
In 2022, Anthropic researchers Toy et al. published Toy Models of Superposition, providing the definitive mathematical answer: Superposition is a geometric compression trick that neural networks use to represent more features than they have physical dimensions.
Why Networks Exploit Non-Orthogonal Space
In 2D space, you can only fit 2 completely orthogonal (perpendicular, zero-interference) directions. But if the concepts you care about are sparse (meaning they rarely appear at the same time), you can pack 5 concept vectors in a pentagon with very low cross-talk:
Not 8,192 concepts, literally millions!
The Johnson-Lindenstrauss Miracle
The mathematical engine behind superposition is the Johnson-Lindenstrauss Lemma. It proves that in high-dimensional vector spaces (d ≥ 4096), you can place an exponentially large number of random vectors such that the dot product between any two random vectors is vanishingly close to zero:
〈 v_i, v_j 〉 ≈ 0 for all i ≠ j
Because natural concepts in human language are sparse, you rarely discuss Shakespeare, golden retrievers, and quantum entanglement in the exact same sentence, the neural network assigns non-orthogonal vectors to each concept. The model suffers minimal interference, enjoying a 10x to 100x increase in representational capacity.
The Interpretability Dilemma
Superposition solves the mystery of polysemantic neurons: individual neurons are not the fundamental units of thought. Individual neurons are merely arbitrary coordinate axes (the neuron basis) cutting diagonally across dozens of superposed concept vectors.
To read the mind of an AI model, we cannot look at raw neurons; we must rotate the coordinate system to find the true, pure feature directions. In Part 6, we discover the universal technology that does exactly that: Sparse Autoencoders.
Ready for Neural Reverse Engineering? Explore the Flagship Treatise
This series translates the core principles of transformer circuit tracing, superposition, and monosemantic feature extraction into accessible visual models. For mathematical invariants, formal proofs, and PyTorch circuit pruning algorithms, consult our foundational monograph: Mechanistic Interpretability Transformer: 5 Empirical Rules.
Mechanistic Interpretability in AI
An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.