When students first encounter the mathematical formula for transformer self-attention, Attention(Q, K, V) = softmax((Q · K^T) / √d_k) · V, it looks like an intimidating jumble of linear algebra. Four matrices (W_Q, W_K, W_V, W_O) are entangled in a single equation. For years, practitioners treated this as an inseparable black-box calculation.
In 2021, Anthropic researcher Chris Olah published a breakthrough mathematical factorization called A Mathematical Framework for Transformer Circuits. He proved that every attention head does not perform one complex task, it performs two completely independent, parallel linear operations: the QK (Query-Key) Circuit and the OV (Output-Value) Circuit.
The Division of Labor
Think of an attention head as a postal delivery worker sorting packages in a busy post office. The worker has two completely separate responsibilities:
WHERE to Look (Routing)
Determines the routing address. The Query asks: “I am a pronoun; where is my antecedent noun?” The Key answers: “I am the subject noun.” The dot product determines the attention weight (how much attention token X pays to token Y). It moves zero data; it only draws the arrow.
WHAT to Write (Content)
Determines the payload. Once the routing arrow is drawn, the Value matrix reads information from the source token, transforms it via the Output matrix, and writes it onto the destination token’s residual stream. It doesn’t care where it goes; it only transforms content.
The Factorization Formula
By factoring the full attention calculation, we can express the behavior of any attention head as two low-rank bilinear forms:
Independent Circuit Matrices
Why This Unlocks Circuit Tracing
Before this mathematical factorization, tracing a neural circuit required analyzing the entire billion-parameter model simultaneously. With the QK/OV decomposition, we can decompose an attention head into two clean linear maps: one controlling pointer movements, and the other controlling semantic data flow.
In Part 5, we confront the greatest challenge in all of interpretability: Superposition & Polysemanticity, why individual neurons don’t represent clean concepts, but juggle ten unrelated ideas at once.
Ready for Neural Reverse Engineering? Explore the Flagship Treatise
This series translates the core principles of transformer circuit tracing, superposition, and monosemantic feature extraction into accessible visual models. For mathematical invariants, formal proofs, and PyTorch circuit pruning algorithms, consult our foundational monograph: Mechanistic Interpretability Transformer: 5 Empirical Rules.
Mechanistic Interpretability in AI
An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.