🌿 CuriousWinds

QK vs OV Circuits in Transformer Multi-Head Attention

Physical mechanisms, mathematical boundary conditions, and architectural trade-offs in QK vs OV Circuits in Transformer Multi-Head Attention.

When students first encounter the mathematical formula for transformer self-attention, Attention(Q, K, V) = softmax((Q · K^T) / √d_k) · V, it looks like an intimidating jumble of linear algebra. Four matrices (W_Q, W_K, W_V, W_O) are entangled in a single equation. For years, practitioners treated this as an inseparable black-box calculation.

In 2021, Anthropic researcher Chris Olah published a breakthrough mathematical factorization called A Mathematical Framework for Transformer Circuits. He proved that every attention head does not perform one complex task, it performs two completely independent, parallel linear operations: the QK (Query-Key) Circuit and the OV (Output-Value) Circuit.

In This Article
3 Curated Inquiries • Computing Systems & Machine Cognition
01 TRANSFORMER ARCH · PARALLEL PROCESSING

The Division of Labor

WHERE vs WHAT

Think of an attention head as a postal delivery worker sorting packages in a busy post office. The worker has two completely separate responsibilities:

Circuit 1 • QK (Query-Key)

WHERE to Look (Routing)

Determines the routing address. The Query asks: “I am a pronoun; where is my antecedent noun?” The Key answers: “I am the subject noun.” The dot product determines the attention weight (how much attention token X pays to token Y). It moves zero data; it only draws the arrow.

Circuit 2 • OV (Output-Value)

WHAT to Write (Content)

Determines the payload. Once the routing arrow is drawn, the Value matrix reads information from the source token, transforms it via the Output matrix, and writes it onto the destination token’s residual stream. It doesn’t care where it goes; it only transforms content.

02 ATTENTION MECHANICS · QUERY-KEY DYNAMICS

The Factorization Formula

W_QK and W_OV

By factoring the full attention calculation, we can express the behavior of any attention head as two low-rank bilinear forms:

Bilinear Tensor Decomposition

Independent Circuit Matrices

W_QK = W_Q · W_K^T   →   (Attention Pattern / Addressing)
W_OV = W_V · W_O   →   (Linear Token Transformation / Payload)
This separation allows interpretability researchers to inspect the QK matrix in isolation to map routing logic, and inspect the OV matrix in isolation to identify what semantic concepts are being transferred.
03 LINEAR ALGEBRA · RANK-FACTORIZATION

Why This Unlocks Circuit Tracing

Before this mathematical factorization, tracing a neural circuit required analyzing the entire billion-parameter model simultaneously. With the QK/OV decomposition, we can decompose an attention head into two clean linear maps: one controlling pointer movements, and the other controlling semantic data flow.

In Part 5, we confront the greatest challenge in all of interpretability: Superposition & Polysemanticity, why individual neurons don’t represent clean concepts, but juggle ten unrelated ideas at once.

Advance Your Knowledge

Ready for Neural Reverse Engineering? Explore the Flagship Treatise

This series translates the core principles of transformer circuit tracing, superposition, and monosemantic feature extraction into accessible visual models. For mathematical invariants, formal proofs, and PyTorch circuit pruning algorithms, consult our foundational monograph: Mechanistic Interpretability Transformer: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Mechanistic Interpretability in AI

An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899