In human cognitive science, observing brain scans is not enough to prove what a brain region does. An fMRI scan might show that your visual cortex lights up when you hear music, but does that mean you hear with your eyes? Or is it merely an accidental correlation? To establish true causality, neuroscientists rely on intervention studies, stimulating a precise brain region with an electrode or examining patients with focal brain lesions to see which specific function disappears.
In mechanistic interpretability, we have an enormous advantage over human neuroscientists: we have total read and write access to every floating-point number inside the neural network. We can perform virtual brain surgery in real time. The primary surgical tool for mapping circuits is Activation Patching (also called Causal Mediation Analysis or Interchange Interventions).
The 3-Step Surgical Protocol
Consider a model completing the sentence: “When Mary and John went to the store, John gave a drink to..” The clean model correctly predicts Mary. How does the model know to choose Mary instead of John? Activation patching answers this with surgical precision:
The Clean, Corrupted, and Patched Forward Passes
Feed the original prompt: “When Mary and John went to the store, John gave a drink to..” Save all intermediate activation vectors at every layer and attention head. Model output: Mary.
Feed the corrupted prompt: “When Alice and John went to the store, John gave a drink to..” Model output: Alice.
Run the corrupted prompt again, but pause at Layer 9, Head 6. Overwrite that single head’s activation vector with the vector saved from the clean Mary run! If the output flips back from Alice to Mary, you have proven that Head 9.6 causally controls indirect object identification!
The Indirect Object Identification (IOI) Circuit
In 2022, Kevin Wang and Arthur Conmy used activation patching to completely map the Indirect Object Identification (IOI) circuit in GPT-2 Small. Out of 144 attention heads, they isolated a crisp, 26-head subgraph working in concert:
Causality Without Guesswork
Activation patching transforms interpretability from speculative philosophy into an exact empirical science. We no longer ask: “Does this head correlate with grammar?” We prove: “If we silence this head, grammar fails by exactly 42%.”
In Part 8, we take surgery one step further: patching live activations and permanently rewriting facts inside model weights with The ROME Algorithm.
Ready for Neural Reverse Engineering? Explore the Flagship Treatise
This series translates the core principles of transformer circuit tracing, superposition, and monosemantic feature extraction into accessible visual models. For mathematical invariants, formal proofs, and PyTorch circuit pruning algorithms, consult our foundational monograph: Mechanistic Interpretability Transformer: 5 Empirical Rules.
Mechanistic Interpretability in AI
An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.