๐ŸŒฟ CuriousWinds

Activation Patching & Causal Mediation in Deep Models

Physical mechanisms, mathematical boundary conditions, and architectural trade-offs in Activation Patching & Causal Mediation in Deep Models.

In human cognitive science, observing brain scans is not enough to prove what a brain region does. An fMRI scan might show that your visual cortex lights up when you hear music, but does that mean you hear with your eyes? Or is it merely an accidental correlation? To establish true causality, neuroscientists rely on intervention studies, stimulating a precise brain region with an electrode or examining patients with focal brain lesions to see which specific function disappears.

In mechanistic interpretability, we have an enormous advantage over human neuroscientists: we have total read and write access to every floating-point number inside the neural network. We can perform virtual brain surgery in real time. The primary surgical tool for mapping circuits is Activation Patching (also called Causal Mediation Analysis or Interchange Interventions).

In This Article
3 Curated Inquiries • Computing Systems & Machine Cognition
01 TRANSFORMER CIRCUITS ยท SURGICAL MAPPING

The 3-Step Surgical Protocol

Consider a model completing the sentence: “When Mary and John went to the store, John gave a drink to..” The clean model correctly predicts Mary. How does the model know to choose Mary instead of John? Activation patching answers this with surgical precision:

Causal Protocol โ€ข Activation Patching

The Clean, Corrupted, and Patched Forward Passes

Step 1: Clean Run (Ground Truth Baseline)

Feed the original prompt: “When Mary and John went to the store, John gave a drink to..” Save all intermediate activation vectors at every layer and attention head. Model output: Mary.

Step 2: Corrupted Run (Perturbed Counterfactual)

Feed the corrupted prompt: “When Alice and John went to the store, John gave a drink to..” Model output: Alice.

Step 3: Surgical Patch (Causal Transfusion)

Run the corrupted prompt again, but pause at Layer 9, Head 6. Overwrite that single head’s activation vector with the vector saved from the clean Mary run! If the output flips back from Alice to Mary, you have proven that Head 9.6 causally controls indirect object identification!

02 ATTENTION NETWORKS ยท INDIRECT OBJECT GRID

The Indirect Object Identification (IOI) Circuit

In 2022, Kevin Wang and Arthur Conmy used activation patching to completely map the Indirect Object Identification (IOI) circuit in GPT-2 Small. Out of 144 attention heads, they isolated a crisp, 26-head subgraph working in concert:

Head Group Quantity Computational Role
Duplicate Token Heads 5 Heads Detects that ‘John’ appeared twice in the text and flags it as a duplicate name.
S-Inhibition Heads 7 Heads Suppresses attention to ‘John’ so the model focuses on the unique name (‘Mary’).
Name Mover Heads 3 Heads Extracts ‘Mary’ from the early stream and writes it directly to the output logit prediction!
03 CAUSAL INFERENCE ยท EMPIRICAL VALIDATION

Causality Without Guesswork

Activation patching transforms interpretability from speculative philosophy into an exact empirical science. We no longer ask: “Does this head correlate with grammar?” We prove: “If we silence this head, grammar fails by exactly 42%.”

In Part 8, we take surgery one step further: patching live activations and permanently rewriting facts inside model weights with The ROME Algorithm.

Advance Your Knowledge

Ready for Neural Reverse Engineering? Explore the Flagship Treatise

This series translates the core principles of transformer circuit tracing, superposition, and monosemantic feature extraction into accessible visual models. For mathematical invariants, formal proofs, and PyTorch circuit pruning algorithms, consult our foundational monograph: Mechanistic Interpretability Transformer: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Mechanistic Interpretability in AI

An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899