🌿 CuriousWinds

Transformer Induction Heads & Circuit Reversal

Reverse engineering in-context copying behaviors, QK-OV circuit decomposition, and attention head subcircuits.

Executive Summary & Key Takeaways
Foundational Synthesis for Practitioners & Research Systems

Transformers encode interpretable circuits from attention weights, not random noise, empirical evidence shows induction heads enable sequence-aware generalization via structured cross-token dependencies, debunking memorization myths.

01 Induction Head Rule

Two-head attention subgraphs predict `[B]` given `[A][B].[A]` via non-local dependencies, achieving O(1) memory for novel associations.

02 Compiler Hierarchy

Training refines weights into circuits: features → arithmetic operations, mapping linear inputs to algorithmic behaviors.

03 Polysemanticity Penalty

Single-layer attention collapses generalization into memorization, forcing O(n) token dependency when induction heads degrade.

04 Circuit Audit Rule

Probe for `[A].[A] → [B]` patterns; maintain <10% head sparsity and ≥4096 residual dimensions.

01 NEURAL SYSTEMS · INDUCTION GAPS

The Black Box Fallacy

Neural Networks as Compilers, Not Oracles

For the first decade of deep learning, neural networks were treated as opaque statistical black boxes. We understood the calculus of backpropagation and the gradient descent that tuned billions of floating-point weights, but we could not explain why a specific matrix multiplication produced a coherent syntactic parse or a correct medical diagnosis. In industry and academia alike, the consensus was resigned: large language models were simply too high-dimensional for human inspection.

Mechanistic interpretability overturns this defeatist assumption. Pioneered by researchers at Anthropic and Redwood Research, mechanistic interpretability treats neural networks not as mysterious biological brains or inscrutable oracles, but as compiled computer programs.

When a transformer trains on billions of tokens, stochastic gradient descent does not build an amorphous soup of connections. Instead, it compiles discrete, human-understandable computational circuits, subgraphs of attention heads and multi-layer perceptron (MLP) neurons that perform precise algorithmic operations like pointer copying, indirect object identification, and in-context induction.

SYSTEM TOPOLOGY & ARCHITECTURE
Four-Stage Pipeline for Automated Circuit Discovery (ACDC)
NeurIPS 2023 · Conmy et al.
Stage 01 · Contrastive Runs
Clean vs. Corrupted Inputs
Feed parallel inputs into the model: a baseline success prompt and a counterfactual prompt where key tokens or references are flipped.
Stage 02 · Causal Intervention
Activation Patching
Replace head activations in the corrupted run with clean-run tensors. Measure recovery delta: ΔRecovery = [Logitclean − Logitcorrupted] / MaxDiff.
Stage 03 · Causal Scrubbing
Path Expansion & Pruning
Zero out non-contributing heads with mean activations. Verify the isolated circuit retains ≥95% accuracy while occupying <5% of total parameters.
Stage 04 · DAG Compaction
Automated Graph Compaction
Compile a verified, minimal Directed Acyclic Graph (DAG) with <30 active computational edges representing the true mechanistic circuit.

1. Activation Patching (Interchange Intervention)

Activation patching is the causal gold standard of interpretability. Rather than looking at passive correlations, the researcher runs the model on two contrastive inputs: a clean input where the model succeeds, and a corrupted input where key variables are flipped.

By surgically slicing into the transformer’s execution graph and copying the intermediate activation tensor of a single attention head from the clean run into the corrupted run, the researcher directly measures causal power. If patching a single head on Layer 9 restores 70% of the clean output probability, that head is a core functional component of the task circuit.

2. Path Expansion & Causal Scrubbing

Isolated heads rarely act alone; they communicate through the residual stream, which is a shared memory bus. Path expansion maps how information flows between specific pairs of heads:

  • Does Head 9.4 write directly to the residual stream to be read by Head 10.7?
  • Or does it modulate the query vector of an intermediate MLP layer?

Through causal scrubbing, researchers systematically scramble all pathways that do not match a hypothesized circuit diagram. If the model’s accuracy on the task remains intact despite extensive scrambling of non-circuit pathways, the hypothesized computational graph is formally verified.

3. Automated Circuit Discovery (ACDC)

Manual activation patching across large models requires thousands of GPU hours. The Automated Circuit Discovery (ACDC) algorithm (Conmy et al., NeurIPS 2023) automates this pipeline by framing circuit discovery as an edge-pruning optimization problem on the computational graph. ACDC systematically tests and prunes edges whose contribution to the loss falls below a strict threshold tau, converting a 32-layer transformer into a clean, human-readable circuit diagram containing fewer than 30 active edges.

04 TRANSFORMER CIRCUITS · POLYSEMANTIC LOADING

Engineering Trade-offs

Polysemanticity and the Superposition Bottleneck

While mechanistic interpretability has successfully mapped induction circuits, indirect object identification (IOI) circuits, and gender-bias subgraphs, researchers face a profound physical barrier: polysemanticity.

In standard computer architectures, a single memory address stores exactly one variable. In neural networks, an individual neuron does not correspond to a single concept. A single neuron in Layer 14 might fire for:

  1. German syntax structures
  2. Photos of golden retrievers
  3. Python `try/except` error blocks

Why do neural networks exhibit this confusing overlap? The mathematical cause is Superposition (Elhage et al., 2022).

GEOMETRIC COMPRESSION
The Superposition Packing Bottleneck
d_model ≪ N_features
Physical Space vs. Concepts

Physical embedding space is constrained (e.g. d_model = 4,096), whereas real-world concepts exceed N > 1,000,000 features. The Johnson-Lindenstrauss lemma enables packing an exponential number of almost-orthogonal vectors into this smaller subspace.

Polysemanticity Consequence

Because features share dimensions, individual neurons activate for multiple unrelated concepts (polysemanticity). Achieving mechanistic explainability requires training Sparse Autoencoders (SAEs) to unpack superposed representations.

Because the number of features in human language (N gg 10^6) vastly exceeds the physical dimensionality of the model (d_{text{model}} approx 4,096), gradient descent packs features as almost-orthogonal directions in the vector space. The network accepts a small amount of cross-feature interference (noise) in exchange for exponentially greater representational capacity.

The Sparse Autoencoder (SAE) Solution

To untangle superposition, the frontier of mechanistic interpretability has moved from inspecting individual neurons to training Sparse Autoencoders (SAEs) on the residual stream. An SAE expands the d_{text{model}} space into an overcomplete dictionary of 32times to 128times more features, enforcing an L_1 sparsity penalty so that only a tiny subset of features fires for any given token.

Early results from Anthropic (2023, 2024) demonstrate that SAE features are remarkably monosemantic: individual features isolate specific real-world concepts (such as the Golden Gate Bridge, transit route numbers, or security vulnerabilities in C code) with near-zero cross-domain leakage.

05 TRANSFORMER CIRCUITS · INDUCTION HEAD MAPPING

Diagnostic Protocol

Auditing Transformer Circuits in Production

When diagnosing unexpected model hallucinations, prompt injection vulnerabilities, or out-of-distribution reasoning failures, apply this four-step mechanistic diagnostic protocol:

  1. Instrument with TransformerLens: Wrap the PyTorch model in an open-source hook framework (such as `TransformerLens`) to cache all internal residual stream states, attention pattern tensors, and MLP post-activation vectors during inference:
 from transformer_lens import HookedTransformer
 model = HookedTransformer.from_pretrained("gpt2-small")
 logits, cache = model.run_with_cache("When Alice met Bob, Bob gave a key to")
  1. Execute Head Ablation Sweeps: Zero out the value output of suspected attention heads (e.g., induction heads in Layers 5, 7) to evaluate task degradation. If copying behavior collapses while general syntax remains unaffected, the failure locus is isolated to the induction circuit.
  2. Map Logit Lens Projections: Project the intermediate residual stream vector at each layer directly onto the unembedding matrix W_U:
DIRECT LOGIT ATTRIBUTION
Logit Lens Intermediate Readout
Intermediate Residual Stream Logit Lens Projection
Layer Predictions = Softmax(LayerNorm(xl) · WU)
xl: Activation state vector of residual stream at intermediate layer l
WU: Unembedding matrix projecting latent representations to vocabulary logits

Identify the exact layer where the model formulates the correct answer versus where a downstream distraction head overwrites it.

  1. Deploy Sparse Autoencoder Clamping: For alignment and safety audits, clamp known unsafe monosemantic SAE features to zero (or negative values) during generation, verifying that dangerous outputs are physically suppressed without retraining the foundational weights.
06 AI SYSTEMS · INDUCTION MECHANISMS

Synthesis

The Dawn of Transparent Silicon Minds

For seventy years, humanity operated under the dogma that artificial intelligence would always be an irreducible mystery, a mechanical genie whose answers could be observed but never verified from first principles.

Mechanistic interpretability proves that intelligence is not an impenetrable mystery. It is a compilation of structured algorithms executing across high-dimensional geometry. From the elegant symmetry of two-layer induction heads that enable in-context learning to the overcomplete feature manifolds extracted by sparse autoencoders, we are developing the debugging tools for artificial minds.

As machine cognition scales into planetary infrastructure, the ability to read the compiled circuits of silicon networks is not merely an intellectual pursuit; it is the fundamental prerequisite for safety, accountability, and truth in computing.

Governing Empirical Axiom

A neural network is not an alien oracle to be worshipped or feared; it is a compiled program waiting to be reverse-engineered.

Curated Inquiry Series • 10 Investigations

Mechanistic Interpretability in AI

An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.

Primary Scholarly Provenance & Empirical Attribution
Olsson, C., et al. (2022). In-context Learning and Induction Heads. Anthropic Research. arXiv:2209.11895
DOI: 10.48550/arXiv.2209.11895
Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread
DOI: 10.48550/arXiv.2209.11895
Conmy, A., et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. Advances in Neural Information Processing Systems (NeurIPS 2023). arXiv:2304.14997
DOI: 10.48550/arXiv.2304.14997
Elhage, N., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread
DOI: 10.48550/arXiv.2209.11895
Nanda, N., & Bloom, J. (2022). TransformerLens: A Library for Mechanistic Interpretability of Generative Language Models. GitHub Repository
DOI: 10.48550/arXiv.2312.00752