🌿 CuriousWinds

Mechanistic Induction Heads & In-Context Pattern Copying

How two-layer attention subcircuits implement in-context pattern replication and prefix matching.

Imagine you invent a completely made-up fantasy word: “Blorpax”. You type the following prompt into an AI model: “In the ancient kingdom, a warrior named Blorpax picked up his sword. Suddenly, an enemy soldier charged at Blorpax. With a mighty swing,..” If you ask the model to predict the next word, it will effortlessly output: “Blorpax”.

Think about how miraculous that is. The word “Blorpax” never appeared anywhere in the model’s 15-trillion-token training dataset. No gradient descent step ever updated a single weight to associate “Blorpax” with warriors or swords. Yet inside the running context, the model instantly learned the pattern and completed the sequence. This capability is called In-Context Learning. In 2022, Anthropic researchers Catherine Olsson, Nelson Elhage, and Chris Olah discovered the microscopic engine that powers it: Induction Heads.

01 1 · TRANSFORMER CIRCUITS · INDUCTION MECHANISMS

The Mathematical Algorithm

[A][B].. [A] -> [B]

, an induction head is a computational circuit that executes a simple, universal sequence-completion heuristic:

Visual Mechanism • The Induction Rule

The [A][B].. [A] → [B] Circuit

Whenever the circuit sees token [A] in the prompt, it searches back through history to find where token [A] appeared previously, looks at the token that came immediately after it ([B]), and predicts that [B] will follow [A] again!

.. [Harry] [Potter].. [Harry] → [Potter]
02 2 · ATTENTION ARCHITECTURE · LIMITED SCOPE

Why One Layer Is Not Enough

The Two-Head Handshake

A single attention head cannot execute an induction circuit on its own. Why? Because a standard attention head can only look at tokens that exist in the prompt; it cannot look at the relationship between pairs of tokens. An induction circuit requires a two-layer handshake between two specialized attention heads:

Two-Layer Circuit Topology

The Previous-Token & Induction Head Handshake

Layer 1: The Previous-Token Head

An attention head in Layer 1 copies information from token [A] and writes it onto the residual stream of token [B]. Now, token [B] carries a tag saying: “I was preceded by [A]”.

Layer 2: The Induction Head

An attention head in Layer 2 attends to any token in history that carries the “I was preceded by [A]” tag. When the prompt reaches a new [A], it grabs token [B] and writes it onto the residual stream as the next prediction!

03 3 · TRAINING DYNAMICS · PHASE TRANSITIONS

The Phase Transition

When AI 'Wakes Up'

During model training, induction heads don’t emerge gradually. They appear abruptly in a sudden phase transition. Early in training, a model’s training loss improves slowly as it memorizes word frequencies. Then, around token 2.5 billion, the training loss suddenly plummets, and induction heads form across multiple layers simultaneously.

This phase change marks the exact moment the neural network transitions from a dumb frequency counter to an in-context learning engine capable of following instructions, translating languages, and solving few-shot examples.

In Part 4, we zoom into the mechanics of the attention head itself: The QK vs. OV Circuit, dissecting how heads decide WHERE to look versus WHAT to write.

Advance Your Knowledge

Ready for Neural Reverse Engineering? Explore the Flagship Treatise

This series translates the core principles of transformer circuit tracing, superposition, and monosemantic feature extraction into accessible visual models. For mathematical invariants, formal proofs, and PyTorch circuit pruning algorithms, consult our foundational monograph: Mechanistic Interpretability Transformer: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Mechanistic Interpretability in AI

An empirical exploration through neural residual streams, induction circuits, polysemanticity, and causal weight editing.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899