One attention head can point at one place. Ask it three questions at the same time and it has to choose. Give it more heads and each one goes to a different place. That is all "multi-head" means — and you can watch it happen.
feed symbols
three questions, asked at every step
At each new symbol the model must answer all three at once: what came 4 back, 13 back, and 25 back. One place to look is not enough.
press Teach — then try it again with 1 head
where each head is looking
Each row is one head. The bars show how much attention it is paying to each earlier position, newest on the right. The dashed lines mark the three answers. After teaching, watch each head park itself on a different line.
What you should find
Set heads to 1 and teach. It picks one distance and gets that question right. The other two collapse toward guessing (25%). It is not stupid — it physically has one place to look, and a single blended answer cannot serve three different questions.
Now set heads to 3 and teach again. All three questions go to 100%, and the head rows separate — they spread out to cover the different distances. Nobody told them to divide the work that way; it falls out of the training, because dividing is the only way to answer everything at once. The split is not always tidy: a head can sit between two answers and still carry enough for both, so you will sometimes see two heads prefer the same distance while all three questions are still answered.
Try 4 heads. Still 100%, and now a head is spare — you will usually see two settle on the same distance. Real models have this too: a large fraction of heads can be pruned with little loss.
heads
4 back
13 back
25 back
what the heads learned
1
~35%
~23%
~99%
one place only: [25]
2
100%
99%
~87%
[4, 13] — third question suffers
3
100%
100%
100%
[25, 4, 25] — covers all three
4
100%
100%
100%
[13, 25, 4, 13] — one doubles up
Measured by this page. Chance is 25%. Your runs will vary — which head takes which distance is arbitrary, and heads sometimes share or double up. What is reliable is the shape: one head cannot serve three questions, and enough heads can.
What a head actually is
A head is three small tables of numbers. One turns the current symbol into a question, one turns every past symbol into a label, and one turns every past symbol into the content to hand back. The question is compared against every label, the matches become weights, and the contents are blended by those weights. That blend is the head's answer.
Heads run side by side, not in sequence, on the same input. Their answers are stuck together end to end and passed on. Because they share nothing while they work, adding heads costs width, not depth — which is why hardware likes them.
In this page each head also learns a plain preference for distance — "look about this far back" — which is what makes the specialization so visible. Real heads mostly match on content: this word looks for its verb, that one for a matching bracket. Same machinery, harder to draw.
Honest limits
The task is built to reward specialization. Three fixed distances is close to the friendliest possible case for showing heads divide work. Real heads are messier, often overlapping, and frequently hard to label at all.
One layer, four symbols, distance-based matching. Real attention stacks many layers, matches on content, and adds an output projection that mixes head results before passing them on. The mechanism here is faithful; it is a simplification, not a miniature GPT.
Every head still stores an entry per symbol. More heads means more memory per symbol and more comparisons per step. That cost is the reason the pond-style fixed-state models exist at all.
Multi-head attention — Vaswani et al., "Attention Is All You Need," NeurIPS 2017. arXiv:1706.03762. Heads run in parallel on the same input and their outputs are concatenated.
The learned distance preference is a relative-position bias, as in Raffel et al. (T5), arXiv:1910.10683, and in spirit ALiBi, Press et al., arXiv:2108.12409.
Redundant heads — Michel, Levy & Neubig, "Are Sixteen Heads Really Better than One?", NeurIPS 2019. arXiv:1905.10650. Many heads can be pruned at little cost, which is what the 4-head run hints at.
Heads with identifiable jobs — Voita et al., "Analyzing Multi-Head Self-Attention," ACL 2019. arXiv:1905.09418; and Olsson et al., "In-context Learning and Induction Heads," 2022. arXiv:2209.11895.
Why fixed-state models exist — Jelassi et al., "Repeat After Me," ICML 2024. arXiv:2402.01032.
Not from the literature. The three-distance task is designed to make specialization legible; it is a teaching setup, not a benchmark. The numbers come from this page's own code and should not be quoted as results. By Daniella M. LaGuerre, August 2026.