Every time you tap a symbol, it is like dropping a stone in water. Ripples spread out. They fade. If you drop another stone before the first ripples die, the two patterns mix together.
Here is the important part, and it is a trade-off, not a superpower. The past symbols are not filed away one by one. Their effects are stirred into one shared pattern. So the pond remembers the effect of the stones, not the stones themselves.
Some things survive that stirring well: roughly what has been happening lately, repeated patterns, timing, overall direction, and anything the machine was trained to hold on to. Other things get blurred away: an exact symbol far back, which detail belonged to which item, or several similar facts that smear into each other.
And the space never grows. Whether you feed it ten symbols or ten thousand, it is the same small pile of numbers. That fixed size is exactly why it is cheap — and exactly why it must throw things away.
A flat pond does not mean "it forgot everything at some moment." It means the earlier taps no longer leave a detectable trace in the state.
A chatbot that must quote your sentence from an hour ago needs a different kind of memory. Attention keeps a separate record for every past position and can go back and look one up, like flipping through a notebook. This machine cannot do that. It writes one running summary and has to decide what matters before it knows what you will ask later.
So it is weak at: copying things exactly, quoting, keeping many separate facts straight, and matching "which name went with which number." It is strong at: long or never-ending streams, sensors, audio, video, general drift and trend, and running cheaply forever.
This is why the serious designs mix the two — mostly this, plus a little attention for the moments when you need an exact lookup.
Any wave can be split into simple ripples of different sizes — big slow ones and small fast ones. Each dial at the bottom is one of those sizes.
The needle shows where that ripple is in its cycle. The ring shows how strong it still is. Add all the dials together and you get the water you see.
Try it: tap one dial. The busy water becomes one clean ring. Tap it again to bring the rest back. That is the whole idea of this design: the parts do not talk to each other, so you can look at them one at a time.
The water sits still between taps. That is on purpose, not a bug.
This machine reads a sequence — one symbol, then the next. It takes one step forward each time a symbol arrives. It does not run on a clock. Screen time and machine time are different things, and mixing them up is the most common way these diagrams lie.
Underneath, the ripples really are always flowing. You just only get to look at them at symbol times. The Δ slider sets how much flowing happens between looks. Bigger Δ means bigger jumps, so things spin and fade faster. Smaller Δ means finer looks, so the past sticks around longer.
Go to the Learn tab. Set K to 12. That asks: which shape did I feed 12 steps ago?
Press Fit with C only selected. This teaches the machine to read the water, but does not let it change how the water moves. You will get maybe 60–70% right. Guessing would be 25%.
Now switch to A, B, C, Δ and press Fit again. Now it can also change how fast each ripple spins and fades — it can design the water to suit the question. Accuracy usually jumps to the 90s.
That gap is the point. Learning how to remember beats learning only how to read what you happened to remember.
But do not read too much into the score. This test has only four possible answers, one fixed delay, and one question. Guessing gets 25%. Getting 90% here does not mean the design could hold a long conversation. A fair test would use many different keys and values, delays that change, questions the machine could not have prepared for, and exact reproduction. That is where a fixed-size memory starts to hurt — and this small demo deliberately does not go there.
Open the Spectrum tab while it is fitting. Each dot is one ripple. How far it sits from the middle is how much of it survives each step. How far around is how fast it spins.
The hollow rings show where each dot started. Watch them move. That movement is the machine redesigning its own memory.
Notice no dot ever crosses the outer circle. Outside that circle, ripples would grow instead of fade, and everything would blow up. The maths here makes crossing impossible, so it simply cannot happen.
The Scan tab shows two ways to work out the same thing.
The top row is one-at-a-time: sixteen symbols, sixteen steps, each waiting for the one before. That is how it runs when you use it live.
The bottom shape does it in four rounds instead of sixteen, by combining pairs, then pairs of pairs. That works because the steps can be grouped in any order without changing the result — like adding numbers. It only works because each step is simple; make it fancier and you lose it.
Both give the exact same answer, and the panel prints the difference so you can check. This is the trick that matters: learn quickly in big parallel batches, then run cheaply one step at a time.
The usual approach — attention, the thing in most chatbots — keeps a record of every symbol it has seen. Twice as much text, twice as much stored. Ten times as much, ten times as much stored. It never stops growing.
This machine keeps the same small amount forever — because it keeps a squeezed summary rather than a record of every symbol. That is the whole bargain: constant cost in exchange for accepting that some detail is unrecoverable.
But be fair about it: the fixed amount starts out bigger. For very short inputs the growing one is actually smaller. It only wins after about 16 symbols. Drag the slider and watch them cross. The honest claim is not "smaller" — it is stops growing.
Everything so far happens inside one layer. But the reason the rest of an AI system exists is the limit you just saw: this memory is small and lossy, so anything it cannot hold has to live somewhere else. Here is that whole world as one piece of ground.
The land is the base model. The frozen weights — the shape of the ground before anyone digs. Everything else sits in it or flows across it.
Ponds are adapters. Dig a different hollow in the same land and you get a different pond: same ground, different shape, different behaviour. That is what a small fine-tune is — the base is untouched, only the shape on top changes. This is why you can keep a library of adapters and swap between them without re-downloading anything.
The grass at the edge is the protocol (MCP and its kin). It is the margin you cross to reach the water — the same approach whatever lies beyond it. Grass stores nothing; it is a way in. That matters: a protocol is not a kind of memory, it is the standard doorway that outside things arrive through.
An inlet from a distant lake is retrieval (RAG). The library of documents is a lake somewhere else entirely, far bigger than any pond. Retrieval opens a channel and lets some of it flow in. This also names the weakness honestly: you only get what you channel, and you have to know which lake to open. Nothing arrives that you did not ask for.
The watershed is the graph. Water does not arrive from nowhere — it comes down connected paths. Structuring your sources as a graph keeps those connections, so what flows in arrives related rather than as a pile of disconnected buckets. That relatedness is the thing plain retrieval loses.
The region is a council. Several ponds, compared. Different bases and different adapters answering the same question, so you can see where they agree and where they part. Note this is a different question from all the others: everything above asks what does the model know; a council asks whose answer do we trust. It is worth keeping those apart in your head, because mixing them is how architecture diagrams turn into soup.
One useful way to read the whole map: these are layers of memory sorted by how permanent they are and what they cost. Baked into the weights is permanent and free to use but slow to change. The pond state is this moment, cheap and fading. The notebook of attention is this conversation, exact but growing. The lake is everything else, unlimited but only reachable on request.
What you have here is one layer. A real model stacks many, with a couple of standard parts between them:
• A normalise step and a shortcut that adds the input back — both just keep deep stacks trainable.
• A feed-forward block (widen, bend, narrow again). Unchanged from a normal Transformer — this part is not replaced.
• A short local filter before the memory, for immediate neighbours, which smooth ripples handle badly.
One thing you can delete: position markers. Transformers need to be told which symbol came first, because attention sees an unordered pile. This machine gets order for free — symbol five arrives after symbol four, and that is what order means here.
One thing worth keeping: a little attention. Memory like this smooths things together, so it is poor at quoting an earlier symbol exactly. The best current designs use mostly memory layers with one attention layer every six to eight.
S4 — Gu, Goel & Ré, "Efficiently Modeling Long Sequences with Structured State Spaces," ICLR 2022. arXiv:2111.00396. Source of the continuous system, the discretization step, and the diagonal-plus-low-rank fix for the cost problem.
Mamba — Gu & Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," COLM 2024. arXiv:2312.00752. Makes Δ, B and C depend on the input, so the model chooses what to remember.
S4D — Gu, Goel, Gupta & Ré, "On the Parameterization and Initialization of Diagonal State Space Models," NeurIPS 2022. arXiv:2206.11893. The a = −ep trick used here.
DSS — Gupta, Gu & Berant, "Diagonal State Spaces are as Effective as Structured State Spaces," NeurIPS 2022. arXiv:2203.14343. The result that a plain diagonal was enough after all.
Jelassi, Brandfonbrener, Kakade & Malach, "Repeat After Me: Transformers are Better than State Space Models at Copying," ICML 2024. arXiv:2402.01032. Proves a fixed-size state is fundamentally limited at copying and retrieval, while a two-layer transformer is not. This is the paper behind the "what it is bad at" section.
Griffin — De et al., 2024. arXiv:2402.19427. Gated recurrence plus local attention.
Jamba — Lieber et al., 2024. arXiv:2403.19887. Interleaved Transformer and Mamba layers.
RWKV — Peng et al., EMNLP 2023. arXiv:2305.13048. Trains in parallel, runs as an RNN.
Unitary RNNs — Arjovsky, Shah & Bengio, ICML 2016. arXiv:1511.06464. Rotation with magnitude 1, to stop gradients dying or exploding.
Vanishing/exploding gradients — Pascanu, Mikolov & Bengio, ICML 2013. arXiv:1211.5063.
LSTM gating — Hochreiter & Schmidhuber, Neural Computation 9(8), 1997.
LRU — Orvieto et al., ICML 2023. arXiv:2303.06349 · RoPE — Su et al., 2021. arXiv:2104.09864 · coRNN — Rusch & Mishra, ICLR 2021. arXiv:2010.00951.
Vision Mamba — Zhu et al., 2024. arXiv:2401.09417 · VMamba — Liu et al., 2024. arXiv:2401.10166.
DLinear — Zeng et al., "Are Transformers Effective for Time Series Forecasting?" AAAI 2023. arXiv:2205.13504. The baseline any forecasting claim has to beat.
Retrieval (RAG) — Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020. arXiv:2005.11401.
Graph-structured retrieval — Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," 2024. arXiv:2404.16130.
Adapters on a frozen base — Houlsby et al., "Parameter-Efficient Transfer Learning for NLP," ICML 2019, arXiv:1902.00751; Hu et al., "LoRA," ICLR 2022, arXiv:2106.09685.
Model Context Protocol — open standard for connecting models to external tools and data. modelcontextprotocol.io.
Councils / multi-model debate — Du et al., "Improving Factuality and Reasoning in Language Models through Multiagent Debate," 2023. arXiv:2305.14325.
The Annotated S4 — Sasha Rush & Sidd Karamcheti. srush.github.io/annotated-s4. Code-level, where this page is picture-level.
hₜ = λhₜ₋₁ + uₜ is associative, partial results compose — (a₂,b₂)∘(a₁,b₁) = (a₁a₂, a₂b₁+b₂) — so the whole sequence resolves in log₂T dependent rounds on parallel hardware. Both produce identical states; the max difference is printed above. This duality is the trick: train like a transformer, infer like an RNN. Break associativity — make the update polynomial in h — and only the slow column survives.