Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.
It's definitely part of many intro explanations, but it sits in a weird position where you have to know MLPs well but not Transformers for this to be enlightening, which is a shrinking set of people nowadays. Either someone is a novice to all this, or know both, so beginner Transformer explanations can rarely assume deep familiarity with MLPs.
But yes, a Transformer block can be thought of as basically input-dependently deciding the weight vector of a dense layer. In classical MLP-like networks there was never any multiplication between input-dependent values (there are exceptions, like Squeeze-and-Excitation layers). Multiplication is always between parameters (that don't depend on the input) and activations that depend on the input. With Transformers attention provides a lot of multiplicative interactions between input dependent activations.
Also obligatory: Schmidhuber talked about this a long time ago.
Welch Labs has a beautiful visualization in their YouTube video about Vision Language Action Models, where the attention of the prompt can be traced to the exact origin pixels in the image from one of those matrices.
Look at this poster [1] (its low-res, full res is paid). Also see this video for context [2] - it talks about deepseek's innovation, but explains attention well.
Above I was pointing to the moment where multiplication AV happens. In normal Dense layer in MLPs when you go through the layer you compute y=Wx, where x is an input and y is output (before gactivation) and W is a weight matrix. This W is usually what's produced through training process. This multiplication AV looks like Wx. If we take one column from matrix V in the poster and call it v, we can pretend for one moment that W=A and x=v [3]. So this multiplication Av works like linear transformation in ordinary network Wx. But in case of attention heads A is not trained directly but produced during inference, and is not trained directly like W is in ordinary network. In this case network is trained to produce A during inference.
The reason it's rarely explained is that Schmidhuber discovered this a long time ago (Fast Weight Programmers). And for whatever reason, people take perverse pleasure in not citing him.
reply