[0:00:00] Over the last couple of videos, we've been trying to understand two-layer attention-only transformers. And as we've done that, this really curious type of thing that we've been calling a virtual attention, it sort of just falls out of the math. And even though it wasn't very important in understanding our previous, the models that we've looked at so far, I do think it's a really interesting idea and potentially a really powerful one that I sort of expect at some point to be important, even though I haven't really empirically observed that yet.
[0:00:30] And so, yeah, I just wanted to briefly chat about them. So, yeah, when we just tried to describe our second layer, our two-layer attention-only transformer, we sort of ended up with these three kinds of terms. So there was the direct path term. There was the individual attention head terms. And then there's this other one that corresponds to going through chains of attention heads. So you go through
[0:01:02] One attention head, and then another attention head, and then you hit the unembedding. And we called those virtual attention heads. And the thing that's really interesting about virtual attention heads, well, so what is an attention head fundamentally? An attention head, I think the fundamental way to think about an attention head is that it moves information from the residual stream of one token to the residual stream of another token. And in order to do that, it needs two things.
[0:01:34] To know which tokens it's attending to and from, and then which subspace it's going to read in and where it's going to write it in the resulting token's residual stream. And these virtual attention heads, even though they aren't represented explicitly in the model like an attention head is, they have exactly the same properties an attention head does. They have an attention pattern, which is just the composition of the two attention patterns.
[0:02:04] If the second attention head looks at token 3 and then the first attention head looks from 3 to token 2, then on net you look at token 2. And so there's this composition of attention patterns that's potentially very expressive. And then they have an OV circuit, just like our regular attention heads, and that OV circuit
[0:02:34] Describes how, if a virtual attention head attends to a given token, how it will affect the outputs. And this sub part here, this product in the middle, that tells us on, you know, what residual stream does it read in and then write out? Or what subspace does it read in from the residual stream right at the beginning? And then where does it write to at the end? So I've sort of been rambling and maybe you're kind of surprised because, you know, I'm just sort of
[0:03:12] Observing this mathematical term, and I don't really have evidence that it's important, and I'm just sort of excited about it and ranting about why I think it's important. So let me try to defend a little bit why I think this structure is potentially so important and so powerful. And I think the key reason for me, really above anything else, is that it's composition. And it's the composition of something that seems to be a very powerful and express...
[0:03:42] Very powerful primitive. And so when you compose them together, it seems very expressive. Let me give you an example of just a simple example of what I mean by composition first. So oftentimes, we'll see attention heads attend to the previous token. And we'll often see multiple attention heads that attend to the previous token. But we...
[0:04:12] Very seldom see attention heads that attend to the second previous token. Very rarely. We'll see attention heads that go and attend to, sometimes they attend to the token two beforehand, but very rarely attention heads that attend sort of in a fixed way to the token two steps before them. And so you might ask, well, why is that? And I think the answer is that
[0:04:43] Well, I think one reason at least is that the model doesn't need to do that in order to express, in order to go and express the ability or have the ability to go and get information of the token two steps before. Because it can use virtual attention heads and it can just chain them together. So you have one virtual attention head and then you have the next one and you attend two tokens back. And so you can go and read information from two tokens back or three tokens back or four tokens back.
[0:05:15] By chaining together this primitive, then you don't have to create a dedicated head. And you can, well, we'll talk about the other benefits in a second, but they're potentially expressible. But we have much richer primitives that potentially can be used. So we have, also we'll see attention heads that attend to the start of a clause or the token that is a transition from an outer clause to an inner clause or to a subordinate clause. And we'll also see sometimes attention heads that attend to the previous sentence.
[0:05:43] And so that also seems kind of like an interesting primitive. And you start combining it with things like attention heads that attend to the verb of the present clause or the subject of the present clause or the object of the present clause, then you can do things like, okay, look at the previous sentence and then look at its subject or its verb. Yeah, or look at the outer clause and look at what the subject was. And that seems like it must be a useful thing to be able to do.
[0:06:09] And those are just combining together a few things. That is, you have more and more of these attention heads that you can compose together. You can express all kinds of things, potentially. And that seems potentially really powerful. In fact, as you have more and more, you have n per layer. So the number of things you can express grows exponentially. Now, of course, most of these compositions are probably not very useful. But in theory, you have...
[0:06:38] You have this ability to have an exponential number of virtual attention heads. Now you might say, you know, Christopher, there's only N previous tokens. And so why would it be useful to have more than N, let alone massively more than N attention heads? Well, you're looking at your attention heads at 10 different places for every sentence. And so you would sort of think of it as being like these semantic positions that are, or these functional positions.
[0:07:08] Which happen to be in one position in one sentence and in other positions in other sentences. But for instance, the subject of the first noun in the sentence and the subject of the previous clause may sometimes be the same token, but they may sometimes be different tokens.
[0:07:36] You know, those are different things. And potentially, yeah, you want to be able to express both of those. And so I think it's potentially powerful that we have exponentially many of these virtual attention heads. Okay. The other final thing is they can, in some sense, be smaller than an attention head. So attention heads move information, right? They read in information from the residual stream of one model of one token and then write it to the residual stream of another token. And depending on the size of the value vector, they have to copy, you know,
[0:08:04] 128 dimensions, or at least you're sort of dedicating the capacity to copy 128 dimensions, or whatever the size is that you dedicate to your attention heads. Well, okay. But what if you only want to copy a single dimension or two dimensions? Then you potentially have to waste an entire attention head to go and do that. Well, with virtual attentions, you don't have to do that. A virtual attention head can move a single dimension without wasting anything. It just goes and...
[0:08:33] Grabs one dimension from one of the attention heads that's going to compose it and one dimension from the other attention head that's going to compose it. And that single dimension gets used and the other attention heads still have all of the 128 minus one dimensions left to go and use for other things. And so it sort of is costless to have tiny virtual attention heads.
[0:08:56] Or the cost is no larger than it needs to be to go and have these tiny little virtual attention heads. And that's potentially quite powerful. Okay, so that's my rant on virtual attention heads and why they seem interesting to me. And probably our next video will be a little bit less theoretical.