{"url": "https://www.youtube.com/watch?v=UM-eJbx_YDk", "page_url": "https://www.youtube.com/watch?v=UM-eJbx_YDk", "title": "2L Attention - Theory [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 561, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-2l-attention-theory-rough-early-thoug", "audio_duration_s": 560.2394375, "worker": "jc-w06", "transcribe_s": 7.8, "model": "large-v3", "segments": [{"start": 0.46, "end": 31.07, "text": "In the last few videos, we've had some success trying to understand one-layer attention-only transformers. And so in this video, we're going to move on and try to study two-layer attention-only transformers. Now, this first video on that topic is going to just be about some theory that we need to build up to do that. But I think it is really quite worthwhile. I found that the ideas that we can build up studying two-layer attention-only models"}, {"start": 31.07, "end": 63.71, "text": "Give us some useful traction on thinking about transformers more generally. And we'll also, when we in the video following this one, get to the point of actually empirically studying what's going on inside these models, we'll find a bunch of things that seem to be really important mechanisms that exist in transformers of all sizes and without any of the simplifications that we're using right now. And so I think it's pretty interesting to study these."}, {"start": 63.71, "end": 91.47, "text": "Recall that in the previous videos, we developed a pretty nice equation for describing transformers. We were able to go and describe them in terms of, well, first we have this direct path term that corresponds to this path here. And then we have these terms corresponding to attention heads, which correspond to all of these paths."}, {"start": 92.14, "end": 121.9, "text": "And we found that there was a really nice interpretation of both of these. This could sort of be seen as being something like bigram statistics. And this term here tells us if an attention head attends to a particular token, what is the effect on the logits? And this tells us where the attention head attends. So that was pretty useful. And another way that we could write that is we could say, well,"}, {"start": 121.9, "end": 156.11, "text": "We could put it sort of in a factored form. We could say, well, OK, the first thing you do is you multiply by we, so first to embed. Then we apply the attention heads, and there's the identity path as well. And then finally, we do the unembedding. So once you've done that, it's pretty easy to generalize it to a larger model or to a model with more attention layers. We're just going to have multiple copies of this term, basically. Now, you might have noticed that every time"}, {"start": 156.11, "end": 184.26, "text": "We talk about WO. It always comes with WV and vice versa. And every time we talk about WQ, it always comes with WK. And because these equations are going to get a little bit larger and more complicated, for simplicity, we're just going to introduce these terms WQK and WOV that correspond to those products. So that's just a little bit of cleanup before we move on to the equations. Okay, so here we have..."}, {"start": 184.26, "end": 223.82, "text": "Our two-layer attention-only transformer. So we start by going and applying the embedding. So that's at the bottom here. Then we're going to go and talk about the first attention block. So that's all of this. And within that, we have the identity term that corresponds to this path. And then all of these attention head terms that correspond to these paths. And then we have the second layer attention heads. And then finally, we have..."}, {"start": 223.82, "end": 257.81, "text": "Our good old unembedding. Now, we'd actually, even though this is sort of an easy way to go and describe it, we'd like to expand this because expanding it will give us, I think, a more helpful way to think about the actual mechanics of the system. And we're going to take advantage of this really nice property. We sort of implicitly talked about it earlier that when we have tensor products, and if you're not comfortable with tensor products,"}, {"start": 257.81, "end": 289.06, "text": "Make sure you watch the video on the theory of one-layer attention heads where we talk about these a little bit more. But when we have tensor products like this, the items on one side of the tensor product, and we multiply them, the items on one side of the tensor product multiply together. And similarly, the ones on the left-hand side end up going together as well. This is called the mixed product identity. And it's really actually the whole reason why I decided to frame"}, {"start": 289.06, "end": 319.18, "text": "This series in terms of tensor products and in the case of attention heads in particular there's a really beautiful interpretation of this which is that if you chain really we're only going to do this when these are attention heads and we're multiplying them together so if you chain two attention heads together the attention patterns combine and create a new attention pattern and the matrices that describe where the attention head is reading from and writing to also multiply together and you get something that looks very much like an attention head so that's the reason we care about that."}, {"start": 319.79, "end": 350.85, "text": "Okay, so now we can write the expanded form of that equation. So we have first, well, we're going to have sort of three types of terms. So we have the direct path term. And this is, we've seen this term all the way back to our zero layer transformer. It's a good old friend, and it just corresponds to this direct path down the transformer. And it tends to represent bigram-ish statistics. Some of the bigram statistics will start to migrate into attention hoods,"}, {"start": 350.85, "end": 383.7, "text": "The kinds of things that it represents are similar to bigram statistics. Then we'll have terms that correspond to going through a single attention head, and that could go through an attention head in the second layer, or could also go through an attention head in the first layer. So these are the effects of attention heads. And then finally, we have what I'm going to call virtual attention heads. So virtual attention heads are when you have two attention heads, the composition of two attention heads."}, {"start": 384.5, "end": 413.39, "text": "And that also has some effect on the output. And so virtual attention heads have this nice property that they have, yeah, we just get them through the mixed product identity that we were talking about earlier. So we get this attention pattern. They have an attention pattern of their own, which is the product of the two attention patterns. And they have a OV circuit of their own that describes if they attend to a particular token, what the effect will be. And it's just the product of the..."}, {"start": 413.39, "end": 441.07, "text": "The first OV circuit and the second OV circuit, or the OV matrices at least. Oh, so that's what we had. Okay, so a question you might, well, okay, stepping back for a second. I think one of the things that's really cool about this is it really allows us to study attention heads in a principled way. So I think a lot of the time, there's been a lot of papers that I think are, they're genuinely super cool papers where people go and study."}, {"start": 441.07, "end": 469.7, "text": "Attention patterns, and they're like, you know, we found an attention head that appears to attend from this to this, and maybe like it attends to the subject of the sentence or something like this, or if it's a verb, it attends to the subject or something like that. But it's actually pretty tricky to know, well, there's sort of a conceptual problem, which is it's very tricky to think about attention heads in isolation. They could be, the attention heads could be reading information for other attention heads, and they could be, you know, they, so they, you know, it might."}, {"start": 469.7, "end": 499.1, "text": "And so it's very easy to be, I think, potentially to be confused about this or at least to worry that you might be being confused about this."}, {"start": 499.1, "end": 528.82, "text": "And it seems to me that, at least for the attention-only case, this framework resolves that. Because if it was the case that the important thing was these change attention heads, then those would be the virtual attention heads. And that would resolve all of these. And the individual attention head terms would sort of end up being small. And that would completely resolve it. So that made me really happy because I think that I felt very uncomfortable that when I've"}, {"start": 528.82, "end": 558.32, "text": "When I've seen people talking about transform interoperability for a long time has been this concern about chains of attention heads and whether attention patterns are really important or whether they're just sort of illusions that are parts of a much longer chain and that we're missing the whole story. And it feels really good to have a framework that puts us on steady ground with respect to that concern. Okay, so in our next video, we'll be able to go and actually start studying these."}]}