{"url": "https://www.youtube.com/watch?v=7crsHGsh3p8", "page_url": "https://www.youtube.com/watch?v=7crsHGsh3p8", "title": "1L Attention - Theory [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 1083, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-1l-attention-theory-rough-early-thoug", "audio_duration_s": 1082.2240625, "worker": "jc-w06", "transcribe_s": 14.4, "model": "large-v3", "segments": [{"start": 0.37, "end": 28.38, "text": "In our previous video, we talked about what's basically the simplest model bearing any resemblance to a transformer you could imagine, the zero-layer transformer. But now we're ready to go and graduate to a more complex model, the one-layer tension-only transformer. Now, it's only a little bit more complex, but it will be able to go and give us some really interesting properties to study, and I think will help us understand"}, {"start": 28.38, "end": 58.82, "text": "Now, we're going to have to work through a non-trivial amount of theory, so it's worth maybe just briefly talking about why this is worth investing in, or why I think there's some payoff to working through this theory. And the biggest reason is that we're going to be able to, in principle, fully understand a toy model. And, well, I guess we often, I feel like we give up on fully understanding neural networks. And here,"}, {"start": 58.82, "end": 84.72, "text": "We're going to be able to go and take something that's sort of a small transformer, and we're going to be able to get to a point where we will just be able to completely understand this model. Now, we'll have to look at some large matrices, and we won't be able to keep it all in our head, but we'll be able to sort of explain all the behavior of this model by consulting some large matrices, and in principle, fully understand it. We're also going to develop some conceptual tools that I think are really useful."}, {"start": 84.72, "end": 107.28, "text": "for reasoning about larger models and will continue to help us as we go forward to larger models. And then finally, actually it turns out that even though larger models and sort of genuine transformers are going to have more complex circuits and more complex behavior, some of the things that we observe in these models"}, {"start": 107.28, "end": 138.13, "text": "These this toy models are going to reoccur and sort of have some echoes and so we'll see them again and that'll we'll sort of get some useful intuition from studying this model. Okay so we're studying a simplified model and as the name suggests the zero layer attention only transformer is uh or sorry the one layer attention only transformer is a one layer model and it's attention only which means that it has no MLPs. But there's two"}, {"start": 138.13, "end": 167.57, "text": "Those are the big simplifications we've made, but there's two other smaller simplifications. We're going to get rid of Liger normalization. And that's just because Liger normalization, we think, probably isn't a critical part of the story, but it would add a lot of bookkeeping and a lot of additional work to think through in our theory. And similarly, we're going to ignore the biases or get rid of the biases, because again, those would add a lot of complexity."}, {"start": 167.66, "end": 203.68, "text": "Now, the model that I'm actually looking at when I, or will be looking at when I give you empirical results later on, and that'll be in another, in the follow-up video to this one, will have, does actually have layer norms and biases, but I'm going to allow them to keep things simple. Okay, so really, if we want to talk about an attention-only transformer, there's three steps, or at a very high level at least, you could summarize it in three steps, which are"}, {"start": 203.68, "end": 237.71, "text": "We're going to go and embed our tokens. So we'll represent our tokens as one-hot vectors, and we'll multiply them by the embedding matrix. Then, for each attention head, we run the attention head and add it into the residual stream. So that corresponds to, we have all these attention heads over here, and we're going to add them into the residual stream that's lying down the middle. And then finally, we're going to go and..."}, {"start": 237.71, "end": 271.01, "text": "Multiply by the unembedding matrix to get the logits. So that's how we'll get the logits, and that's the output of our model. Okay, so, of course, in order to actually understand this, we're going to need to dive into the attention heads in a lot more detail. And a relatively standard way to describe the attention heads might be something like this equation that we have at the top. So what is it saying? Well, it's saying something like, first,"}, {"start": 271.01, "end": 302.19, "text": "We produce the value vectors by multiplying the residual stream by WV. Then we go and we multiply and we go and weight all those value vectors by our attention matrix. So that moves, yeah, goes and combines the value vectors for different tokens. And that weighted combination gets multiplied by WO so we can add it into the residual stream. Okay. Well,"}, {"start": 302.58, "end": 332.78, "text": "That is a formal definition of an attention head. But it's actually a kind of tricky definition to think about. And I think it obscures a lot of important facts that attention heads. And it's just kind of tricky because on the one hand, we have matrix multiplies we have to think about. On the other hand, we have this weighted combination that's kind of orthogonal to them. And it's kind of complex. And there's another way we could describe it, which is using tensor products."}, {"start": 333.04, "end": 361.94, "text": "So you might not be very familiar with TensorFlow products, but they're actually, they're very convenient. And one way to think about them is that they allow us to accomplish the thing that we often accomplish by broadcasting or thinking about multiplying certain dimensions when we program in TensorFlow or NumPy or PyTorch. So yeah, it would be very common for us to say, okay, well, you know,"}, {"start": 361.94, "end": 391.04, "text": "First, we're going to go and multiply the vector at each position for each token by wv. Okay, well, the way we'll write that is we'll say that wv is on the right-hand side of the tensor product and identity on the left-hand side. And that just means we're going to multiply every, when we just have a matrix on the right-hand side, it just means we multiply the vector for every token by that matrix. Okay, well, the next thing we need to do is go and weight things by..."}, {"start": 391.04, "end": 421.01, "text": "The attention pattern. Okay, well, that's instead of going and multiplying the vector for every token independently, now we're going to go and independently mix every component across tokens. So we're multiplying across the token dimension instead of multiplying across the vector dimension or the model dimension or something like that. So we're going to represent that by multiplying on the left-hand side."}, {"start": 421.52, "end": 450.85, "text": "Okay, well then we need to go and multiply by WO and multiply, we're going to multiply every vector, the vector for every token by WO. So that's like the thing that we did at the beginning earlier when we multiplied on the right-hand side. We're going to multiply on the, have WO on the right-hand side and that just means we multiply the vector, every token's vector by WO. And one thing that's nice about tensor products is they have this identity that all the things on the same side"}, {"start": 450.85, "end": 481.14, "text": "Just multiply them together, all the things on the same side combine, and all the things on the other side also combine, and they're just completely independent. So if we want to understand what the, if we want to multiply these together and combine them, we look at, for the right-hand side, we'll go and we'll multiply by WO, and then ID, and WV, and ID collapses, so we just get WO, WV. And on the left-hand side, we have an ID, and then an A, and an ID, so we just get an A."}, {"start": 481.14, "end": 508.99, "text": "We just have the attention pattern. So what that's really saying is that the attention patterns action is sort of independent of the W-O-W-V action. And they're sort of separate things. So that's one way you could think about it. Another way to think about this is, what is an attention head fundamentally doing? An attention head moves information from the residual stream of one token."}, {"start": 508.99, "end": 539.42, "text": "to the residual stream of another token and when it does that it has to pick some subspace of the the the attention that it's moving information from it has to read some subspace and it has to write that to um that's the information that it read the vector that it read to this a different subspace in the residual stream of the of the the second token well w-o-w-v describes which attention head or sorry which which subspace you read from and write to and a"}, {"start": 539.42, "end": 570.51, "text": "Describes which token information moves from and to. So a move describes what the token that gets read from and written to, and w o w v describes the subspace of the token that we're reading from that we read from and where it gets written. So that's a pretty cool way to think about attentions. Okay, so now we can go and plug that into the previous definition we had of an attention layer where we were going and adding all the different"}, {"start": 570.51, "end": 598.8, "text": "Attention heads and then going and adding them to the residual stream. And there's another way we could write that, which is if we fix the attention pattern. Now, the attention pattern is computed in a very nonlinear way. But if we pretend that it's a constant or that it's fixed, well, then all of this is linear. And so we could go and write it as a"}, {"start": 598.96, "end": 632.8, "text": "A linear transformation that acts on X0, where we have an identity plus all of these, well, they're tensors in the mathematical sense. And we can even plug that into our, you know, plug all of our equations together. And we get this final end-to-end description of a transformer in some sense. So the identity term becomes W-U-W-E."}, {"start": 632.8, "end": 665.02, "text": "You can think of that as, well, it's gonna end up being kind of like bigram statistics. And for every attention head, we have this term here. And we'll return to this product on the right-hand side. It's really interesting. And in some ways is one of the most informative things we can use to think about in this model. Okay, so that first term,"}, {"start": 665.02, "end": 688.96, "text": "It corresponds to this direct path down here, this one here, and it's going to tend to represent bigram-like statistics. Remember, in our previous video, we saw an identical term in our zero-layer transformer, and it was exactly the bigram, well, an approximation of the bigram log likelihood. That was exactly what it was trying to do. And here, we're going to see that it's"}, {"start": 688.96, "end": 720.77, "text": "It's going to do something a bit similar. Some of the background information will move into the attention heads, but the remainder will continue to be on that direct path. But we also have terms corresponding to all of these other paths. So those are all going to be in that sum on the right-hand side. And each one of these parts of the sum is a tensor product. And this describes where the attention heads attend."}, {"start": 720.77, "end": 750.64, "text": "And this describes if the attention head attends to a given token, how does that affect the logits? So if we attend to a given token, how do we affect the logits? That's very interesting. That sort of tells us a very large story, a large portion of the story of what this model's behavior is going to be. Now, the thing that we're missing is we still need to understand how the attention patterns get created."}, {"start": 750.64, "end": 767.55, "text": "And then we'll basically have a complete story. I mentioned this earlier, but it's worth noting that if you fix the attention pattern, it's linear. And that's something that we'll be able to get a lot of leverage out of. And in general, I think..."}, {"start": 767.55, "end": 796.98, "text": "Anytime you can go and split a function, a very complex function, but then you can split it into two things where you hold one thing constant, it's a very simple function. If you hold the other thing constant, it's a very simple function. That's a nice point of leverage. And here's an example, a case where we have that kind of leverage. So that's very nice. Okay, so how is the attention pattern computed? Well, the attention pattern computed is computed by going and dot producting keys and queries. So we dot product the keys and the queries."}, {"start": 796.98, "end": 828.26, "text": "But how are those computed? Well, the queries and the keys are going to be computed by taking the residual stream and multiplying by WQ or WK. And the residual stream is just the token times the embedding matrix. The token is represented as a one-hot factor. So if we combine this together, we get WKWE and WQWE. And if we combine all of that together, we'll find that the attention pattern"}, {"start": 828.26, "end": 859.76, "text": "This is a form of a softmax of the tokens on both sides and then this matrix in the middle. And that looks very similar to that matrix that we saw earlier. And we'll see that it's really telling us which tokens want to attend to which other tokens. Okay. Yeah, so we have these two really interesting matrices. And they're both vocabulary by vocabulary sized matrices."}, {"start": 860.85, "end": 891.18, "text": "And they seem to really be at the heart of this model's behavior. Okay, so why do we have those matrices? I can see a little bit mysterious, like the products of four different matrices together. And why those matrices and what exactly do they mean? Well, okay, let's start with the first one. We're going to call that the output value circuit or the output value matrix."}, {"start": 891.44, "end": 920.75, "text": "And that's its form. And what's going on here is if we say, okay, well, fundamentally what it means is it's saying, if you attend to a token, this is how the logits will be affected. And if you want to understand what that effect is, well, okay, we start with the token that we attend to. We'll call that the source token. And we go through WE. So we embed the token."}, {"start": 922.54, "end": 952.5, "text": "And then we have to convert it into a value vector. So we convert it into a value vector by multiplying by WV. So we've gone through WE, we've gone through WV. And then we have to go, the information gets moved by the intention pattern, and then it gets hit by WO. Okay, so we have to go, because we were going to go and add it back into the residual stream, so we need to multiply by WO. And we get hit by the un-embedding, and that causes a change to the output. And in the simplified model, that's..."}, {"start": 952.5, "end": 983.54, "text": "It's exactly linear. So this matrix here tells us what effect every token will have if we attend to it on the outputs. Okay, but the second circuit, this query key circuit, it tells us which tokens want to attend to which other tokens. Okay, so if you, again, let's start with the token that's being attended to. And if we run it through the embedding matrix and then multiply by WK, we get..."}, {"start": 983.66, "end": 1017.94, "text": "The key. And if on the other side, we're on the token that's going to do the attending, we go and multiply by WE and then by WQ, we get the query. And so those two paths then meet in the middle and we have to transpose this side because to go and make things work. And that gives us this matrix, which tells us how much every token wants to attend to every other possible token. Now, this is ignoring..."}, {"start": 1018.03, "end": 1047.28, "text": "The attention pattern also cares about positional embeddings or some other, probably, you have to account for position in some way, and there's now a variety of mechanisms for doing that. I'm going to elide that for now, and we'll see there are a few attention heads that will care about position, and we'll need to go and just sort of put that off to the side. But you could make this, if you were just dealing with positional embeddings, you'd add another term, something that describes the..."}, {"start": 1047.28, "end": 1075.52, "text": "Which positions want to attend which other positions and which positions potentially want to attend other tokens. That would probably be very small. But if we ignore the positions, that's exactly right. Okay, so that is the theory that we're going to need to understand one layer attention-only transformers. And if you're interested, the next video will actually leverage this then to go and understand"}, {"start": 1075.52, "end": 1080.11, "text": "One layer, yeah, understand the behavior of a one layer attention only transformer."}]}