{"url": "https://www.youtube.com/watch?v=ZBlHFFE-ng8", "page_url": "https://www.youtube.com/watch?v=ZBlHFFE-ng8", "title": "1L Attention - Results [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 963, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-1l-attention-results-rough-early-thou", "audio_duration_s": 962.7109375, "worker": "jc-w06", "transcribe_s": 14.2, "model": "large-v3", "segments": [{"start": 0.75, "end": 30.56, "text": "In our previous video, we developed some theoretical results for reasoning about transformers. And in this video, we're going to take advantage of those to try and understand an actual one-layer attention-rolling transformer and to really get to a point where, in principle, we could fully understand it if we were willing to put in the effort. So that's kind of exciting. Now, to recap, the key idea from the previous video was just that we have this..."}, {"start": 30.56, "end": 61.55, "text": "These two circuits, the OV circuit, which tells us how an input, a token, if we attend to it, will affect the logits. And the QK circuit, which tells us which tokens want to attend to which other tokens. And they correspond to these matrices, which we have over here. So that's the key idea that we need. And so the OV circuit."}, {"start": 61.84, "end": 92.38, "text": "It sort of describes how the source token, the token we attend to, affects the output, and the QK circuit, how it affects which tokens want to attend to it, so the destination. So one links the destination token to the source token, and one links the source token to the output token. And so a very natural thing to do is to go and say, well, okay, let's think about a single source token, since that's the one that both of these circuits connect to. And let's ask, first,"}, {"start": 92.38, "end": 123.7, "text": "Which tokens want to attend to that token? And if it gets attended to, what is its effect on the output? And so, for example, we can see that the token Perfect, well, the token R wants to attend back to Perfect. And then if we attend to it, we'll increase the probability of Perfect. And also, to some extent, Super and Absolute. And that means that we increase the probability of"}, {"start": 123.7, "end": 152.96, "text": "Of what I'll call a skip trigram, where we see the token perfect, and then we jump over some tokens. And then we see the token R and increase the token perfect, increase the probability of the token perfect being the next token. So we sort of say perfect, later on R perfect. Or it can be other things too, because it could be the other tokens that attend back and the other tokens that get increased in probability. So perfect, look super, and things like that. Of course, you could also look for the tokens that get decreased in probability. I think the ones that..."}, {"start": 152.96, "end": 183.98, "text": "You can increase your probability of the most interesting. And we can do this for a bunch of things. So large, a bunch of tokens want to attend back. Increase the probability of large and small. And that results in increasing the probability of skip trigrams like large, using large. Large contains small. So it's kind of this copying aspect to it or copying or using similar words where something potentially quite far back in your context can now influence."}, {"start": 183.98, "end": 213.87, "text": "Your next token for numbers we see two and the token one wants to go and attend back to that and predict two so we say two and then one two but it can also be two has three and there's obviously lots of other skip diagrams that are getting affected. The next couple are interesting so this lambda one is really related to LaTeX which is a typesetting language for scientific documents and lambda is a keyword that's"}, {"start": 213.87, "end": 244.48, "text": "It has to be preceded by a backslash. So we see all these cases where there's backslashes that want to attend back to it and then increase the probability of Lambda. So we have Lambda, backslash Lambda, but also other things that could be in LaTeX backslash escaped in our special commands like operator. This NBSP one is from HTML. So in HTML, there's some characters that you can't write or you're supposed to go and write them using ampersand and then a special..."}, {"start": 244.48, "end": 275.15, "text": "Special code or words that represents it. And so NBSP is non-breaking space. And so if we've seen it once, maybe next time we see an ampersand, we think we're going to say, go and write a non-breaking space character. And there's other things that could be like this greater than is another one that it could be. And great, this is maybe a more normal thing. Although I feel like this is related to titles, like titles of books or something like The Great Gatsby or something like that a little bit."}, {"start": 275.15, "end": 304.72, "text": "The uppercase, the uppercase, great. Great, the great. Okay, but just stepping back, it's worth noting here that we're just, we didn't run the model. We just went and took its weights and we're reading algorithms off its weights. And these algorithms, you know, of course, there's a similar, we'd have to do this for every attention head, but they're sort of telling us the whole story of the model without running it. And we can just understand it by studying these matrices."}, {"start": 304.72, "end": 333.57, "text": "Now, the matrices are fairly large, and it could be quite a bit of work, but in principle, we could fully understand this model. Okay. Now, that copying is one example of the kind of thing that it does. It does some other fun things. So this first example here on this new slide is very similar to the previous one. We're going to just copy Indy. Yeah, sometimes a single word, the tokenization system will split it into two tokens. And so we might have..."}, {"start": 333.57, "end": 364.58, "text": "Have the name Cindy, and sometimes they get split into a C and Indie. So we see Indie, and then we want to go and say Cindy, and that's two tokens, the C and the Indie. And so we have C attend back to Indie, and then we increase the probability of Indie, or possibly all uppercase Indie. A fun variant of that, and I actually didn't see this in the model that I was using to prepare this talk. Unfortunately, I didn't have this. These are actually from a different one layer model."}, {"start": 364.58, "end": 392.8, "text": "But I think it's a very interesting behavior. Sometimes you'll have a word and you look at the version where it's tokenized as a single word, but it could also be tokenized as two tokens. And so then you have the tokens that could be the first part of that, like a space P or a P, attend back to the full word and then increase the probability of its continuation or things like."}, {"start": 392.8, "end": 421.2, "text": "Py, the plural version, which I guess is sort of very close to it being a continuation. And we can also have just similar words that, so we could have spikes and that starts with SP instead. So this isn't quite literally copying, but it's also very interesting. And all of these other ones are examples where a word can be tokenized potentially in two parts. Ralph, R, Alf, Lloyd, L, Lloyd, PIXMAP, PIXMAP."}, {"start": 421.2, "end": 450.05, "text": "And then it tries to go and correct that. But do other fun things as well, like, you know, all uppercase ALF. I guess with Lloyd, the reason we're seeing Athrin and a C over here is because there's probably some famous Catherine that's either their last name is Lloyd or associated with Lloyd in some way. You might wonder about why is there a Q for the PixMap? Well, there's a programming language called Qt that has a class called QCanvas. And so..."}, {"start": 450.05, "end": 479.78, "text": "And presumably it interacts with PIX maps. So we have all of these sort of interesting stuff hiding in these matrices if we look at them. Now, I think one thing that's a really, an observation that had me kind of excited at these is, if you look at neural networks and especially if you look at language models, sometimes the cases where they predict strange things are very inscrutable and you just have no idea why in the wide world where the model's just that."}, {"start": 479.78, "end": 509.31, "text": "An interesting thing that I found looking at these attention heads is that they're kind of forced to introduce bugs. There's places where in order to go and sort of encode as many possible of these useful skip trigrams, they necessarily add some bugs. So for instance, for the Pixmap one, we want to be able to go and say p xmap to go and complete Pixmap and q canvas to go and say q canvas later. Those are both valid things."}, {"start": 509.31, "end": 538.4, "text": "Those all make sense. But then that necessarily entails that we must also have P canvas because they have to go together in any order. And that seems like a bug. As far as I can tell, there's no P canvas object that's in some programming language or being used. And so that seems to just be a bug that follows from the way that we're encoding things. And similarly with Lloyd,"}, {"start": 538.4, "end": 568.13, "text": "Lloyd and Catherine both seem like reasonable things that could be increasing the probability of, but Claude and Lathran both seem like they're probably bugs that are just coming from the way that things are being encoded. So that's kind of neat that we can understand these bugs and how they naturally follow from it trying to do something reasonable and then just having limited expressivity. So we've seen a couple sort of abstract patterns and it's maybe worth calling out. So one is that you see some token."}, {"start": 568.13, "end": 598.45, "text": "And then you see a token that might precede it and predict the same token. So you see two, then you see one, and you think, ah, probably I'm going to say two again. Another pattern is that you see a token, and you see a token that might precede a similar token. So two, and you see hands, and you think, ah, maybe we're going to talk about a number again. I'll increase the probability of three. It's not literally copying, but it's very similar. There's this version where you see a token that..."}, {"start": 598.45, "end": 629.2, "text": "can be tokenized in a split way and then you see the first part of the split version and you predict the second part it's sort of an interesting pattern and then there's a variant on that where we see a split token or a token that could a word that is represented as a single token but could be split and we see the first part and then we predict a variant of the second part like all the all uppercase version of it now this might give you the impression that"}, {"start": 629.2, "end": 657.46, "text": "All of the attention heads are copying. And that impression wouldn't be far from the truth. From the model that I was looking at as I prepared this talk, 10 of its 12 attention heads are really doing this sort of copying behavior. But there were two that weren't. And those two actually are kind of interesting because they're primarily positional. So remember the QK circuit that we're describing for simplicity and because people handle positional."}, {"start": 657.46, "end": 684.54, "text": "Information in various ways, depending on whether they're using positional embeddings or rotary attention or things like this. We elided the positional part, but some attention heads care a lot about position. And so we have some attention heads here that primarily are going to care about position. And so one interesting pattern there is"}, {"start": 684.54, "end": 710.37, "text": "You'll have these attention heads that mostly attend to the present token. And you'll say, you know, it seems you might ask, well, why in the wide world would it do that? You know, anything that it could do by attending to the present token, it could also go and do by just going and writing down that, you know, we have that terminal equation that just goes directly down the residual stream and where the embedding directly affects the unembedding. So it could do that. But you have these token, these attention heads that mostly attend."}, {"start": 710.37, "end": 737.86, "text": "to the present token but sometimes attend a little bit further back and often in cases where the tokenization like sort of spirit the thing that sort of spiritually you think this is bigram statistics really lives with a token that's a little bit further back so first let's look at one where it's just literally doing bigram stuff so we see the token corresponding as probably our present token and we increase the probability of two and two and four and markup and width so"}, {"start": 737.86, "end": 768.78, "text": "Corresponding to, corresponding for, corresponding markup, corresponding with. Those are all very reasonable things to go and complete and those seem very bigrammish. But then we have an example of the token co-ints. And presumably that's part of coincides. Actually, in this case, if you do look at the QK portion of this, if you look at the circuit, there's a positional component and a QK component. And the QK component."}, {"start": 768.78, "end": 798.51, "text": "The token component is having the token IDEs attend back to Coinside. But the Coins part is a lot more informative than the IDEs part maybe. And so about what the next token is going to be. And so looking at Coins and being able to predict width or closely, Coinsides width, Coinsides closely, that maybe is more useful. Similarly, if you have the token couldn't, well, just seeing the apostrophe T,"}, {"start": 798.51, "end": 835.28, "text": "And it gets split into two tokens. And just seeing the apostrophe T doesn't really tell you what's going on. So maybe it's more helpful to go and look at the couldn't part and say couldn't resist, couldn't compete, couldn't stand, couldn't identify. And similarly, when we have shouldn't, looking at the shouldn't part is maybe more informative than the apostrophe T because you can then say shouldn't have, shouldn't be, shouldn't remain, shouldn't take. And those are actually kind of different maybe than the most likely things for couldn't."}, {"start": 835.28, "end": 865.01, "text": "In summary, our one layer, I guess there's a few interesting points. So first of all, these one layer attention only transformers can be understood by studying these OV and QK circuits. And that's really neat because understanding models is hard. And we often don't understand, well, I think there's often a lot of questions about whether we can understand neural networks. And this is an example where you can succeed. So that's fun. Really, in principle, we could fully understand this model."}, {"start": 865.01, "end": 891.42, "text": "Transformers desperately want to do meta-learning. I think this was a very surprising point to me. The model goes and dedicates 10 out of its 12 attention heads to going and doing meta-learning-y stuff. It wasn't at all obvious to me that the model was going to do that. In fact, I would have expected it to mostly be doing sort of n-gram statistics-ish stuff. But even though the model's architecture is actually"}, {"start": 891.42, "end": 920.86, "text": "Quite poorly suited to doing meta-learning. We'll see that it has much better meta-learning abilities later on. I think that, yeah, it's dedicating a ton of its capacities anyways to going and accomplishing that. The constraints of the system force us, force the model to introduce bugs. So going and trying to represent just sort of perfectly reasonable things, these skip trigrams, intrinsically forces..."}, {"start": 920.86, "end": 950.1, "text": "The model that have certain bugs that are very inscrutable if you just look at the resulting behavior. And so that's pretty interesting. And then finally, attention-only transformers are linear if you freeze the attention patterns. And again, this trick of functions that are simple if you can go and hold one thing constant, even if the whole function as a whole is very complex, that's a very nice point of leverage that I always appreciate when I find something like that."}, {"start": 950.86, "end": 962.38, "text": "Well, that's the talk. In the next video, we're going to go and discuss a way to sort of automate and much more quickly analyze one layer, similar models to this."}]}