{"url": "https://www.youtube.com/watch?v=g4xV2mPi8aY", "page_url": "https://www.youtube.com/watch?v=g4xV2mPi8aY", "title": "Attention - General - Indirect & n-gram Attention Heads [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 1710, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-attention-general-indirect-n-gram", "audio_duration_s": 1709.568, "worker": "jc-w06", "transcribe_s": 19.9, "model": "large-v3", "segments": [{"start": 0.11, "end": 29.28, "text": "Well, our last couple of videos gave us tools to go and talk about attention heads and large models and try to go and figure out what's going on with them. And so in this video, we're going to start looking at them. Now, large models have a lot of attention heads and a lot of them are pretty interesting. So before we start looking at them, it's worth maybe just sort of trying to organize them a little bit. And what we've done here is we're just going to make"}, {"start": 29.28, "end": 57.62, "text": "And we're going to start"}, {"start": 57.62, "end": 83.33, "text": "Well, before actually we talk about the categories here, I think actually the most important thing to mention is, you know, there's lots of tensions we're not going to talk about because we don't have, you know, particularly strong theories. So there's all this space here sort of for categories that we're not going to be talking about because we don't really understand this attention. So we're going to talk about attention where we do have theories, but it's worth keeping in mind that there's lots of things that we aren't talking about."}, {"start": 83.33, "end": 112.66, "text": "For the attention heads where we do have theories, there's really two kinds of theories we might have. So first we might have attention heads where we have really a theory based on their attention patterns. We can say, oh, you know, they tend to attend to X or they tend to attend to Y. And that's something that we've seen in a lot of previous work and has been really interesting. The other thing we can do."}, {"start": 112.66, "end": 140.27, "text": "is we can sometimes go and develop functional theories. And so what I mean by that is we have all, actually, I think more attention heads really where we can have functional theories than ones where we only have theories based on attention patterns. And what I mean by having a functional theory is that we think we understand at least part of what the model, the attention heads role in the model is. So what does it do? When it attends somewhere,"}, {"start": 140.27, "end": 168.4, "text": "How does that affect the model's behavior that it attended there? The way we'll do that is we'll look at the direct effect of attention heads on logits. You can look at the output of the attention head and just run it through to softmax and see how it affects the softmax. We can look at the direct effect. We developed this tool, the logit attribution pattern in the previous video. That's what we'll be using to do that."}, {"start": 169.1, "end": 195.17, "text": "Now, in a lot of cases, we won't be able to develop a functional theory and that's probably because the attention heads have an indirect role. We might also just not have understood things correctly, but I think mostly this is especially for early attention heads where it makes sense that lots of other layers are going to go and process things and later MLPs are going to process things and combine things with other information and later attention heads might move it elsewhere."}, {"start": 195.17, "end": 225.12, "text": "Just looking at the direct effect seems like it's probably not telling the story and we probably don't understand things very well. So there's going to be a lot of attention heads like that, but we'll still be able to go and observe really interesting things like all this previous work where we'll observe interesting grammatical structures and patterns and interesting things like that. And then for the ones where we do have a functional theory, there's going to be two major subcategories. So first, there's going to be a lot of attention heads."}, {"start": 225.12, "end": 254.21, "text": "That really seem to be doing, or at least our theory is that they're doing something like, something kind of like n-gram statistics or sort of skip n-gram statistics where they're just sort of saying, you know, given that there's this head in the last couple of tokens, or this token in the last couple of tokens, these words are more likely. And so, you know, people talk a lot about, you know, language models are just statistical pattern matchers."}, {"start": 254.21, "end": 283.31, "text": "You know, I think people probably mean different things by that, but I think these kind of n-gram like heads are really like, you know, the most extreme version of what you might, what you might mean by that or implementing the most, the most extreme sort of simple version of what you might mean by that. Seem to be at least. The other category is these copying or copying heads and in particular this subtype of copying heads called induction heads. And those really"}, {"start": 283.31, "end": 313.12, "text": "are actually one of the most interesting things that we've discovered. They seem to be tied up in meta-learning and really, yeah, just really related to a lot of the most interesting things about transformers. And they're so interesting, we're going to give them their own video. So if you just want to hear about them, you can skip ahead to the next video. But for this video, we're going to go and focus on the ones that we can understand in terms of their attention patterns."}, {"start": 313.12, "end": 342.91, "text": "We'll often use the info-weighted patterns to investigate those and the attention heads that sort of have this n-gram like behavior. And then we'll talk about the other ones in the next video. Okay. So to start with these ones that have, yeah, that we understand in terms of their attention patterns that sort of seem to go and do something natural and interpretable in terms of their attention pattern, but we don't really have as much of a"}, {"start": 342.91, "end": 372.51, "text": "A functional theory. Well, I guess one thing to say is just that this is very similar, I think, to things that people have found in a lot of previous work. And frankly, we're going to be doing really a much less thorough job than a lot of these papers. So maybe it's interesting in that we're going to be able to confirm or sort of have slight variations on things that people have seen. But a lot of this is going to be very similar in flavor to these previous."}, {"start": 372.51, "end": 401.65, "text": "Yeah, I'd really encourage you if you're interested in this kind of thing to look at them. But yeah, maybe we'll just look at a couple of these heads and sort of get a sense of some of the things that we discover for attention heads in terms of their attention patterns. So this first attention head has a very, it's actually I think a very, very simple behavior."}, {"start": 401.65, "end": 432.59, "text": "Almost always, there's a couple exceptions, it almost always attends to the previous token. So just to back up for a second, we have this interface that allows us to explore attention patterns and also info-weighted attention patterns and attribution patterns and things like that. And we need to give some sample text to poke around, and because it's fun, we're going to go and use the first paragraph of Harry Potter. But obviously we could use anything."}, {"start": 432.59, "end": 463.33, "text": "Yeah, there's a few other things that I'll sort of point out as we play around with it. If you watched previous videos, you've seen the same interface. Yeah, so when we hover over a token, it gets a square around it, denoting that it's the present, the token that we're focused on. And we can see that this attention head, all the attention goes to the previous token. Now there are a few exceptions to this. Really, I guess there's really one exception to this, which is sometimes when you're on a period, you attend to"}, {"start": 463.33, "end": 491.92, "text": "The period instead of to the previous token. So usually you attend to the previous token, but sometimes when you're on the period, you attend to yourself. And that happens for some periods that happens here. I think it happens here. But there's other periods where for some reason it doesn't happen. So this period tends back. So it seems like it's kind of seems maybe arbitrary whether it does it or not. I don't really have a theory for why. But I think it is kind of natural to"}, {"start": 492.11, "end": 516.54, "text": "To not attend backwards on periods. And because, well, you could imagine that a big part of the role of these might be to help construct sort of trigram statistics. So to go and say, well, you know, given the previous two tokens were this, you know, the next token is likely to be this. And, you know, I guess what it means for these to be the two previous tokens."}, {"start": 516.54, "end": 546.22, "text": "It's very different when you're on the other side of a period. And sometimes a period is a much bigger gap than the normal gap between tokens. And so you could imagine that given that, yeah, it's helpful to sort of ignore the token or treat it differently when there's a period in between than you would a normal previous token. Now, it's worth mentioning that this kind of attention, this kind of attention that looks at the previous token is"}, {"start": 546.22, "end": 573.76, "text": "Very similar to things discussed in other papers. Both Voida et al. and Clark et al. and probably a lot of other papers have observed the existence of what they call positional attention heads that attend to tokens at a fixed position relative to them. And sometimes that's attended to the previous token. In their case, they also have attention heads that attend to the next token."}, {"start": 573.76, "end": 599.89, "text": "Attention heads that attend to the next token, you can have that in, they're looking at BERT, and you can have that in BERT. In our model, since it's an autoregressive model, it's just looking at the previous tokens, trying to predict the next token, it can't look at future tokens, and so we don't observe these attention heads that look at the next token, only attention heads that look at the previous token. One other thing that's worth mentioning is we don't observe attention heads."}, {"start": 599.89, "end": 626.75, "text": "So we have attentionants that attend one back, but not attentionants that attend two back or three back or something. Of course, there's attentionants that attend further back, but they're not attending to a fixed number of positions. They're doing something that's more involved in what particular word they're attending to. And you might find that surprising. I found that kind of surprising initially, but I think what's going on is that if you have"}, {"start": 626.75, "end": 656.26, "text": "Multiple previous token attention heads. And these models tend to have a lot of them. You can chain them together. So two previous token attention heads can function as a two token attention head. Two tokens back attention head. So, okay. In any case, that's a long rant about previous token attention heads. Now, another really interesting thing that maybe has a slightly similar flavor is"}, {"start": 656.26, "end": 687.15, "text": "Well, maybe rather than spoil, let's just poke around for a second. We'll discover it together. So there are a lot of attention heads that just attend to the present token a lot of the time. There's an exception. But this attention, most of the time, attends to the present token. Okay, but there is an exception, and the exception is what makes it so interesting. And I think the easiest way to go and see this is to"}, {"start": 687.15, "end": 717.02, "text": "Look at which tokens get attended to, and maybe especially if we look at the infoated ones. Okay. So for instance, let's look at Dursley here. And we can see that all of the attention is focused on the D and none of it's going to the later tokens within that word. So we're using BP tokenization and that means that words often get split up. And so something that's interesting is we'll see if we"}, {"start": 717.02, "end": 742.14, "text": "If we look at it, that all of the tokens within Dursley attend to the D. So D attends to itself, and then all the later tokens attend to the D. Okay, so that's an interesting hypothesis. So maybe let's look at other Dursleys. There's another Dursley. It's three tokens again. It's all attending to the D. Was, tends to it a little bit as well, but not as much. Mustache is split into multiple tokens, and they all attend to the M. Dursley again. And so let's look at..."}, {"start": 742.14, "end": 773.78, "text": "There's other things going on here. Let's look for a second at words that are split into multiple tokens. So Dursley is one. It all tends to D. Mustache. Dursley. Dursley, that's plural. Dursley is a possessive. Drills. There we go. We have drills and the ills are attending to the DR. We have grunnings. That's one word split into a bunch of tokens. They all attend to the G at the beginning. Potters."}, {"start": 773.78, "end": 801.76, "text": "Yeah, so we're seeing that in a lot of cases. Now, there's some cases where that's not what's going on. Like, thank you, for instance, is, we'll see that you is attending to thank. So really, in some sense, thank you is sort of being treated as one word, or very much, very is attending to itself, and much attends to very. And so there's all these things that they're, the models"}, {"start": 801.76, "end": 830.5, "text": "If you take this theory that this is helping the model sort of undo tokenization and combine things together that are split into multiple tokens that the model sort of wants to think of maybe as one unit, there's some sense in which maybe it thinks that very much is sort of one unit, just like all the tokens within Dursley are one unit. And there's a lot of cases of this. Privet drive, one unit. Number four, one unit. So there's lots of things like that."}, {"start": 830.5, "end": 855.66, "text": "So I think this is really interesting because often tokenization seems very unnatural and awkward and is splitting things up in really weird ways. And it seems like the model has to be somehow undoing that. And you'll see a lot of attention that seem like they might be involved in that. And of course, those attention heads, you know, this is a very early thing that you want to do in the model. And then you can go and use it for your later."}, {"start": 855.66, "end": 887.01, "text": "In later MLPs, probably you have to sort of process things to understand maybe more of the semantics of the unit that you're combining together, and then you can use it to do other things. Okay, our next attention head. Okay, so if you just look at it like this, it's a little bit hard to figure out what's going on. Often we're attending to EOT. Something that will make it a little bit easier is if we make our default view seeing how much every token gets attended to rather than"}, {"start": 887.01, "end": 916.61, "text": "So here we see a lot of tokens are attending to, I guess, probably one theory you might have is verbs. That might be an initial theory, but opinion is a counterexample. But we still have a lot of attention focused on the EOT, for instance, or this period here. So that's kind of weird. But we know that often attention heads attend to places sort of as arresting behavior. And so if we look at the infoid pattern, which is just"}, {"start": 916.61, "end": 944.24, "text": "Scale everything by how much information is copied from them. This gets a little bit sharper again. We're still tending a little bit to EOT, but not as much. It's become weaker. This is really speculation, but the thing that I've come to think about this attention and some related ones is that they're about words that might be setting the stage for a clause."}, {"start": 944.27, "end": 972.94, "text": "So that can be, sometimes that can be conjunctions that are just, you know, literally joining together two clauses. But in a lot of cases, they're words that might continue either explicitly or implicitly with a to or a that. So proud to, say that. It might be your opinion that, you might fear that. You might, well, if it's just an explicit clause, but you might say that. You might know that."}, {"start": 972.94, "end": 1001.12, "text": "You might want that. You might reason that. Or you might have reason to. You might want that. You might shudder to. You might think that. You might have seen that. You might pretend that. I guess bear... Yeah, you couldn't bear to. Or bear that. I guess you're sort of aligning it there because it's sort of a..."}, {"start": 1001.12, "end": 1029.1, "text": "Something that we do sometimes in English, but there's sort of as implicitly something like that. And then, yeah, there's a bunch of other, yeah, found out that. So a lot of these really seem to be places where that kind of thing is going on. And I think it's actually very natural, you know, and this is just a hypothesis. And we're sort of just giving anecdotes from poking around at these attention heads. But in this case, we'll have more rigorous investigations of some of the other stuff."}, {"start": 1029.1, "end": 1052.59, "text": "I think it is very natural to think that this kind of attention head might exist, because in some sense, these words are sort of setting the scene for the next clause. The kind of words you're going to use in a clause about something that you're proud of are very different from the kinds of words you're going to use in a clause describing what your greatest fear is that, what you fear that."}, {"start": 1052.59, "end": 1082.06, "text": "A clause about something that you know is very different from a clause about something that you want or that you shudder about. Or if you think something, you're kind of caveating it a little bit and you're going to say something different. So these kinds of words, they're not exactly conjunctions strictly, but they're sort of setting the stage for a clause, and I think that seems very useful maybe. Who knows? I'm not a linguist, so maybe there is a formal linguistic term for this. I just don't know what it is."}, {"start": 1084.02, "end": 1113.9, "text": "Okay, and here's another one that's going to be easier if we switch to looking at where the attention is going and looking at an info-weighted pattern. And yeah, this one, there's probably a very natural theory that's going to jump out at you. If you look, it's almost all articles. The, a, a, the, a, the, the, the, the, a, a, the. Now there's a few exceptions."}, {"start": 1113.9, "end": 1136.45, "text": "There isn't an article, but it's a determiner. Determiners are this generalization of articles where if you have a possessive word somewhere, it replaces the article or if you have a quantifier or a word like this, they replace the article. Now, there's an exception here of"}, {"start": 1136.45, "end": 1167.25, "text": "I mean, it's sort of of A, so maybe it's still those two together are sort of a little bit acting. That is an exception to that theory. But this attention hand really seems to want to attend to previous articles or previous determiners. And yeah, that's kind of interesting. You know, another attention hand that sort of is mimicking grammar, modeling grammar. Here's another one that does the same thing, but..."}, {"start": 1167.25, "end": 1197.01, "text": "It's actually even, well, it has had some things where it attends to something that isn't a determiner, but it's also much more general. So not only do we have words like a and a and the and there are those we saw before, but other words that are determiners like here we got possessive, the Dursleys, that's taking on the role of an article for sister. No, so quantifiers like that can act as determiners."}, {"start": 1197.01, "end": 1228.78, "text": "Another is another word that's taking on the role of a determiner. Several is taking on the role of a determiner there. What else? In that case, that isn't. But there are cases where, like if you said that boy, that would be taking on the role of being a determiner or this boy. There should be cases where that's happening. Yeah, like here, this boy there."}, {"start": 1229.14, "end": 1260.98, "text": "Now, there's a lot of words here where that doesn't make sense. So there's lots of exceptions like with or because or things like this are not determiners. But this attention head seems to preferentially attend to determiners. Now, another interesting category, again, let's look at where this attention head is attending back to. And let's switch to an info-weighted view. And we'll see that this one really..."}, {"start": 1260.98, "end": 1287.15, "text": "Likes conjunctions, although, which, but, but, and, and, which, but. So it tends to like conjunctions. We also see that it sometimes likes things that, yeah, the periods or the first token of a sentence a lot of the time. And you could imagine that, you know, in cases,"}, {"start": 1287.15, "end": 1323.95, "text": "You know, conjunctions are linking clauses, and in cases where there isn't one between two clauses, maybe you attend to the first token or the period, and it's very faint, so you aren't removing that much information from them. And here's another one doing something similar. Because, with, which, although, as, and, but, and, which, and, but, and, because, period, semicolon, in fact, but. Those are all conjunctions."}, {"start": 1325.07, "end": 1350.19, "text": "Okay, so those are some interesting attention heads that we found. There's lots of others, but I'll go and leave it there. And let's go and talk about the n-gram heads. So the other category of attention heads that we're going to talk about in this video, and then we'll talk about the copying and induction heads in the next video, are what we're calling sort of n-gram like heads."}, {"start": 1350.19, "end": 1380.69, "text": "Or sort of maybe another way to describe would be like short-term statistical heads or something like this. And so I think it's just easiest to talk about this with an example. So what we're doing here is we're looking at the logit attribution pattern, we're just grabbing cases where there's larger values in it. So these are cases where some word, yeah, where the attention head by attending somewhere increases the probability of another word and it just directly has that effect on the softmax."}, {"start": 1381.84, "end": 1411.73, "text": "Yeah, so let's just read through these. And they're sort of this funny property that, especially when it's big, they almost feel like they rhyme. Say they were. Expect to be. Because they just. Although he did. Had nearly twice. Time craning. And even just, like, time X-ing. Opinion was. Finer anywhere. Was somebody. That would. Think could."}, {"start": 1411.92, "end": 1439.25, "text": "Past tense, didn't. What would, if arrived, that had, but never, never seen. So this is really speculative, but the thing that I always think when I see these heads is there's just some way in which it really feels to me like there's sort of this rhyming feeling to them when I read off the attributions."}, {"start": 1439.25, "end": 1460.02, "text": "And I really strongly suspect that there's something about just very local statistics of language. It's like bigram with jumps between it, or maybe sometimes it's mediated to some extent by what the word that's tending back to that token is. And I only developed this theory after"}, {"start": 1460.02, "end": 1488.93, "text": "Studying the small models, if you watched our videos on the one-layer attention models, you saw that there were these skip trigram heads where we could really understand them, and this is exactly what they were doing. And so my suspicion is these are just the large model versions of that, and maybe they're a little bit more sophisticated. Maybe the choice to attend back is based on several tokens rather than just the immediate token, and maybe the effect of the token that you're attending to is mediated by other tokens around it."}, {"start": 1488.93, "end": 1518.21, "text": "But that's kind of the hypothesis. So let's look at a few more of these. People too, with nonsense. Spent time, overfence. Spy on, on neighbors. Bare if, for years. Keeping away, with child. And especially look at these big ones. Like the ones that have really big effects. Spent time. Those could just be side by side. For years."}, {"start": 1518.21, "end": 1547.23, "text": "This could just be side by side. So there's really this feeling to them. Again, this is just a hypothesis, but there's really this feeling to them that they are encoding something about short-term statistics. Yeah, again, this is going to be very similar. Big man, A large, was blonde, the amount, time x-ing, we've talked about that one, over fences, we've talked about that one."}, {"start": 1547.23, "end": 1577.12, "text": "With nonsense. We've talked about those. Oh, I'm on the other side. A son. Somebody discover. For years. Have sister. Her husband. A son. Another reason. Keeping away. With child. There's just some feeling to them that they really seem like... I don't know."}, {"start": 1577.12, "end": 1607.74, "text": "Is it interesting going through these? This one maybe is even a little bit less compelling. I think I set a lower threshold for whether to include things here. Well, it makes sense that thank you maybe increases the probability of a period. People, I don't even know, like people subjunctive. Anything or with nonsense was director, although did in useful time axing. We've talked about that."}, {"start": 1607.74, "end": 1642.83, "text": "Boy, anywhere. Yeah, these seem a little bit weaker, but let's maybe just focus on the big ones, like think could, but hadn't, for years, possible, well, possible be, possible period, what neighbors, but had, but never, seen period, this was, keeping away, with child. Yeah, so these n-gram heads,"}, {"start": 1642.83, "end": 1671.39, "text": "Again, they're just speculation, but it really seems like there's something about local statistics that some of these attention heads are including. And one thing I think is interesting is it means these attention heads, just like if you look back to the first video and you look at the skip trigram heads that we were able to really reverse engineer and understand exactly, these sort of n-grammy heads are, you shouldn't expect them to have"}, {"start": 1671.39, "end": 1697.55, "text": "You know, obvious interpretable attention patterns because they're sort of less about the function of the words and it's more like this word happens to be suggesting or this token happens to be suggesting that these other tokens might be more likely and now it seems like they could be injected here. And so the attention patterns are very opaque a lot of time for these attention hints. But I think that there's sort of a natural explanation for that."}, {"start": 1698.19, "end": 1707.79, "text": "So in the next video, we'll be talking about copying and induction heads, which I think are probably the most interesting thing we've discovered. So I'm excited to chat about them with you."}]}