[0:00:00] In our last video, we talked about some of the attention heads we found in large transformers. But we've saved the best for last. And in this video, we're going to talk about the most interesting attention heads we've really found, copying heads and a subset of copying heads called induction heads. Yeah, so the last video we talked about some other things, but in this video, we're going to talk about them. And they're very interesting for a number of reasons.
[0:00:26] But probably the biggest one is that they seem really at the center or like a very important part of the story of how large models accomplish meta-learning. So this is the phenomenon where somehow you can show large transformers examples of things and they can learn in their context to do new things. Or put another way, if you have a longer and longer context, transformers tend to do better and better at predicting the next token. And so that seems like a really interesting phenomenon about.
[0:00:54] Transformers and induction heads and copying heads are going to give us a hint as to how they do it. So it's worth noting that copying heads are the more general kind of head, and then induction heads are a subset of copying heads. So all induction heads are copying heads, but not all copying heads are induction heads. Now, we've talked about copying heads and induction heads before. We've seen them, well, copying heads, we've seen as far back as the one-layer transformer we studied right at the beginning of the series. So we saw
[0:01:24] In one of our first videos, you have these heads that will attend back to a token and then increase the probability that the next token is the same token or a very similar token. That's what we call a copying head. You attend to a token and then you increase the probability that that token is going to occur again. The characterizing thing about these models are these attention heads, these copying attention heads.
[0:01:51] Or at least one very nice way to characterize them is that the OV circuit, the output value circuit, had positive eigenvalues. So you could just go and look at the eigenvalues and go and discover all these copying heads. Now, unfortunately, that isn't possible for large models because at least it isn't trivially possible because the MLP sort of makes it, MLP layers make it a lot harder to look at things this way. But one sort of
[0:02:21] Very theoretical way, like you could sort of imagine if these heads were in only in attention-only models, you could define them as having positive eigenvalues. So it just means that they increase the probability of the tokens they attend to being the next token. Now, induction heads are a subset of copying heads. And they seem to allow models to do meta-learning much more efficiently and extract.
[0:02:50] A lot more information from the earlier context. And it seems like what they do, so here we have just an example from the first paragraph of Harry Potter, which you've been using as an example text because it's fun, where what they do is they look at the present token and its context, and they look for previous places where similar things have happened, and then shift one forward and look at, last time I saw this.
[0:03:20] What happened next? And then they go and predict that whatever happened next that time is what's going to happen next this time. And so if you really sort of pull these apart, that's what they're doing. And the simplest version would be to just look at the present token and look if that was the previous token, but often they'll actually use a little bit more of the surroundings. One way you could think about this is it's really a very simple learning algorithm that's running on top.
[0:03:49] Within the context, it's a meta-learning learning algorithm. It's basically nearest neighbors. You're looking for the last time that something similar happened and then you're predicting the thing that happened then. If you find multiple matches, you'll average them together and increase the probability of all of them a little bit. We have nearest neighbors going on as we learn. That's pretty cool. One thing that's worth mentioning,
[0:04:16] Is that induction heads, because they have to move information forward like this, induction heads only become possible starting in the second layer of a model. So the first layer model can't express them because they need to use an attention head to go and construct the keys for the induction head. Okay, so those are copying induction heads. We've talked about them in previous videos before, although we're going to talk about them in more depth now. But the really interesting thing is
[0:04:46] In addition to these sort of very strict, straightforward copying and induction heads, there seem to be a large number of what you might call soft versions of them. Different variations that are in some sense doing copying or induction, but not literally copying tokens. So we'll look at a couple of examples of those. And I think they sort of paint a picture of how, you know,
[0:05:15] Doing really strict copying and induction wouldn't be sufficient to accomplish a lot of behaviors we see in these models, but perhaps these variants can allow a much wider range of meta-learning behaviors. That's very interesting to me. To give two examples, we'll first do an example of a soft copying head, and then we'll do an example of a soft induction head. An example of a soft copying head.
[0:05:44] will be a head that goes and maintains sort of pronoun agreement or inference between sentences or between previous words. So it'll look back and try to, you know, whenever it needs to go and say something that implies either gender or plurality, it'll go and look at previous context to go and try and determine, you know, what is the right pronoun to use or what's the right conjugation of a verb to use.
[0:06:12] That might just be, you know, if you're about to go and say a pronoun, look at the previous pronoun. That's a very common one. But maybe also look at other words that might suggest things. So you could maybe go and, you know, if you're going to have to say a pronoun, maybe you go and you notice that you just said Mrs. And maybe you're like, oh, it's probably going to be she. Or, you know, if it's going to be a possessive pronoun, maybe you look back and you saw she, and now you're going to say a female possessive pronoun. Or maybe you just saw an and, and that implies plurality, and then you say they. So anyways, we'll talk about that more in a sec.
[0:06:42] For the soft induction example, we found an induction head that does translation. So it's like induction, but it's across languages. So it's not literally copying things, not literally going and looking to the last time that it saw the same thing and then predicting the next token. But instead, it's looking for places where, you know, like we say maybe, I am, and then you look and see.
[0:07:10] What happened next that time? And then you're going to translate that into English and increase the probability. So yeah, that's a very powerful thing to be able to do. So yeah, let's actually go and look at these attention heads. So we've talked about this interface before, but to very, very briefly recap, what we have here is
[0:07:42] We just have a piece of sample text to go and demonstrate this on. This is the first paragraph of Harry Potter. And we have the attention pattern of the head. And if we hover over a token, we can see where it attends to. By default, it's colored by the maximum value that it's attended somewhere. And we can swap it around so we can color it by how much every token gets attended to and then see which tokens attend to them. And we can also, if we want, rather than look at the attention pattern, we can also look at the
[0:08:12] Infoated attention patterns. So that's just weighting things by how much information is being copied. And we can also color things by the attribution patterns. So that's how much every token affects every other token. And this is really giving the game away here. So what we're seeing is these are all the tokens that really have their probabilities increased by this attention head. And if we hover over them, we'll be able to see what tokens did that.
[0:08:40] Remember that in our normal attention pattern, we go from source to destination. But in the attribution pattern, we go from source token to output token. So we're highlighting the token that is affected and looking at the token that it attended to. So here, they is affected, it's increased in probability. That means that that attended back to were and attended back to and, and both of those increase the probability that the next token should be they.
[0:09:10] And that makes sense because both were is telling us that there's a plural conjugation and and is linking together two subjects and saying that there's going to be a plural. Now, one thing that's worth noting is this would have been much, much harder to go and see if we weren't using the attribution pattern. If we were looking at the attention pattern, okay, so it would be that. Well, we can kind of see that it's attending to there, but it's also attending to a bunch of other tokens.
[0:09:36] And it's much less obvious what's going on. You know, I don't think I'd be able to figure it out just by looking at this. But when we switch the attribution pattern, things become very clear, both in even just looking at which tokens are affected, which tokens get attended to, and then especially looking at the links between them will really, really help us see. So we can look at more examples. Here's another they. Again, it's now we're going and having this they.
[0:10:04] Well, it's the period that's attending back, remember, but they is increased in probability by they. We use they as our previous pronoun and were and and. Those are all suggesting plural. What else? I need to unlock that. Here's another they. It's the same type of thing. People is plural. They is plural. We're looking less at the earlier ones now. What about, here's a he. Well, we said he, mister, man. Those all implied that there might be, we might need male pronouns in the near future.
[0:10:35] Her, there was a she and a Mrs. Probably if we look at that she, it's looking at, yeah, Mrs. increased the probability of it. And a big part of what I find so cool about this, oh yeah, Dursley's is plural, right? So even if we didn't have they, that sort of is telling us that they is primarily coming from Dursley's being plural. You know, this is...
[0:10:59] This is in some ways like keeping its ability to create sort of coherent text, but there's a really simple story, which is just that it's really looking at the word that's just what the present plurality and gender might be and taking advantage of that. So that's a very simple thing that it's doing, and it's just really easy to see once we can look at the direct logit attribution pattern. So it's a big part of why I find that tool really helpful.
[0:11:29] Okay, a similar type of thing is tense. It's really, this is going to be the same story. So let's just jump into the attribution pattern. And this is going to have a little, I think this might have a little bit of plurality mixed in as well. So here, for instance, why do we say that this is, you know, this is a past tense imperfect word? Well, the reason for that is we're looking at
[0:11:53] The previous world. Okay, that makes sense. So that's really telling us what the tense is. And each one of these is going to tend back to the last verb, maybe the last couple of verbs, and look at their tense and then increase the probability that we're in the same tense. Maybe it's not that concerned about whether it's perfect or imperfect. Yeah, now there's some singular versus plural as well. So where can we see that?
[0:12:25] Yeah, was, had, had, was, those are all, oh, I guess had is actually plural. So that's an exception, but yeah, had, had versus, yeah, there's another was. And so there's maybe a little, maybe there's less of a plurality thing than I thought there was. There's other ones where I've seen that maybe more crisply.
[0:12:54] That's what I do get for going and doing this a bit more spontaneously. Okay, so those are two examples of copying. And the reason we're calling those copying is they're sort of soft copying. We're copying, you know, a couple dimensions of the word, maybe just one dimension, like tense or like plurality or gender, maybe a couple dimensions in the case of the gender and plurality one. So we're not copying exactly the word, but, you know, conceptually, we can't.
[0:13:25] We can't do the eigenvalue decomposition for this one because it's entangled with NLPs, but conceptually, if we could, what we'd expect to see is a couple very large positive eigenvalues and then a lot of smaller eigenvalues because there's a couple dimensions that we're really copying and then there's maybe other ones that we care less about. This isn't an induction head because it's not, as far as we can tell, looking at the previous word to decide what it's going to do. Instead, it's really influenced by the
[0:13:57] Yeah, it's really influenced by just looking at the recent words and waiting for a time when it needs to go and say a verb and congenate it correctly, or waiting for a time it needs to say a pronoun and using the correct pronoun. Now, the translation head is going to be an induction head, and it actually does a small amount of induction, even within one language, but it's really translation where it starts really acting.
[0:14:26] Before we dive into that, I think it's worth just seeing this is actually an attention head that would be really, really hard to understand if we were just looking at the attention pattern. I think it's worth just sitting with that for a second because I think it drives in why these tools of the intuited attention pattern and the attribution patterns that we introduced in previous videos are useful for exploring attention heads. One thing to notice that almost all tokens attend to EOT.
[0:14:55] Almost every token attends to EOT. So this is, you know, a lot of previous papers have noted these attention heads that will attend to the first token or to separator tokens or things like that. And the speculation is that it's a default. And we'll be able to show that because when we switch to the info-weighted pattern, that'll go away. So it's only with the attention pattern that we see that.
[0:15:21] And later on, it will start attending to periods as well. So that's sort of another default place that the attention head can rest. And we're actually not the first ones to observe that not only does it rest on the EOT tokens, but often punctuation will also get sort of used for that purpose. I think it's, I think it's Voida at all that observed that as well. So maybe it was Clark at all. Yes. Okay. So, but there are a few exceptions.
[0:15:51] Okay, so the first thing to do, though, is we should actually really go and, yeah, so if we look at this, for instance, if we switch to looking at the source, we'll see that almost all the attention is going to EOT, and we can hover over EOT and see everything that attends to it, so it's almost all tokens. But a few tokens are attending elsewhere, especially to the period, that's everything over there, but there's a few other ones that are looking at very specific things. So a way that we can clean this up and really get rid of the confusion with those resting positions is we could switch to the info-weighted pattern.
[0:16:21] All of a sudden, we don't care at all about EOT. We can see we don't care at all about the periods. It's just these other words where we're actually taking information from them and moving it elsewhere. So that means that we don't have to guess what a resting position is and what isn't a resting position. We can just actually know and understand that those positions didn't matter. They were just resting positions. No information was being moved from them. But the thing that will be really informative is if we switch to the attribution pattern. And maybe let's switch now again to looking at targets.
[0:16:52] And so, for instance, but. Well, we're looking at the previous token was a comma, and we're looking at other places, especially where there was a comma. Maybe this is an exception, but in most cases, looking at places where there's a comma, and then we went but again. Okay, but they. And again, we have but they. Comma but they. Comma, but, and then they. And then we predict they by looking at those previous copies.
[0:17:23] There's other ones. Was. So Mrs. Potter was. Mr. Dursley was. But the place that this really gets interesting, these are relatively small attributions. It's doing a lot less induction than other induction heads that we've seen. But the place this gets really interesting can't be demonstrated on the Harry Potter text. And this is an important thing to keep in mind.
[0:17:55] Sometimes an attention head's interesting behavior only surfaces if you start looking for rare examples where it does a lot of work. It might be quite uncommon. This is an interface that's going to allow us to poke around a little bit and go and type whatever text we want and see how attention heads behave.
[0:18:26] So let me go and put in some texts in multiple languages. Maybe I'll go and I'll put this on multiple lines so that it's a little bit clearer. That's not right. I'm not even splitting the languages up right. Okay, so we have French on the top line. Je m'appelle Christophe. Je suis du Canada. J'aime la terre. And then we have Spanish, which I'm not going to try to pronounce. And then we have the English.
[0:19:01] And one thing that you can immediately see is if you look at just the attention pattern. So what you're seeing here is for every token, where does it attend? Well, there's these two off diagonals which correspond to attending back to, I guess, the... Yeah, okay, so this here is attending back to English and this is attending to Spanish. So the second, yeah, so it's like it's...
[0:19:34] Yeah, I guess this is... Well, okay, let's look at the actual tokens to go and get this. I'm getting ourselves a little bit confused, but the diagonals correspond to attending back. Okay, so this isn't going to show us the attribution, unfortunately. It's just going to show us the attention. But in some ways, it will make this a little bit clearer because part of the really cool thing is that we're going to do induction. So here, we're on name. Not only do we want to look...
[0:20:05] Back to the previous sentence and find the translations of name or my name. So je m'appelle, mi nombre, unfortunately I didn't learn Spanish, is. We want to look one forward and see what the next token is going to be so that we can predict is. And so in the case of appell, it's just folded in to the one verb. But here it's a standalone verb.
[0:20:35] Yeah, it is. Okay, well, we can continue moving forward. Okay, well, again, they've managed to find the next, not the same token, but the next token. And looking at the next token is going to allow us to predict that it should be Christopher. Great. And we can keep going. Now we're going to predict that there's going to be a period. Then the period, we go and look one forward. Okay, I. And, you know, again, these are places where the words are pretty different.
[0:21:02] In terms of surface form, GE and AI are very different. But yeah, the model understands that we're doing translation. I guess here we're actually going and having a little bit of a mistake where period GE happens there, but also here. So both of these are period AI. And so now it's sort of skipping a little bit forward and maybe we might sort of erroneously increase the probability of like.
[0:21:32] As well, because we're looking at a fairly short horizon. And we can keep going. Am. Okay, from. And we're getting right there. And a little bit on doing Spanish as well. From Canada. Great. Period again. And it keeps going. And we can do this also with the Spanish part. There's a little bit of it not working that well at the start. But then this head takes off.
[0:22:02] And yeah, now we're correctly going and predicting the next token. So that's pretty cool to me. You have an attention head that's specifically doing translation. And not only is it doing translation, but it's an induction translation head. So that's really cool. Now we can do this with more languages. So I studied Latin in high school.
[0:22:32] I just looked at my Latin textbook and I got the first few sentences of the first project. So, Quintus est puer Romanus, Quintus in Apulia, Habitat, Apulia est in Italia, Scintilla est famina Romana, and Casa Laboret. Quintus is a Roman boy. Okay, so we haven't quite got there, but now we're starting to see. It's starting, right, to look at the previous instances of Quintus and look at the is.
[0:23:04] Great. So we're starting to get an is, okay, Quintus, est, and yeah. So it's not perfect, but it is getting some of this alignment, which is pretty cool. And presumably, you know, Latin is a lot less common in the training set. But we're, you know, now we're starting to pretty confidently go and attend to like, you know, Apulia, est, is. So unfortunately, we can't see attribution here, but we do see
[0:23:35] If we look at this, that it does in a lot of cases successfully increase the probability of the correct next token. So that's, that's pretty cool. Okay, so back to our slides. And yeah, so this phenomenon of soft copying and soft induction seems really important and really generalizes our understanding of these ideas of copying heads and induction heads. And, and so you could sort of think of the space.
[0:24:07] Of copying and induction heads as you have these, these two sort of this larger category of copying and then the subcategory of induction heads. And you can split them into soft versus literal. Now, the soft ones, it's really hard to, in any sort of automated scalable way that I see at least, you know, going automatically detect soft copying because it's sort of a broad category. Maybe if we had the
[0:24:37] Access to the eigenvalues, we could go and do that by looking for large positive eigenvalues, but only some. And similarly for induction, it's hard in these models without these models that aren't just attention heads. But the literal ones, we could just behaviorally look for attention heads that tend to increase the probability of the same token they attended to.
[0:25:06] I think that we sort of categorize something as a copying head if more than 50% of the time when it increases the probability of the token, it was attending to the exact same token. And then of course, I mean, there's lots of cases where a token is extremely similar. So, you know, 50%, you know, maybe like, like for instance, Dursley versus Dursley's or things like this, you know, just very, very slight variations. And then literal induction, the same thing.
[0:25:36] But the token, not only does it need to be the same token, but also the previous token needs to be the same. So these are very strict rules for detecting them. And by that, it's actually about 4.5, 4.6 percent in this pretty large model are doing copying. And similarly, in a slightly smaller model, about 4 percent.
[0:26:05] And then induction, we're seeing that about 3% of the attention heads in the large models seem to be doing literal induction and about 2% in a somewhat smaller model. And of course, you know, for, for these ones, there's, there's probably quite a large number of soft variations. I think, you know, we probably, I think we see maybe more, more of the soft ones than we do the literal ones. So that suggests that actually a pretty substantial fraction
[0:26:35] of attention hints are doing sort of copying and induction-like behavior. And that's pretty cool. And I think we don't know how much of meta-learning is due to these, but it seems like they are really a significant part of what the model does. And it seems like they are at least an important part of the story of how meta-learning works. So that's something that I find pretty exciting.
[0:27:07] It's worth noting, this is rare, but we do see some anti-copying heads. So anti-copying heads and sort of, I think often anti-copying sort of induction heads. So they're like an induction head. They look for places where something similar happens, that are similar to the present, but then they decrease the probability of the exact same thing. And one theory for this is like if you're making a list of items, for instance.
[0:27:35] um like maybe you're doing a list of um i don't know something you can like get models to go and produce things like a list of fruits or a list of um presidents of the united states or something like this you want to say something similar for every item in the list but you don't want to say the exact same thing and so uh anti-copying maybe can help increase the probability of that or you know if you've
[0:27:58] There's a lot of cases where I think you don't want to say the exact same thing and you can increase the probability by decreasing the probability of that. The next couple of videos will really be zooming in on induction heads and maybe it's just briefly saying, why do we think it's worth doing that? Well, one thing is that induction heads actually seem to create a phase change in model training where we'll be able to find this bump.
[0:28:26] on the loss curve. And it will just correspond to the discovery of induction heads. And I don't think there's any other case where we know of, you know, like something that's on the scale of the loss curve, which is sort of the most abstract level you could look at a model. And we can link it all the way down to the circuits level and understand what's going on. So it's really this like, you know, I think that those of us who have been involved in kind of this more circuit-sy reverse engineering
[0:28:55] Models sort of line of research. I think a question has been, you know, we're sort of, I don't know, like if you're trying to make an analogy to physics, we're kind of doing like, you know, this very fine grain, you know, like, I don't know, like mechanics of like.
[0:29:11] Molecules banging into each other or something. And there's a lot of other really cool research that's sort of more like statistical physics and very abstract. And, you know, here's a place where we have a bridge, like a really a pretty substantive bridge. So that's really cool. And probably the next video will be on that. I think another reason this is really exciting is, you know, we... This is the first time... Oh, I didn't finish typing the slide. I'm sorry. But this is...
[0:29:36] I think, you know, it's a theory of what's going on in meta-learning. And meta-learning seems to me like the most mysterious part of neural network behavior or of large language model behavior. How is it that these models are, you know, they're able to learn examples that we show them in context. How the hell do they accomplish that? And so, and I think a lot of safety concerns too that I hear. I think maybe the biggest sort of scariest theories I've heard of how models could really radically
[0:30:06] You know, generalized to new data in really scary ways have to do with meta-learning and how models are accomplishing meta-learning. So there's this idea of mess optimization that you could imagine possibly being what's going on. And if we can say how meta-learning works, we can kind of address that. And then the final reason that I think this is really cool is we can actually predict meta-learning scaling laws from this. So the fact that there's sort of this remarkable fact that as context length,
[0:30:36] It goes on, we get better at predicting tokens. And we'll be able to go and predict that because we now have a theory of what's going on in meta-learning and can then reason about what it implies. So those are all really cool and we'll be able to talk about them very soon.