[0:00:00] Hey, this is Chris, and today I want to talk about how induction heads seem to cause meta-learning. This is preliminary work, but I think it's one of the coolest things that we seem to be discovering about transformer circuits, and so I'm really excited to chat about it. If you haven't watched my colleague Catherine's previous video on how induction heads seem to cause a bump in the loss curve, you should really check that out before watching this video, because this video isn't going to make a ton of sense without it.
[0:00:30] Now, meta-learning is probably, at least to me, it's the most impressive thing about large language models. It's really the thing that's most surprised me about them and about, I think probably, in some ways, maybe the most surprising thing about deep learning in the last number of years. And in fact, it's, I think, so striking that the GPT-3 paper decided to go and make it really the title of the paper is that
[0:01:00] Language models are few-shot learners. So the idea here is when we train models, normally we expect them to learn over training. But with these large language models, we're starting to see that you can go and put examples in the prompt or in the context and they'll learn as they read so they can go and do later things in the context that they couldn't do earlier in the context.
[0:01:26] And this has led to all sorts of interesting things. Like you can go and do so-called prompt design, where you go and you train models or you design these little prompts that instead of training the model, you can sort of get it to do what you want by just designing the prompt. And in fact, there was this little meme with Andrej Karpathy that maybe the future of computing is that we'll just design the prompt. I think that's somewhat joking. But I think the fact that we're sort of joking about things like that sort of
[0:01:56] I think is sort of striking about how interesting a property meta-learning is. And it's deeply mysterious. Like we design neural networks so that they'll be able to go and learn. But somehow when you train large enough language models, you seem to get meta-learning for free. And so that's really interesting. And I really want to know what's going on. So there's a previous paper. This is the Kaplan et al.
[0:02:27] Skilling loss paper. And I think it has a really clever idea for how to think about meta-learning. So normally when we think about models learning, we can look at a loss curve. And you could think of this as the curve describing how the model learns over time. So we have our training step. And as we train the model, our loss goes down.
[0:02:54] And actually, in this loss curve, it's for a two-layer transformer. We can see the induction bump from the previous video. So the idea that Kaplan will have is that you can also go and look at this not as a function of training step, but as a function of token index. You can take the fully trained model right at the end here, and you can ask, how good is it at predicting the first token?
[0:03:22] It's not very good at predicting the first token. How good is it at predicting the second token? Well, it's also not very good at predicting the second token. These are the unigram statistics and then the bigram statistics and the entries of those. But then as we go on, we start to get much later in the prompt. You've already squeezed out all of the short-term distances, but the loss continues to go down.
[0:03:51] And so if the loss is continuing to go down, somehow that means we're getting information from earlier tokens and using it to go and predict later tokens. And so this curve in some ways, it's a little bit like a learning curve, but it's us learning over the context. It's us learning from earlier tokens in the context, how to go and predict later tokens in the context. Now you have to average over lots of examples to see this, but if you average over lots of examples in your training set and then just look at token index, you start to see.
[0:04:20] This curve. And I think this is a little bit of a grandiose way to describe it, but I'm not the one who came up with the idea, so I feel free to describe it that way. You could sort of think of this as a meta-learning curve. So you could think of this as being your learning curve, and you could think of this as being your meta-learning curve. It's describing how you're going and how your meta-learning is progressing. But you could think of both of those as really being slices or projections of a more general 2D picture. So here we have
[0:04:51] Our model over training. So here's snapshots that we saved as the model was training. And we can also look at the token index on this axis. So if we go and we average over this axis and projected all down, we'd get the loss curve. And if we take the final slice here, we get the curve describing how good we are at predicting different tokens.
[0:05:24] At the end of training. So this is sort of a generalization of the diagrams we saw before. And it allows us to get this nice 2D picture and get an overview of what's going on. And yeah, so I think the most obvious and striking thing when you look at this diagram, or at least to me the most striking thing, is that there seems to be kind of a discontinuity. We can zoom in on it. And that corresponds to the bump that we saw earlier and that we discussed in the previous video.
[0:05:53] Where induction heads form. So we can see induction heads forming in this plot. So something that we can then do is we can say, okay, we'd like to somehow summarize how good the model is at meta-learning into a single number. That would be a really useful thing if we could do it. And one way we could try and do it is we could go and take how good we are at predicting, say, the 50th token.
[0:06:25] We can look at that for different points in training. So we can write at the beginning of training and after a little bit and so on. And when we learn about induction heads, we see a little drop, but not a very big one. And then we can go and take how good we are at predicting a token much later in training or much later in the context. And we can see we're better at predicting it and we have a much bigger drop.
[0:06:55] If we subtract those two numbers, you could think of that as kind of being the amount of meta-learning we're doing between token 50 and token 500. So this is in some sense a measure of meta-learning. And the lower it is, the better we are at meta-learning. And we see that there's this really abrupt drop, which happens right at the bump, right at the point where we think that induction heads are forming.
[0:07:24] So that's really interesting and suspicious. And I don't know about you, but this is like, you know, this is a really sharp curve. Like it's almost, you know, it's not exactly like this. There's a little bit of maybe bending here, but it's almost just an abrupt discontinuity. I mean, it's pretty discontinuous. So that's kind of an unusual thing. And yes, seeing things like that, it's very striking to me.
[0:07:54] Okay, so one question you might ask is, you know, how significant is this, right? Like, how big a deal is it that we're getting, you know, this negative 0.4 nats of meta-learning? Our losses is measured in nats from information theory. What does that mean? And I think there's a few ways you can interpret it. So in the case of this model, it's an 11th of the loss.
[0:08:23] So one way you could think about it is that means that when we predict token 500, it's almost as though we get to go and predict, like maybe every 11th token, we're able to go, we get sort of get to for free, just magically know the right answer and say it was 100% probability. That would cause the same, you know, losing reduction of an 11th of our loss. Another interpretation of it is,
[0:08:54] It's 0.4 NATs per token. But NATs are kind of a tricky way to think about things. I prefer to think about things in information theory in terms of bits. So let's convert to bits. That's about 0.5 bits a token, which is about one bit every two tokens. And remember that a bit is enough information to distinguish between two things. So it ends up meaning something like you could sample the model twice and take the better token or take the better sample.
[0:09:24] Every two tokens. That would be another thing that it would mean to go and drop by 0.4 nats. So I think it really is a very meaningful drop. And we were already pretty far out in the context by token 50 and we're still getting that much more. So that's pretty impressive. Okay. So another way we could picture this is in some ways the important thing is how much more information are we as we move
[0:09:54] To later tokens, how much more information are we squeezing out and reducing our loss by? And before we discover induction heads, we see that, you know, we pretty early on, you know, it basically becomes zero and we're not getting very much more. And then all of a sudden, so this is, sorry, this is the derivative with respect to log token log, because it seems like,
[0:10:25] It seems like probably the order of magnitude of tokens that you have before you seems to be the more important thing. If you look at these curves, they seem very linear on a log axis, but not on a linear axis. We'll take the derivative with respect to the log token. It seems like before we discover induction heads, maybe the first 20 or 40 tokens, 50 tokens, you're still getting pretty significant learning.
[0:10:54] Or pretty significant reductions in your loss. But it pretty quickly levels out after token 50. And as you go further down, you're not learning much. Whereas after induction heads form, we continue to see these really significant reductions in loss. So that's really interesting. And then if we take a further derivative and we ask, okay, well, you know, how is our ability to go and increase? So we have our first derivative which tells us as we go this direction,
[0:11:23] How is our ability to predict tokens going and increasing? And if we take a derivative this way, we can ask, how is our ability to go and predict tokens improving? Our ability to get better at predicting tokens, how is that improving as we go and we move this way? And we can see that there's, you know, I mean, you can just visually see it here, but it's nice to confirm. There's this extremely, extremely, you know, sharp discontinuity that really, really dominates the story.
[0:11:52] To first order, the story of meta-learning is that in the span of a few snapshots, which is really just a few hundred steps of training, we go from not being very good at it to being quite good at it, and that's really the story of meta-learning. Now, this is for a small model. This is only for a two-layer attention-only transformer where it's really easy to study things. But we can also look at large models, and large models also seem to have a bump.
[0:12:21] It's right here. And that's at about, and now we're measuring in tokens. And so that's at about 10 to the nine tokens. And it turns out that if you go and measure the same notion of, of doing a difference between how could you out-predict these two tokens to go and look at, get sort of a meta-learning score, right at 10 to the nine tokens, we see an abrupt drop, an abrupt discontinuous drop.
[0:12:51] And it seems to not matter that much how big your model is. Across several orders of magnitude, they all drop in a pretty similar way. Now, right at the beginning, the large models are better at predicting than the small models are. But after the drop, the meta-learning scores really dominated just by which side of the drop you're on, which side of this bump you're on, and not matter that much.
[0:13:18] How large your model is. That's kind of the opposite of what I usually think about model size. Usually I expect model size to just make everything better. But here we have something that to a pretty significant extent is it's just the same amount regardless of model size. So that's interesting. In any case, that's kind of rambling a little bit. But I think that the striking thing is that this is still happening in large models. And so the thing that we, you know, to me that's crying out when I see this pattern is what the hell was happening when we see this drop?
[0:13:45] Now, of course, I've sort of already given away the theory we have, which is that it's induction heads. But, you know, we'll talk about that more in a sec. But, you know, I think that it's really, you know, maybe one reason why I'm so struck by this is I don't know about you, but I'm used to seeing weird things in loss curves. In fact, this is like a pretty smooth loss curve, all things considered. You know, it's not that unusual to have random bumps and quirks in your loss curve.
[0:14:14] This thing that seems at first glance like a relatively not that big a deal thing and everyday phenomena turns out to be doing something like really the most interesting property of models to me is undergoing this phase change at that point. It's dramatic. That's really interesting. The theory that we're going to have is that this is driven by induction heads. Now, remember that an induction head,
[0:14:45] It's kind of doing a nearest neighbors lookup. So it says, okay, this is my present token. Well, actually, maybe I might even include some previous tokens. But let's just focus on the present token for a second. So here we have the first paragraph of Harry Potter, because I find it fun to use the first paragraph of Harry Potter. And we have the token D. Okay. So now we're going to go and look for the token D, but shifted one forward. So we're going to look at what...
[0:15:13] What happens next? And then we're going to look at what happened next and increase the probability that that is the next token. So we did this kind of nearest neighbors lookup over our present token and surrounding context, and then looked for something similar, shifted one forward, and used that to go and predict the next token. And so this kind of nearest neighbor search over our context is really the
[0:15:45] The thing that induction heads add. And you have to remember that in order to do this, you have to have a previous attention head that can go and do this shifting. So you need to have two attention heads in your circuit to go and implement this. First, you have to have one attention head that can shift information forward. And then you have the attention head that goes and uses that shifted information to go and match something to the shifted version and then go in and increase the logic. So that's what an induction head is. And it's very natural that something like this could be used
[0:16:16] For meta-learning, because it's allowing you to search and find similar situations and then do what happened last time when you had a similar situation. Okay, so the hypothesis is that induction heads drive meta-learning. And we're going to have a few pieces of evidence. I think the big ones are for small models, well, small attention-only models, we can really show that induction heads are causing the bump and they're causing meta-learning.
[0:16:45] And then for larger models, the story is going to be a little bit harder. We'll still be able to say that meta-learning forms at the bump. And we'll still be able to go and say that induction heads form at the same time as the meta-learning forms. But it'll only be a correlational argument. So this does leave open some possibilities that maybe there's additional things that are happening at that bump, at that weird transition.
[0:17:12] And it could be the case that large models are different. So we can sort of rigorously understand this in small models. In large models, we just have correlational evidence. And it could be that there's other things. We'll talk about that more in a minute. Okay, so for small models, for small attention-only models, one thing that's nice is you can just sort of mathematically define induction heads. Because remember that they have, if you've watched our early videos where we have this really nice
[0:17:41] Analytical framework for understanding these small models. An induction head is one that has positive OV, output value, eigenvalues, and also has positive QK eigenvalues attending to the previous token. So that gives us a nice mathematical definition and we can turn that.
[0:18:01] Into a score. We talked about this more in the previous video with Catherine, but you can turn this into a measure of how much something is an induction head by looking at the fraction of the eigenvalues that are positive and then going and taking a minimum from OV and QK. And so here what we have is we've colored each one of these lines is a induction head. And we're seeing if you knock it out, how does the meta-learning score change?
[0:18:32] A positive value means that the meta-learning score increases, but remember that negative value, it's a delta and loss. A lower value meant that we were better at meta-learning. So having a higher value here means that the induction head is improving our ability to do meta-learning. Having a lower value means that knocking it out actually seems to, well, knocking it out improves it, so it's sort of doing the opposite.
[0:19:01] And we'll talk about that in a little bit as well. And then they're colored by the score that tells us how much they are induction heads. So the obvious story here is, well, we have this one head that suddenly starts to become an induction head and becomes right at the bump, starts to go and cause, yeah, really, really drive meta-learning. I should say this line here is the start of the bump.
[0:19:30] And this line here is the end of the bump. And there's some other heads that also become induction heads and contribute to meta-learning. Now there's this weird thing where they all sort of, basically all of the heads seem to sort of rush to contribute to meta-learning and then stop and then some recover, the induction heads recover. We don't know exactly what's going on there, but the theory is probably that they...
[0:19:59] You know, once they figure out they can do induction heads, a lot of attention heads try to do it because it'd be very useful. And then there's one winner or a few winners and the others go and do something else. Now, there's one outlier here, which is sort of an induction head. It's a little bit borderline. Maybe it's sort of weekly scores on this induction head score. So it has some negative eigenvalues, some positive ones, and it's really not doing meta learning. So we don't know what's up with that.
[0:20:29] Another thing you might ask is, how can there be heads that are sort of doing the opposite of meta-learning? Well, my guess is that these are heads that when you knock them out, damage the model more generally. And damaging the model more generally might just mean that you're predicting tokens more uniformly. So that would be, sorry.
[0:20:58] Knocking out more might mean that you're, I mean, I guess it's sort of clear that this theory is very speculative and it's not something I thought through very carefully, but it could be that this destroys your ability to do n-gram statistics or something like this. And so the only thing that remains maybe is your meta-learning and it's able to go in and contribute more to the loss. So that might be one theory. Like if you're bad at predicting early tokens, then the meta-learning score would increase.
[0:21:26] But yeah, I think the significant story here to me is that very clearly the induction heads are the ones that are driving meta-learning. And because this is an attention-only model, there isn't anything else that could be driving meta-learning. So we know that in this case, the model is really, yeah, the meta-learning is really coming from the induction heads. Yeah, so that, so yeah, okay, so there's a few other arguments for why you might think that
[0:21:59] In the case of small attention-only models, the induction heads are really driving this meta-learning change. Another one is just that we really understand these models and there isn't really anything else that seems like it could be driving meta-learning. Also, we really carefully in the previous city were able to analyze the induction head bump, which seems to correspond to this meta-learning phase change.
[0:22:27] In that case, we were able to do things like look at individual token losses and study how those tokens were changing. And those were both cases that sort of seem like you could interpret them as kinds of meta-learning, and they all very naturally flowed from having induction heads. So if you haven't watched the previous video, again, I'll encourage you to go and watch it.
[0:22:52] Now for large models, I think this is a trickier argument. When you have thousands of attention heads and a large model that's more expensive to run, it starts to become annoying to go and do ablations of every single head. So we're left instead with a more correlational argument. We can see that meta-learning develops right at token 10 to the 9 or right around token 10 to the 9. And we can see that that's when
[0:23:22] Induction heads form. And this is across a wide range of model sizes. The score is the score of the attention head that most matches an empirical estimate. So we can't use the, remember that previously in the small models we could, I mentioned we could, we could go and detect induction heads based on their eigenvalues. That doesn't work in large models because of MLPs. So instead, we have to go
[0:23:51] And come up with some empirical way to detect induction heads. And what we do is we just create a string of some random tokens and we see if it's able to go and look at the next token. And because it's a sequence of random tokens, the only way that it could sort of go and look at the token that previously followed the present token is if it's actually doing induction. It couldn't be relying on other statistics because it's a random sequence. So it really has to be doing
[0:24:19] This kind of search through the context and then look what happens next. That's the only way. And these attention heads are really accurate at it, right? Like some of these are getting up to like, you know, suddenly in the 90 percentiles. And then they start to fall. I think that's because there's multiple induction heads that are taking on different roles. And so, you know, maybe one matches when...
[0:24:43] There's like multiple tokens within a word and another one matches in other cases and so on. So then they start to become, you know, only look at a smaller fraction of those. But yeah, so at 10 to the 9 they form, and that's when the meta-learning forms as well. So that's kind of a natural argument, right? That sort of seems to me, you know, it would be a very shocking correlation to just happen by happenstance.
[0:25:12] Now, it could be that there's additional things happening there, but we know in the small model case that this is really the thing that's driving it, and it certainly looks very similar here. We know that the same thing's happening and the same result's happening, and so it's natural to draw that inference. Now, could it be the case that there are additional things happening here? Yes, it could be, and we'll talk about that more later, but it could be that there are additional things driving.
[0:25:37] Meta-learning and large models that happen to form at the same time as induction heads and that we haven't yet noticed. And we'll talk about how plausible that is later. But yeah, to first order, I think it looks like the, yeah, it looks like induction heads are probably a big part of what's going on. One other thing that's maybe worth mentioning, this is really an aside, it is kind of cute. This kind of, there's sort of this oscillating feeling around the induction head bump.
[0:26:08] It actually seems like that might be real. It's not something that we studied carefully, but it does seem, it's the same thing that we were seeing earlier with having some attention heads sort of overcorrect or something and have to bounce around a little bit. That may or may not be, yeah, it's a theory. Okay, so our hypothesis is that induction heads are implementing
[0:26:37] Meta-learning and Transformers. And that's kind of a wild claim to me. So I think it sort of is clear or it sort of makes sense that induction heads could go and predict later tokens or predict copied tokens. So like here, we have the first paragraph of Harry Potter. We're at this token D. We look back. We see that there's an errs. We're like, oh, it's going to be errs again. And we increase the probability of that token. Then we're...
[0:27:07] On the token Errs, and we look back at Errs, and we see, ah, the next token was Li. And we predict Li, and we get a better loss than that. So it makes sense that induction heads can copy repeated chunks of text, both very short turns of phrases or names or things like that, and also longer things. And I think it also makes sense that they can, like we saw in one of the previous videos, that there was...
[0:27:36] A cross-language induction head that looked for words that are in different languages and then looks one forward and allows you to go and generate translated versions of text. So there's other things like this. And it makes sense that those would cause your loss to go down later in your context. But is it really plausible that those could be explaining meta-learning? These large language models, they can do all these things that...
[0:28:04] Are different from, you know, they aren't just going and root copying things. They learn functions and they can go and do all sorts of, you know, interesting generalizations on top of the thing that they learned. So, yeah. So could it be that these induction heads are doing the same thing?
[0:28:33] Well, okay, so one thing you could think of is that maybe the more general thing that induction heads are doing is they're a kind of meta-learning nearest neighbors. They're doing a nearest neighbor search over the context. And that could be, you know, going and matching literal previous token and literal next token. But they can also be softer and they can also copy more flexible things. We saw in a previous video, not an induction head, but a copying head that was copying.
[0:28:59] Things like gender and tense and plurality to go and keep sort of manage agreement between adjacent sentences and stuff like this. Well, you could have induction heads be soft as well. The translation head was one of those. And it turns out that these soft induction heads can also help you learn functions. And that's the thing that really made me start to, maybe at least it was the thing that knocked down the last barrier to me of...
[0:29:27] Thinking that induction heads might actually be the thing that's driving meta-learning. Sorry. What we have here is a little toy problem that I set up. We're trying to simulate a pattern matching problem that we know for sure the model has never seen before because I made it up and it's very silly.
[0:29:58] And will allow us to go and explore the role of some softer induction heads in going and learning functions. Okay, so the puzzle here or the problem here is sometimes the first word is a color and sometimes it's a month. So sometimes it's a color, sometimes it's a month. And sometimes the second word is an animal and sometimes it's a fruit. And then we interpret each one of those
[0:30:28] That's a binary variable and we do XOR. So green lizard is false because it's a color and an animal. And similarly, a month and a fruit is false. But the other combinations, so a color and a fruit will be true. And a month and an animal will be true. Okay, so...
[0:30:58] This is a pretty arbitrary function. And the thing that we'd like to see is when we are on a colon, the model should look for a previous instance that is the same, conceptually the same as our situation, and then go and look at what the answer is. So here we have a month animal, and it attends back to a previous month animal and says, ah, it was true.
[0:31:25] In fact, it's looking for one where the month is the same. So it's April and April. And I guess dogs and cats are kind of similar. So maybe that's why it likes it. It's also looking at March lizard a little bit, which is also a month animal. But it's putting less weight there because I guess it's less of a close match from its perspective. It's working on high-level linguistic features. That makes sense. Okay, and if we go to January wolf, this is also a month.
[0:31:54] Animal, and it wants to get his one that's January as well. So it goes and it looks at true. And here's another one, looks at true. Over here we have March, Lion. I guess Wolf and Lion are kind of similar, so maybe he likes that. And here we have a Wolf. It's true. But it's successfully going and looking at these cases. Okay, let's look at one that isn't. So here we have a color and a fruit. So Purple, Lemon. Well, okay, we attend to Red, Cherry. That's true.
[0:32:24] We go to purple apple, that's true. We go to gray lemon, that's true. So we're looking at these previous cases of this function. Okay, here we have another, oh no, here we have a month fruit. And when we look here, we have October cherry, and it's false. Now this one is actually an error.
[0:32:53] It's looking at the wrong thing. It's probably because we have September here, and we have September here, and it likes that. And then we also have cherry on the previous line, so it's attending to the wrong place, but it's noticing that there's a cherry. Oh, I guess in this case there was also a cherry on the previous line, so maybe that's why it got confused. But it's giving most of its probability to a correct case, and it's just putting a little bit of probability on this incorrect one, so we're still going to get the right answer. Oh, I guess we're also putting some...
[0:33:24] Some probability here, so yeah. The one that's getting the highest probability is still correct. Let's keep going. What about, yeah, we could look at this one now. Okay, we have Red Cherry, true. Gray Cherry, true. Gray Strawberry, true. Red Apple, true. So all those are correct cases to attend to and use. And now we're back at a month.
[0:33:55] Animal. And yeah, we're going and seeing July fish is the one that's getting the most. April cat, that's another valid one. We're giving a little bit of probability, actually quite a bit of probability to this one, which is not a correct case to attend to, but mostly we're attending to the correct cases. So we're still going to go and get the answer right. So the thing that I think is so cool about this is this is a completely made up nonsense function and the model has learned it.
[0:34:24] By going and using previous examples. And it's, you know, it's not just, well, okay, I guess I'm sort of making a fool of myself a little bit, but it's not just, you know, I'm showing you ones later on, but it's also early, right? So here we have Red Frog, and then we're looking to Green Frog and Blue Lizard. Those are valid cases. We're putting a little bit of probability on one of the wrong ones.
[0:34:54] In this one, we put quite a bit of probability on the wrong one, but we're still putting probability on two correct ones, so that might still win out. This also works early in the context. Works better later in the context, but also works early in the context. We can look at the effect on the logits and see when are they made more probable. We can see that this one is made more probable by a correct match and by correct matches and correct matches. There's all of these
[0:35:24] And it's working. So it really does seem like these soft induction heads are able to go and allow for the model to learn functions like this, which is to me really cool. Because the ability to learn functions and pattern match like this, I think was one of the things about large language models to me that was sort of most demonstrative that they were really
[0:35:53] Doing in context meta-learning. And so now we have sort of a theory of, of why that's the case. Okay. So that's that. Hopefully that makes it at least seem plausible that induction heads could be doing at least a lot of meta-learning. Could other things be contributing? Yes. So it's certainly possible that in large models, other things could be contributing. Now.
[0:36:21] I think there's two kinds of barriers to that as a theory. One is it really seems like they would have to be forming more or less simultaneously with induction heads. We know that when induction heads form, that is when the meta-learning forms as well. If there's something else, it seems like they have to form there as well. It also seems
[0:36:51] Like we need to explain why, so we know that in the small attention-only models, meta-learning is really driven by induction heads. We can sort of just look at everything and go and show that it's the induction heads that are driving things. And so I think you have to explain then why is it that first of all, whatever you're hypothesizing has to not form in these small models. That seems reasonable. It could be that there's things that just don't form in small models and do form in large models.
[0:37:20] But then why is it that we get this constant amount of meta-learning regardless of model size? Why is it that the large models aren't doing better at meta-learning? Now, in some sense, there's a way in which this is a little bit deceiving, which is they're getting the same number of bits from meta-learning, but their loss is lower.
[0:37:49] Maybe a harder bits of information. And we know that large models have more induction heads and more soft induction heads, like the translation head or the pattern matching head. Those don't exist in really tiny models. So I think that's a little bit of a strange argument that I'm making here, but it is very suspicious to me. The same amount of meta-learning gets learned in all models. And that does seem like an argument for, I have no idea why that should be the case, but to the extent that it's true, it does point in the direction that somehow whatever's going on in all these models is the same.
[0:38:20] But who knows? I think that is very much an open question. Okay. Now, another question that I've been wondering about is, is it possible that induction heads are the reason transformers do better than LSTMs? Okay, so let me unpack that a little bit because that's going to be sort of an out of the blue question. Well, Kaplan et al. notice that, this is the scaling laws paper from 2020, and they have a plot that, yeah, it's not very emphasized, but they're the ones we got this whole idea.
[0:38:50] Of going and looking at loss as a function of token index from. Well, they make a really interesting observation, which is if you look at transformers, the loss continues to go down. But if you look at LSTMs of the same size, for a while they do kind of similarly, but then they flatten out. And they tend to flatten out somewhere before.
[0:39:20] Token 100. So we have token 100 here. And somewhere before then or around then, I think at least a little bit before then, they flatten out. Okay. Well, that's kind of interesting because we saw something flatten out earlier. We saw that before induction heads form, we flatten out, you know, we start to flatten out probably around token 40 or 50. And, you know, by token 100, we're really starting to flatten out. We've a little bit, get a little bit more here.
[0:39:49] But we're really significantly flattening out and are basically flat. We're significantly flat, I think, from somewhere between token of 50 and 100 onwards. So that kind of rhymes with this result. It's certainly not error-proof, but it is really suspicious to me that LSTMs are happening to flatten out roughly the same time.
[0:40:18] Transformers before they develop induction heads do. And there's another reason why I think it's a little bit suspicious, which is induction heads are fundamentally a mechanism that it seems like transformers couldn't implement because they're searching over the previous context. And so you can't do that with a constant amount of time per step because it's...
[0:40:47] Probably you have to do some compute for every previous step. Or another way to think about this is that going and doing search for every token over all previous tokens, natively that's going to be the ON squared. But transformers only have ON compute. And if you were really clever, maybe you could make it ON log N. But I think probably...
[0:41:12] You know, neural nets aren't discovering those algorithms. And even if they were, it's still larger than ON. So it really seems like that's something that it seems like LSTMs couldn't be doing. And so we know that before induction transformers are relying on these simple copying heads. And those do seem like something that LSTMs could be doing.
[0:41:37] Something analogous to, you know, just being like, oh, you know, I saw this token previously, probably it's going to happen again. And so it really, yeah, I don't know. Again, this is speculation. I don't think we rigorously know this is the case, but it is kind of suspicious. And, you know, I do wonder. So that was a fun little thing. One other thing that I think is worth talking about is, well, okay, so at least,
[0:42:07] Now, okay, I think this is a little unclear, but if you put your context on a log axis, these curves, these meta-learning curves, they do look pretty linear, especially if you put your loss on a log axis as well. So it does seem like meta-learning within a context is probably modeled pretty well as a scaling law. And there's a paper by,
[0:42:38] by Marcus Hutter from DeepMind. And he gives sort of an argument that we should really expect models to follow scaling laws, at least with respect to data, that as you go and you give them more data, you should expect them to follow a scaling law. And something I found interesting as I was thinking about this is, you know, these models, they follow, okay, so we seem to be observing scaling laws. Well,
[0:43:08] Actually, I think the case of meta-learning with induction heads matches the theoretical argument that Marcus is giving better probably than the standard case of going and training models. The theoretical stop in the Hutter paper is that you're observing a bunch of data points.
[0:43:38] If you've seen a particular problem before, then in the future you get zero loss on it. So you're observing the sequence of training examples, and then if you see the same thing again, you go and you get it perfectly, and otherwise you get a bad loss. You get the same bad loss. So that seems very similar to having a context and seeing a bunch of tokens. And if you see a token that matches...
[0:44:06] Something earlier in your context, you can look at what happens next and get a very, very small loss. That seems very similar. And so you can really take just all the arguments that are in this paper and they run through for the case of meta-learning with induction heads. And so we should really, really expect scaling loss. So I thought that was kind of fun. Okay, so in summary, meta-learning seems to form really abruptly in the span of a few hundred steps.
[0:44:36] And induction heads form at the same time. So that's interesting. Induction heads can be seen as a kind of nearest neighbor meta-learning algorithm. And we saw that they could even, even learn functions. And in addition to copying text. And induction heads seem to substantially cause meta-learning. Certainly they form at the same time as meta-learning and in small models they can really be shown to be causing it.
[0:45:06] Yeah, I don't know. I guess I'm just excited about all of this because it seems like it's getting really, you know, potentially some deep insight or getting at some of, you know, what seems to me one of the biggest mysteries of these large models. And so that was really exciting to me. In any case, thanks for listening.