[0:00:00] Hi everyone, this is Catherine and I'm excited to be sharing with you today an observation that we've made in some of our smaller transformer models and some analyses we've done that give us a sense of what's really going on there.
[0:00:14] So unlike some of our previous videos that take a specific piece of theory or a specific small detail that we've observed, this video will take a macro scale observation and slice and dice it in a lot of different ways from different perspectives using many of the tools that we've introduced so far. So without further ado, I'm excited to introduce this phenomenon to you and explain what we understand about it so far.
[0:00:42] So here's the phenomenon. We call this the bump. So I'm showing you two loss curves here from small transformer models. The one in red is a one layer attention only transformer. The one orange is a two layer attention only transformer. And what immediately stood out to us is that at a certain point, their trajectories diverge and the two layer starts getting dramatically lower loss and performing better than the one layer beyond that point.
[0:01:11] And we first became particularly interested in this after Chris made an accidental discovery about these trajectories. So he had launched a server using Garson, which we've described in previous videos. So he launched a server with a model that he didn't realize wasn't done training yet. So he was looking at a two-layer model from quite early in training.
[0:01:33] And was surprised to discover that it did not have any induction heads, which was disappointing because that's what he wanted to study. So he quickly realized his mistake and then
[0:01:45] waited for the model to train further and the two-layer model later in training did of course have induction heads and he could perceive with his analysis. But this was quite interesting to observe that the two-layer early did not have induction heads and later did and in fact these two observation points were on either side of the bump.
[0:02:05] So this led us to develop a hypothesis that this divergence in training trajectories between the one layer and larger models had something to do with induction heads. So that's the main hypothesis that we will be exploring today. Another reason that we found this hypothesis plausible, that the bump has to do with the discovery of induction heads, is that induction heads can only form if there are two or more attention layers. I'll review why that is.
[0:02:34] And I'll also say that, you know, this was our initial hypothesis was that discovering induction heads is what's going on. My current view is a little more nuanced. I think there's something about really specializing as induction heads, really fully coming into their role that's important to the story of what's going on here.
[0:02:55] Let me introduce the rough structure of how I'll present our findings. So we're going to answer three different questions that you might have about this phenomenon. The first one is generality. So are we looking at the same bump in different models? Is this a generalizable phenomenon?
[0:03:16] The second question is behavior. So the model makes different predictions before and after the bump. What specifically changes about those predictions? Is it just overall doing better across the board at every token? Or are there particular types of tokens that are changing in terms of the model's predictions about them?
[0:03:35] Finally, we'll get into the circuits level. So what's going on structurally in the model's weights and matrices that produces the prediction changes that we are observing.
[0:03:47] So you might wonder why this is interesting. I think this is interesting for two big reasons. The first is it's evidence that the model's behavior can change quite suddenly in just a few hundred steps from before to after the bump. The model's behavior changes drastically and in a really noticeable way. It shows up on the overall loss curve, but even more strikingly when we drill down even further. And I think understanding sudden shifts.
[0:04:14] is safety relevant because one type of concern you might have about larger models is that they're behaving in ways that you approve of and then up until a certain point where they start executing a different strategy that you don't like. And so if these kinds of dramatic shifts in how the model understands the world and how the model's internal structure works can happen so suddenly that type of behavior
[0:04:44] Something we really would like to understand and to know how common that is and how you can see it coming and that kind of thing. I also find this interesting because we can connect the circuits level analysis with that macro scale learning dynamic level of analysis. Something that shows up in the overall loss curve in the macro scale behavior of the model. We've managed to connect all the way down to specific circuits and specific attention heads, at least in these small models and with parallels to large models.
[0:05:13] And that's pretty exciting. That's more end to end than many other things we've been able to do so far. So let's go question by question. So I'll start with generality and show you how this bump occurs across different models. So in terms of generality, I'll first show you these small models that we work with. So this is the same graph I was showing you before with the one layer and two layer attention only model.
[0:05:41] And I'll show a couple more attention only models on the left here. And I'll also show a few models with MLPs on the right. So here I've added the rest of the models in different colors. And you can see that the one through four layer attention only and with MLPs all show some kind of a bump. It shows up at different places for each model.
[0:06:03] There isn't really a particular size trend. So the two layers somewhat intermediate, the three layer develops the bump later, the four layer earlier. We haven't gone into much depth about why specific models develop the bump at specific times, but it does seem to be a pretty general phenomenon.
[0:06:22] And we see something like this in our larger models as well. So here are models with millions to a few billion parameters. And these also show a bump. Here the x-axis is tokens rather than steps. But at about the same number of tokens, there is the same bump.
[0:06:44] And I won't go in this video into explaining why I think this is the same phenomenon, but in the next video on meta-learning that Chris will present, he'll provide some evidence for why this is the same phenomenon that we see in our tiny models as well. So let's go into behavior. So in this, I'll be looking at how the token predictions change. So, and particularly the data that I'll be analyzing is the token by token loss trajectory.
[0:07:15] So you're probably familiar with the average loss curve, which is an average over many different tokens. But that overall summary is, of course, made out of thousands of individual tokens, distinct and discrete loss trajectories that have their own behaviors that, when averaged out, look like the loss curve we're all familiar with. For much of this analysis, I'll be working not with the average, but with that token by token broken down
[0:07:44] Individual loss trajectory set or matrix. And we'll be asking which specific tokens losses are the ones that improve at the bump. So let me just show you a visual sense of what I mean by that token by token loss. So here's the familiar average token loss curve over the course of training step. The loss improves on average.
[0:08:10] Now of course this is made up of individual tokens, so I'll show you an arbitrary token from the Harry Potter demo text. And you can see that the model's performance on this example token looks very little like its overall average. I'll smooth this out just to make it a little easier to see. And you observe that the model performs quite poorly on this token over its entire training lifetime.
[0:08:37] And in fact, it doesn't seem to get much worse. This isn't too surprising, not only because a two-layer model is just not very smart, but also because some tokens are just intrinsically very surprising and no model can do particularly well at predicting them.
[0:08:55] So I'll also add in a couple other arbitrary tokens from that same demo text. And you can see that they have very different personalities. There's a lot of idiosyncratic variation in terms of how well overall the model performs and how that performance changes over time. So these variable and idiosyncratic token trajectories are what we'll be analyzing in the rest of this behavior related analysis. So one other thing I'll also point.
[0:09:25] I'll be drawing some lines on this plot. The left line will be at the start of the bump. The right line, the dotted line, will be at the end of the bump. And then we can see where that turns up in some example individual token trajectories.
[0:09:47] So here are those same two lines and you can see that these tokens that I've selected show different behaviors over the course of the bump. So this token gets suddenly noticeably worse and remains worse after the bump. So it has become somehow more surprising or harder to predict and other tokens seem to do better.
[0:10:09] Another interesting thing is that before the bump starts, there was some kind of flattening out or plateauing, maybe even a little worsening before this bump related improvement kicked in. So what I'd like you to take away from this is that individual token trajectories contain a lot of information that can help us understand what particularly is going on at the time of the bump.
[0:10:35] So I'll analyze these individual token trajectories with three different ways of summarizing them. The first one is PCA. So I'll look at the principal components of that snapshots by tokens matrix. I'll also build something that I'm going to refer to as the meta-learning index, which is a measure of how well a model is performing at in-context learning, how much better it does at later tokens than earlier.
[0:11:04] And then I'll also look at this before and after token loss difference. That is to say the overall difference between the token losses before the bump versus after the bump. If you subtract those behaviors, what's that delta? What's that delta vector in token space? So we'll go one at a time and break this down.
[0:11:27] So in terms of PCA of the snapshots by tokens matrix, I just want to make sure to clarify what that matrix is. So I think many viewers will be familiar with PCA as a way to analyze or get a quick understanding of the main dimensions of variation in a dataset. And so the dataset we're talking about in this case is the token behaviors of all of these different models.
[0:11:56] All of these models, I mean, we not only have eight different architectures, we have four attention-only architectures, four architectures with MLPs, but also each of these trajectories is made up of many individual snapshots. I have 200 different snapshots over the course of training. For each of the eight models, they're saved every 50 steps. So overall, that's going to be 1,200 snapshots whose token behaviors we'll be checking out.
[0:12:25] Each model has been shown 10,000 different contexts and a random token has been selected for each context. So overall, this is a 1,200 snapshot by 10,000 token matrix that we'll be doing PCA on. Here's the result of that in the first and second principal component. So the thing that immediately jumped out to me is that this bump
[0:12:53] is very prominent so I've sort of marked these locations right at the bottom of the swoosh and these locations at the top of the swoosh and those correspond quite precisely to the start and the end of the bump on the overall loss curve.
[0:13:12] So you'll also notice that the one-layer models never undergo that swoosh. They just remain in that original sort of trajectory direction. And it's the two layers and further that are undergoing this substantial transformation, mostly along the second principal component.
[0:13:30] I won't be analyzing in much more depth what these principal components are. The main thing that I want you to take away from this is that straight away in the most significant dimensions of variation of this data set, the bump is extremely obvious, perhaps even more obvious than it is in that average loss view. So this tells us that clearly something very major is changing at this point of the model's development.
[0:14:03] So the next lens that I'll introduce on understanding that token wise loss matrix is meta learning or in context learning. So.
[0:14:13] Meta-learning is the phenomenon that later tokens are easier for the model to predict. By meta-learning, we mean sort of learning how to learn. So the model has learned how to take information from early in the context and adjust its later predictions. So as a measure of how good at this the model is, we'll take the difference between its loss on earlier tokens and loss on later tokens. So specifically, we'll take the loss at token 500.
[0:14:43] Out of a 512 length context and subtract off the loss at the 50th token. I'll point out we're using token 50 rather than something like token 5 because those first few tokens are intrinsically very hard to predict and quite noisy. So the 50th token removes some of that.
[0:15:01] variation while still giving us a measure of how good the model is at learning from early tokens to better predict later tokens. So this is the index that I'll be plotting.
[0:15:15] Just another way of visualizing what that measurement is. At the top here, that's the loss on the 50th token for a particular model. Then the bottom in blue is the loss on the 500th token, which is better and increasingly better over model training. It's getting better at meta-learning over the course of training. And this difference here.
[0:15:38] Uh, is that meta-learning index that I'll be graphing and in the future upcoming video on meta-learning, Chris will break this down in much more detail as well. So here's that meta-learning index. Again, I've pulled out just the one layer and two layer attention only, and I'll show the rest in a moment. Um, as you can see, the one layer attention only model has
[0:16:02] Never really learned particularly much meta learning and the two layer really nails it. In particular, these two lines I've drawn here are the start and the end of the bump. And you can see that regime is where the model really, really comes into its own in terms of its ability to use earlier tokens to perform better on later tokens. This seems to be a major thing that that bump related transition is all about.
[0:16:32] Here are the rest of the models. I haven't drawn those bump start and end lines just to make it a little easier to see, but what I'll draw your attention to is that they all undergo, other than of course the one layer, they all undergo this steep transition between being mediocre at meta learning and being quite skilled at it.
[0:16:53] The location of this transition does correspond with the bump quite well. As before, the two layers somewhat in the middle, the three layers later, the four layers earlier, as we saw before. This is another lens that tells us that the bump is a
[0:17:09] Drastic difference. It's a very striking change in the model's behavior. Perhaps again, as with PCA, it looks more striking in this view than it did in the overall loss curve. So something very significant is changing about how the model's behaving.
[0:17:26] So the final lens that I'll take on behavior is this before and after token loss difference. So that's just a simple measure of what is different in a snapshot after versus before the bump. So I'll be taking two different snapshots. All the data I'm presenting has already been smoothed. So some of the noise is reduced, but I'll just take a sort of smoothed.
[0:17:52] Loss snapshot here and a smooth loss snapshot here. And this is that full 10,000 token multi-dimensional, multi-token loss. And I'll just be taking the difference of those.
[0:18:07] So that's, as I said, a multi-token loss. I've shown here loss on the Harry Potter demo text. Now, of course, this isn't 10,000 random tokens, but you can do the same operation on a demo text like this. So the difference I'll be taking is between these token-wise losses after the bump and the token-wise losses before the bump.
[0:18:30] If we look at these raw losses, it's a little difficult to immediately see what's different, although you can see some straight away, sort of with the naked eye. I'll call out just briefly.
[0:18:42] That this phrase, Mrs. Dursley, which is one of the most common repeated phrases in this demo text, the lighter color here in this case is lower loss. So the model after the bump is visibly doing better at this repeated token sequence than it was before. But I won't linger here. I'll go ahead and take that difference and then the deltas will be more visible.
[0:19:10] So here is that difference of the after versus before. And in this case, the red colors is better performance on these tokens. I'll also say I've taken this example from the four layer attention only model because it was a crisper and clearer depiction than some of the others. So that's a little bit of cherry picking for the sake of clarity, but the other models did show a very similar, albeit somewhat fuzzier pattern.
[0:19:40] So the main thing I want to draw your attention to is these repeated token sequences that we're so familiar with in this particular demo text that we use.
[0:19:50] Often because of these repeated sequences. So, of course, after the token space D, the last time it saw that it was followed by URS. And so after the bump, it is much better than before the bump at determining that that repeated sequence recurs again. Dursley again shows up here. And then Dursleys has been observed at this point, which improves.
[0:20:15] Later as well. So that common repeated sequence it's performing better at. It's also performing noticeably worse at things that break that pattern. So previously when it saw Mrs., that of course was followed by Dursley. And so when Mrs. is followed by something else, it is much more surprised by that after the bump because it has learned to repeat itself. And so times where it doesn't repeat and breaks the pattern, it gets here in darker green a lower loss.
[0:20:45] And then other of our familiar repeated sentences, such as a small sun, which had already appeared earlier in the text. We're also seeing improved loss on that as well. And of course, the potter's starts to repeat. And those repeats, particularly once they've been repeated over and over again, also intensify. So this tells us, at least in a quick glimpse, that the major difference in the loss curves
[0:21:14] After versus before the bump is on these repeated token sequences. So let's just review quickly what we saw in the behavior of the model on those 10,000 random tokens by 1200 snapshots dataset. So the first thing we observed was in the first two principal components, the bump shows up very distinctly as a major swoosh, a major change in those major dimensions of variation of the model.
[0:21:44] We also looked at this meta-learning index, which shows us that the two layer and higher models become much more adept at using early tokens to better predict later tokens. And that improvement happens over the course of the bump itself. And then we looked at this delta between the after and before bump.
[0:22:07] So I'm going to move into circuits and mechanism in a moment. I want to just make sure to review what we've already covered in previous videos about induction heads as it's important context for understanding the circuits that we see.
[0:22:35] You might remember from a previous video that the main hallmark of an induction head is two things. The first is in the OV circuit. So that's the output and value matrix. The OV matrix has positive eigenvalues selecting for similar output. So showing here, if you have the token potters, so pot.
[0:23:02] followed by the token TERS. Then the attention pattern from pot will look back to a previous instance of TERS and the output will be made similar to what's being attended to. So that similar output shows up as positive OV eigenvalues. The other, the second hallmark of an induction head is that this attention from
[0:23:29] POT back to TERS, the key information is carrying the similarity to the previous token. So the previous token was POT, which is similar to where the attention is coming from. So that similarity in the QK circuit is allowing it to attend to tokens whose previous token was similar to the current one.
[0:23:54] So those two different properties, the OV, which gives it copying like behavior and the QK, which gives it specificity to where the prior token was similar. Those two are the hallmark of an induction head.
[0:24:10] Importantly for why this was at first a plausible hypothesis to us as to why the bump was going on is that induction heads can only form if there are two or more layers of attention. That's because the information in the key needs to get moved from a previous position to the current position. So jumping back for a sec, the key from this URS token needs to have all of this information moved into it by previous attention.
[0:24:40] So,
[0:24:58] I'm showing you here some example attention heads. These are from the two layer attention models, and I'm showing these eigenvalues. So in green, as I said, the positive QK eigenvalues are responsible for that specificity to repeated sequences where the previous token was similar. And then the OV eigenvalues are giving it the copying like behavior where it outputs something similar to the token that it's attending to.
[0:25:26] And the one thing to notice is that there are several very highly induction-like heads in the two-layer attention model. This is, of course, its second layer. And I'll also draw your attention to this score. So I'm going to use this score in a couple later plots. What I'm measuring here is more or less what fraction of the QK and OV eigenvalues are positive.
[0:25:55] And specifically, I'm taking the minimum of the QK and the OVs because if only one of them is positive and not the other, that doesn't count fully as an induction head, which needs both of those hallmarks. So that minimum positivity score of the QK, OV eigenvalues, I will be using to color some future plots to highlight which ones are the most induction-like heads.
[0:26:23] So let's get down to the circuits level. I think this is where it gets particularly exciting. And one reason that we are quite excited about this phenomenon and our findings is that we can connect the circuits level all the way to that high level behavior. So the two different lenses all present on the circuits here. The first is these eigenvalue trajectories of the induction heads over the course of the bump.
[0:26:48] I'll also do an ablation study where I remove the contribution of different heads and I'll compare the deficits after removing each head to the specific difference that we see in that before and after token loss delta. And what we'll see here as I've been
[0:27:11] Gesturing at so far is the development of these specialized induction behaviors where induction heads have really come into their role in doing induction like having induction like eigenvalues and driving induction like behaviors. So just to quickly remind this OV eigenvalues are the similar output. The QK eigenvalues are similar previous token. So I'll be presenting two different.
[0:27:40] Measures of how induction-like a head is. The first I highlighted was that score. So that's sort of the min positivity, like the minimum over QK and OV of the fraction of positive eigenvalues. I'll also look at the trace, which is just the sum separately of the QK and OV eigenvalues, which is telling us just how positive really are these eigenvalues.
[0:28:05] So this first set of graphs I'll show you is that sort of minimum positivity score, what fraction of the eigenvalues are positive. And I've shown you here for just the single most induction-like head, but I'll add all 12 in a moment once I've explained what this graph is showing.
[0:28:22] So this is our most induction-like head. I've colored it dark purple because by the end of the trajectory, it is over 99% positive eigenvalues on both dimensions. And you can see over the course of its training,
[0:28:38] It becomes more and more induction-like. And during the course of the bump, it really comes into its own. It goes from maybe being 80 to 90% positive eigenvalues to more than 99% positive. So it really specializes as an induction head over the course from the start of the bump to the end of the bump.
[0:29:00] So I'll now add the rest of the 12 heads. Sorry, first I'll just show a different, just a different head. As you can see from this eigenvalues plot, it is not particularly induction-like. I've colored it light yellow because by the end of its trajectory, it is not very induction-like. And you can see it goes through some oscillations over the course of training where it becomes varyingly positive or negative, but it ultimately, after the bump,
[0:29:30] Settles into having a not very induction-like role. Okay, now let's put the rest of the heads on this plot. So this is all 12 of the final layer of the two-layer attention-only model. So there's a couple things I want to draw your attention to here. One is, again, these heads that end up being highly induction-like, they sort of continue to rise coming into the bump.
[0:29:57] In terms of how induction-like they are. And what we see during the bump is they really solidify their role. They go all of them from being maybe 80 to 90% positive eigenvalues to more than 90, more than 95, in some cases more than 99% positive. So overall there's a subset of most induction-like heads that really settle into that role.
[0:30:21] I'll also point out that many, many heads seem to be moving in an induction-like direction coming into the bump. And then as these heads specialize, many of them turn back around and go to do something else. So this turning back around is part of why I'm describing this as a specialization process, where rather than a whole bunch of heads all deciding to do a little bit of induction,
[0:30:50] A few heads really specialize into that role and the other heads go and do something else. So I'll also show the...
[0:30:59] Three and four layer attention only models in a second. I want to just point out that I'm not going to show the models with MLPs because this analysis in terms of matrix eigenvalues is not well defined once MLPs are in the mix. So I'll stick to the attention only models for now.
[0:31:21] There are ways to, using some heuristics, map how induction like a head is, but I haven't done this analysis at this time.
[0:31:31] So here's the three-layer model. I've again taken the last layer because that's where I observe these highly induction-like heads. So again, the same two phenomenon that I find interesting seem to show up where the most induction-like heads were already starting to move towards that behavior before the bump.
[0:31:52] And at the same time, there were a number of heads with somewhat induction-like behavior before the bump, and over the course of the bump, many of the heads just go do something else.
[0:32:10] Very similar in the four layer. I'll also point out these ones don't seem to have as much oscillation as the two layer. I don't really know what this oscillation is about, but that might be something we learn more about in the future. Jumping back to the...
[0:32:25] Four layer, same two things. There's a collection of very induction-like heads that really solidify their role over the course of the bump. And there's a cohort of heads that had started to do somewhat induction-like behaviors and the ones that don't specialize just turn around and go do something else. So that's a pretty consistent pattern that we're seeing in this data.
[0:32:50] So I mentioned there were these two different scores I was going to show. One was that induction score, which is the fraction of positive eigenvalues. We're now going to look at the trace, which is the sum of the eigenvalues. So that's how strongly positive are these values. So here I've plotted for one particular head. Again, this is head 8, which is the most induction-like head in the two-layer attention-only model.
[0:33:15] And I've now, rather than taking the minimum or combining the QKs and OVs, I've left them separate. So we can see in the solid line, the OV, which is that copying-like behavior. And in the dotted line, the QK, which is that similar previous token behavior. This is on a symmetric logarithmic scale, a symlog scale.
[0:33:40] So one thing to notice is that this had developed a copying-like output pattern fairly early. What it really specialized into was that second order QK behavior. Over the course of the bump, it became much, much stronger at doing that previous token, that previous token specificity, which is responsible for that repeated sequence of token type behavior.
[0:34:08] I'll point out this additional deviation here. It's kind of a second bump. It's not that important for us right now. This is where we turned off weight decay in the training process. So it somewhat seems to have saturated a particular value. And then when we removed weight decay, it continued and saturated the new value. I'll mostly crop this part out because it's not as important in the graphs going forward. So the next graphs that I'll show
[0:34:38] I'll put the OV, the copying-like behaviors on the left, and I'll put the QK, which is that induction-specific previous token similarity on the right. So here we've got all heads, again, copying-like behavior on the left, previous token specificity on the right.
[0:34:58] And what really jumped out to me is that the copying-like behavior didn't change very much over the course of the bump, for the most part. It seems like some heads did change what they were doing, but they mostly specialized away from copying-like behavior in that range. What's very striking is it's this second order QK behavior that substantially intensifies over the course of this development. And here I've colored the heads by
[0:35:28] At the time, their overall induction score as previously. So you can see this one that ends up, this is again head eight, very dark purple, very induction-like. It's sort of developing that induction-like nature over the course of the bump. So here we see there's a collection of induction-like heads that really learn this highly similar.
[0:35:51] Previous token behavior over the course of the bump. And there's other collections of heads that just go do something else. We haven't analyzed in great detail what are these other second-order QK behaviors. It's less clear, but it seems like the process of having some heads specialize in induction-like second-order QK behaviors kind of frees up other heads to go do something else.
[0:36:18] So one other view on these eigenvalues that I'd like to show you is a little movie animating their development over time. So for a particular head, let's again look at number eight, which is the highly induction-like head in the two-layer attention only. We've got the QK eigenvalues on the left and the OV on the right. And then we've seen these two plots before. So this again is that fraction of positive eigenvalues again on here on the left.
[0:36:48] is the trace, the overall sum of how positive the eigenvalues are on that log scale. So I'll just show the movie through once and then I'll slow it down a little bit and show you some of these particular points.
[0:37:03] So if we watch this movie, the first thing we notice is the OV eigenvalues develop and then the QKs. So that's a bit of an interesting trend that we see also replicated in these graphs where that copying-like behavior comes on first and then the QKs come online. So if we watch this one more time, we notice there's this phase in which we're not seeing a lot of second-order QK behavior at all.
[0:37:27] And then it emerges quite suddenly. And it sort of emerges in those sort of two pulses as it reaches different levels before and after we turn off weight decay.
[0:37:41] So I've showed you the eigenvalue trajectories. The final thing I'm going to show in this video is an ablation experiment that gives us another view on these heads. So the question that we're looking to answer with this experiment is, how much is a given head responsible for that bump-specific change in behavior? So the way we'll do this is we'll run the model with one head at a time zeroed out. So it's contribution to the residual stream.
[0:38:11] We'll be replaced with zeros and in that way we'll get a new measure of the model's losses on various tokens.
[0:38:20] If it didn't have that head contributing, we'll sort of measure the change in token losses from the baseline. So we'll take that difference. We'll look at these in its raw form first. But then the more interesting analysis is if we compare the ablation related change in behavior to the bump related change in behavior. This will give us a sense of which heads are driving that bump specific behavior change.
[0:38:49] So just to get us oriented as to what the raw data is, the lower line here in gray is the unoblated two-layer attention-only model as compared with the one-layer. And then all of these other lines are if you remove each of the 12 heads in the last layer of the two-layer attention-only. As you can see, some heads have a larger impact than others. They induce a greater deficit when removed.
[0:39:17] So the difference between each of these ablation lines and the baseline is this change in loss that we'll be working with. So I'll show you those in terms of what the raw data looks like.
[0:39:32] The average magnitude of the deficit that is observed when you remove each head at different points in training. So as you can see at the time that the bump happens, head seven and eight are the most important to the model's overall loss. The model performs with the largest average deficit when those heads are removed. Other heads don't seem to induce a very large deficit.
[0:39:58] This plot doesn't show any particular correlation between how induction-like a head is and on net how large a deficit is observed when it's removed. So I think this plot is not particularly important for our current analysis. I just wanted to make sure to show it to give a sense of what this raw data looks like.
[0:40:19] I'll also show this one other way where if instead of coloring each line, you know, in this case solid purple based on how induction-like it is at the end, I can also color it based on how induction-like it is at various points in training. This is also just to drive home that there's no particular relationship between how induction-like a head is and how large its overall on average contribution is to the model's loss.
[0:40:48] However, more interesting is if we compare the token-specific deficits to that before and after bump difference. So just to remind you, this is the after minus before individual token losses, which if we do on 10,000 random tokens, this is a length 10,000 vector that characterizes what the specific nature of that bump improvement is.
[0:41:16] So if we compare that with the specific deficits seen by each ablation, then we get a much clearer story. So if we just zoom in on this most salient curve for a moment, this is again head eight in the two-layer attention only, which we've seen time and time again as the most induction-like head.
[0:41:36] And what you'll notice is that particularly around the end of the bump, when head eight is ablated, the deficit induced is highly similar to the bump related difference. Another thing you'll see, given that I've colored these based on the change in how induction related they are over time, is that all of these heads that end up
[0:42:03] Producing a bump related effect or a bump like effect are sort of coming online as they become more and more induction like over the course of their development through the bump. They become the heads that are really responsible for or really driving that bump specific performance improvement. So this is, I think, one of the clearest demonstrations that the induction heads are responsible for that bump related improvement.
[0:42:34] So just to summarize what we've seen so far, let me just review what these questions were or these topic areas were that we wanted to check out. So the question of generality, we were able to see that the same bump turns up in multiple models. Behavior-wise, we could see in a couple different ways that the bump represents a large change in behavior when analyzed on that.
[0:42:57] Token by token level. And what the model seems to be improving on is repeated sequences of text. And then at the circuits level, we've seen a specialized development of induction heads over the course of the bump. And then some suggestive evidence that this frees up other heads to do other interesting kinds of second order QK behaviors.
[0:43:19] So just reviewing some of these key figures, here is our evidence of generality where across different model sizes, the bump was shown in a somewhat similar position. And we also saw suggestive evidence that Chris will show more of in the next video that larger models have a similar effect too.
[0:43:38] We also looked at behavior in these three different ways. So first we looked at the first and second principle components and saw that immediately in the PCA view, the bump turns up as a very drastic change in the trajectories in the two layers, but not the one layer. And then we also looked at this meta-learning index. So how much better the 500th token is being predicted than the 50th. And over the course of the bump, meta-learning improved dramatically.
[0:44:07] We also looked at this before and after token delta vector and saw that these repeated sequences of text are where the specific improvements are showing up and then breaks in the pattern also seem to be performing slightly worse, if anything. We also looked at these eigenvalue trajectories, so we saw that
[0:44:31] These most induction-like heads are settling into a highly induction-like pattern and other heads are specializing away from that pattern into something else. Looking at the OVs and QKs specifically, we saw that the copying-like behavior remained fairly steady and it was really these second-order QK behaviors that were developing during the bump.
[0:44:54] Finally, we looked at this ablation study, which tells us that removing these induction-like heads induces a deficit in the model that is highly similar to the before and after bump difference in token loss predictions.
[0:45:10] So just recapping one more time why we find this interesting. For one, it's evidence that the model's behavior can change very suddenly over just a few hundred time steps. And we'd like to understand how these sudden shifts in the model's weights and the model's behaviors show up.
[0:45:28] Thank you. Thank you. Thank you. Thank you. Thank you.
[0:45:57] This is an example of us making that connection and learning something cool about both in the process. So I hope you enjoyed this video and I encourage you to move on next to Chris's video about meta-learning that will connect induction heads to meta-learning and also go in more depth into some of the phenomenon we've seen today. Thanks so much.