{"url": "https://www.youtube.com/watch?v=NnSgSZXqGmQ", "page_url": "https://www.youtube.com/watch?v=NnSgSZXqGmQ", "title": "Attention - General - Summarizing with NMF [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 2203, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-attention-general-summarizing-with-n", "audio_duration_s": 2202.84225, "worker": "jc-w01", "transcribe_s": 52.5, "model": "large-v3", "segments": [{"start": 0.53, "end": 27.63, "text": "The goal of these videos isn't just to understand toy transformers. We want to be able to understand transformers as they're used in practice, the kind of large models that are getting all these really exciting results lately. And in our previous video, we sort of addressed one of the challenges to that, which was talking about how switching from attention-only transformers to a real transformer introduces various theoretical challenges and how we can work around them."}, {"start": 27.7, "end": 58.06, "text": "But there's also a really practical challenge, which is these models can be huge. Even if we're just talking about the attention heads, you can easily have thousands of attention heads in large models, and you might want to be studying multiple models as well. And so this video, which you can skip over if you want, is about a trick for sort of summarizing and getting a big picture overview of the kinds of things that exist in a model, at least the kinds of attention patterns that exist in a model."}, {"start": 58.16, "end": 86.53, "text": "And attention circuits. Yes, we're going to try and get an overview. And the trick we're going to use is something called non-negative matrix factorization, or NMF. And I'm Chris. I'm the one who is recording this particular video. And I have to confess, I'm kind of obsessed with NMF. People who..."}, {"start": 86.53, "end": 115.09, "text": "If you are familiar with my previous work or know me personally, may know that really one of my first reactions to almost any problem is to try and go in and apply NMF. And at least in computer vision, it's been really effective. So one example that I'm quite proud of is in one of our papers, the building blocks of interpretability, we used NMF to decompose the activations of a vision model."}, {"start": 115.09, "end": 139.22, "text": "And so, for instance, here we have a picture of a dog and a cat. And there are about 500 neurons at every position. It's a continent. So at every position, there's like 500 neurons. And that's a lot of neurons to look at. But we can just go and use NMF to go and reduce this into a small number of factors that describe the activations at every position. So we see here."}, {"start": 139.22, "end": 170.4, "text": "There's like one factor that corresponds to the dog's ear and the head and the snout and the bodies of both the cat and the dog and the background and the cat's head. And we can kind of summarize things. And so that was really effective. And pretty consistently, NMF is just a really powerful tool if you want to go and summarize things. So yeah, one of the first things that I did when I started working with Transformers was to try and apply the same approach."}, {"start": 170.4, "end": 202.67, "text": "It seems to work quite well. So just stepping back a second, non-negative matrix factorization, it's just a kind of linear dimensionality reduction. And really all we're saying is we have a matrix and our matrix has some width and some height. And we try to describe it as a product of two smaller matrices. So one of them will be tall and thin and one of them will be narrow and wide."}, {"start": 202.67, "end": 233.26, "text": "And they multiply together to go and produce the big matrix. And then it's non-negative because we're going to go and require that both of these matrices that we write as the product be positive. So if we tried to go and do this without any constraints, you'd get principal component analysis. These would end up maybe not basis aligned, but you'd end up extracting if you have this as, say, K dimensions on the small side that we've created."}, {"start": 233.26, "end": 262.62, "text": "You'd get the first K principal components. But when you go and you add the positivity constraint, you get something else. And it's just kind of this way in which NMF tends to be interpretable. And this isn't just a property in machine learning. You actually see this in the sciences where often in, say in physics, people will prefer to use NMF sometimes. I think one really striking example of this is if you take a data set of, and I'm not a physicist, so."}, {"start": 262.62, "end": 292.42, "text": "I might misstate this, but if you take a data set of the spectral emissions of different stars and you apply NMF, you'll go and have all the different star types go and just fall out as the factors of NMF, whereas that won't happen if you do PCA. And you'll see this in other cases that, you know, in physics, people often, there will be cases where people prefer to use NMF. And yeah, my experience is just in interoperability. This works."}, {"start": 292.42, "end": 321.01, "text": "Works really well. And we could talk more, perhaps I'll produce another video sometime about why you should expect NMF to work particularly well. But NMF is often a very, very nice way to handle things if you want to do some kind of dimensionality reduction. So the idea here is we're going to take all the attention patterns in our model, maybe even all the attention patterns from multiple models. And so each attention pattern is a tokens by tokens matrix. And then for every attention head,"}, {"start": 321.01, "end": 349.94, "text": "We have one. So we get a three-dimensional tensor. And the question is, how can you apply NMF to something that has three dimensions? It's a matrix factorization technique, not a tensor decomposition technique, at least on its face. And what we'll do is we'll flatten the tokens and the two tokens dimensions into a token squared dimension, into a single one. And then we'll apply NMF. And NMF gives us two factors, the pattern factor and the heads factor. So the pattern factor is going to be a tokens times tokens."}, {"start": 349.94, "end": 380.14, "text": "We'll go ahead and turn that in unflattened into a tokens, tokens, components. So it's sort of just like originally we have an attention pattern and then rather than having an attention pattern, like a tokens by tokens pattern by for each head, we have one for each component. And then each component on that, that should be components, not factors, but each component is a vector over heads. So we have one factor that describes where each component looks and one factor that describes"}, {"start": 380.14, "end": 412.53, "text": "Which heads correspond to each factor, which component? And this is a little bit of a technical point. And if you don't follow that, that's totally fine. But there is this interesting observation that actually, this is actually kind of quite principled to do. And in fact, if you think of all these equations we've been writing that describe transformers, we always have these terms that look kind of like this, where we have a sum over attention heads, and then we have this tensor product of the attention pattern."}, {"start": 412.53, "end": 442.61, "text": "And WOV, the OV for each matrix for each head. And it turns out that you can approximately rewrite that. It's kind of a change of basis, but a change of basis for tensors instead of matrices in terms of these components. And it'll only be approximate if the, you know, as you increase the number of factors, they'll get better and better. But you have sort of this attention factor that sort of, you can think of as being an attention pattern that exists for each component. And then you can go in and construct an OV."}, {"start": 442.61, "end": 472.27, "text": "This is really cool because it means that we have potentially, even though we could potentially have a very large number of heads, or if we start working with virtual attention heads, we could have an exponential number of heads, we have a way to potentially keep trying to re-express things in terms of a smaller basis. In any case, this is kind of a technical point and not immediately relevant."}, {"start": 472.82, "end": 499.78, "text": "So I think the more immediately cool thing is that we can go and take all of the attention heads in the model and all of their attention patterns and express in a single image a sort of summary of the model. So here we have, this is actually not from the model we're going to be looking at. This was one of my first experiments doing this with GPT-2. And you can just go and take all the attention patterns."}, {"start": 499.78, "end": 527.5, "text": "And you just go and compress them all. And here we've mapped each one to a color. And we can sort of get this overview of the kinds of the largest scale attention structures. And you can ask what those are. So for instance, those green stripes turn out to be periods. Now, you'll recall from the previous video that often when we see an attention head attending to periods, that's because it's acting as a resting position."}, {"start": 527.5, "end": 557.01, "text": "Some of the structure that you see if you apply this natively turns out to be, you see a lot of interesting structure, but it turns out that structure is kind of superficial. But we can do better than that because we can go and apply it not just to the regular attention patterns, but we can also apply NMF to the info-weighted patterns and to the attribution patterns. And that will be really informative. So let's switch to an interactive view because that'll be a lot more effective for looking at this."}, {"start": 560.21, "end": 589.95, "text": "That's not what we have. Okay, so this is actually going to be probably a fairly extended chat about the kinds of things we can see in these NMF factors. And so feel free to end if you just wanted to understand sort of what kinds of things exist in these models. But what we have here, we'll just start with an info weighted pattern. So we take all the info weighted patterns in the model."}, {"start": 589.95, "end": 619.84, "text": "For every attention head. And in fact, we're doing this for two large models. So it's even for two models. And we're just going to reduce it to the 20 factors that, yeah, that do NMF with 20 factors. And so we get 20 different, yeah, we get 20 components that have expressed different common patterns that the attention heads have. And some are not that interesting. Like even though we did info waiting, we see that there's one factor that really wants to attend to the EOT token."}, {"start": 619.84, "end": 647.01, "text": "And that gets, that's there. But there are a lot of really interesting ones. So one that's kind of cool is we see there's a fact, a component that corresponds to attending back to the sort of to persons or pronouns or something like this. So here we have son, boy,"}, {"start": 647.01, "end": 677.63, "text": "And one way that's often effective to understand these things is we can just look at every token, how much it's attended to. And you can see that more or less, these are all people. In some cases, they're nouns. Here we have blonde over here, but it's probably blonde could be also a noun that would be a person. So that makes sense. And we'll see more when we look at attribution that things like this."}, {"start": 677.63, "end": 708.78, "text": "are really helpful because you might want to predict later pronouns or like the gender or plurality of a pronoun maybe, or even just attributes when you're describing someone might be helpful to go in and remember what sort of person you might be talking about. We also see ones that correspond to because, although, but."}, {"start": 708.78, "end": 736.37, "text": "There's an interesting, this is actually a very common thing, that you'll see, I don't know what the linguistic term for this is, but you'll often see components and attention heads that seem to attend back to the last thing that could either be introducing a that clause, a verb or something else that creates a that clause. They were proud of that, and it doesn't actually do that."}, {"start": 736.37, "end": 765.82, "text": "They could have said proud that or maybe proud to. Say that. With that. You could think that. You could have an opinion that. Could be useful that. You might fear that. You could have a reason that. You might want that somebody do something."}, {"start": 765.82, "end": 795.2, "text": "Yeah, I guess Shutter, maybe it's really two, but it's sort of functionally similar. You might think that. So you have all these things. You might expect that. In a lot of cases, you allied the that informally. So like when we say opinion there was, it's really, you know, it's their opinion that there was. So it's all these places where something like that could be introduced. And you could imagine that's really important because then..."}, {"start": 795.2, "end": 826.96, "text": "In the next clause, if you look at these and you look at which tokens attend to them, it's really the next clause. And the kinds of words that might be in that clause really depend on what is the context of this clause. So that's a very useful attention pattern, maybe. Attending to the previous token. So I should look at it so that we're looking at where we're attending back to. So that's a very common thing. This one will be easier if we look at where."}, {"start": 826.96, "end": 857.18, "text": "Which words are getting the attention? Again, this is attending to people, but it's really attending to any word that implies a gender or whether something's plural or maybe a noun. We see there's lots of people. There's lots of pronouns, titles. Mrs. implies female. Mr. implies male. So that's helpful. They implies plural."}, {"start": 857.18, "end": 888.58, "text": "And yeah, we'll see later on that things like this, you know, her implies female against sister implies female. Where is this? But the Lee's and Dursley's implies plural. And we'll see this is super important because there's actually a fair amount of entropy in guessing, you know, what is the right pronoun? What is the right plurality? What is the right conjugation of a verb? So attention heads like that and attention patterns like that are very common. Of course, we see attention heads that are related to induction."}, {"start": 888.58, "end": 918.21, "text": "So it's probably easier to look at this way that we go and we were on D, we look at previous cases where D happened, look one ahead, and we see lots of attention patterns like that. And NMF will start to split them up because there's lots of induction heads that might be doing very slightly different versions of induction. Here's a factor that maybe is approximately looking at verbs. It could be that it's maybe trying to get at the tense of verbs, or maybe it's just useful to know what the last verb is."}, {"start": 918.21, "end": 946.51, "text": "There's some exceptions, like obviously director isn't a verb. Maybe if we had more factors, this would split apart a bit more and we'd get something that's more cleanly a verb. Yeah, so those are factors that I thought were pretty interpretable and interesting. Now, instead of doing 20 factors, we could do 40 factors with the introverted patterns. And I wonder if there's some here that are worth mentioning. A lot of these are..."}, {"start": 946.51, "end": 976.24, "text": "Pretty similar to things that we saw, like there's one that is cases where there could be a bad, it's maybe a little cleaner. Say that, expect that, it was their opinion that, it might be useful that, you might think that, you might pretend that, you might think that, it's possible that, I guess shudder to, you might."}, {"start": 976.24, "end": 1006.43, "text": "You might want somebody to do something or want that something happens. You might know something, know that. You might have a reason. Yeah, reason that. Or you might, if reason was functioning as a verb, it might be, yeah, reason that. So all of these words that are sort of introducing a clause, a particular kind of clause, I guess. This is interesting. You'll see sometimes attention heads that look for the..."}, {"start": 1006.43, "end": 1035.12, "text": "To the first token of words. So we should look, switch to a mode where we're looking back. So ordinarily it attends to the token that's on. But when you're in a word or a compound word, it'll often attend to earlier tokens in that. So here we have, we're on potters. It's attending to the first token. If we go to Dursley, it's attending sort of to misses and the earlier tokens in it, to the D there."}, {"start": 1035.12, "end": 1060.21, "text": "Mustache, we're tending to the earlier tokens in mustache. So that's a fairly common pattern as well. Here again, we have something that's maybe the person, the gender plurality. We have lots of tending to pronouns. Man is another word that sort of implies something about gender maybe."}, {"start": 1060.21, "end": 1087.22, "text": "And things that apply plural. And this is going to be really useful again for predicting conjugation, for predicting pronouns later on. You'll be able to go and recover some entropy that way. There's a lot of induction hints here. I'm going to skip over that. Here's one that may be doing tense. Again, when we're just looking at factors like this, until we get to attribution, we can't really say"}, {"start": 1087.22, "end": 1116.22, "text": "For sure what the function is, but, you know, this looks like it might be, you know, we have an imperfect past tense, imperfect past tense. Um, we have some, some perfect, uh, tenses. We have, uh, some subjective, some subjunctives. We have some, some gerunds, um, and that really seems to be what we're, what we're mostly attending back towards. Um, uh, could also be telling us maybe a little bit about the, uh, the conjugation of the verb, uh, like."}, {"start": 1116.22, "end": 1147.23, "text": "Is there a singular or plural subject in some cases? Yeah, bunch more induction heads. Oh, this one's cool. That's worth talking about. You'll often see these determiner attention heads and components. So here, a determiner is kind of a generalization of the idea of an article. So we'll see a lot of cases where we have, you know, a, the, a."}, {"start": 1147.23, "end": 1176.11, "text": "Yeah, thus we have these perfect or these, yeah, we have these different articles. We also have her sort of functioning as almost like an article for sister, right? It's sort of replacing an article. So that's another type of what people call a determiner. Several is kind of functioning in a slightly similar role there. Sometimes no functions in a similar role or such a sort of functioning in a similar role to that."}, {"start": 1176.11, "end": 1205.34, "text": "Anything is kind of replacing an article there as well. Yeah. So you have these things. It's not exact. The possession here is sort of replacing an article for sister. Now, there's some things here where that doesn't make sense. Although in a lot of cases, it's in places where there's a missing article, but where there's sort of an aligned article in some way."}, {"start": 1205.34, "end": 1239.31, "text": "This doesn't make sense exactly. You'll see attention heads that do this more cleanly when we start looking at attention heads, perhaps. Conjunctions, we have because, although, but, and, but, and. I guess in some ways, a semicolon is a little bit similar to a conjunction. And then it's joining together two clauses, and, which. Nouns, yeah, that maybe is a little bit of a noun component."}, {"start": 1239.7, "end": 1270.0, "text": "Yeah, prepositions here, in, in, for, maybe some things that are not quite that mixed in as well. Okay, so the thing that we can do though, so far we've looked at introverted patterns and compressed them down to a small number of components. We're summarizing thousands of attention heads here just in a few components and sort of getting an approximate sense of some of the things that might exist."}, {"start": 1270.0, "end": 1302.11, "text": "We can also look at attribution patterns, and that's nice because it'll give us a more functional explanation of these. So yeah, one of the most common things is going to be induction. So for instance, here we have a, we're predicting D. So remember, because we're now on an attribution pattern, it's no longer the token that's attending back."}, {"start": 1302.11, "end": 1332.5, "text": "It's the token that's having its logic affected. So this is saying that by attending back to all these Ds, we increase the probability of this D here. And so it's really, it's actually this Mrs. token attended back to Mrs. and then the D over here, and then increase the probability of this D. Okay, so induction's really common because it's just such a big part of what's going on when you look at least at these direct attribution patterns. When you look at..."}, {"start": 1332.53, "end": 1363.18, "text": "You'll see a whole bunch of induction patterns. So that's one induction pattern. Here's another induction pattern. Here's another induction pattern that's getting the middles of the words. And it's getting split apart because different induction hints are giving slightly different weights to slightly different parts of the word. And then NMF pulls them apart. But yeah, so you'll see a whole bunch of factors that are involved in induction. And when we go to a larger number of factors, that'll get even bigger."}, {"start": 1363.18, "end": 1394.29, "text": "We'll see things related to other kinds of copying as well. So like this one's really simple, but here we're copying that dash. So we have good for nothing. The dash increases the probability the other dash is getting copied. One thing that's really cool and we will see a lot of when we start looking at attention heads as well is attention circuits that are responsible for maintaining"}, {"start": 1394.29, "end": 1423.28, "text": "Gender and plurality and agreement around those. So the colors that we're seeing here, we're seeing words where their probability was increased by this attention factor, attention component. And we can look at what it was doing. Well, it looked at, especially at were. Were is a, yeah, it's conjugated so that we know it's plural. So that's helpful."}, {"start": 1423.31, "end": 1453.44, "text": "This day, yeah, so this is actually a little less clean than I was expecting it to be. The fact that it's affecting these tokens is very clean, but perhaps this component is a little less clean than I was expecting. We're seeing some cases where it's still fairly straightforward, but maybe let's just very quickly explore this. Yeah, so was, yeah, okay, so sun, ham, those are both really implying."}, {"start": 1453.44, "end": 1483.63, "text": "Male, Singular, They, the Dursleys, New, and we're also actually a little bit of attention back to the day there as well. But it wouldn't surprise me if there's some virtual attention heads or information being moved around in a way that's making this a little harder to see her sister. There's another component. We'll get a cleaner component also when we go to the 40s, the 40 factorization."}, {"start": 1483.73, "end": 1515.47, "text": "Yeah, they say, this one's also not that clean. There's probably a bunch of virtual attention heads going on. Yeah, I bet. Yeah, so like here, well, I guess Shuttered might contain a bit of information about this, but it could also be that Dursley's is clearly plural and some of it got folded into there by another attention head and then we're picking it up. Yeah, they were going and seeing."}, {"start": 1515.47, "end": 1545.2, "text": "Seeing a previous day, but it's, yeah, usually this is cleaner. So we'll come back to that in a later one. It looks like there's still another one. So let's have a look at this one. Yeah, child. So this is maybe not just gender. It might also be more generally sort of information about recent people who've been talked about. So, you know, we have some things maybe that are related to age and being a child, and that helps us predict that this is going to be child here and not just."}, {"start": 1545.2, "end": 1575.84, "text": "Yeah, or her sister, we're just literally copying the word sister a bunch of times. Mrs., we've seen Mrs. before. So this is something sort of in between a more simple copying head and something that maybe, Claire, is about gender or plurality or something like this, or maybe a copying head that's specifically involved in people. Okay, so that's another kind of interesting thing, and we'll get a better example of that in a minute."}, {"start": 1575.84, "end": 1603.92, "text": "I don't know if we talked about that one. There's another one. So again, we're getting a whole bunch of components that are involved in thinking about gender, plural, things like that. That's quite common. Now, one other important type of attention circuit that we see is what I've called an engram circuit or an engram attention maybe component. And these ones are kind of mysterious. And I have a..."}, {"start": 1603.92, "end": 1633.26, "text": "Pretty strong hypothesis about what's going on, but I'm not sure. So the observation is, if you look at these, you'll see that they, you know, they're sort of just jumping across words in a kind of, you know, unpredictable way. And some earlier word just increases the probability of this word. But if you start reading them aloud, you'll notice that they sort of, they almost rhyme, like her husband. Well, there's some way in which, like those could just be side by side and they would increase in probability."}, {"start": 1633.26, "end": 1666.7, "text": "With Child, Keeping Away, Another Reason, What Neighbors, Four Years, Time X-ing, In Strange, Over Fences. And there's a bunch of other ones. Let me go to another Ngram head. Yeah."}, {"start": 1666.7, "end": 1698.58, "text": "What would? It's so blacked out there that you can't even see it. It's so intense. But yeah, what would? That's actually that would. So those are both sort of clearly very common phrases. Like it's very common for would to come after over what. Past tense didn't. They want. Semicolon one doesn't make as much sense. Keeping away again. Another reason again."}, {"start": 1698.58, "end": 1733.46, "text": "And we saw that in a previous one. With nonsense. Say were. Anything or. Let me go and find. There are a bunch more of these. Here's another one. Yeah, anything or. Have large. Of time. Finer anywhere. So in any case, I've been ranting about this for a little bit, but the hypothesis is."}, {"start": 1733.46, "end": 1759.57, "text": "But these are just, they're kind of like bigram statistics, but they're like bigram statistics at a distance. Now, that's probably a simplification. Oh, this is copying the as. That's kind of a simplification. And it's probably not quite that literally what these attention heads are doing. It's probably not quite that simple. But we saw things like this a little bit, even in the one-layer model. And yeah, I suspect that there's..."}, {"start": 1759.57, "end": 1790.26, "text": "There's more. So let's switch to the 40 layer or the 40 factor one, again, attribution. So this is now we have 40 components and each one we're seeing which tokens affected which other tokens through some set of attention heads. And yeah, just continuing on the Ngram one for a second. Again, it's gonna be a lot of the same thing. With nonsense, four years, with child."}, {"start": 1790.26, "end": 1821.17, "text": "Keeping Away, Another Reason, Her Husband, Four Years, Somebody Discover. These are kind of weak, although they're kind of interesting. Let me go and switch back to this view so that I can see some other ones. A Sun, On the Neighbors, Over Fences, The Mount, In Strange."}, {"start": 1821.26, "end": 1852.85, "text": "You know, makes me think of God works in strange and mysterious ways. Sorry, there was a technical glitch there. But yeah, let's look at some other ones. So those are the n-gram ones. We see a bunch of things involved in tense and plural. So here, had, we're looking at all these previous words that are past tense. Was, we're looking at previous words that are imperfect past tense. Singular, I guess. You might wonder about and."}, {"start": 1852.85, "end": 1884.05, "text": "But remember that and is often going to imply plurality, so it sort of makes sense that's getting caught up in the circuit as well. There's a whole bunch of past tense. By the time we get here, it's more diffused because there's so many past tense words that we can look at. Yeah, misses, well, it's more about titles there, so that's not quite as clean. And there's a bunch again of these tense copying sort of attention heads."}, {"start": 1884.56, "end": 1916.3, "text": "I'm just going to keep going on this for a little bit because there might be people who are finding this interesting, but please feel free to drop off and just move to the next video at any point. Yeah, where is another one that's involved in that? Well, we're calling this a person copying one, but it's like son increases the probability of child, sister of sisters, things like that. That's not quite gender copying. Maybe I'm surprised."}, {"start": 1916.34, "end": 1944.45, "text": "It seems like there's a little bit caught up in that one. Oh, here we go. Yeah, so here we have Mrs. increases the probability of she. Okay, so that is sort of introducing a clause that could be subjunctive. But okay, here are a bunch of, the Dursley's is plural, Potter's is plural, increases day. So that's kind of a classic type of plurality sort of consistency. All of these."}, {"start": 1944.45, "end": 1976.77, "text": "Yeah, the Dursley's plural, they plural, Potter's plural increases the probability of they. It's a new clause starting, new clause starting. Yes, Mrs. implies female, sister's female. So we're seeing all these things. And of course, you know, why, okay, so why is this head doing both tense and sort of gender plurality agreement? Well, you know, those are often kind of intermingled."}, {"start": 1976.77, "end": 2004.32, "text": "Since you're trying to predict verbs, the conjugation of the verb depends both on the tense and the plurality and gender of the subject, maybe. Probably just the plurality. So that makes you want to look at pronouns. Once you're looking at pronouns, you can easily copy gender information. And you can predict later pronouns. So those sort of naturally go hand in hand a little bit."}, {"start": 2004.32, "end": 2035.47, "text": "What about other things? Well, again, there's a ton of heads involved in induction. They're pretty simple and we've talked about them a lot, so I'm going to skip over them, but induction or factors involved in induction. And it's just that induction is getting split apart in a way that's a little sad right now because that's just part of what happens when you have a whole bunch of components and induction heads are so big and they attend in slightly different amounts to different examples. It naturally wants to do that."}, {"start": 2036.43, "end": 2072.99, "text": "Okay, so what do we have here that we haven't talked about? Maybe, is there anything here that we, a lot of these are just copies of things that we saw previously. Oh, there's one that's maybe specifically involved in copying titles, Mrs., Mrs. Yeah, I think I'm gonna call it a day there. So maybe let's just very, very briefly summarize. If we look at,"}, {"start": 2072.99, "end": 2103.12, "text": "Particularly if we look at the attributions, basically the direct attribution effects of attention heads have a pretty simple story. There's a bunch of heads that are doing engram-y things or these sort of bigram skip type things that sort of seem kind of local and seem to just be about the, there's lots of words that have affect the relative probability of words nearby and we can just in a pretty simple way model that. There's a bunch of these induction and copying heads that's just tons and tons"}, {"start": 2103.12, "end": 2131.33, "text": "And tons of induction type stuff going on that allows us to go and just copy previous words and increase the probability of when something similar happens, the same tokens appearing. Then there's a bunch of heads involved or attention components involved. That's not one. In copying people, gender, plurality, handing..."}, {"start": 2131.33, "end": 2160.66, "text": "Verb subject agreement, handling verb, you know, consistent tenses between sentences. So there's a lot of stuff like that. And honestly, that's a pretty big fraction, at least of the interoperable stuff that seems to be going on when we look at attribution. When we looked at raw attention patterns, we saw some stuff that isn't as emphasized when you look at attribution. Like we saw this determiner component that was really cool, or the preposition component."}, {"start": 2160.66, "end": 2188.32, "text": "Those didn't have as dramatic an effect when we were looking at the attributions. They're probably a little bit smaller in terms of their effects or primarily having indirect effects. So maybe their effects are more moderated by later attention heads and MLPs rather than having a big direct effect."}, {"start": 2188.32, "end": 2200.27, "text": "But yeah, so that gives us a sense of some of the things that we should expect to find when we are poking around in these models. And in our next video, we'll actually look at the attention heads and sort of move past these factors."}]}