{"url": "https://www.youtube.com/watch?v=8wYNsoycM1U", "page_url": "https://www.youtube.com/watch?v=8wYNsoycM1U", "title": "MLP Neurons - 40L Preliminary Investigation [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 3355, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-mlp-neurons-40l-preliminary-investigat", "audio_duration_s": 3354.92, "worker": "jc-w06", "transcribe_s": 45.6, "model": "large-v3", "segments": [{"start": 0.53, "end": 32.1, "text": "So one of the directions that our research at Anthropic has involved has been just trying to understand what different neurons and different layers of transformers are doing inside the MLP layers. And it's been something that we haven't investigated as much as attention heads and attention circuits. And honestly, I think we've found to be more difficult than we were initially hoping. But we have learned some pretty interesting things. And so..."}, {"start": 32.1, "end": 63.3, "text": "Hopefully in this video we can we can talk about some of them and maybe they'll be useful to people who are investigating similar things elsewhere. So I'm lucky to be joined here by my colleagues Catherine Olson and Nelson Elhage and we'll be just going and chatting through some of the things that we found. So yeah I thought it would be helpful maybe just talk a little bit. So this is sort of really just a preliminary investigation and maybe just to go over a few basic points I jotted down a few things."}, {"start": 63.3, "end": 89.23, "text": "But as we talk through these, maybe Catherine and Nelson, if there's anything you want to add, just jump in and say that. All of these results are going to be from a kind of moderate-sized model with 40 layers. We find that there's a lot of differences between features that exist at different layers. But probably the exact number of layers doesn't matter. And I think our intuition is that in most cases, if you look at models with different numbers of layers, you'll find"}, {"start": 89.23, "end": 119.57, "text": "Similar neurons roughly in terms of the fraction of the way through. So if you're at 5% depth often, that's what we see generalize. We'll be investigating MLP neurons, not neurons on the residual stream. So I guess, in some ways, neurons on the residual stream is probably an oxymoron, or probably not the way that we would use the terminology. I think when we talk about neurons, we mean something that has an activation function. And since the residual stream is just"}, {"start": 119.57, "end": 143.62, "text": "Linear functions being added into it or linear projections being added into it. You sort of can't think about it that way. We'll be talking about a number of different depths in the model. And yeah, usually our process is that we start with data set examples. And then we start going and after that going and interactively playing with the neuron and when it activates to sort of verify our theory. Chris, what's a data set example?"}, {"start": 143.62, "end": 171.89, "text": "What's a dataset example? That's a great question. Well, Catherine, how about you tell us what a dataset example is? Sure, absolutely. So we take the neuron that we're looking at and throw a bunch of different examples at it, just tons and tons of different possible contexts that's in the dataset that it was, the same distribution that it was trained on, and just see what it fires most on or where its sort of highest activations are. And we look at the top couple, like sort of asking like, what is this neuron most like?"}, {"start": 171.89, "end": 202.48, "text": "Or where does it show the highest activity? And so we just look at those pieces of text where we saw the most activity from it. And I think it's not that we think that data set examples are dispositive of what a neuron does, but they're going to help us build a hypothesis and sort of narrow down the space of things that it could be doing. Do you also, I feel like you probably are the person who's spent the most time studying neurons. So if there's anyone who should really comment on what our process has been."}, {"start": 202.48, "end": 219.5, "text": "To try to understand neurons, really, I feel like you should feel like you can elaborate on that. Yeah, yeah. I mean, one thing I'll just say is, as Chris said, starting with these positive examples is only the first part of the picture. I feel like I didn't really get traction until I could start to look at negative examples as well."}, {"start": 219.5, "end": 246.53, "text": "So when I say positive examples, I mean you can see that this neuron really likes texts of this type. But then you have to ask, well, does it like everything of this type? Or are there things like this that actually are excluded? So you need both to know what's ruled in and what's ruled out by sort of the rule that we might use to explain the behavior. And so there's a couple of different ways you can get this sort of ruling out behaviors as well. So you could look at tokens."}, {"start": 246.53, "end": 272.42, "text": "But all very exploratory. It hasn't been that."}, {"start": 272.42, "end": 286.22, "text": "Rigorous yet so much as trying to feel out the space. And I think one thing that's just worth being kind of super explicit about is we tend to talk about like what a neuron likes or what a neuron is looking for."}, {"start": 286.22, "end": 313.74, "text": "We're looking at a point in the model where an activation function, a non-linearity, is applied, typically a relu or some variant of it. This model uses gelu. And so when we say a neuron likes or a neuron fires, we typically mean that it has a positive activation. That scalar value has a positive activation, which, if we're using relu, means that that value is going to be passed through unchanged. If it doesn't like it or doesn't fire, it's somewhere in the negative regime."}, {"start": 314.26, "end": 327.78, "text": "We tend to use this sort of anthropomorphizing terminology, and I think it's sort of intuitive sometimes when you're looking at these neurons, but it's just worth being super explicit. We're just talking about like the value of that scalar at that point."}, {"start": 327.78, "end": 349.58, "text": "In this exploratory phase, we've largely been looking at very sparse neurons that fire infrequently throughout the whole body of text that they've seen because that's an easier place to get started. Again, that's going to be a strong bias in what we're reporting here, but that was a choice we made to get a foothold. So there's only a small number of different tokens that they fire on."}, {"start": 349.58, "end": 379.34, "text": "Yeah, and I guess a related thing is because these neurons are sparse, and both it's the case that the typical neuron in the model is quite sparse, and also that we've often preferentially looked at sparse neurons."}, {"start": 379.34, "end": 409.94, "text": "Very sparse and trying to understand, like you could sort of think of the case where it doesn't fire as the default. And so you might ask, why is it that you're looking at the examples where it fired rather than the cases where it didn't fire? Well, you can sort of think of it not firing as default and it firing as the exception that you're trying to understand. Whereas if you had a neuron that was sort of equally often say to had activation to zero and one, you might not be able to do that as easily. One final point that's maybe worth making is just that"}, {"start": 409.97, "end": 436.56, "text": "At least for me, I've previously done this circuits-type work in vision, where we're explaining the fact that vision models have a number of neurons where it's at least theoretically plausible that you could look at every single one. And large transformer language models, this isn't necessarily true. And so we're showing you some neurons, but it seems almost impossible that we would look at every single neuron and some kind of strategy."}, {"start": 436.56, "end": 467.94, "text": "If we want to scale, this is going to require us to do something else beyond that. OK, so with that said, I think we can dive in. And I think that it's interesting that there's some layers where we've had a lot more luck understanding neurons than others. And broadly, I think our biggest cluster of successes and our earliest successes were these relatively early layers. Roughly, especially in the 10% to 25% depth regime, we had a lot of luck."}, {"start": 467.94, "end": 500.9, "text": "But the earliest layers we found quite challenging. And then we found that features became harder and harder as you got deeper past that. And then towards the end, and especially the last layer, we also had a lot of luck. And so those seem to have been the places where we've had the most luck understanding neurons in these models, or the most success. Cool. So I think we can just dive in. And layer 5, so in this model, that's about 12% depth. I think we found a lot of pretty interesting features."}, {"start": 500.9, "end": 527.87, "text": "Catherine, do you want to comment underneath these? Yeah, I mean, I think this is sort of just a quick signature of this sort of positive description. What does it fire for? That these sort of layer five-ish neurons, again, that's sort of in the 10% through the model kind of regime. There's sort of like little semantic, semantically similar phrases. So the like, I would like, it would be nice, I will like neuron in the top left there."}, {"start": 527.87, "end": 554.14, "text": "There are other cases where the word like shows up and it doesn't fire. So if you just say, I like pizza, that doesn't work. You say, I would like pizza, then that works. And there are some ways to sort of fool it or trip it up and sort of write something that contains like, I'll come up with a specific example later, but kind of mash up the words and then it does fire, even though it doesn't mean semantically the same thing. So they're not like."}, {"start": 554.14, "end": 586.8, "text": "Ultra, ultra precise at picking out things with a meaning like I would like. But they do rule out other like I like or I love that doesn't have this same sort of future tense preference kind of expressing thing. One thing I found pretty interesting at least is some of them respond across multiple languages to short phrases that mean the same thing. So for instance, there's this like this one here that at the bottom left that"}, {"start": 586.8, "end": 617.68, "text": "We have not only, but we have also in French non-semence or non-sense, or there's another one that responds to four reasons, but also pour essa raison. Yeah, so I thought that was kind of an interesting pattern. I've wondered if it might be helpful to conceptualize these as being like, I think linguists have this idea of a, I don't even know how you pronounce it, but a phrasny or a phrasny."}, {"start": 617.68, "end": 651.58, "text": "of like words that just frequently co-occur in just statistically in English way more often than they should. Say it again. Phraseeme, like a phraseeme. Phraseeme, a phraseeme. Excellent. Yeah, and so I have wondered if these might be kind of like that. But the fact that it's across languages and it's grouping similar things together seems like maybe thinking of it as like very small semantic units is a better framing. And yeah, this is just looking at one of them, but"}, {"start": 651.58, "end": 671.22, "text": "This is the N different or N separate or N independent. And you can see that it's firing on the word independent or different or distinct, but in every case, there's a number immediately before it. Okay, Catherine, you should definitely discuss this because I think this was one of the things that I was totally stumped by and Catherine figured out."}, {"start": 671.22, "end": 696.56, "text": "Yeah, I'm happy to take it away. So I'll describe this briefly and then I'll probably share my screen and show sort of a bit of the sort of discovery process. But at a high level, this neuron showed up in the data set examples preferring the word install very strongly. If you look at other tokens that it fires on, it might more weakly fire on other words that seem somewhat computer related, you know, package was the other one that showed up."}, {"start": 696.56, "end": 726.02, "text": "Fairly often. But if you look at these dataset examples, they're not very computer related. And so when playing interactively with it, I can get it to FHIR if it's for a computer related word, specifically install or more weakly maybe package, but the context doesn't suggest that. So maybe I can just steal presenting for a second, show some of that. Cool. Sounds great. While Catherine's switching, I was really quite confuzzled by this neuron."}, {"start": 726.02, "end": 751.23, "text": "Because it seemed to be firing for all these words, but very mysteriously, it would only fire some of the time. And you'd try writing sentences with the word install on them, and it wouldn't fire. And so, yeah, I was pretty confused by that. Yeah. So let me pull up. So this is a little dashboard that we have for checking out what neurons are doing. So I'm going to go to the dataset examples page, which is where we generally start."}, {"start": 751.23, "end": 778.93, "text": "When trying to understand what a neuron is doing. And so you see if you scroll through, it's the word install. I'll just scroll through very quickly first and then we can pause for a moment. So these are all the word install. Everything is install. But it's not. Go ahead. One quick addition on that is it's the token install, which isn't always necessarily a standalone word. So because these models split things into tokens and that doesn't always line up. And in some cases here we have install as part of installments."}, {"start": 778.93, "end": 800.34, "text": "But it can't see, since it's autoregressive, it can only see the past. And so it doesn't know that it's going to be installments. It just knows that it could be install or it could be installments. Right. So in this case, as I was describing, this is paying a fee in installments. It's not installing software. If I scroll up, I think there was another."}, {"start": 800.98, "end": 830.35, "text": "Oh, yeah. The weekly french fry-dizer, Fries, evident in the earliest installments, recent installments in this series. So this is some kind of cartoon series that comes out in installments. So you can look at this, and then we have a way to pull this up in an interactive neuron explorer. And I think I didn't show you, but another view that has been useful to me is just other than install, what are the other tokens? So that's how I discovered that..."}, {"start": 830.35, "end": 860.03, "text": "Package is another thing it sometimes fires on. So here we have an interactive editor where I can try different texts to try it out on. And I can plug in the neuron over here on the side that I'm looking at. And so we can just try something out, like this idea of like, please open the Debian package installer. Or like, please use the, yeah, so that didn't work at all. Use the command line."}, {"start": 860.03, "end": 890.54, "text": "There's no activation at all. So let me just show you. I have this activation view. You can look at the different tokens. And if there was an activation, it would be more brightly colored. If I just make that just the word install weekly. So you get an activation of one, which is something. It's something. But if I go to one of these sort of non-computer related contexts and say the construction people, right? The work site ready to install. If I can finish the sentence and they'd love that."}, {"start": 890.54, "end": 916.02, "text": "So that's an activation of almost 13. So order of magnitude higher on the spectrum. And then these words like package that are not its preferred word, it's a little harder to evoke an activation. I definitely won't get it if I say something like, you know, download the file for the package. That's, I'm guessing, going to be nothing at all. Yep, that's..."}, {"start": 916.02, "end": 935.73, "text": "Negative, so it doesn't clear the positivity threshold. But if I tell some story about like, Debbie's guests brought numerous presents to her birthday party. She gazed upon a golden wrap package. I bet I'm gonna get a little."}, {"start": 935.73, "end": 951.12, "text": "Activation there, yeah. And so I get a little bit of response to this similar word, but also with this sort of outside the computer context. And just a quick note about these sort of magnitudes that Catherine is highlighting here. Sort of like, you know, why..."}, {"start": 951.12, "end": 977.79, "text": "Why, you know, this is 1.2. We color that as very pale. What's going on there? In general, it's sort of very hard to know what the magnitudes of these activations mean in any absolute terms because they'll only ever be interpreted by being multiplied through some weights. And so kind of, are they big? Well, like that depends what weights they're multiplying by. So we scale these neurons by the largest activation we've ever seen that neuron fire on in the data set, sort of under the assumption that."}, {"start": 977.79, "end": 1006.19, "text": "We don't know what these scales really mean, but any individual neuron probably has some scale. And so if we scale it against its biggest ever activation, that at least gives us some sense of what's big for that neuron versus for other neurons may be different. Yeah. I also want to point out I didn't know this was going to happen, but I got a tiny activation for the word wrapped, which I actually haven't seen before. So on the fly, learning new things about neurons."}, {"start": 1007.15, "end": 1035.94, "text": "Anyway. Catherine, just before you do that, I wonder if you want to say anything about the, sorry, I'm also trying to figure out at the same time how to take back the presentation mode, but I feel like this has sort of been a recurring theme and motif of these neurons that sort of appear, like they're sort of like X, but context says that it's not X or something like this."}, {"start": 1035.94, "end": 1059.71, "text": "And so maybe if you just want to, I'll talk about that motif for a second. Yeah, happy to. I mean, maybe I should jump to one of the later neurons that has this as well, or do you want to sort of talk through the later neurons and then I'll pick up the thread? Yeah, I think just like you briefly flagging this now so that when we come to other examples, we can sort of refer back to it. Yeah, so this is just a motif we've seen that will come up later in this presentation."}, {"start": 1059.71, "end": 1089.01, "text": "is neurons that are sort of saying well you might expect this word but actually it isn't or you might expect that this is a computer meaning but the rest of the context suggests that it's not so there's kind of a you might expect but actually type of interpretation of the activation pattern we're seeing and it's been quite common it's cropped up a number of different times where the neuron doesn't seem to be best explained is expressing just this thing is present"}, {"start": 1089.01, "end": 1121.22, "text": "But it's better explained as expressing you might expect this thing to be true or to happen next from everything else you've seen, but actually something else is going on. That kind of motif that I think of as a somewhat either inhibitory or sort of push-pull kind of motif is pretty common. So moving a little bit deeper into the model to layer 10, so that's about 25% depth. I think another interesting type of neuron that we started to observe more of were neurons related to copying."}, {"start": 1121.22, "end": 1149.46, "text": "And if you've watched our other videos on our studying of attention heads, we know that attention heads are often really concerned with whether text is copied and looking for previous places it might be copied from and trying to predict what might come next. And we have induction heads and stuff like this. But there seem to be a lot of neurons involved in this as well when you switch to full transformer models and sort of looking for different kinds of copying. So you can have"}, {"start": 1149.46, "end": 1178.22, "text": "And we have one neuron that seems to be sort of immediately repeated text. So here we have well, well, well. So well is repeated three times. Or here we have, and this is a little bit trickier, but there's a reviewer, reviewers as reviewers. And the second reviewers is a sort of short-term copy as well. And so it's firing there. But then we have also these longer term ones where if you look at this first paragraph, it's copied."}, {"start": 1178.22, "end": 1210.32, "text": "in the second paragraph here. And I think that this happens sometimes, for instance, like if an email thread is in the training set or a forum thread and people are quoting earlier people on the forum, or if you have a news article that does like an excerpted quote to emphasize it, that's also in the main text. These are all situations where you can have paragraphs or large chunks of text repeated. And so having neurons related to that maybe makes sense."}, {"start": 1210.32, "end": 1239.71, "text": "And another famous neuron that we found in this layer was another one of these neurons that Catherine figured out that really had me stumped. And this was a particularly interesting neuron, because as I recall, we were looking at the neurons that were both extremely sparse and had extremely high magnitude activations. And this was the neuron in that layer that scored highest on that metric. So that was why we were looking at it. And when I started looking at it, I was really stumped."}, {"start": 1239.71, "end": 1269.41, "text": "It sort of feels obvious in retrospect. But yeah, Catherine, you should just tell us the story if it's not. Yeah, happy to. So this is another one where the interactive interface was very helpful because I was seeing it showing up in sort of index kind of mode. So let me just grab, if you don't mind, presenting again. Here I have the data set examples pulled up."}, {"start": 1269.41, "end": 1295.04, "text": "Wait for that to start. Cool, I think we're good. Yeah, so I was seeing it show up in these sort of like appendix, you know, index at the end of a text kind of thing, or it's like a table of contents. So you might think it's just tables of contents. Here's like a, you know, self-positioning 204. So is it just somehow lists of stuff?"}, {"start": 1295.04, "end": 1323.66, "text": "But then if you start playing with these and you just start making lists of stuff, it doesn't always react. So let's see, this is 10, 11, 5, 8, 6. So if I do, this is a random list of made-up surnames like Darby, Dodson, Edwards, Franklin. I'm kind of making this up. Yeah, it's starting to fire here. But if I did Darby, Franklin."}, {"start": 1323.66, "end": 1345.81, "text": "I don't know, random names. It's starting to fire a little here because I've started to go alphabetically. So that was the sort of genesis of this hypothesis that it's alphabetized lists. And if we go back to these data set examples, all of these are alphabetized lists. And the more, you know, it starts to, it's a little fuzzy."}, {"start": 1345.81, "end": 1375.7, "text": "It's not like, you know, it doesn't have access to the characters, right? So it has some sense of alphabetical ordering. It's not perfect. So you can, again, fool it or get it to mess up with respect to the story we're telling about it by making lists that aren't actually quite alphabetical, but it kind of thinks they are. And you can see it's sort of trying to figure out like, okay, is the word summer school the next entry in the list? Maybe, maybe this is a list that goes particle, summer, symmetry."}, {"start": 1375.7, "end": 1406.59, "text": "No, no, no. And then it gets back on the train. It's like, oh no, this goes particle, and then there's a couple other lines, and then particles. And so you can sort of watch it figure out or try to come up with an interpretation of like, is this a list? And if so, is this the alphabetized next entry? And have this kind of smooth prediction of whether it should be considering this the next item in the alphabetized list. I feel like this was a place where having the interactive interface for exploring this was just"}, {"start": 1406.59, "end": 1426.99, "text": "Really, really crucial and seemed to really be the difference between us getting this. I think especially like approaching it in a sort of adversarial way and trying to really see if it holds up when you go and you play with it or not seemed like it was important."}, {"start": 1426.99, "end": 1450.24, "text": "So here's an example where it's starting to get on, but it doesn't seem to think the dates comes before donuts. So I think another thing that I have been grappling with, especially with this neuron, is that the kind of work that we're doing is coming up with stories, explanations, hypotheses for how we might, in a compressed way, describe the selectivity of the neuron."}, {"start": 1450.24, "end": 1478.94, "text": "The fact of the matter is that like the only ground truth model of what the neuron is selected for is the neuron itself. So, you know, if the neuron doesn't fire on dates, that's the real truth of the matter of what this neuron is actually doing. And the fact that I think that's a mistake because I think dates comes before donuts and after coconut. That's, you know, whether you can say that the neuron is messing up relative to this simple rule that otherwise explains its behavior or whether you can say like the rule is just inaccurate."}, {"start": 1478.94, "end": 1509.14, "text": "Actually, it's doing something more complex and more sophisticated. It's actually kind of a philosophically challenging question as to which of these views is more useful, I guess, for what we're trying to do. One other thing that I feel like was pretty helpful with this neuron was initially we weren't formatting the white space in the text correctly. And I think we also weren't collecting our data set examples over that many examples or that larger fraction of the data set."}, {"start": 1509.14, "end": 1538.77, "text": "And improving that, I think, made it easier just to go and see patterns and data set examples. And so I think investing in infrastructure has really helped with this kind of thing. Yeah, this view previously would have just been, the white space was not correct. And so it would have just been a whole pile of words, which was much more challenging. Steal back presentation. Go for it. I guess maybe just one other quick context, quick comment while Chris is stealing."}, {"start": 1538.77, "end": 1566.7, "text": "Catherine's mention of sort of sometimes you come up with these rules that are like nearly right, but imperfectly predictive. That reminds me a lot of my experience taking my first ever linguistics class where there's like a lot of sort of pop knowledge about like think about rules in English language or places where there are seeming regularities or irregularities. And in a lot of cases, the like pop rule is like."}, {"start": 1566.7, "end": 1592.66, "text": "Decent or the pop pop culture rule is like yeah it's just arbitrary but then linguists have like gone much deeper and they're like actually this rule that you thought was about the spelling of the word is about the phonetics of the word and if you like look at the phonetics everything is totally regular it's just that the spelling has been corrupted because English spelling has such a weird and arbitrary history and that can be a kind of similar story of like if I have a rule and it's almost right it's often worth"}, {"start": 1592.66, "end": 1613.2, "text": "There's a natural temptation to be like, ah, the model is trying to do X badly. And it's worth thinking hard of like, actually, it's trying to do X prime and it's doing it quite well. That's right. Both stories can be true in different cases, but it's almost always worth like, I don't know, I find it's..."}, {"start": 1613.2, "end": 1639.66, "text": "There's some instinct of like, oh, it's just a model. It's like trying to do something, but it's bad at it. But these models are very capable, and it's often worth trying harder to find the real rule. And then I think the other intersection is that I remember Chris mentioning going off down a total linguistics Wikipedia rabbit hole and then coming back with half a dozen new hypotheses for various neurons or attention heads."}, {"start": 1639.66, "end": 1664.93, "text": "And it's been interesting to note that like in some cases the sort of innards of these models are like modeling language in some ways like better than sort of I consciously understood language. I clearly unconsciously understand language like quite well. You're very good at it. You're using it right now. Yeah, right. But there's like a whole set of rules that the linguists have figured out that like I didn't learn or forgot in my one linguistics 101 class."}, {"start": 1664.93, "end": 1693.42, "text": "And that's actually been, in some cases, a surprisingly rich source of hypothesis generation. I think a related thing is just, I think people, I think it's sometimes underappreciated how much of the challenge of understanding these models can be that we just don't have the right hypotheses. Because I think I often see work where people sort of like, you know, they go and search for some features that they think are going to exist in the model and see if they can find."}, {"start": 1693.42, "end": 1723.97, "text": "I want to just riff on what Nelson was saying about the discovery process where it's tempting to say"}, {"start": 1723.97, "end": 1744.56, "text": "Oh, this neuron is kind of doing short phrases of this type. Got it. Got it. Okay. And move on. But in my experience with the install neuron and the alphabetical list neuron, with the first two where I really felt this, there's a kind of click. There's a kind of aha where you stumble on the exact right hypothesis and suddenly everything is very clear. And all the weird..."}, {"start": 1744.56, "end": 1771.31, "text": "Quirks of like, but why aren't these very computery? Just instantly snap into clarity. I have some experience a couple of years ago, you know, for a phase in my life, I was a MIT mystery hunt competitor in, you know, you get together with a team, big team of dozens of people, and you're trying to understand the meaning of a random pile of letters. And at some point you have the aha, and you have the hypothesis of what the puzzle constructor."}, {"start": 1771.31, "end": 1803.3, "text": "Okay, so we can also jump to the end. And I think the last layer is somewhere we've had a lot of success as well. And the reason for that, I think, is"}, {"start": 1803.3, "end": 1829.89, "text": "MLP neurons in the last layer have a unique property that they linearly affect the logits, and that is the only thing they can do because they just get added. They go through their down projection that gets added to the residual stream, and that immediately hits the unembedding. I guess there's a layer on there, so maybe it's very, very slightly nonlinear, but it's effectively just a linear effect on the logits."}, {"start": 1829.89, "end": 1852.51, "text": "And they can't go and affect other neurons. They can't be going and affecting things through attention tents. They can just linearly affect it. And so you might have, in some cases, have trouble understanding when the neuron fires. But you can absolutely understand what the neuron does when it fires. Catherine, I think you analogize these sometimes to motor neurons. Yeah, yeah."}, {"start": 1852.51, "end": 1879.79, "text": "Half of a PhD that I then quit, I did a bunch of neuroscience. And so the sensory neurons, motor neurons analogy tends to make a lot of sense for me where computational neuroscience has had the most success more on the sensory side or more on the motor side of analyzing sort of the behaviors and selectivities of neurons or clusters of neurons. I think this is a similar case because we know what the output is. Then we can sort of directly track."}, {"start": 1879.79, "end": 1900.54, "text": "The contribution of these neurons behaviors to the output, which gives us a grounding that in these 40 layer models, when I was trying to understand layer 20, I just had no footholds at all. I had no way of telling what the right frame of reference was. But for last layer neurons, the correct frame of reference has to be very close to"}, {"start": 1900.54, "end": 1931.6, "text": "What next token is emitted or predicted by the model? What slate of anticipations does it generate? That has to be just structurally close to the right way to think about them. And this is sort of strictly true for the last layer, but sort of a softer version of this might be true for the last couple of layers. I mentioned earlier that with this sort of inhibitory motif, some of the"}, {"start": 1931.6, "end": 1940.34, "text": "Some of the ones near the last layer, I think, also are interesting, if there's a good point for me to show a couple of those too. Awesome. And I think we've actually found"}, {"start": 1940.34, "end": 1964.66, "text": "So for neurons in the middle of the model, our usual assumption is that because of the kind of residual nature of transformers, every neuron is added directly into this residual stream. And then that value can either interact with later neurons or attention heads, but also that value is kind of there forever. Someone might subtract it out, but then they're sort of"}, {"start": 1964.66, "end": 1990.24, "text": "And so every neuron, you can look at what its direct effect on the logits is. And I think we've found that often, even if that direct effect is a small part of the story, it's sort of in some way related to what it's doing. Sort of something about the training process or the sort of way these models build is that it will affect tokens that are kind of in some way related to what it's doing. So even for middle layer neurons where that's not"}, {"start": 1990.24, "end": 2017.79, "text": "It was potentially not even mechanically a large part of the story. There's still sometimes information or hints there about, ah, this neuron makes those tokens more likely. That helps me generate hypotheses. Yeah, that's a great point. And in general, it's just a very nice thing to be able to go and quickly look at and have available. And people who are watching our other videos know that we've been doing this a lot for attention heads as well."}, {"start": 2017.79, "end": 2048.16, "text": "Anytime something writes directly to the residual stream, you can be asking what the long-term effects of it are. And this relates to, I guess, there was a user, Nostalgebrist, who wrote a post on the logic lens interpretation of Transformers. This is a little bit in that vein. But yeah, so here we have one neuron. And if you look at when it increases tokens, it increases"}, {"start": 2048.16, "end": 2075.17, "text": "All the tokens that it increases start with a vowel, and it decreases the probability of tokens that don't start with a vowel. And often it fires on an an, and so an an is going to indicate that the next word must start with a vowel, because otherwise you'd use an a. There might be other cases where it also fires, but all the really high magnitude examples that I could find when I was preparing this slide were like that."}, {"start": 2075.17, "end": 2104.77, "text": "I also think this is just another great example where if you hadn't seen the dataset examples on the left side, and you're just looking at this pile of, like, increase-decrease tokens, you'd be like, amalgam? Embroidery? Is this about, Austria, is this about, like, feathers or decorations or what is, you know, range, combination, pair? Like, there's also some additional, like, semantic thing that might be part of the story, but you could also miss this sort of high order bit."}, {"start": 2104.77, "end": 2132.69, "text": "is the vowels versus consonants. Like if I look at these lists, I do get a sense that there's something about rich textured nouns versus mathematical nouns, but that's not the high order bit of what's going on. And as soon as you see those activations on and you understand that the primary thing to understand here is the vowel consonant. Yeah, I think it's sort of"}, {"start": 2132.85, "end": 2159.89, "text": "It also lends, sort of sheds light on sort of on when you're trying to understand or interpret these neurons, it's important to have a sense of where they are in the model and sort of what that relates to your space of hypotheses. Because, you know, if you just look at these data set examples, it'd be very easy to guess, you know, ah, this just fires anytime it sees the word and. But that hypothesis makes no sense because that, the model could do that."}, {"start": 2159.89, "end": 2188.38, "text": "A, the model could do that way earlier on. And in general, we tend to assume that these models are well optimized and reasonably efficient. And they're not going to be spending too much of their capacity doing things that are obviously useless or that they could have done earlier. And so it doesn't make sense for it to be doing something so trivial this late in the model. But also, there's no computation after this neuron. So like, ah, this is the word Anne."}, {"start": 2188.38, "end": 2218.27, "text": "Like that fact doesn't really, like that fact is, it's too late to know that fact. And so it is the case that this neuron very often fires on that token and very often fires on others, but sort of the logic effects tell you this, the kind of the much better story and give you a much better story that fits into where the neuron is in the model. And so the space of things that like are sort of informative to talk about this neuron. I think this also relates to why it's useful."}, {"start": 2218.27, "end": 2242.16, "text": "To invest in understanding the architecture of these models. I was having a conversation with somebody else the other day, and I think it's really valuable to invest, before you go and spend a lot of time poking around with these models, to invest in understanding what kinds of features exist at different layers, or what the architecture of the model is, and how things are wired up such that you..."}, {"start": 2242.16, "end": 2264.5, "text": "The observation that the only thing that an MLP layer can do is go and affect the logits is something that you can get when you've invested in that. So Chris, can I grab presenting for a moment? Because I just, you know, discussing... Yeah, go. Let me just grab and then I'll... Although I think we're almost at the end. We're almost at the end. So we can also just go through this and then... Either way, you have the presentation now."}, {"start": 2264.5, "end": 2296.74, "text": "Just as Nelson was saying, this neuron is too late for it to be firing on an. If it's actually just predicting when there's going to be a consonant, I can make this whole string with just context, meta-learning, where the word or is always followed by something that starts with the vowel. And sure enough, it picks it up. And now it knows that it should be up-weighting the vowel neurons. So I just threw this together right now, just there. That's phenomenal. There you go. Back to you. Thank you. Well, I can ask to."}, {"start": 2296.74, "end": 2332.5, "text": "I really need to learn this trick of how to resume presentation. How do I, there we go, resume presenting. Okay, so, oh, yeah, so there's, we're in fact not quite at the end. So what about neurons in other layers that we found harder to understand? So one layer in particular that we found kind of tricky to understand, and perhaps that you'd sort of think would be the easiest layer to understand of all of them, has been the first MLP layer."}, {"start": 2332.53, "end": 2362.34, "text": "Part of the reason for that, well, I guess there's two things that are very striking about this layer. One is that the attention heads immediately before it, which are the only thing the neurons can be computed based on, along with residual strain, are extremely diffused. Rather, in large models, this isn't true in small models. As models get larger, those attention heads just start being smeared out over basically all previous tokens. And the neurons become exceptionally sparse. And so in theory, you could actually, like these neurons have a very nice form."}, {"start": 2362.34, "end": 2389.25, "text": "Where you could just mechanistically understand them. You could expand them back, and you could be like, well, this is what it's looking for through this attention head, and this is what it's looking for through this attention head. But because the attention heads are these extremely blurred things, that's been a little tricky to reason about. And a lot of the obvious hypotheses you might have for this layer don't seem to be true. So I think the thing that I first imagined the first layer would be doing when I started looking at it is piecing together"}, {"start": 2389.25, "end": 2419.57, "text": "Subword tokens into whole words. So when a word gets split into two tokens, it would be very natural to have neurons that try to go and fix that and turn them into coherent words. And that doesn't seem to be what the first layer is doing, or at least not a large fraction of it. And something that really helped me think about this is there's this paper from Google Pair by Konadol in 2019. And they have this really nice diagram."}, {"start": 2419.57, "end": 2450.26, "text": "Where they take the activations, and these are, I guess, residual stream activations, for a single token, like die. And then they go and they just do a t-sting or a umap. And they find that there's one cluster that corresponds to the German article D, and another one that relates to people dying, and another one that relates to dice. Because those are all different interpretations of the token D, or die."}, {"start": 2450.86, "end": 2480.7, "text": "And it made me wonder if maybe something similar might be going on because a very sort of an attention that's blurred out and looking at all previous tokens might be actually pretty well positioned to help you guess what language you're looking at. And so maybe you could go and correct your interpretation of tokens. And it seems like that might be what's going on. So if you either do like a PCA of the MLP, the first MLP layers activations for a single token, or you do a UMAP, you'll notice that..."}, {"start": 2480.7, "end": 2510.61, "text": "You get clusters corresponding to languages and sometimes even to like particular types of speech within that language. What do you mean by emotional, Chris? I'm not sure I recall this. Yeah, yeah. It's like just there's some sentences about death or some samples about death that are like from a novel where somebody's like speaking very emotionally about death. Lamenting, yeah. Lamenting or it's just like kind of, I don't know, kind of intense language in some way."}, {"start": 2510.61, "end": 2541.7, "text": "And very personal. And there's some that's much more depersonalized, statistical or clinical. And that seems to be the primary dimension of that cluster over there. So based on that, you can then start looking at neurons in MLP0. And here's one, for instance, that seems like it might be related to, it only fires in contexts that are Spanish. But the tokens, it doesn't fire on every token."}, {"start": 2541.7, "end": 2571.12, "text": "That's Spanish. And in particular, it seems like it may be fires on tokens that could be English tokens or maybe could be tokens in other languages. And this is a little bit cherry picked because I wanted to find a couple of interesting examples. And there's some places where it fires on tokens where it's less clear this could be the case. But for instance, the token fund, well, fund is an English word as well. Like fund is in funds. You could also have fund is in fundamental. In is a standalone English word."}, {"start": 2571.12, "end": 2601.81, "text": "Also is used in lots of other contexts. Europe is an English word. Indef could be part of indefinite. Import is an English word. Fin is an English word. And a lot of these other things you could imagine that maybe there's some English word that can be tokenized such that it's some portion of them. And so there seem to be a lot of neurons, at least a fair number of neurons in MLP0 that seem like they could be interpreted in this way where they"}, {"start": 2601.81, "end": 2631.14, "text": "They are words that are disproportionately, yeah, seem like they could be in one language, but context makes them seem like another. And so that's one hypothesis. This is that sort of inhibitory pattern. Like you might think it's English, but actually it's Spanish. So don't be fooled. The context says something different than the token itself. And there seem to be a lot of these for different languages and different pairs of languages in MLP0. Again, I think this is still sort of relatively, yeah, not as pinned down as all the things that we've studied."}, {"start": 2631.14, "end": 2657.54, "text": "But it seems like that might be what's going on. Yeah, just one thing I'm noticing for the first time staring at this example is like on your last line there, the word embargo is also a straight up English word and it doesn't fire on that. So it's sort of quite clear that we don't fully understand it, but it's- Oh, that's a good point. It clearly has that flavor, but there's work to be done to try to... And I think, again, this gets to the like..."}, {"start": 2657.54, "end": 2681.71, "text": "Is the model doing X badly or is it doing X prime, which we don't yet understand well? I think our guess is that if we find the right explanation, it will, as Catherine said, kind of click into place. And but we're still figuring it out and kind of it will, I think, likely be the case that we sort of don't solve this one, but that we'll have some later revelation about some other neuron. And then we go back and look at this one and we're like, oh, yes, it was clear all along."}, {"start": 2681.71, "end": 2709.84, "text": "Yeah, this is really in that phase of like, it seems to be doing something like the following question mark, as opposed to like, we got it. And I think that is partly because we have some of the positive story. We don't have some of the negative stories. We don't know when is it something that is an English token or word, but it nonetheless didn't fire. What's the story with that? And I think I have nothing to offer for this guy in that regime. One thing that I might be tempted to suggest is it could be about just..."}, {"start": 2709.84, "end": 2738.99, "text": "The relative probability of these tokens in English and Spanish. And so embargo might be more common. I don't know. I don't think that would be an interesting thing to overlay. It is more common in Spanish. I thought one final thing maybe to just talk about is what have been the challenges to this kind of investigation. So I think having good tools, I think, has been really important. I think that's probably a fairly straightforward one. I think people."}, {"start": 2738.99, "end": 2769.07, "text": "Yeah, I think this challenge of not having the right hypotheses to distinguish between. I think we often think about hypothesis-driven science, where you have two theories that could be true, and you're trying to distinguish between them. But I think we just have this enormous space of hypotheses that we can't enumerate, and we don't know what the right ones might be. And that's much more often the regime that we're operating in. I think that's tricky. And then there's this EOT token issue. And Nelson, I wonder if you want to just briefly comment on that. Yeah."}, {"start": 2769.07, "end": 2792.93, "text": "This might be worth a whole video of its own at some point, but maybe the two-minute version. Yeah, it's hard to know. There's a whole thread here, and it's hard to know how much is a quirk of some details of our architecture. But when we train models, we always give them this leading token that we confusingly call EOT for end of text, even though it's always the first token in the text."}, {"start": 2792.93, "end": 2817.1, "text": "We always give it this first token, which we find is useful for attention heads to attend to, among other reasons. The EOT token, if you look at its activations in the residual stream, is very unique. It doesn't look like anything else. It has these very large magnitudes. We expect it to be weird, because it is a very weird token. But we also find that"}, {"start": 2817.1, "end": 2842.53, "text": "Because of an artifact of how our attention mechanism works, the model tends to replicate this EOT token so that other random tokens gain very similar activations to EOT at sort of some, not exact, but some periodicity through the model. And there's not, I don't think an enormous amount of capacity, but some number of..."}, {"start": 2842.53, "end": 2870.16, "text": "Neurons and attention heads that are seem to like mostly be dedicated to constructing and then tearing down when it's time to make a prediction, these fake EOT tokens. And one of the salient things is that these so-called fake EOT tokens, what we call fake EOT tokens, have very large absolute magnitude of values in the residual stream and activations in some point."}, {"start": 2870.16, "end": 2888.91, "text": "And as we mentioned earlier, one heuristic we sometimes use to search for things is looking for things that are sparse and have high magnitude. And so those tokens just sometimes cause false positives. And they tend to be very uninterpretable because we believe them to be this artifact of the model."}, {"start": 2888.91, "end": 2913.22, "text": "Architecture where it's trying to preserve information that it otherwise wouldn't have access to. And so you get these very large activations that have functionally nothing to do with the texts that they're attached to. And we spent a while on some red herrings until we understood this phenomenon because we were like, these things are so much larger than any other activation. Like there has to be something really important going on here."}, {"start": 2913.22, "end": 2940.48, "text": "If you look at certain statistics of the sort of whole model, these guys dominate, these guys are the whole story. But then you look at them and they're attached to, we understand a little bit now about where they're attached to, but they're attached to functionally random positions and resist any easy attempt to understand what's going on there, which I think also speaks to Chris's point of if you don't have the right hypotheses, things..."}, {"start": 2940.48, "end": 2949.9, "text": "And or if you have the wrong hypothesis and you're intentionally or not too attached to it, it's sort of possible to just be completely baffled."}, {"start": 2950.45, "end": 2978.32, "text": "I will say, I mean, I thought it was useful. You said sort of random tokens or, you know, arbitrary tokens. But one discovery that seemed to sort of help move towards that story or help fit into that story is one common place. We often use the first paragraph of Harry Potter as our demo text. And there's the word didn't and hadn't show up at two places in this text. And they're tokenized, I think, DIDN as one token."}, {"start": 2978.32, "end": 3005.12, "text": "Or HADN as one token, and then apostrophe T is like a whole token which the model can very, very accurately predict. And you see DIDN, the chance that the next thing is apostrophe T is like through the roof. And so that's a token, apostrophe T is a token that the model would use as one of these fake EOTs because it sort of has capacity, is our story, to put this other random stuff there without throwing itself off because its prediction task is so easy."}, {"start": 3005.12, "end": 3031.12, "text": "That it knows what's coming next. I guess I stated that kind of, it's the token before the apostrophe T. Yeah, thank you. Everyone's sort of looking a little gloomy as I'm speaking because I'm saying the wrong thing. The D-I-D-N, right before the apostrophe T, is where it piles this furious EOT information because our story is because it can't throw itself off that badly because its task at that point is so easy."}, {"start": 3031.12, "end": 3051.66, "text": "This turned out to be a general story. There's kind of a couple of other patterns. And this story seems to be that it's always in a place where usually due to some quirk of tokenization, the prediction task is like brain dead easy. Sort of the model can tell very early on that there's really only one thing that it could ever possibly be."}, {"start": 3051.66, "end": 3072.83, "text": "And then it's like, great, I don't need to perform any computation here. I already know the answer. And so I can reuse this capacity for this scratch space to work around a quirk of my architecture. And I think that's kind of one of the, I think that might be the reason why this might be worth a whole video is I think the specific phenomenon is."}, {"start": 3072.83, "end": 3102.37, "text": "Is potentially just sort of architecture quirk, but I think it's an interesting view into the like kinds of behaviors that these architectures are capable of evolving and sort of how they allocate capacity to different tasks in different positions. Well, I want to push back a little bit on this being, I think this is an arbitrary quirk, but I think it's an arbitrary quirk that probably a lot of people at different labs have in their models right now. And so I think in particular, if you have a model that is using block sparse and you are seeing really weird activations."}, {"start": 3102.37, "end": 3128.16, "text": "This is a pretty live hypothesis that you should have. And I guess that if you're trying to study large models, there's like a 50% chance you're using blocks first. I think we've also seen evidence of it in models that are partially dense and partially local attention. And I feel like I suspect every large model is using some combination of one or both of those strategies. And so, yeah, it may be."}, {"start": 3128.16, "end": 3141.58, "text": "It's a kind of quirk, but you're right. I think it is a quirk that is probably a wide enough class that many other transformers have it. Universal quirk. Chris, do we have time to make one more comment about infrastructure?"}, {"start": 3141.71, "end": 3170.66, "text": "Yeah, I think the only thing we have left is, is there anything else we want to talk about? Well, if you go back to the previous slide, I just wanted to remark on that first line you had, which is the infrastructure, the tooling is pretty crucial. So, you know, throughout this video, I've pulled up this interactive text box where I'm typing stuff and trying things out. And I built that before Nelson joined and built Garcon. And if you want to learn more about Garcon, there's a previous video on it, which is fantastic."}, {"start": 3170.66, "end": 3196.62, "text": "You know, when I first built this interactive text box, I would just spin up a Python, you know, Colab notebook and put a model in the notebook and then run a little flask server in the notebook and then pull up a separate, you know, page to then be speaking to my notebook. And that was, it's a little clunky. It's not great. Whereas Garcon allows us to put these models on our cluster. And so now I just have a sort of static website that's..."}, {"start": 3196.62, "end": 3228.56, "text": "Sending these texts that I'm typing to the model in the cluster. And I don't have to be hosting it on my own dev box. And it's a lot easier. It's been fantastic. And so thanks to, you know, I'm glad Nelson joined. And maybe I'll just make a little pitch that if you're an engineer and you want to work on this kind of stuff, we are hiring engineers. And it will make our lives a lot better in these exact ways. So think about it. Well, thanks everyone for joining us. Catherine, Nelson, do you have any last things you want to add in?"}, {"start": 3230.64, "end": 3259.01, "text": "I just want to say I'm excited about more people out in the world doing this kind of work. It's fun. It's very much sort of like exploratory kind of science. I think machine learning can get really bogged down in make the number go up, you know, make a state of the art whatever to classify whatever. And that's fun. But this is also fun. It's a different kind of fun. It's a different kind of science. And I think it's a kind of science that's lacking in the machine learning landscape."}, {"start": 3259.01, "end": 3283.63, "text": "And I would love to see more folks doing this. And as I said, having the right tools and infrastructure behind you is a large part of it. And I built my first JavaScript interface a few months ago when I joined. It's not that hard to learn. So even if you think that you're not this kind of person, if you can code and you can learn new stuff, this kind of work might be more accessible than you think. And it's fun."}, {"start": 3283.76, "end": 3298.61, "text": "Yeah, and I think on that note, I think there's a paper I think that I want to give a shout out to. It came out, I think, late last year was Transformer Feed Forward Layers are Key Value Memories. I think we haven't actually..."}, {"start": 3298.61, "end": 3316.61, "text": "I've been kind of directly influenced by that. I think we sort of read it closely midway through this work for the first time, which is maybe remiss, or at least I came to it sort of after doing this work. But among the other features it contains, they did some kind of similar work of sampling a bunch of neurons, farming them out to people in their labs, forming."}, {"start": 3316.61, "end": 3347.7, "text": "These kind of hypotheses of what's going on. And they document a bit about what they learned about the different kinds of neurons and different layers and have some cute examples of sort of a very similar kind of work to what we're talking about here. They were working on a somewhat smaller model, but that's the one other instance I'm aware of very similar work being published. And so I thought it was worth a shout out. Awesome. Well, yeah, thanks to everyone who joined us. And thank you, Catherine and Nelson, for taking time to chat about this."}, {"start": 3348.14, "end": 3350.03, "text": "Yeah. Thanks Chris."}]}