{"url": "https://www.youtube.com/watch?v=e0u7ZSAPZAA", "page_url": "https://www.youtube.com/watch?v=e0u7ZSAPZAA", "title": "Softmax Linear Unit (softmax(x)*x) Neurons - Preliminary Results", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 6346, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing. SoLU videos may be led by Nelson Elhage/others - Olah's share uncertain.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-softmax-linear-unit-softmax-x-x-neuro", "audio_duration_s": 6345.4911875, "worker": "jc-w05", "transcribe_s": 85.0, "model": "large-v3", "segments": [{"start": 0.11, "end": 27.78, "text": "Well, today we're here to talk about some experiments we've been doing at Anthropic on how to make MLP layers more interpretable. And we think we found something that works quite well. And so I'm excited to talk with everyone about it. And I'm here with my colleague, Nelson Elhage, who has really been the person driving these experiments. So we're going to go and just sort of chat about what we found. Welcome, Nelson."}, {"start": 27.78, "end": 56.21, "text": "Yeah, I'm excited to share these initial takes on some, I think, early but pretty cool results we have. So just, yeah, so I think the TLDR is just that changing the activation function of the MLP neurons can make neurons much more integral. And one thing I think is sort of important to note here is this is really"}, {"start": 56.21, "end": 84.75, "text": "An interim update on early research. So we're, you know, this isn't something that we're highly confident about. There could be mistakes. There could be things that we're misunderstanding. But these sort of seemed like sufficiently exciting results that we wanted to share them early on rather than waiting until we could go and produce a really highly polished paper. So expect this to be kind of rough and sort of take things that we say with a grain of salt because this is early work. Sort of relatively low confidence on any assertion that we make here being sort of ultimately correct. But we think they're probably directionally onto something."}, {"start": 84.98, "end": 116.27, "text": "Yeah, and I think hopefully as we talk about this, we can sort of be clear about what level of evidence we have and what we found that has made us believe this. This is in collaboration with many of our colleagues on the interoperability team, really, really everyone, and also other colleagues at Anthropic. So really, none of these experiments would be possible without tooling and infrastructure and conversations and support from other people. Yeah, okay, so I think the high level..."}, {"start": 116.27, "end": 144.62, "text": "Problem is we'd like for neurons to be interpretable. And that's actually like, I guess maybe one very basic thing is, is that even a reasonable thing to hope for? Like, is it reasonable to ask for individual neurons to be interpretable? And for that matter, I think we'd claim that that isn't the case for the residual stream. So yeah, maybe Nelson, maybe I could ask you, like, why is it one might even hope for neurons to be interpretable?"}, {"start": 145.39, "end": 174.88, "text": "Yeah, I mean, I think the main reason for sort of neurons in particular is because there this one point in the model where we certainly with we apply this non-linearity to typically a gel U or a rel U in a transformer. And we even apply it to sort of these like specific scalar values. And so like the actual like, you know, value of the floating point number there has some significance. The model is, you know, branching on that value."}, {"start": 174.88, "end": 180.69, "text": "And so we sort of, that gives it a privileged basis is the way we describe that."}, {"start": 180.69, "end": 209.97, "text": "And so that value means something. Whereas values in the residual stream, we always read and write to them through essentially an arbitrary linear projection. And so you can rotate the entire residual stream or apply any linear transformation and then apply the corresponding transformation everywhere else in the model and get an identical transformer. And so there's really no reason to expect the value, definitely no reason to expect the individual dimensions of the residual."}, {"start": 209.97, "end": 226.62, "text": "Stream to be meaningful. You might hope that you could sort of find some basis where it's interpretable, but even that there's reason to believe that it sort of would be tricky or would be clear or that sort of that might not be the right way to think about it. And kind of, you know, for neurons where we're like."}, {"start": 226.62, "end": 251.23, "text": "Applying this non-linearity to this specific scalar, like that number is in some sense kind of very non-arbitrary or at least potentially very non-arbitrary that there's real meaning encoded in that specific number that we're, you know, we're doing a comparison and we're, that comparison or that, you know, if we're doing a ReLU, we're doing this comparison, sort of, you know, are we greater than or less than zero?"}, {"start": 251.23, "end": 278.74, "text": "And that, you know, we might expect that to be, you know, is some signal or some feature present? And it turns out that in vision models and a lot of Chris's past work, I think that's sort of not always, but often true that these nonlinearities correspond to, you know, we sum up a bunch of evidence and if it's above some threshold, we sort of some feature, we say some feature is present. And if it's not, we kind of just round to zero and don't emit that feature at all."}, {"start": 278.93, "end": 309.09, "text": "And I think that actually brings us to the second mystery, which is why is it the case that sometimes this gives us really interpretable neurons and features where we can really understand what's going on and sometimes we can't. Oh, and actually, before we move on, maybe, Chris, do you want to unpack a little bit what you mean when you say interpretable here? Yeah. It's a slightly subtle thing. I think that's a great question. So I think what we really mean is that we can develop some hypothesis."}, {"start": 309.09, "end": 332.85, "text": "In the case of a neuron, for instance, that we can develop some hypothesis that explains when the neuron fires. So this neuron is firing in response to some articulable feature that we can describe in the text or implementing some articulable output behavior. And we can generate that hypothesis. And then when we test it, it continues to be true."}, {"start": 332.94, "end": 358.56, "text": "I think of it a little bit as the features or the features the model is working with in some sense are recognizably aligned with the kinds of features that we as humans understand. I think the one place where this gets really subtle from my perspective is that sometimes"}, {"start": 358.56, "end": 388.56, "text": "We have features that they're just things we haven't thought of, but they're actually natural in retrospect. So in vision models, for instance, we found these high-low frequency detectors that there's a whole paper on them if you're interested. But they seem to be something that you wouldn't have thought of beforehand. But in retrospect, they make a lot of sense. And so I sort of think that even features that are sort of kind of alien, if in retrospect, once you understand them, they're easy to understand. I think that should count as an integral feature. And I think that makes it much harder to judge quickly."}, {"start": 388.56, "end": 412.21, "text": "When you have a neuron that you can't understand, is it that you just haven't figured out the right hypothesis yet, or is it that something fundamentally weird is going on with that neuron? So I think that is a tricky thing about all of this. Yeah, and I think that, Alex, I think that and the reference division models leads into sort of the question of why aren't neurons always interpretable? Because I think sort of..."}, {"start": 412.21, "end": 438.29, "text": "The poly-semanticity gets into this clear example of a thing that is not well interpretable, or at least not in this sense. Yeah, so there's lots of neurons in these models. We see this both in vision models and even more strongly in language models, where there are neurons that, even when you look at them really carefully, they don't seem to correspond to anything in particular that's interpretable."}, {"start": 438.29, "end": 468.38, "text": "I think a very common behavior is that you see neurons which fire seemingly to a union of unrelated things. So the famous example in vision that we, well, famous to us, is this neuron that fires in response to cats and fronts of cars and paws, cat faces. And those are just like pretty unrelated things. But for some reason it fires to that union. And so,"}, {"start": 468.38, "end": 495.54, "text": "We sort of called this phenomenon polysemanticity. It fires in response to multiple unrelated things. And it's hard to systematically study this because we don't have really a mathematical definition of it. Like there's no automated way in which you can say that a neuron is polysemantic, or at least no one that I know has developed such an automated way. And I think we don't even really have a very good theory for..."}, {"start": 495.54, "end": 527.5, "text": "Well, we have a theory for why this happens. We don't have a way to test that theory very well. And really, the only reason that it sort of is the theory that we take seriously is that we don't have a serious, yeah, good competing theory. And so there's this one theory that we'll get to in a second called superposition that is a theory for why this happens, but we don't really know. And by the way, I should flag here that this has been discussed a fair amount in the original circuits thread. So that's another place that you can go if you're interested in this."}, {"start": 527.5, "end": 554.94, "text": "Apparently an image did not come through to the slide, so I guess we will not be showing this. But maybe Nelson, do you want to just talk a minute for a minute about what we mean when we sort of have a superposition hypothesis and what the theory is for why polysematic neurons might be forming? Yeah, I think the one perspective that I think"}, {"start": 554.94, "end": 574.29, "text": "I don't remember who in our team observed this way, but it's the idea that the neuron is encoding. If an individual neuron is a interpretable feature in some sense, like this neuron is firing if we're talking about dogs, you can sort of"}, {"start": 574.29, "end": 603.22, "text": "Think of that as like this one hot encoding of the feature space. And I think you, like one hot encodings are sort of like super nice to work with in a lot of ways. They're very simple, but they're also in some sense inefficient. And I think, you know, one hypothesis is sort of the model is sort of developing a like richer sort of like potentially still, you know, treat each feature sort of relatively sort of conceptually as kind of present or not present. But then it's like using the"}, {"start": 603.22, "end": 629.57, "text": "Using sets of neurons to get the power set of all of these neurons to represent your feature space. And so the presence of a dog is indicated by the following four features all lighting up. Or maybe the four features lighting up and then also some other features not lighting up. And if you're doing some sort of encoding like that, then potentially you can just pack much more information into the same number of dimensions."}, {"start": 629.57, "end": 656.16, "text": "And potentially, you run the risk of some interference of some encodings are incoherent or you can encode conflicting things. But then presumably, the model just arranges so that the possible incoherent states are very unlikely or very unlikely on the training distribution. But so the theory is basically you're trying to encode more information or more features than"}, {"start": 656.16, "end": 684.83, "text": "There's one hot, one feature per neuron encoding allows. And it seems perfectly possible that you could do that. And these neurons that seem to be firing over unions of unrelated sets are probably what you would expect to see in that case. And so we haven't really done at all the work to nail down that this is exactly what's going on. But it seems like a plausible hypothesis, as Chris says."}, {"start": 684.83, "end": 714.67, "text": "We sort of believe it because it is plausible and we don't have a better hypothesis, but that's very different from us feeling confident or asserting that we've demonstrated that this is what's going on or anything like that. One other piece of evidence for this is that in the original circuits thread, there were some circuits we traced out where we'd see things like, oh, here are car connector forms, and then it gets smeared over a bunch of unrelated neurons. And so you can sort of see this in the forward direction of the weights as well. And that sort of seems like some evidence this is happening."}, {"start": 715.66, "end": 734.24, "text": "Yeah, and I guess one maybe useful intuition here is that the more a model has the capacity to represent lots of useful features, but not enough neurons, the more that sort of seems like a tempting strategy for the model. So if there's a million..."}, {"start": 734.24, "end": 767.95, "text": "Useful linguistic features to go and represent, but you only have 10,000 neurons. Let's say that there's only 20,000 useful features, but there's 10,000 neurons. It's really tempting to go and try and somehow squeeze all 20,000 of those features into those 10,000 neurons, even if that means that you're going to do it in a slightly less principled way. And these sort of effective if statement-like things that you're using sort of are going to have interference between them and so on. Maybe it's still a worthwhile cost to pay, perhaps."}, {"start": 768.05, "end": 791.9, "text": "Yeah, Nelson, can you talk us through what change we make to go and make it? Yeah, so we've been thinking a lot about this problem of understanding neurons in these transformer models. And we've been doing some experiments with doing architecture tweaks, not for the sake of capabilities, but for the sake of can we produce models that"}, {"start": 791.9, "end": 821.28, "text": "perform comparably well, or potentially, you know, we take a little bit of a performance hit, but we find it much more interpretable in some way. And at Chris's suggestion, the experiment, or we ran a couple experiments, but the sort of one that seems most promising is we replaced the gel U inside the MLP with softmax of x times x. And so we do a softmax like over the D MLP dimension. So typical transformers."}, {"start": 821.28, "end": 849.98, "text": "Architecture is you have your demodel dimension. You run that through a linear projection to get up to four times demodel neurons. That four is a hyperparameter, but four is the standard choice for most of these. And then normally you would just run a pointwise gel you over all of those. We've been running one big softmax over all of those four times demodel neurons. Then take that result and do pointwise multiply it with the activations."}, {"start": 849.98, "end": 866.5, "text": "And then we add another layer norm there. And then we do the down projection to another demodel vector and add it back into the residual stream like normal. The layer norm there turns out to be pretty important. We tried..."}, {"start": 866.5, "end": 884.46, "text": "Without that layer norm, and the model still trains, but you end up with much worse performance. And when you investigate, you end up with, it looks like a lot of the neurons just end up dead. And we don't totally understand why the layer norm fixes that, but empirically it does. If you train without that, a lot of the kind of..."}, {"start": 884.46, "end": 914.29, "text": "The pre-activation values are just deeply in the negative regime. You do a softmax where some of the things are deeply negative and some of them aren't. All of the deeply negative ones go to zero. They get no gradient signal, sort of looks a lot like a dead ReLU neuron. There's no obvious reason that we're aware of why adding layer norm would fix that problem, but we can sort of tell second order stories. But the main thing is that empirically it does. And the kind of motivation for this, as we see in the bottom, is that"}, {"start": 914.29, "end": 924.8, "text": "Sort of want to drive the model towards these sort of sparser representation of features. And if it wants to encode a feature, we want to discourage it from"}, {"start": 924.8, "end": 951.94, "text": "Smearing that feature across a lot of different neurons in a lot of different ways. We want to align these features and computations. One way you can try to do that is push towards sparsity. You only get a few or small number of positive values or non-zero values in your activation function. And Softmax just naturally, architecturally, pretty aggressively forces its input towards sparsity."}, {"start": 951.94, "end": 976.34, "text": "Flushes most of the smaller values very close to zero and then extends up the larger values. We add in the times x. Chris can correct me. That was also Chris's suggestion, but I think there's a couple of ways you can think about it. One is it potentially pushes you a little bit more towards sparsity because it makes this kind of quadratic in some sense."}, {"start": 976.34, "end": 1004.9, "text": "A little bit more dynamic range potentially to allow the neurons to act in because softmax is constrained to 0, 1. The layer norm will then expand that out again. But if you add the times x, it's sort of a little more sensitive to the input. And so potentially you get sort of more dynamic range in these sort of condition statements that the model is doing. And then it also gives you this second path for gradients to flow along in the back propagation. And we were hopeful it might."}, {"start": 1004.9, "end": 1033.06, "text": "Stabilized training, although it turns out the model trains pretty well with or without it. I think two other intuitions for it are, one, gel neurons are basically sigmoid of x times x. And so in some ways, softmax of x times x is sort of a natural generalization of it to multiple dimensions. I think another reason why it might be natural is that if you're really trying to resist small values,"}, {"start": 1033.06, "end": 1052.78, "text": "Sort of multiplying two small numbers together gives you an even smaller number. And so there's sort of a way in which softmax of x times x, if x is very small, is going to drive you even closer to zero and really penalize small values even more strongly, perhaps. Yeah, I think this gets complicated, right? Because if x goes to the negative regime,"}, {"start": 1052.78, "end": 1081.66, "text": "I guess assuming that there are positive numbers, if X goes to the negative regime, softmax of X is essentially zero. And so multiplying it by a large negative number doesn't actually push it more negative. Yeah, it's one of yours. Maybe it goes to not quite zero, and so it ends up a little bit larger in magnitude than it otherwise would be. But it's sort of, yeah. Yeah, I guess it's the case when they're all in the slightly positive regime, and that sort of assumes that X is actually centered at zero, which it might not."}, {"start": 1081.66, "end": 1113.42, "text": "I think, ultimately, we did some experiments with both. They work pretty comparably. We think softmax of x times x ended up being a little bit more interpretable in our experiments. And so that's the main that, plus all of these somewhat hand-wavy motivations are why we're focusing on that one and sharing that one. Yeah, so Nelson, how much more?"}, {"start": 1113.42, "end": 1140.38, "text": "Yeah, so the sort of, as we said, the punchline is that these models seem drastically easier to develop theories of. We shared another video a few months ago where we talked through some of the neurons we'd found in our GELU models. We sort of highlighted a couple of neurons that we thought we kind of understood or could characterize, but mostly highlighted how hard we had found that process. The Softmax X..."}, {"start": 1140.38, "end": 1165.26, "text": "times x neurons at least in our models are just really drastically easier. I've done a bunch of experiments of sort of taking models of identical scale and architecture except for the activation function and randomly sampling 10 or 20 neurons from each and spending about 30 seconds with each neuron. I did this a couple times. I wasn't super rigorous, but the sort of"}, {"start": 1165.26, "end": 1194.03, "text": "Something like 10 to 20% of the Gelu neurons sort of have a clear theory that jumps out of like, ah, yes, it's sort of like pretty clear what information this neuron is trying to compose. And it was something like 70 to 80% of the Softmax X ones have a pretty clear theory. And this is, again, this is like 30 seconds roughly of looking at each neuron. This is a sort of very early first pass when we're just trying to convince ourselves of whether or not there's something here at all."}, {"start": 1194.03, "end": 1221.3, "text": "But the numbers really did jump from 10% to 20% interpretable to 10% to 20% mysterious on a first glance. It's really just sort of completely flipped our experience of looking at these models. And I think when you were describing this to me originally, I'm sort of going to put you on the spot and please feel free to push back. But I think you also said that many of the neurons that you didn't classify for the softmax x times x model as being like."}, {"start": 1221.3, "end": 1247.1, "text": "Having a really clear theory that jumps out in 30 seconds. There were hints of a theory that maybe if you'd looked longer. Yeah, I think that's right. For the quick experiment, I was making a binary judgment on, do I feel like I have a highly plausible theory? But for the ones that failed that test, the softmax x times x1s were much more likely to be like, well,"}, {"start": 1247.1, "end": 1273.33, "text": "I don't have a plausible theory, but there's sort of some obvious pattern here or some obvious structure that I don't fully understand yet, but I bet with some more exploration would kind of yield a pretty crisp story. So it's pretty plausible that 70% or 80% is sort of more of a lower bound on things rather than a... Yeah, that's certainly the optimistic take, and I think very plausible, yeah."}, {"start": 1274.0, "end": 1303.6, "text": "Rough numbers, this sort of, you know, but the sort of the really, we'd been looking at these models and we're like, yeah, these Softmax X1 seem pretty good. And kind of, you know, I just wanted to convince ourselves of like, are we onto something here? Are we just misleading ourselves? And so we sort of did these sort of cheap randomized experiments. And it was just like, I kind of stopped the experiments relatively early on because the difference was just like so sharp that it was just very clear that."}, {"start": 1303.6, "end": 1334.37, "text": "That this is a drastic step forward. Yeah, my experience also just, I didn't do anything quite as carefully controlled as what Nelson was doing. But I just started looking at these models and it was a night and day difference. And I was just extremely, extremely struck by it. So now we can only really speak to our models. And so there's, you know, other people train different models. And yeah, I guess something that we want to caveat is that we, you know, we can't say for sure that other people's models don't have lots of."}, {"start": 1334.37, "end": 1354.38, "text": "I think what we're training is a pretty standard transformer, but sometimes different code bases have different type parameters or something, and maybe this is sensitive in some weird way. Do you want to comment at all on that, Nelson? Yeah."}, {"start": 1355.02, "end": 1381.9, "text": "I always am very uncertain with these results, sort of how aggressively to assume that they generalize outside of our code base or not. Like there are a couple of papers out with various takes of trying to interpret or understand neurons in transformers with varying degrees of success. It seems really hard to say."}, {"start": 1381.9, "end": 1407.5, "text": "To say for sure if this is our architecture, but we're pretty sure we're doing something pretty vanilla and we've had a lot of luck and the difference here is really dramatic. We'd really love to hear from other teams, I think, who have sort of done, tried to do close studies of neurons in a sort of somewhat similar way and whether they've had similar experiences or not. I think if anyone is inspired to replicate this experiment."}, {"start": 1407.5, "end": 1426.35, "text": "That would be really exciting to us to get a side-by-side from another team. These results for us show up. We'll talk later, but the difference between Softmax and Gelu is pretty obvious even in a one-layer transformer. So if someone is interested, I think this is..."}, {"start": 1426.35, "end": 1455.68, "text": "This experiment can be totally reproduced on a single GPU in a couple of days. If someone's excited, you don't need a big science cluster to contribute evidence for whether or not this phenomenon happens in a different code base. One other thing I might add is I think we have some weak evidence just from talking to other groups that at least some other groups are having trouble"}, {"start": 1455.68, "end": 1485.09, "text": "We're figuring out MLP neurons and haven't had a lot of traction with that. And I haven't really heard, none of the groups that I'm in contact with have told me at least that they've had, that they've got traction. So I think that there's, that's sort of not, not dispositive for like what the distribution of successes around groups have been. But I think that, I think that it's at least not unique to us that neurons aren't very integral in your standard model. Yeah."}, {"start": 1485.09, "end": 1512.02, "text": "Okay, yeah, so we've talked about how different it is. Nelson, can you talk about if it affects performance? And also, I guess you were saying that layer norm is really important for that. Yeah, so we talked about how different it is, and I think it's sort of really both of us and other members of our team have all looked at these and sort of, I did my sort of fairly randomized experiments, but everyone on the team sort of independently, I think, you know, and those sort of agreed that these are just night and day different to look at."}, {"start": 1512.02, "end": 1541.41, "text": "The question is, does it affect performance? The short answer is there might be a slight hit from these, but it seems to approximately perform as well as the GelU models. The trends aren't super clear as we change model size. We've done a couple of different sizes. It seems to take a little bit of a hit, but it's pretty small."}, {"start": 1541.41, "end": 1564.72, "text": "High-level takeaways, it's roughly within noise of performing as well as a GelU model, which in some ways is a striking result to me. In some ways, it's unsurprising in that I think most activation functions that are vaguely reasonable people have tried, RelU, RelU squared, other things. There's some difference, but they all roughly work."}, {"start": 1564.72, "end": 1594.72, "text": "The sort of softmax across the MLP dimension feels in some ways very different. You're really aggressively encouraging sparsity. And so maybe it's surprising. I was a little bit surprised that it seems to work about as well, but it seems to work about as well. Okay, so we're going to talk through some different models now just to give some examples of the kinds of really exciting things that we found. And I think we're going to structure this as first talking through a one-layer model."}, {"start": 1594.72, "end": 1624.03, "text": "Because one layer models are sort of very simple and being able to understand neurons in them is actually very exciting. And then we'll talk through some larger models and talk about more interesting neurons. So the neurons we find in the one layer models are going to tend to be simpler. And then the ones that we find in larger models are going to tend to be more complicated. And one thing I think I want to, one kind of, I think, interesting bit of context here is we're going to highlight a lot of cool things we found in these softmax models. For many of these neurons, we had sort of"}, {"start": 1624.03, "end": 1653.25, "text": "We had postulated these neurons and we had gone and searched for them in varying manual and automated fashions in our GELU models and just sort of couldn't find them. We're like, there's sort of some computation. Obviously, the model is doing it because it's sort of fundamental to parsing or understanding the language. And also, it doesn't seem to be, it's sort of some weird superposition or something messier is going on because we like searched for the neurons, we can't find it. And then in these Softmax models,"}, {"start": 1653.25, "end": 1680.82, "text": "These things just fall right out in very straightforward ways as best we can tell. Again, everything we present is based on probably a few minutes of poking around with our interpretability tools. We haven't exhaustively done ablation tests or attribution or counterfactuals or anything to conclusively prove what's going on."}, {"start": 1680.82, "end": 1709.47, "text": "The first pass theory of what's going on is very clear for all of these in a way that really wasn't true in the GelU model. So yeah, let's dive into a concrete example. Yeah, so we studied one-layer attention-only models pretty thoroughly in our first paper. We presented some really nice mathematical analysis of how to think about them and empirical studies of what attention heads are doing."}, {"start": 1709.47, "end": 1737.33, "text": "Then for this talk, we took the one-layer models with the MLP layers, so sort of the next smallest model transformer that you might imagine. And it seems like in the Softmax X models, the individual MLPs are relatively interpretable and pretty easy to come up with theories about. So let's dive into some of the ones we found. Yeah. So Nelson, one thing we're saying is that these neurons implement interpretable rules. And maybe with a concrete example, you could sort of talk about what we mean."}, {"start": 1737.33, "end": 1751.6, "text": "By this idea that they're implementing interpretable rules. Maybe you can talk through what this neuron is doing, and then you can talk through that. Yeah. And again, I think it's always a little subtle to talk about interpretable because it sort of, in some sense, grounds out in this like."}, {"start": 1751.6, "end": 1781.23, "text": "Things that make sense to human researchers, which is definitely not a concept I know how to make super rigorous. But there is definitely a, at least for first pass, there's a very strong, you know it when you see it. Either these things are doing something that you recognize or they're doing something totally alien. And this is a pretty cool neuron that seems to be doing a sort of activating in all caps text and increasing the probability that the next token will be in all caps."}, {"start": 1781.23, "end": 1808.82, "text": "And so that's, I think, a clear rule that makes sense. If you're currently in caps in text that is all caps and maybe has been for the last couple of tokens, then probably the next token is also in all caps. That's a rule. And it appears, again, we haven't traced out all the circuits. This isn't rigorous. But it sure appears that that's exactly what this neuron is doing."}, {"start": 1808.85, "end": 1833.71, "text": "And what we're showing on the right over here is we're showing sort of the direct contribution of this neuron to the residual stream. You can think of this if you're familiar with the Logit lens. It's sort of the Logit lens applied only to that neuron. If you just, more mechanically, this neuron has some..."}, {"start": 1833.71, "end": 1853.71, "text": "Output vector in the down projection or the output projection from the MLP matrix. We just take that vector, multiply it through the unembedding, and that gives us a vector over the vocabulary that says, to what extent does this neuron, when it fires, increase the logit for every given token?"}, {"start": 1853.71, "end": 1877.6, "text": "So on the left here, we've looked at some of the data set examples, examples from our data set where this neuron fired most strongly, where its activation was the most positive. And we can see that it seems to fire in places where we are writing in all caps."}, {"start": 1877.6, "end": 1896.26, "text": "And there's like some variation. We don't totally understand, but the high level story is pretty clear. It's firing in all caps. And then if we run it through the unembedding on the right, and we look at the tokens that it most increases the logic of, they are all tokens that are in all caps."}, {"start": 1896.26, "end": 1918.1, "text": "Potentially there's some deeper structure there, but there's a pretty clear first pass rule. And the tokens that it decreases the probability of are all in lowercase. And so there's a clear sense in which mechanistically the thing that this neuron is doing is it's increasing the probability of all caps output and decreasing the probability of lowercase output. And we haven't verified, but..."}, {"start": 1918.1, "end": 1945.17, "text": "This is a one-layer model, so it would be pretty easy to use the techniques from our first paper and trace through in the other direction. And that's work that we hope to do probably for some of these neurons in the future. I think we would bet with high confidence that we would just see that this neuron is looking for current token, all caps, last couple tokens, also all caps in some combination. Can you comment at all on, I think there's some ways in which"}, {"start": 1945.17, "end": 1971.39, "text": "Neurons in a one-layer MLP model are maybe especially integral and especially easy to understand as rules like this. Yeah. So if you looked at our first paper, we completely factored these one-layer models. The attention heads run completely in parallel. And in those models, they feed into the"}, {"start": 1971.39, "end": 1998.83, "text": "Directly into the unembedding. If you add in the MLP layer, you get the exact same analysis for the MLP neurons. The input to the MLP neuron is the sum of the direct path for whatever the actual token value is, plus for each head, the output of that head. And each head factors into this skip bigram detection. It looks at the current token, looks at"}, {"start": 1998.83, "end": 2026.22, "text": "The current token chooses which token to attend to, or which token or tokens to attend to, and then the tokens that are attended to produce value vectors and sort of multiply through the OV circuit. And so you can multiply out these, for each of these neurons, these sort of tables of, well, for the current position,"}, {"start": 2026.22, "end": 2050.3, "text": "The first order effect of the current position is the contribution of the current position to the input vector for the neuron is really just that value is directly present. And then it also mediates the attention matrix, but we sort of abstract out the attention matrix. And usually we find those are pretty interpretable. We can sort of describe what's going on there."}, {"start": 2050.3, "end": 2077.18, "text": "For every token that we attend to, there's some value that that token adds to the input value for the neuron. So we can sort of look at this table of for any current token, here's what the input value sums up. And then for every head, for every attended to token, here's the value. And then we just sum those all up, and that gives us the input to the neuron."}, {"start": 2077.18, "end": 2101.73, "text": "I guess there's also something special about neurons in the last layer of a transformer are easier to interpret. And one-layer models, all neurons are, by definition, they're both the first layer and the last layer. And so they have these special properties. We talked about these logit values on the right, but the special thing about the last layer is that these logit values are the only thing these neurons are doing."}, {"start": 2101.73, "end": 2111.04, "text": "In the general case, a neuron is writing a value into the residual stream. That value will both still be present at the end of the model for the unembedding."}, {"start": 2111.04, "end": 2136.21, "text": "It might have been modified or deleted, but sort of conceptually the unembedding is this big sum that will always and forever have that term in it. But also that value can be read by later attention layers or MLP layers. And kind of in general, we expect that most neurons aren't writing directly to the embedding as the main thing they're doing. They're producing values that are consumed by later neurons, later attention heads."}, {"start": 2136.21, "end": 2165.78, "text": "The last layer, the only thing they're doing is writing into the residual stream, into the unembedding. And so this table on the right expanded out over the entire vocab is a complete description, in some sense, of the effect of this neuron. There's no subtlety or mystery there. It's a large table, but it's just a table for every token. How much do you increase or decrease the logit for that token?"}, {"start": 2166.8, "end": 2191.63, "text": "Cool. Okay, let's look at another such neuron. Can you comment on this one? Yeah, and this one I think is in some ways kind of very spiritually similar. The last one was talking about all caps text. This one is identifying we are in base 64. Base 64, you're likely familiar, is sort of a common encoding for encoding arbitrary binary data into..."}, {"start": 2191.63, "end": 2210.67, "text": "64 character alphabet, but sort of most of which are the letters and numbers and then plus and slash. And if we look at the data set examples, empirically, it mostly fires in snippets of the text that are the base 64 encoding of some value."}, {"start": 2210.67, "end": 2240.24, "text": "And if we look at on the right, we look at the effect on the residual stream, it is boosting all of these kind of short, random seeming tokens that are kind of likely to show up in the way that Base64 ends up tokenized and then decreasing the probability of all of these sort of nice, normal, sensible English words that are very improbable to find inside a sequence of encoded binary data. And then I guess."}, {"start": 2240.24, "end": 2270.02, "text": "There's sort of one exception that's kind of surprising, which is mem is sort of being increased by 0.14. But I guess the hypothesis there would be these models are kind of small and sometimes just weird. They're trying to compress lots of things into a low dimensional space. Sometimes it's just weird products between things. Yeah, that's our best guess. Whenever I see those things, I feel like I always assign some probability math to there's something subtle going on that we don't understand yet."}, {"start": 2270.02, "end": 2294.27, "text": "I think there's also a very good chance that just you know these sort of doing very right you know this this vocab is of size something like 60,000 in our models and this model has a d model of a couple hundred and so you're just you're compressing very large spaces into into low dimensional spaces and occasionally they'll kind of be errors but but in net they'll they'll aggregate out to sort of"}, {"start": 2294.27, "end": 2321.26, "text": "Small probability mass of these actual errors is the first guess for things like that. This one, I think, is a little more specific. The last ones were a broad flavor or context of the text we're in. This seems to be a number followed by a comma is where we fire."}, {"start": 2321.26, "end": 2350.77, "text": "And when we fire there, we mostly predict that we're going to see a three-digit number because if we're in a number and we have commas, we separate, in English, we separate chunks of three digits. So if we have a number and a comma, it's pretty likely that we're in a larger number. We're also like, it's kind of cute if you look, the sort of nice round numbers, like 0, 0, 0, and then 500, 600, 700 are more probable. It's sort of..."}, {"start": 2350.77, "end": 2375.14, "text": "Obvious in hindsight that the triplet 000 is probably drastically more common than any other triplet in text, and this neuron knows that rule to some extent. But also, I bet if we zoom down, essentially every other three-digit number appears somewhere near the top of that list. I bet that the 000 is especially common when you see a number and you have a comment, because it's often..."}, {"start": 2375.14, "end": 2410.3, "text": "You're expressing a dollar amount or something like this. And you can see our first list here even has an example of that. All of these numbers are rounded numbers of hundreds or thousands. And again, we have a couple examples in the weights that make less sense, or at least don't fit quite as cleanly into this theory. But it seems like most of them do. OK, so those were a few examples from small models. And hopefully, that demonstrates what we mean by small models."}, {"start": 2410.3, "end": 2438.45, "text": "Not only are the neurons interpretable, but they often correspond to these sort of clean rules that sort of demonstrate, like, in this behavior, go and affect your output predictions in this way. But in larger models, we'll see that there's more complex things going on. So that'll be kind of exciting to dive into. And one theory that we've sort of developed is that often early layers are decoding tokens. So they're sort of turning tokens, which might"}, {"start": 2438.45, "end": 2468.13, "text": "Sort of split things in weird ways into semantic units. And then later layers can re-encode tokens so they can turn sort of more semantic units back into tokens. And so some really concrete things that we'll see on the early layer side are recognizing things that are, you know, token X is in language Y. So we'll see a bunch of things where like, you know, some token occurs in a lot of different languages and means different things to those languages. So it gets different, there's different neurons for them."}, {"start": 2468.13, "end": 2497.65, "text": "And we'll also see tokens that correspond to detecting multi-token words, or I'm sorry, neurons that correspond to detecting multi-token words and neurons that correspond to detecting compound words or proper nouns. Yeah, so let's actually look at those. So I think the first one we can talk about is these tokens and language neurons. And I think it's interesting that, I mean, going all the way back to 2019, there was..."}, {"start": 2497.65, "end": 2526.86, "text": "And this nice paper from Conan et al, where they they were talking about they just looked at a single token and how it was sort of represented a transformer. And they found that even though, you know, a token like like DAI, D-I-E, initially has the same embedding. After a few layers, you start to go and see the different interpretations of it. And especially this case where it's in a different language. So it's in German and where it's acting as an article gets separated from the English ones."}, {"start": 2526.99, "end": 2556.96, "text": "And they also saw this sort of split between death and playing with die. And so we actually see something really similar. Nelson, do you want to comment on this? Yeah, and I think one cool thing is sort of we replicated that previous kind of experiment of you do a UMAP of some specific token in different contexts. And sort of our GELU models behave kind of similarly over the first couple layers. Those tokens kind of move out in space."}, {"start": 2556.96, "end": 2584.13, "text": "But we tried to figure out mechanistically, how is it doing that? And we got a little bit of traction, but it seemed to be a pretty messy story. And then in the softmax models, the first order story, at least, seems the pretty good reason to believe that there's just one neuron for each language comma sense pair that detects a token comma sense or token comma language pair that"}, {"start": 2584.13, "end": 2609.14, "text": "You know, fires on that token when we're in the appropriate context and writes in some value that serves to separate those in activation space. And so losing that same example of the token die or D, we find here three different neurons that all fire on that token, but only fire in the context of an appropriate language. We were able to find..."}, {"start": 2609.14, "end": 2634.1, "text": "With sort of a little bit of searching, you know, Afrikaans, German, and Dutch all had these die or D tokens that fired on that token in the respective language. And so it's sort of a behavior that, you know, we know is there and sort of, I think there's a good reason to believe it, that any transformer that's exposed to these languages has this behavior. But in the Softmax ones, it's sort of the behavior."}, {"start": 2634.1, "end": 2661.18, "text": "It's just very easy, at least to first order, to understand how it's happening. It kind of factors very cleanly. Yeah. And I think a thing that's interesting here, well, maybe there's a few things that are interesting. One is we didn't find an English version of this. And so it seems like often in these cases, the default, and I call this default just because I think our data set has a lot more English in it than it does have any of the languages."}, {"start": 2661.18, "end": 2686.14, "text": "That the most common case doesn't get a neuron and then sort of cases that are less common get neurons to sort of differentiate them from the most common case or something like this. And so, yeah, we didn't see that. I think another thing that's interesting here is sometimes in smaller models we'll see neurons that are like tokens which might occur in English but are actually occurring in Dutch or something like this."}, {"start": 2686.14, "end": 2716.06, "text": "And here we're seeing something kind of different. I think even in the small softmax x times x models, we saw just really clean versions of that. And now we're seeing these individual token in language y. And in order to do that, you need to get to a slightly larger scale of model. So yeah, I thought it was pretty interesting that we were finding these kinds of neurons. It's not just natural language, though, where this happens. So it also happens with a less than sign. Nelson, do you want to comment on these?"}, {"start": 2716.06, "end": 2745.7, "text": "Yeah, I mean, I think sort of it points to language is a sort of very natural or intuitive sort of thing for us to reason about of, you know, yeah, like kind of, you know, the word D or the D-I-E is sort of obviously a different word in different languages. But sort of the phenomenon here is some much broader context disambiguation thing of sort of, you know, programming or technical terms or punctuation sort of also mean different things in different contexts."}, {"start": 2745.7, "end": 2774.9, "text": "It's likely a sort of very similar phenomenon to language disambiguation to separate those. So we see this here with less than signs being used in XML versus in IRC versus in Python. And you actually see this for lots of things. If means something very different in Python than in English. Colons in Python versus colons in a timestamp mean different things. So there's lots of cases where."}, {"start": 2774.9, "end": 2795.18, "text": "Where other tokens like this mean different things in different languages. Yeah. And I think it's sort of not really relevant to our point, but I find these studies always kind of fun because of what they teach you about what's in your data set. Because these sort of data sets are enormous gigabytes or terabytes and generated by some very automated processes."}, {"start": 2795.18, "end": 2824.56, "text": "So if any of these are questions you could have asked, but sort of the, you know, when you start looking, doing this kind of interoperability and looking at individual neurons, you're like, oh, I guess there's a lot of IRC transcripts in our model because there's sort of self-evidently some amount of capacity sort of specifically devoted to parsing or processing those. And is that good? Is that bad? I don't really know, but it's kind of fun to stumble on those and sort of get a better feel for what sort of start to get a better feel for the kinds of things that's in your data set."}, {"start": 2824.56, "end": 2858.35, "text": "Your model has learned. Another fun example is question mark in C, where it's used as part of the ternary operator. Claims in the context of patents. Or three in base 64. So three means different things. If it's in a decimal number or hexadecimal number or in base 64, and you can have different neurons for those. Cool. So the next sort of family of early layer neurons that we found."}, {"start": 2858.35, "end": 2881.62, "text": "We're what you might call multi-token neurons that correspond to multi-token words. Yeah, maybe it's worth a bit of context here, although maybe it's clear to most people. We use a byte-wise BPE tokenization pretty similar to the GPT-2 one, although not exactly the same one."}, {"start": 2881.62, "end": 2909.01, "text": "Individual words, especially longer, rarer words, get split into tokens in idiosyncratic ways sometimes. And I would expect the details to vary if you're using a drastically different tokenization, although I expect the phenomenon is pretty fundamental to the constraint of tokenizing text into discrete tokens."}, {"start": 2909.01, "end": 2936.27, "text": "That's the context for some context for the specific tokenization we're using, which deeply informs the details of all of these examples. So for example, trending gets split into two tokens, trend and ing. And so there's a neuron that fires on trending as a whole. So presumably it's looking at the previous token, checking that it's trend, and the present token that it's ing, and then fires. And it's able to"}, {"start": 2936.27, "end": 2951.5, "text": "And like one way you can think about it is it's almost functionally making the vocabulary larger by going and creating special tokens that are, or special neurons that sort of correspond to cases where the tokens correspond, where there's a particular sequence of tokens."}, {"start": 2951.5, "end": 2974.56, "text": "Yeah, and I think it's also, you can think of this as in some ways actually very similar to the language disambiguation problem of you have this token ING that kind of that, you know, the embedding just has one value for ING, but like what ING means is actually sort of entirely determined by the context that it's in. In this case, the context is the sort of..."}, {"start": 2974.56, "end": 2999.17, "text": "Previous one or two tokens and in the language, it's sort of the broader language context, but it's sort of, it's a very similar problem in some sense of, you know, the tokenization and the embedding means that we just get one vector, but actually we want a lot of different meanings or shades. And so we stitched together some circuits that end up in these neurons to peel those apart and sort of presumably."}, {"start": 2999.17, "end": 3028.72, "text": "Every other common word that has a gerund form ending in ING that gets split into multiple tokens, most of those also have neurons, I would imagine. We haven't done an exhaustive study, but... I did a search, though, where I looked for first layer neurons that fire primarily on ING, and it turns out there's like dozens and dozens and dozens of such neurons. So there are lots of words that end in ING that the model wants to dedicate neurons to, and indeed it does."}, {"start": 3028.72, "end": 3056.98, "text": "And again, it's this token comma sense or context or meaning. Again, for all of the specifically important cases in that table, we find a neuron. And that's really neat. One thing I'm realizing might actually be surprising to people is this idea that a first layer neuron that can only use attention heads can determine the language. Maybe it makes sense that it can check where the previous neuron is. But maybe could you comment on how it might be able to determine what language it's in?"}, {"start": 3059.12, "end": 3087.68, "text": "Yeah, I mean, we haven't studied this for sure, but I sort of seems, you know, I would guess that you probably use a sort of relatively diffuse attention pattern. You know, you arrange to attend to the last, like, couple of words or, you know, last sort of several words. It's sort of relatively easy to do in most positional encoding schemes. And then, like, sort of likely..."}, {"start": 3087.68, "end": 3113.55, "text": "Every token has sort of a subspace in the embedding that indicates which language it's in. Or maybe just sort of there's a like, this token frequently appears in Afrikaans direction. And your OV circuit for that attention head moves that into some subspace in the present token. And then you can just look at that."}, {"start": 3113.55, "end": 3143.41, "text": "You know, combined with, you know, maybe you also have some weight for the current token for sort of is this even plausibly Afrikaans is encoded in some direction. And you sort of sum the is the current token even plausibly Afrikaans and the do the last several tokens kind of very likely Afrikaans, which is in some other direction in your kind of current position, courtesy of that attention head. You sum those together and then, you know, if that value is large."}, {"start": 3143.41, "end": 3164.56, "text": "Probably the sort of current token is also Afrikaans. So I think another interesting one, in some ways the second one is even more interesting because it's got different tokenizations. Yeah, I really, and we've seen this for some other words as well. It's a very common thing. It's a very common pattern."}, {"start": 3164.75, "end": 3178.67, "text": "Often certain words get tokenized subtly differently depending on context. So you can see the first one, there's the capital C and our vocab doesn't have a single token for civilian with a capital C."}, {"start": 3178.67, "end": 3207.97, "text": "The second one, there's this parenthesis in a similar way to the GPT-2. We glue spaces onto the start of tokens. So if there's a space, it gets attached to the next word for the purposes of tokenization. And so space civilian and then civilian without a space end up tokenized differently. And so all of these are very clearly the same word, but they end up as a subtly different sequence of tokens."}, {"start": 3207.97, "end": 3227.73, "text": "And presumably for a large enough model, it doesn't want to reason about these as separate tokens. It wants to, at some point early on in the model, kind of have the civilian direction or the kind of the word civilian or the concept of civilian kind of unified across all of these. And we see here."}, {"start": 3227.73, "end": 3259.55, "text": "This one MLP that sort of implements at least substantially that process of recognizing all of these different tokenizations. And then it's going to write the same output direction because the neuron only has one output direction that it writes for all of these. And so sort of after this point, presumably the model can use that direction to abstract over all of these tokenizations that got split due to sort of annoying details of the tokenizer."}, {"start": 3259.55, "end": 3288.62, "text": "So on the flip side, at the end of the model, it's going to need to do the opposite. And this is a really interesting thing that we've been observing, where there's often a duality between the first layer of the model and the last layers of the model. Yeah. Yeah, because the autoregressive objective and the tokenization, the model has to, at the end of the day, predict the next token. But presumably,"}, {"start": 3288.62, "end": 3315.46, "text": "Assuming these models are thinking about the world in a semantic sense that is at all recognizable to us, we wouldn't expect that they're manipulating in token space for all of their logical reasoning or intermediate computation. It's much more likely, we imagine, that they're operating in some much more semantic latent space since the last couple of layers are like, all right, well, I'm..."}, {"start": 3315.46, "end": 3340.62, "text": "Thinking this concept or this word should come next. I guess now I have to decode that into the constituent tokens. And we see evidence of some specific examples of some version of that process where the word nappies here and the word straggler or maybe stragglers are split or rare enough that they don't get their own tokens."}, {"start": 3340.62, "end": 3369.18, "text": "And we see specific tokens that sort of fire on when you've started saying that word on ensuring that you accurately finish saying that word. And these don't fire. There are kind of other words that start with ST that this neuron doesn't fire for. It's sort of specifically for the context of I've started saying this particular word and I need to accurately say the rest of it."}, {"start": 3369.18, "end": 3399.04, "text": "One way that I might think about it is it almost creates conditional logits. So suppose that you want to say something like a diaper. You could say diaper. You could say nappies. You could say probably there's other words that are in that space. But because nappies is tokenized as multiple tokens, you first need to say n. And then you can go and say continue with nappies. And so if you have a neuron where you make it so that it can only fire on the token n,"}, {"start": 3399.04, "end": 3434.83, "text": "And then it continues to go and finish nappies. Or if you have a token that only fires on ST and then can continue stragglers, it sort of gives you functionally a new option in your output space that only exists when the conditions for it are sort of available to you. And that seems very powerful. Yeah, okay. So we also see some neurons related to multi-token numbers. Yeah, and I think..."}, {"start": 3435.47, "end": 3459.9, "text": "Numbers, I think, are kind of in some ways a particular victim of our tokenization scheme. They sort of often end up parsed in ways that kind of appear very arbitrary. And we see various neurons that are sort of understanding or reconstituting those in some way. This first one here sort of fires on"}, {"start": 3459.9, "end": 3488.46, "text": "The sort of 7x or 8x token when they're immediately following a 4. So they're sort of parsing this specific range of 20 numbers between 470 and 489. This neuron identifies those and fires them in some way. So it maybe starts to give us some hint of how this model is representing numbers or quantities internally."}, {"start": 3488.46, "end": 3517.73, "text": "It wants that roughly 20-digit range to get some signal associated with them. The second one, it's a little bit less clear semantically what it might mean, but mechanically what it's doing is when there's a three-digit token following a one-digit token, then it lights up. It's not yet intuitive to me why you want that or exactly why it's useful, but it's pretty clear empirically what it's doing."}, {"start": 3517.73, "end": 3549.97, "text": "Trace or explore it downstream. I guess one theory you might have for why that would be useful is depending on how many digits exist in the previous token and how many digits exist in the present token, you need to do some work to parse. If you want to figure out am I the third digit in or what value is the third digit in from the left, then splitting those cases out is potentially quite useful. So that might be one hypothesis, but we don't know. On the flip side,"}, {"start": 3549.97, "end": 3582.86, "text": "This is maybe a little less compellingly like a multi-token thing, but there are some really interesting neurons about how right at the end for dealing with outputs or dealing with predicting the, yeah, going in and controlling the range of tokens you predict next. Yeah, and the sort of first one like really wants the next digit to be a three, it seems, but it's sort of..."}, {"start": 3582.86, "end": 3610.9, "text": "If I had to guess, the first one is not necessarily what we'd expect to find in arithmetic cases where we're specifically doing some computation. But maybe there's some other context where we're pretty sure the next digit is the three. And maybe also there's some other digits after that. It's hard to say without future study."}, {"start": 3610.9, "end": 3641.95, "text": "I guess one way I might think about this is for a lot of purposes, including potentially arithmetic, you might want to be able to predict different tokens or different digits of a number differently. So you might want to be able to like, you know, you do your arithmetic and you calculate what the first digit should be and what the second digit should be and so on. You might want to be able to go and predict that."}, {"start": 3641.95, "end": 3662.19, "text": "It's a little awkward because you're sort of predicting left to right and it's sort of not like, yeah, it's not clear that's the way that you would want to do things, but it might be useful for something like that. And on the flip side, our next neuron actually, instead of predicting the first digits, seems to be more about predicting the last digits."}, {"start": 3663.28, "end": 3687.07, "text": "Yeah, and we actually, we didn't kind of show anything about this, but we found this neuron most frequently showing up in running lists where you're counting or enumerating some items. And so it sort of seems like potentially this is sort of counting our position mod 100, that if we're sort of..."}, {"start": 3687.07, "end": 3707.94, "text": "Counting things and we're somewhere in the hundreds or thousands. We're about to hit the 13th item in particular, and so it'll show up. And I think the fact that the negative examples are all, it really doesn't want 12 makes some sense. If you're counting up and you just said 12, you're not going to say 12 again."}, {"start": 3707.94, "end": 3737.38, "text": "Potentially, sometimes you'll say 14. And in fact, 13 is probably relatively common to skip as a number because it's sort of unlucky. And if we're highlighting the stories of a building, maybe skipping from 12 to 14 is totally reasonable. And so this neuron is less confident that we aren't going to say 14. But if our sense is right that it's sort of appearing when we're counting up, it makes sense that it probably mostly fires when we just said 12."}, {"start": 3737.38, "end": 3768.42, "text": "It thinks we're going to say 13, and also it thinks it's really unlikely that we're going to repeat 12. Was there anything else you wanted to say on that? No, let's just move on. So I think another interesting category that I think isn't really that different from the multi-token words is compound words and abstraction. And we found some fun examples, including a machine learning neuron, which I was pleased to find. Nelson, I wonder..."}, {"start": 3768.42, "end": 3792.34, "text": "Yeah, I don't know. Maybe you have some comments on both on how this really isn't very different from the more general phenomenon of multi-token words. Yeah, these seem, I think, again, a very similar phenomenon or plausibly even justified in calling them the same phenomenon as parsing multi-token words."}, {"start": 3792.34, "end": 3820.75, "text": "There's maybe some question of shading or degree of maybe the word die in English and the word deep in German perhaps are more different than the word learning versus the phrase machine learning. But there's a very similar thing of machine learning is really a single concept or unit in some sense."}, {"start": 3820.75, "end": 3849.65, "text": "Some languages like German are fond of forming these compounds by just gluing the words together without a space. And machine learning as one unit is in some ways not that different than trending as just another entry in the vocabulary. And the fact that there's a space there is a weird quirk of our orthographic conventions rather than it is something fundamental about what's going on here."}, {"start": 3851.09, "end": 3879.81, "text": "Yeah, that seems right. Seems right to me. So yeah, we also have forum policy, lemon juice, animal welfare. There's lots of neurons like this. We just picked a couple of random fun ones. All of these are pretty common units that commonly show up together, have some semantic meaning that is somewhat different or, at a minimum, much more specific than just the final word or just the individual word. And the model is picking those up and"}, {"start": 3879.81, "end": 3903.95, "text": "Marking that down somewhere in the residual stream for the rest of the model to reason about. Yeah, so on the flip side, again, we're going to continue seeing this duality or continuing to see this duality between the first layer and the last layer. We're seeing some neurons that are about going and outputting certain sequences of words that sort of maybe mean something together."}, {"start": 3905.78, "end": 3933.94, "text": "Yeah, I think the last one here is maybe the sort of simplest of United Nations is a sort of very clear entity. Those words have a specific meaning together that is, you know, different than kind of just the composition of them separately. And then there's sort of very similar, I think, to our earlier nappies or stragglers example. We have this neuron that exists to sort of, you know, help say the phrase United Nations as a unit."}, {"start": 3933.94, "end": 3962.61, "text": "Although it wants to increase the probability. It increases the probability that the next word is national more. Yeah, and that United National does not, to me, seem like a super likely thing. So maybe there's some nuance we don't yet understand yet. Yeah, I looked at the dataset examples as well. And empirically, when it fires nations is the next word. So I was wondering if maybe it's just a weird thing about the embedding and unembeddings. Or it could be that the bigram statistics are going and correcting it. Yeah."}, {"start": 3963.06, "end": 3988.99, "text": "Yeah, and I think kind of one thing that is always important to keep in mind is that these numbers are kind of added into the logits along with kind of all of whatever the rest of the model is doing. And so these numbers aren't really, like in some sense, aren't really, sort of aren't like absolutely meaningful. And kind of especially because the logits then get run through a softmax."}, {"start": 3988.99, "end": 4015.87, "text": "If it is the case that the model has a prior that the word national is incredibly unlikely here, then you can actually add a pretty large value into it and after the softmax have very little effect. And so these numbers are very informative, or at least often very informative, but they're hard to interpret with too much nuance outside of context."}, {"start": 4015.87, "end": 4043.79, "text": "Yeah, and then the first two examples here are helping produce these common phrases, per cent or lost count, lost track. There's a little bit of nuance that we, I think, don't fully understand yet of this helping to produce both parts of the phrase to some degree. The first one is clearly involved in producing the lost count or lost track."}, {"start": 4043.79, "end": 4073.09, "text": "But it also helps generate both the loss and the count or the track to some extent, which are sort of clearly part of the same phrase. I think I would not have guessed that you'd have kind of one neuron in the very last layer contributing to both of those logits. But empirically, it's there. It's doing that. There's probably something interesting to study about what's going on there. It sort of, again, points to"}, {"start": 4073.09, "end": 4104.88, "text": "We're not claiming to fully understand these neurons at all, but these are all ones that are sort of, some large part of what they're doing is pretty clear, and there's sort of these strong footholds for digging in on them to a degree that was unusual in the GelU models. Yeah, I think this idea of having footholds or having places we can get traction into exploring models is really useful. You might not be able to understand absolutely everything at first, but having a path,"}, {"start": 4104.88, "end": 4124.24, "text": "Yeah, and I think having a theory that is approximately right or informative, even if it turns out to be wrong, is often a great point to start."}, {"start": 4124.24, "end": 4155.78, "text": "Expanding out circuits and looking at where they're coming from and then sort of, you know, develop the rest of the theory and the rest of the nuance. Whereas if you sort of start from a place of total confusion, it's kind of much less clear where to go from there. Okay, so next we're going to talk about some neurons that are more, a little bit deeper into the model still. I think a lot of these are more on the earlier side, but they're not as much the first layers as, you know, early mid layers or something like this. Or even mid layers."}, {"start": 4155.78, "end": 4184.35, "text": "So this seems to be an ANSI escape sequence neuron, and Nelson corrected me on even what it was called. And I know about these from when I'm writing my bash prompt and I need to go and insert them, but I don't know anything else about them. So maybe I'll let Nelson comment on what this neuron is doing. Yeah, I mean, these are the sequences that are used to tell your terminal to render text in a..."}, {"start": 4184.35, "end": 4210.14, "text": "Different color or bold or this sort of also can be used to move the cursor around and do other manipulation. I think probably many viewers like Chris will have ever encountered them because they were Googling how to make your prompt read or something and you just sort of copy pasted some sequences. This sort of surprisingly rich and complex set of things that they're capable of doing."}, {"start": 4210.14, "end": 4239.95, "text": "Most of these look like they're just populating colors. I think the three there probably is set foreground color from memory, and the next digit is the color. But if I'm wrong, please don't at me. I'm going from memory. So I thought it was just really cute that this is like, it's a very specific thing. It's like a thing that definitely, I feel like there's a way in which my brain has looked at these enough times."}, {"start": 4239.95, "end": 4270.93, "text": "And even though not that often, but they like immediately jump out to me as, oh, they're those things that I stick in my bash prompt. And I feel like it's kind of, I don't know, it's kind of interesting that the model also has a neuron for that. Yeah, it's sort of clear evidence of sort of some context or shade of meaning or kind of knowledge that is present in the model, at least to some degree. I think we can conclude this, but there's a similar neuron."}, {"start": 4270.93, "end": 4301.89, "text": "When you're doing a Python print statement or formatting things or in C you're doing a print statement and you have percent D or things like this. I don't even know what those are called, but these like percent something to go in and insert something. There's also a neuron that does those and so other similar types of things. Another one that goes into text, the end of an ASCII or table. If you're using"}, {"start": 4301.89, "end": 4332.43, "text": "Yeah, ASCII art to go and draw a table. You can, yeah, there's a neuron that fires at the end. Oh, I thought this was a really cool one. So this one fires on numbers, but it doesn't fire on all numbers. It fires on numbers and maybe, I guess, also some other words, but numbers that indicate, that context indicates are a number of people. So the main banquet room can be set up to seat 150 guests."}, {"start": 4332.43, "end": 4362.08, "text": "Family suites that can sleep as many as six. And there's no, like, it's not that there's any, it's just implied that those are, you know, I guess it's like, if you talk about seating or sleep or things like this, it's implying accommodating. It's implying that a number is going to be referring to a number of people. Yeah, this is a really, I think a really cool one and sort of,"}, {"start": 4362.08, "end": 4389.52, "text": "There's sort of clearly some somewhat non-trivial kind of reasoning or context clue here that sort of, if you had posed the question, I probably would have said, yeah, kind of obviously a large transformer sort of has that capability in some way just sort of based on its output, but that we sort of start to get a handle on how it's doing this sort of relatively nuanced thing of..."}, {"start": 4389.52, "end": 4418.06, "text": "Tracking counts of people in some context. And you see other interesting versions of this. So you see, for instance, like neurons that fire for numbers when they're the score in a game. And again, it's not like you can't just tell from the previous word. You sort of need to know from context that, you know, they're up three or something like this means that three is a score in some sport or something like this. So there's lots of things that are kind of at the sliver."}, {"start": 4418.54, "end": 4441.73, "text": "Yeah, and I think to sort of one thing, sort of just to touch on the stuff that we said earlier about sort of just having footholds on these models, I think sort of having a neuron like this opens the possibility of, you know, that neuron is ultimately reading from one direction in residual space and writing to one direction. And so you can imagine sort of."}, {"start": 4441.73, "end": 4469.82, "text": "This sort of gives you a foothold of like at this particular point in the transformer, those directions in residual space mean something like, you know, we're talking about people or numbers of people and you could imagine tracing those forward and seeing what other neurons and attention heads interact with those. And, you know, kind of actually doing this in a large model is non-trivial, but it kind of gives you a very concrete foothold to start to..."}, {"start": 4469.82, "end": 4498.96, "text": "to explore outward and try to find the fuller circuits that are doing these bits of abstract reasoning. And so even if we don't have the complete story or our story isn't right or isn't exactly right or is later falsified, having these footholds to start to trace out feel really exciting for starting to stitch together stories of what's going on in broader parts of these models. Yeah, that feels very true."}, {"start": 4500.78, "end": 4529.71, "text": "Another interesting one is whitespace when you had indentation. So I think this is actually a little subtle, but it's like, you know, here we indent and it fires. Then you unindent, then it doesn't fire. Then you indent again, then it does fire. You have a new function, you indent, it fires. You indent again in an if statement, you fire. You unindent for an else, you don't fire. You indent again, you indent again. Then you unindent, then you indent again. Then here..."}, {"start": 4529.71, "end": 4559.47, "text": "The whitespace is increased, but it's not really indenting, at least not for Python's purposes. It's just going in for this function, the arguments are indented, so we don't do that. Then I guess this is still the same level of indentation, so maybe it fires there. Then try, you go and indent, you fire, you unindent for accept, then you indent again. Then again, these are a function statement or a function call, so you don't really fire for them. So it sort of seems to be doing something actually quite sophisticated about the..."}, {"start": 4559.6, "end": 4568.38, "text": "Yeah, about helping parse indentation or something like this or parse what's nested under what in Python. Yeah, I'm kind of..."}, {"start": 4568.38, "end": 4592.53, "text": "Pretty sure from memory that the actual the Python parser deals with indentation by sort of injecting virtual tokens into its tokenization stream of sort of there is a token for the indentation increased and the indentation decreased and then sort of once you have those like parsing you know sort of you can use a kind of normal parser and treat those the same way that you would like curly braces in C."}, {"start": 4592.53, "end": 4621.33, "text": "And so sort of this, again, super speculative early evidence, but sort of hints at the model potentially doing a sort of surprisingly similar thing to reasoning about scope in Python. That would be really delightful if it turned out they were analogous. Who knows? It would be delightful if true. Here's another one that's like a number."}, {"start": 4621.33, "end": 4651.02, "text": "A particular interpretation of a number. So in a date stamp, you know, this is the date portion, and it just always fires on the date token and date stamps. DNA sequences. Often there's like this particular format, way that you format sections, where like if you have a list of things and you want to, or like a title. So if you have a list of things and you're giving like a little title to each item in the list, you might go and like do, you know, number, then like title, and then period. And so you have the"}, {"start": 4651.02, "end": 4684.32, "text": "Okay, so."}, {"start": 4684.32, "end": 4712.03, "text": "There was this paper that I actually was lucky to work on with a couple of colleagues on studying multimodal models, in particular CLIP. And the lead author was Gabriel Goh. And we found some really interesting, highly abstract neurons. And something I've been wondering since really like one of the first experiments I did when I started poking around in language models was whether we could find similar neurons because this was sort of a vision model."}, {"start": 4712.03, "end": 4738.14, "text": "That was trying to align itself with a language model in some sense. And it developed these really highly abstract neurons that I thought perhaps would exist in language models. And previously, I just wasn't able to find them despite significant effort. So we found neurons that correspond to various famous people and also neurons that correspond to various geographic regions. And even though we couldn't find that in the previous models,"}, {"start": 4738.14, "end": 4765.09, "text": "Things like this have been popping out really quickly and really easily just poking around in our models now. So we found neurons related to various famous people, Joe Biden, Hillary Clinton, Donald Trump, Martin Luther King, just lots of things, various US presidents, various famous figures, various world leaders and politicians. And you pick it, there seem to be lots of neurons."}, {"start": 4765.09, "end": 4792.91, "text": "Yeah, I think one thing I want to highlight here is in some ways these are sort of similar to just these sort of multi-token words, right? Joe Biden, you can think of as sort of a multi-token word meaning Joe Biden. But if you look at the third example here, for instance, sort of very clear from context that we're talking about Donald Trump, but the word Donald doesn't actually appear here. So I think at least some of these are..."}, {"start": 4792.91, "end": 4822.13, "text": "Potentially both flavors exist of ones that are sort of just doing, you know, gluing together a sequence of tokens. But some of them are also, I think, sort of a bit more contextual of parsing context to a somewhat more subtle degree to associate that, you know, we're talking about this specific individual, even though we didn't say their full name in the usual order. Yeah, and we've actually seen a lot of these be a few layers deeper in rather than the first layer."}, {"start": 4822.13, "end": 4852.21, "text": "I think Hillary Clinton is also kind of interesting because it's already firing on Hillary. Sorry, you were going to say something. Yeah, I think the fact that these are a few layers in kind of accords with the sense that they're not kind of doing anything incredibly sophisticated, but there's a bit more abstraction going on here than kind of just gluing tokens together. We've also found geographic neurons."}, {"start": 4852.21, "end": 4880.51, "text": "So one way you can sort of illustrate these is you could just feed in a list of country names and then color a map by how much they fired. That's something we did in the multimodal neurons paper. And this seems to be like a sort of France-Europe neuron. And you can see it also, I think this is French Guinea over here. Maybe I think French something. And you can see it's mostly firing in Europe. And so there seem to be these neurons that are at least corresponding to sort of geographic sets of"}, {"start": 4880.51, "end": 4909.62, "text": "of nearby geographic country names. But I think more, you know, if the multimodal model is anything to go by or the experiments there maybe are also more generically encoding these countries. It's not really something we've investigated yet. Okay, so far we've talked mostly about like even if they're more sophisticated neurons that are really about detecting like a particular thing but or like a particular like"}, {"start": 4909.62, "end": 4936.19, "text": "Object or like what a token means in some particular context. But there's also these neurons that seem to be more about phrases sort of as you get towards the middle of the model. So one sort of family of these that I found especially interesting is these sort of like descriptive clause neurons. So for instance, if you're describing music, you know, you might say, well, you know, they sang in box, moss."}, {"start": 4936.19, "end": 4965.22, "text": "In B minor with the Boston Symphony Orchestra conducted by, with, and all of these are sort of things that are sort of adding descriptive information about some musical performance. Or, you know, I was dressed up in a short sleep weather in a winter campaign army jacket. Or, you know, you might have another one that we have a taste, smell descriptive clause. And it, you know, it has a salty, earthy taste."}, {"start": 4965.22, "end": 4993.97, "text": "So all of these are sort of like particular kinds of descriptive clauses and there's a neuron for each one of these and you just look through them and like you know the the taste smell one it's just always referring you know every time it fires it's it's referring to like some place where somebody's describing a taste or a smell. So yeah I thought that was really interesting and or like maybe this last one's the most interesting like there's you know with the word Russia emblazoned across the front. Well so it sort of likes to describe when there's"}, {"start": 4994.86, "end": 5024.96, "text": "When there's writing on some object. And you can imagine, I often wondered, I think this was actually something I was impressed by at one point, where a model can answer, and maybe the place where I was most impressed by this was with multimodal models, but can answer questions like what's written on what object or something like this. And having features like this is kind of cool. Is there anything you wanted to say about these, Nelson?"}, {"start": 5024.96, "end": 5045.6, "text": "I don't think so. I think we haven't studied them deeply. I do think it is cool the phrases and snippets and contexts get pulled out. And I think the way these things are a little bit"}, {"start": 5045.6, "end": 5075.94, "text": "continuous or smeared across the context dimension sometimes. I think the model is often doing things that are less crisply token aligned than we might imagine, especially in the middle of the model. It can be, in some sense, reasoning or acting across a bunch of positions at once. One thing that I think is actually interesting is these often stop and start at interesting points."}, {"start": 5075.94, "end": 5102.32, "text": "Well, here, with the word, and then we actually get to the thing that's written on the object, you know, you don't go and have this, and you continue firing again later on. And so, or like here, you know, well, okay, maybe in a winter campaign army, and then jacket doesn't fire. And so it's like the, it's like the, often like the, or like, you know, has a salty earthy, and then you don't fire on taste. So there's something interesting about."}, {"start": 5102.32, "end": 5135.49, "text": "When these fire and when they don't. And it wouldn't surprise me if there's a different neuron that fires in those places where. Yeah. Yeah. And I think it's sort of sometimes also kind of interesting to do some things that some feel a little odd. I think makes sense when you remember that these models are kind of autoregressive. Like the brew has a salty, earthy taste and sort of very plausible that we were going to continue talking about the taste. And so sort of when the model sees that and"}, {"start": 5135.49, "end": 5166.96, "text": "It sort of seems like it's sort of, you know, the and is still firing because it doesn't know what comes next. But then as soon as we stop talking about the taste, then it fires. And so sort of maybe fires for like one token longer than you might expect. But I think if you just think about the autoregressive nature, it makes a bit more sense there. We're back after an interruption where the internet cut. But yeah, we can talk about some more interesting neurons."}, {"start": 5166.96, "end": 5195.5, "text": "OK, so there's a lot of other sort of interesting sort of clause-like neurons. Do you want to talk about any of these, Nelson? Yeah, I think these are kind of similar to the last one. I think some of them are sort of not obvious things that I would have predicted existed, but sort of make sense, driving directions, sort of clearly recognizable thing that it sort of is somehow"}, {"start": 5195.5, "end": 5226.67, "text": "Important to flag all of historical periods, time ranges. Cause of death feels like a very natural unit. You encounter that a lot in text. It is a specific thing. And there's a neuron that seems to fire over those. But yeah, I don't know that there's much more that I have to say about these."}, {"start": 5226.67, "end": 5256.72, "text": "Well, I'm not a linguist, so maybe I'm misusing this, but you might describe them as like discourse markers. So like little turns of phrases people say to emphasize the significance or importance of what they're about to say, like what's so amazing about, you know, or, but what's really remarkable is, or what is really striking is, you know, all of these things are just, they, they, they aren't so much carrying content as they're like, they're carrying like a"}, {"start": 5256.78, "end": 5281.55, "text": "They're helping highlight the importance of something or something like this. Or conversely, like there's another neuron that fires for things like some would later say or one may loosely call or it might be more accurate to, you know, all these things are sort of like they're like hedging a little bit or like distancing yourself a little bit from the thing that's being said."}, {"start": 5283.82, "end": 5305.25, "text": "Yeah, I'm not sure if you have any other thoughts on these. Yeah, I haven't spent much time looking at these. I think they, I think sort of, I don't know, one thing I always just think about when I'm seeing these is sort of the kind of"}, {"start": 5305.25, "end": 5333.78, "text": "Dual challenge that the model always has when it's dealing with these is sort of like when it's in the middle of this phrase, it sort of has to predict the next token. But then it kind of also like, you know, after the phrase, it probably really wants to treat these to some extent as a unit to sort of, you know, abstract the, you know, that claim was hedged sort of and extract that fact as a sort of kind of relatively single thing where relevant."}, {"start": 5333.78, "end": 5363.34, "text": "And I don't have a kind of concrete story of how these neurons relate to that, but sort of keeping those sort of dual challenges of, you know, it does have to deal, it has to consume these and it has to output them token by token. It also wants to abstract them, sort of sometimes helpful for thinking about what these neurons might be doing or why they might be there. Yeah, it makes me think a little bit of those neurons we were kind of surprised by for outputting multi-token sort of words or outputting"}, {"start": 5363.34, "end": 5393.55, "text": "Sort of compound words like United Nations or percent where it was like increasing both per and percent. You know, if you're firing sort of over like this more contiguous range where there's sort of some higher level action that's expressed in multiple words or like higher level concepts expressed in multiple words, maybe it's then also natural that it might push you a little bit later on to have neurons that respond to like that general category of things. And then maybe that makes it more natural. Interesting though."}, {"start": 5395.76, "end": 5424.82, "text": "Okay, so that's what we're going to say about specific neurons. Hopefully those were interesting. There were some more general phenomena that we observed sort of as we were poking around at these neurons that maybe are worth highlighting for a minute. And I guess a lot of these will be sort of us drawing analogies between things that were found by the original circuit thread in vision models and then things that we're finding here. So in vision models, we were often able to go and organize neurons into sort of families of"}, {"start": 5424.82, "end": 5448.19, "text": "of interesting, of neurons that seemed very closely related. So here we have like some high-low frequency detectors, some brightness gradient detectors, some curves, some black and white versus color detectors. And in some sense, these are sort of all like variations on the same thing that you could sort of think of as one type of neuron. Like the high-low frequency detectors are in different orientations. The curve detectors, each one sort of detecting roughly the same thing in different orientations."}, {"start": 5448.19, "end": 5478.91, "text": "The black and white color detectors are mostly the same thing in different orientations, like black and white on one side, color on the other. There's a few that are center color and black and white around them, are all black and white. Or like the gradient, the brightness gradients are mostly in different orientations, but some are like dark in the middle and bright on both sides. But they're sort of, in some sense, all the same family of thing. Well, yeah, and I guess another interesting observation of these is many of them, not only are these families, but they're often like organized on"}, {"start": 5478.91, "end": 5503.9, "text": "You know they're they're equivariant in some sense so they're they're they're like there's some type of transformation that sort of they're all like transformed versions of the same thing so like they might all be rotated versions of the same thing or in more abstract ways you you might see neurons that are like the same thing but with a short snout or a long snout or dogs versus humans or visual perspective on the same thing or things like this these are sort of more abstract."}, {"start": 5503.9, "end": 5534.77, "text": "And the thing that's exciting about this kind of stuff, or like the one reason you might really care about this kind of stuff, is that it can really simplify your model if you can go and find, if you can say, well, it seems like there's a lot of neurons here, but really they're just one thing parameterized in this way, that might help you with going and, yeah, with being able to understand these things. And so I think we've started to see some things like this a little bit in language models. Nelson, maybe you want to talk about this."}, {"start": 5535.6, "end": 5558.93, "text": "Yeah, I think this notion of equivariance is a little more general, but I think of it as this notion of symmetries or some way that we can collapse the neurons down into a conceptually smaller set of neurons and all of their symmetries, some way that we can collapse them down."}, {"start": 5558.93, "end": 5587.68, "text": "These like token X in language Y or even more broadly sometimes sort of token X in context Y kind of gives us one of these sort of symmetries of we can kind of have one rule or one conceptual rule that generates or describes a larger number of neurons and so sort of gives us a way of compressing these neurons in sort of conceptual space or in reasoning about them space."}, {"start": 5587.68, "end": 5606.22, "text": "And as Chris was saying, some of these can actually already be pretty large, like TokenX and LanguageY. I don't know how many tokens there are that are ambiguous between language, but there is a fair amount of you have this potentially."}, {"start": 5606.22, "end": 5635.98, "text": "This kind of, you know, X by Y matrix. And then I think the, especially for kind of parsing words, you know, common endings to words like, you know, ED or ING in English, those often get split for a large number of words. And so you kind of, you know, parsing all of these words, you sort of end up with sort of, there's a whole bunch of neurons that you can kind of just think of as like added vocabulary members in some way."}, {"start": 5635.98, "end": 5658.99, "text": "And kind of generate these huge sets of neurons that you can kind of abstract away into one conceptual family or rule for thinking about and sort of start to eat away at the scale of these models in your description of them by kind of exploiting these patterns. Yeah, yeah, I think that's right. And I think that's cool about this language."}, {"start": 5658.99, "end": 5684.53, "text": "TokenX in language Y1 in particular is that it's almost giving you this 2D grid where like you could imagine language, you know, like a column for each language and a row for each token. And there's holes in that. Like if you, you know, we've, these are four neurons we found. We in Dutch, we in German, best in Dutch, best in German. In some of these cases, they may not be like a standalone word. They might be a token that's in other words. But."}, {"start": 5684.53, "end": 5713.28, "text": "You see a lot of neurons like this. And if you start to look at them, often there's the same token occurs in many languages, and each language has many things. And you can almost start to imagine this 2D grid with gaps in it. And that's kind of interesting to be able to organize your neurons like that. And I think, as Nelson says, the key reason that this is exciting is large language models have millions, tens of millions of neurons maybe."}, {"start": 5713.28, "end": 5740.98, "text": "And probably that'll, you know, maybe there'll be larger models in the future. So, you know, how in the wide world can we deal with models that have that many neurons? Well, I think our hope has to be that there's some structure or organization or way to group or organize these neurons such that we don't need to think of every single neuron independently, but instead we can go in and somehow organize things."}, {"start": 5746.9, "end": 5762.1, "text": "Yeah, so this is another interesting phenomenon. Yeah, Nelson, I don't know if you want to take this or if I should. Yeah, I think this is, as we look at larger models, we sort of see sometimes that"}, {"start": 5762.1, "end": 5783.95, "text": "A kind of one neuron in a smaller model seems to sort of split into more specific cases in larger models. We showed a number of these specific examples. For the one layer model, we found just this base 64 neuron that seems to first order just note we're in base 64 and predict more base 64."}, {"start": 5783.95, "end": 5809.73, "text": "But then for large models, we actually found neurons for every character, or at least many common characters or tokens that appear in Base64. And there's neurons for each specific one. And then kind of similarly, small models sometimes have just a like, this is a US politician neuron, or this is a token that commonly appears in English and Dutch."}, {"start": 5809.73, "end": 5837.58, "text": "But maybe even more commonly appears in my corpus in English, but that we're in a Dutch context. And so probably it should be interpreted as Dutch instead. There's just kind of one token that does that in a kind of probably kind of rough way for one neuron that does that in a kind of rough approximate way for many common English tokens. And then those fission into these more specific ones."}, {"start": 5837.62, "end": 5867.22, "text": "And so that kind of gives us another sense of these sort of patterns of these large models. We can group the neurons in these ways. We can try to understand them. You could imagine trying to maybe in a kind of mostly automated way build these sort of these trees of these fissionings of these neurons. We don't have kind of concrete leads on how we'd do that, but it seems conceptually at least plausible."}, {"start": 5867.22, "end": 5897.2, "text": "And that might give you another kind of powerful way of kind of trying to categorize these and group them and understand them. And I think it's just also sort of a kind of cool demonstration of kind of what these models are doing as you give them more capacity. Yeah, it sort of, it paints an interesting picture of what it means to have these clean scaling laws where, you know, loss goes down as you make models larger because it sort of just like,"}, {"start": 5897.2, "end": 5926.54, "text": "Like maybe there's some sense in which at a micro scale what's really going on is all these neurons are fissioning into finer ingrained distinctions and you know in aggregate that creates a very smooth decrease in loss but it's actually like all of these like very micro like actually very discrete changes to the model in some sense and it's a cool cool way to imagine things. Okay so I think that's everything we uh we I yeah these slides plan to cover. Nelson is there anything else you want to chat about? Um I don't think so."}, {"start": 5926.54, "end": 5953.52, "text": "I think we've spent a lot of time on these. I think we're pretty excited about the degree to which I think there's a bunch of research directions that we tried unsuccessfully on our model that we're excited to revisit again. And it seems like a lot of ideas that didn't work now just work. And so there's a lot of exciting directions."}, {"start": 5953.52, "end": 5982.34, "text": "This is very early days for these results, but we wanted to share this video because I think if other teams are trying to do similar things, if you're able to tweak your activation function in that way, there's a good chance you'll get similar results. And if you do or you don't, we'd love to hear about it as well. And I think there's a good chance we'll be releasing more of these and probably eventually a paper as we..."}, {"start": 5982.34, "end": 5997.54, "text": "Get more of these results and make some of these observations more concrete and rigorous or otherwise. But for now, we're just excited by how fruitful these models are to look at. Yeah."}, {"start": 5997.54, "end": 6029.95, "text": "Okay, Nelson, I'm going to put you slightly on the spot before we close. You know, you have this blog post about how computers can be understood, and sometimes you, like, rant on Twitter about, like, you know, computers can be understood, damn it. I believe, and I continue to be rewarded. So, you know, the question I want to ask is, you know, can neural nets be understood? Or where are you on your view on whether we can make a similar claim about neural networks? I think it remains a kind of profoundly open question of how well we can understand neural networks."}, {"start": 6029.95, "end": 6051.7, "text": "I think believing that it is an open question, honestly, probably makes me weird in the field. I feel like the modal opinion is to just forget about trying to understand these, treat them as black boxes, and make them go. But I think it seems very clear from"}, {"start": 6051.7, "end": 6077.55, "text": "I think both the past circuits work that you and others did and from our initial results that these neural networks are far more understandable than you might believe and that it's possible to make progress and to reason about them. And I'm very excited about the project of"}, {"start": 6077.55, "end": 6102.37, "text": "Doing our damned best to understand neural networks and maybe we'll fail, but we haven't failed yet. We have not yet exhausted anywhere near the level of effort that we would have to exhaust for me to conclude that it's impossible. And honestly, I would never say that it's easy, but it does really feel like."}, {"start": 6102.37, "end": 6130.78, "text": "Many of the things that we try work, at least in part, and bear fruit. And so it sort of, you know, continues to be worth trying. And I'm very excited about the prospect of getting to a world where we do feel like we can kind of mostly understand what's going on inside of these. And it seems very possible. How much has this result sort of updated you on that? Like, I feel like for me, at least, before this result, it sort of felt like, you know, I was determined to try."}, {"start": 6130.78, "end": 6157.42, "text": "And I believed abstractly that we were quite likely to get traction. But it didn't feel like I could really approach an arbitrary behavior on a neural network and think that I had a good chance of figuring out what was going on. And I feel like that's actually maybe changed for me with these results, where I now think, give me an arbitrary behavior, and I have a decent chance of something. I'm curious if you have any sort of ."}, {"start": 6158.32, "end": 6184.54, "text": "I think the Rome paper especially gave me a lot of confidence that it was going to be possible because they sort of, you know, those methods definitely, I think, sort of were complicated and messier, but sort of, you know, gave me some confidence and we were able to replicate some of those results to sort of, you know, teasing apart these MLP layers. Sorry?"}, {"start": 6184.54, "end": 6209.09, "text": "Oh, this was our colleague Neil Manda who's been going and doing some cool work replicating Rome. Neil has done some cool work replicating some of those. I don't feel like this has drastically shifted the sort of weight that I put on whether it's possible. I do think it has drastically shifted my expectation of how easy it might be. I think sort of before these results, I would have said like, I think it's..."}, {"start": 6209.09, "end": 6226.26, "text": "I think it's likely that we'll be able to tease apart the MLP layers, but we sort of might need pretty high powered like mathematical and software engineering tools to sort of get at them."}, {"start": 6226.61, "end": 6256.91, "text": "I sort of put decent probability mass after this result on if you name an abstract behavior and if the team sits down with kind of our current tools and our current understanding for a couple of days, there is a decent chance we will emerge with a sort of roughly accurate first order description of that behavior. Like not certainty, but I now I put like appreciable probability mass on that in a way that I didn't before. Well."}, {"start": 6257.23, "end": 6285.22, "text": "I think it's pretty exciting. And yeah, we'll leave it there. But yeah, very excited to hear from anyone. Yeah, maybe the one other thing that I think sort of we take for granted, but is sort of, I think our first two papers, we did a pretty good job of sort of understanding attention to our own satisfaction of sort of feeling like we know what's going on there. I think with this result, with these models, it's sort of the MLP layers seem like by and large, like,"}, {"start": 6285.22, "end": 6311.97, "text": "Pretty easy to understand. A lot of the obvious things mostly work for giving you a good foothold on them. And attention and MLPs are all there is in these models. And so these models are very large. And you get emergent behavior out of composing simple primitives as a general theme of complexity. But it feels like we have a very good handle on attention. It feels like we have a"}, {"start": 6311.97, "end": 6343.65, "text": "Pretty promising handle on MLPs. And at the end of the day, that's all there is. And that sort of is not necessarily sufficient that we can understand these models. And I'm definitely not going to assert that it's easy because they're very large and you end up with emergent behaviors. But it feels like a very promising place to sit to sort of start going deeper. I think so. Onwards. Yep. All right. Thanks for joining us, everyone. OK."}, {"start": 6343.65, "end": 6345.07, "text": "Thank you, Chris."}]}