{"url": "https://www.youtube.com/watch?v=DA3AA1w5TC0", "page_url": "https://www.youtube.com/watch?v=DA3AA1w5TC0", "title": "Softmax Linear Unit #2: Update on Preliminary Results", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 1589, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing. SoLU videos may be led by Nelson Elhage/others - Olah's share uncertain.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-softmax-linear-unit-2-update-on-prelim", "audio_duration_s": 1588.8213125, "worker": "jc-w12", "transcribe_s": 39.0, "model": "large-v3", "segments": [{"start": 0.3, "end": 29.34, "text": "All right, we're back with another research update on the softmax of X times X neurons that we talked about last time. We have made a bunch of kind of new, some number of new discoveries and learned some new things that we wanted to share. And I wanted to also, we'll introduce Tristan here, is the newest member of the interpretability team who's been helping out a lot with this research. And so he'll help recording this video. Hi."}, {"start": 29.34, "end": 55.98, "text": "We always want to emphasize our usual caveat that we sort of plan to release a paper eventually. And kind of by that point, hopefully we'll be on really solid footing for everything that we share. But for right now, these are kind of early results. We're pretty sure that we're onto something, that there's something interesting here. Take everything we say with a little bit of a grain of salt and sort of these are, we think these are right, but these aren't super confident results. We don't necessarily understand everything."}, {"start": 55.98, "end": 85.26, "text": "We're sharing this because we think they're useful and might help other people who want to explore similar areas. But don't assume that these are completely verified yet. We've mentioned to some folks that we have a paper. We're working on a paper on this. We still are, but we pushed it back a little bit due to some internal prioritization and due to Chris being out on vacation. Hope that will come out in the next coming months at the latest, but timeline is a little less close than it once was."}, {"start": 85.26, "end": 112.21, "text": "And so we're dropping another video to talk about a bunch of things we've learned since then in case anyone else is following along and interested in working on this. I think the first thing we want to talk about is we did a sort of I talked last time about doing an informal looking at sets of random sets of neurons from different models to try to get a sense quantitatively of how interpretable they are."}, {"start": 112.21, "end": 139.82, "text": "We turned that into a somewhat larger scale, actually randomized and blinded experiment. And I'll let Tristan talk through what he did because he was also one of the key experimenters on that project. Sure. So basically, we each had a list of neurons that just told us what the model size was and what layer it was. And we each went through a bunch of these neurons."}, {"start": 140.27, "end": 160.8, "text": "And spending about like one to two minutes per neuron looking at the max activations and coding them as kind of like interpretable or non-interpretable. And then once we were all done this list, we could take them all together and use a program to unblind them and look at"}, {"start": 160.8, "end": 186.38, "text": "Our collective results to see if we could determine a difference of how we were coding the neurons as interpretable between the models with the different activation functions. And then we also noticed that looking at some summary statistics, the models where we added the layer norm to the Gelu seemed a little bit more sparse than the pure Gelu models."}, {"start": 186.38, "end": 216.72, "text": "Part of the fundamental theory is that the sparsity was part of the cause of the increased interpretability. So we added just a Gelu plus layer norm model to one of the model sizes of the randomized experiment. Next slide. Yeah. And so this was the criteria that we were using. We told experimenters to spend one to two minutes on each neuron and kind of count them as interpretable if..."}, {"start": 216.72, "end": 243.6, "text": "Sort of 80% or so of strongest firings felt like they had a articulable rule or category or description. And for every neuron, we had 20 highest activations that we observed in a scan over a subset of the training data set. The interface we were looking at was sort of something like this. We just see the neurons. We color the text by the activation."}, {"start": 243.6, "end": 265.46, "text": "This particular neuron, I think we would have coded as interpretable. I struggled to clearly articulate the rule, but there's a clear words protection from or against, like when you're talking about shielding or protecting or defending from or against and on that word."}, {"start": 265.81, "end": 287.65, "text": "The overall results we have plotted here on a graph of model size on the x-axis. We're on a log scale here. This model over here is a one-layer model. That's a tiny little model, but we talked about the one-layer models last time. They're kind of interesting because we're really pretty close to being able to fully characterize them."}, {"start": 287.65, "end": 317.1, "text": "And I think the high level story here is the kind of initial numbers I gave last time were a little optimistic in my informal experiments, but there's a pretty clear effect here that is robust across a decent range of model size of the Sol-U models are 20 points, 25 points, percent more neurons are kind of easily interpretable under this metric."}, {"start": 317.1, "end": 346.51, "text": "The GelU plus layer norm model, interestingly, kind of sits right in the middle there. We haven't really done any more follow-up investigations, but that's an interesting observation that that also seems to shift on this metric. And then the one other observation is that the largest model we looked at here, which is about a 50 billion parameter model, the difference really drops off. And that's barely more interpretable than the GelU model."}, {"start": 346.51, "end": 359.86, "text": "Something happens with scale that we don't understand yet. And Tristan, maybe you want to talk about the qualitative experience of looking at those models really was, I think, also quite different. Yeah, so there was like..."}, {"start": 359.86, "end": 385.18, "text": "It was blinded, so we haven't gone and looked back and double-checked this intuition, but we did kind of notice that there were kind of different types of interpretable or something, where some patterns were just very clear and stark, like the example in the screenshots we showed, where it would fire very strongly on one token and basically not at all on others. And others were kind of like..."}, {"start": 385.18, "end": 404.72, "text": "a mishmash where it would show like middling activations for a whole bunch of things just in the general vicinity of the tokens that fired strongly on and our interpretability rule was based on the strongest firings um but some of these neurons were just like substantially harder to interpret because of the like"}, {"start": 404.72, "end": 428.82, "text": "Middling firings and a little bit more confusing and muddled. And based on our informal poking around, we hypothesized that those were the Jellu neurons that when they are interpretable, they're more confusing. And also another thing we noticed is that a lot of the times we were coding things as..."}, {"start": 428.85, "end": 457.76, "text": "Uninterpretable. It just had one or two polysemantic cases. So we would say, ah, this is the neuron for this specific word plus the concept of not knowing something. And because the random German word that it fired on, in addition to the concept of not knowing something, it would either be one of those two all the time very starkly, but that would be coded as uninterpretable."}, {"start": 457.76, "end": 479.89, "text": "Our other poking around unblinded later seems to suggest that that might happen more with the SolU models than with the GelU models, where when the GelU models are uninterpretable, it seems like they might just be completely muddled. And the SolU models are more often polysemantic over two or five things."}, {"start": 479.98, "end": 500.85, "text": "Yeah, I think the other thing that I would remark on is that the 64 layer models, I think this is true of the 50 billion parameter. I think this was likely true of both models, although again, I haven't done too much unblinded poking. Some of the neurons are clearly just doing much more sophisticated or nuanced things."}, {"start": 500.85, "end": 532.11, "text": "And so there were many cases that I coded on the borderline of not quite sure if interpretable. I'm going to break no because I can't quite explain what's going on here. But there's clearly some structure or some logic to it. And so I think it's possible that there is a real difference here, but we would just need a different methodology or different tests to show it. It's also possible that we would need additional techniques."}, {"start": 533.49, "end": 562.02, "text": "Yeah, just sort of summarizing what we said. I think there's a pretty, the win for Solu has held up. The advantage for Solu has held up in this blinded and randomized. This sort of definition of interpretable from this experiment is definitely kind of weak and has lots of limitations. We're spending one to two minutes on neurons. We're only looking at the largest activations. So I wouldn't claim, for instance, that we kind of completely understand those neurons just because they pass this test."}, {"start": 562.02, "end": 574.75, "text": "But I think it's good evidence that there is a real effect here. The effect is way weaker on the largest models. So there's additional questions there. And kind of moreover."}, {"start": 574.75, "end": 602.29, "text": "We have a lot of specific questions, but we're really enthusiastic about the proof of concept of finding model architectures that perform about the same as the standard architecture, but are more interpretable in some way. And that is going to be a productive avenue for trying to understand these models is, in addition to analyzing the architectures that exist, trying to find variants that are just easier to interpret or work with in some way."}, {"start": 602.29, "end": 628.03, "text": "Yeah, I want to emphasize the thing where like these tests are like really limited and we don't think that like when we do this interpretation that we've like fully interpreted the neuron and we'll talk a bit more about that kind of thing later. But it's an exciting like quick measure we can do to see that we're kind of like this metric that we previously couldn't budge on at all. Like these neurons, even with this really..."}, {"start": 628.03, "end": 658.53, "text": "Weak notion of interpretability weren't particularly interpretable. We finally have a research direction that seems to be pushing this, and that's what makes us excited. But it's excited that we are moving in this direction with this new activation functions. Yeah. And the next thing we wanted to dig in a little bit on was the performance of these models and especially on the impact of the layer norm. We mentioned last time that we're using this"}, {"start": 658.53, "end": 686.34, "text": "SolU activation, we're calling it, the softmax of x times x, and that we then apply a layer norm after that. The layer norm is kind of critical to making these models perform well. One layer models actually work fine either way, but as soon as you get to even a four layer model, which is only 10 million parameters in our setups, performance heavily tanks without that layer norm."}, {"start": 686.34, "end": 715.9, "text": "And we have some real open questions about the implications of that change on the interpretability. A lot of the neurons we showed last time when we were doing a lightning tour of cool neurons, we found in models that were trained without layer norm. And so those models were taking a substantial ML performance hit. We have some concern that the layer norm models are not quite as nice. So we can just throw up some loss curves quickly."}, {"start": 715.9, "end": 742.16, "text": "This is a suite of three models trained with the SolU activation function compared with the GelU function. The solid lines are the GelU, and the dashed ones are the SolU. And you can just see that the loss is substantially worse for the SolU models. We're really taking a big hit to the ML there."}, {"start": 742.74, "end": 772.69, "text": "But then if we instead use the SolU followed by the layer norm, we actually see that the SolU models do, in many cases, better early in training. And then by the end of these particular training runs, they're basically identical within noise. There's sort of very small differences that we think are basically noise. And so the layer norm models really do"}, {"start": 772.69, "end": 799.44, "text": "Match the performance of the baseline model. We don't really understand what's going on there. We do note that the models without the layer norm seem to have a lot of effectively dead neurons. There are a lot of neurons where after the softmax, their activation is essentially always zero or extremely small."}, {"start": 800.02, "end": 829.47, "text": "And then we have some concern that, again, that those models are less interpretable, that some of the results we showed last time might not hold up as well with the layer norm. The randomized experiment that we did, those were those layer norm models. So we were comparing models that are basically apples to apples for ML performance and for compute. And so we are confident that there's a real effect here."}, {"start": 829.47, "end": 857.54, "text": "But yeah, some of the neurons we showed last time that were really cool were those models without the layer norm. We haven't exhaustively studied, we haven't deeply studied the ones with the layer norm yet, but we have some evidence that you're getting a little bit of polysemanticity back. And it makes some sense that the layer norm might enable the model to scale up sort of small post softmax activations and kind of..."}, {"start": 857.54, "end": 883.98, "text": "Smuggle signal even through those low activation areas and sort of retain the non-sparse activation patterns and the activation patterns where multiple neurons are acting in concert. And we'll talk a little bit more about that in a moment. Yeah, so I just want to give some quick concrete examples of some stuff we've seen."}, {"start": 884.14, "end": 892.77, "text": "We were looking at, we tend to look at the maximum data set, the data set examples that maximize activations for some neuron as they're kind of."}, {"start": 892.77, "end": 915.41, "text": "First pass, look at what's going on. So this is some lightly cherry picked, but these are basically the maximum activations for one particular neuron on one model. And looking at this, it really seems like that neuron basically fires on the second token of Pasadena. It's one of these neurons that we talked about last time that is sort of..."}, {"start": 915.41, "end": 932.64, "text": "What we can do, though, is if we..."}, {"start": 932.64, "end": 958.24, "text": "We run some collection over our data set, and we collect a bunch of examples of firings for that neuron at different ranges of activation. So we've collected 20 buckets here. At the far right, we have tokens that activate this neuron in the top 5% of the range of activations."}, {"start": 958.24, "end": 987.02, "text": "And then we chose this one because we can kind of write this regular expression that's very easy to test automatically, testing for whether it fired on Pasadena. And we can actually see that the neuron most frequent, that in this second till max bucket, 50% of tokens where the neuron fired with that activation were firing on Pasadena. The other 50% were something else. And it's actually even lower at this top bucket."}, {"start": 987.02, "end": 1013.28, "text": "And then we can go in and eyeball these and just like write some more regular expressions trying to pattern match what's going on. The second one is a giant messy Unicode character class that is testing for any character in the CJK unified ideographs regions of Unicode in case you're a Unicode nerd and that's meaningful to you like it is to some of us. And so basically we see..."}, {"start": 1013.28, "end": 1028.19, "text": "There's a couple of things going on. And also, I think kind of more interestingly, they vary with strength of activation. Like these slightly middling regimes are firing for what look like PHP stack traces."}, {"start": 1028.19, "end": 1050.53, "text": "Pasadena kind of spikes also here. The very strongest, many of them are actually something that I don't speak any Asian languages and some tokenization problems make these hard to interpret, but there's something going on that I can't understand in those Asian languages. And then there's also just a couple of other oddities that show up across the board."}, {"start": 1050.53, "end": 1073.68, "text": "I think even though this is a neuron that our experiment probably would have coded as interpretable because you look at these examples and they're so crisp, there's clearly polysemanticity going on in at least some of these neurons in kind of confusing ways. And if you look at the max data set examples, you're especially sometimes liable to be confused."}, {"start": 1073.78, "end": 1098.43, "text": "And yeah, again, we kind of have this concern that because the layer norm allows small post softmax activations to get scaled back up, it might be exacerbating this phenomenon relative to softmax and sort of, you know, maybe vanilla SolU works is great for interpretability, but the performance hit, you take a real ML performance hit."}, {"start": 1098.43, "end": 1128.74, "text": "And maybe Layer Norm works great from ML, but is restoring our policy semanticity. We're really not sure about that, but we just want to caveat that that's one open direction that we want to develop some confidence on. And on that, we actually, having done this, we did a couple more experiments and realized some sort of surprising empirical and algebraic results that I'll toss to Tristan to talk about. Sure."}, {"start": 1128.74, "end": 1158.42, "text": "The first thing, we were looking at the activations in a notebook and just looking at the, on a bunch of dataset examples, the activation before the SolU and after the SolU, but I believe before the layer norm. That's right. And we found that they, like, corresponded very closely to exp of x times x."}, {"start": 1158.58, "end": 1186.72, "text": "And we thought like, huh, that's interesting. I wonder what will happen if we just train a model with X with X times X as the activation function instead of SolU, whether it'll have approximately the same interpretability properties and ML performance. So we kicked off that model training on a Friday. And then on a Monday, we came back and Chris..."}, {"start": 1186.72, "end": 1216.05, "text": "commented on our slack post about this that, next slide, it turns out that you can mathematically prove that layer norm of x with x times x is equivalent to layer norm of softmax of x times x and Chris didn't reproduce the proof but I then went and like went through the math myself and reproduced the proof and the basics of it is that softmax is a"}, {"start": 1216.05, "end": 1234.4, "text": "This top line of formula here is kind of the formula for softmax of x times x, and it basically corresponds to an exponential on the top divided by a normalizer, which is a sum of exponentials, and then we multiply it by x in our version."}, {"start": 1234.4, "end": 1263.49, "text": "Normalizing term on the bottom is constant with respect to which neuron it is. So we can kind of think of that as a constant alpha multiplier on X times X. And then if we substitute that into the formula for layer norm, we get this like alpha XI, wherever we previously had alpha X or just XI. And then if we just do a bunch of algebraic simplification, we can pull out the alpha all the way."}, {"start": 1263.49, "end": 1291.47, "text": "To the end of either fraction and then completely cancel it out. So this means that all of our experiments where we did a solute followed by a layer norm were actually mathematically equivalent to doing a layer norm of just X of X times X. And sure enough, our models that were trained with X of X times X that we've already kicked off had basically the same ML performance and basically the same interpretability properties."}, {"start": 1291.47, "end": 1319.9, "text": "And if we look at what this formula actually is, I forget if I, I don't think I have this on a slide, but it basically corresponds to a different kind of normalization than softmax and also this mean normalization. And we kind of have a theory that the mean term might end up negligible because these exponentials tend to push things to extreme values and there's just so many neurons."}, {"start": 1319.9, "end": 1347.57, "text": "It might not end up negligible. It might. But if it does, then it ends up being basically the same as the softmax formula, but using a L2 norm on the bottom instead of an L1 norm. And if we go to the next slide, these are our kind of like speculations about what this result means. So the L2 norm is known to incentivize sparsity less than the L1 norm. So that might be a like..."}, {"start": 1347.57, "end": 1378.88, "text": "Implication for why we might see more polysemanticity in the layer norm version. And the mean term might end up negligible, but it might not end up negligible, and that might also have implications. And then the intuition, we might have presented this intuition for why Softmax works, of pushing it to only fire on certain values. Sorry. Sorry, I lost my voice."}, {"start": 1378.88, "end": 1405.97, "text": "Can you take over, Nelson? Yeah, yeah. So I think our original intuition for Softmax is it sort of incentivizes you to, or not incentivizes, it just structurally pushes to only have a small number of strong firings for any given position because Softmax constrains you to have a distribution that sums to one and pushes you towards the extremes."}, {"start": 1406.54, "end": 1436.11, "text": "And so it turns out that potentially having the exponential times X, which also pushes you towards the extremes, that might be a big effect or a similar effect to the actual normalization. And then, yeah, there's the question of with the layer norm, you're normalizing by the L2 norm or by the variance. There's this mean adjustment, which may or may not matter."}, {"start": 1436.11, "end": 1466.14, "text": "And it's just a little bit. So yes, I guess in some sense, the layer norm models aren't really doing a softmax in any sense. They're doing a differently normalized softmax. And then there's this question of the L2 norm versus the L1 norm that we're dividing by. The L2 norm does tend to, I think, encourage sparsity less."}, {"start": 1466.14, "end": 1493.33, "text": "And yeah, we were kind of a little embarrassed to realize that in some sense, we were writing softmax in our code, but in some sense, the fact that it was a softmax was irrelevant. It was just this exponential times x and then normalizing it differently. And if you do the softmax normalization first, that's kind of irrelevant for those models. Anything else you wanted to add, Tristan? Nope."}, {"start": 1495.31, "end": 1524.93, "text": "All right, and that was our last slide. I forgot to make a final slide for discussion. But yeah, these are just our quick updates on this since we told people that a paper would be coming out. It still is, but it got delayed, and so we wanted to share these interim results. We're kind of more confident than we were, or we were always pretty confident, but we now kind of have a little bit more rigor in convincing that there is some real effect here."}, {"start": 1524.93, "end": 1549.9, "text": "There's this big open question of kind of how deep the effect goes, if maybe we've made the strong activations interpretable, but there is a lot of potentially still a bunch of meaningful poly semanticity hiding in the lower activations and the layer norm version might be worse for that than the vanilla softmax."}, {"start": 1549.9, "end": 1576.29, "text": "But just doing a vanilla softmax does currently tank ML performance. Another open question is whether there's some other tweak that we could do that might get the best of both worlds, or whether there's even more exotic architecture tweaks than just changing the activation function like this that would also encourage interpretable neurons. And yeah, we'll..."}, {"start": 1576.29, "end": 1586.9, "text": "We're continuing to research those and we'll have a paper out at some point that will contain all of this along with anything else we've learned between now and then."}]}