[0:00:03] Oh, it's a pleasure to be speaking to you this morning, morning in San Francisco. And I was hoping... Oh, is this better? I don't think I have any control over that. Great. Well, it's a pleasure to be speaking to all of you today. And I wanted to talk today about two high-level questions. So the first is, how can we understand neural networks? This is a mechanistic interpretive workshop. The second is...
[0:00:35] How can we discover things that we didn't expect to see when we're trying to understand neural networks? How can we not only look for things that we guess might be there, but how can we be surprised? This feels like a pretty exciting moment to be speaking to all of you. As I was preparing this talk, I was sort of reflecting. And it felt to me like, you know, perhaps two years ago, I thought that maybe the capstone of my career and all this work on interoperability that I had been doing might be an explanation for why we couldn't.
[0:01:04] Thanks to the work of a lot of people in our community, including a lot of people in this room, that no longer seems true. So I think this is a very exciting moment.
[0:01:33] Now, what do we mean by mechanistic interoperability? I actually think the pragmatic answer is that mechanistic interoperability is an umbrella for a wide cluster of different things. And I think that's really great. But I wanted to sort of articulate sort of my own view at what the sort of core molten nexus of mechanistic interoperability is. And I don't mean by doing that to sort of narrow the tent or limit mechanistic interoperability, but I think it's useful to talk about.
[0:02:06] Think about regular computer programs. We write, you know, we might write planning docs and sort of describe what we're trying to do. And then we can write source code that is a human understandable description of that computer program. And then we compile it into a binary. But when we have neural networks, we, you know, we maybe have our data set and training objectives, and then we train it and we get this sort of, you know, compiled binary neural network, the parameters. But there was never sort of a human understandable algorithm in the middle.
[0:02:35] And so the goal of mechanistic interoperability is to figure out, you know, go and create something that takes the place of source code. And I think that's actually a very deep analogy. So if you think about computer programs, regular computer programs and neural networks, you know, maybe a computer program is a variable, a neural network has a neuron or a feature. If we think about the program state vector, we have the activation vector in a neural network. If we think about the processor, we have the network architecture in a neural network. If we think about the compiled binary, we have the network parameters. But there's nothing analogous to the source code. So we'd like something...
[0:03:04] That can take its place. And so how can we do this? Well, I'll tell you how I think about it. So a lot of my thinking actually was originally informed by thinking about continents, and especially this model Inception v1. And in Inception v1, many of the neurons actually just correspond to interpretable features. Like there's a car detector neuron that you can just look at. And it seems to genuinely be a car detector. We also have the weights, which you can sort of think of as being the binary instructions that we want to understand.
[0:03:33] And the weights connect together different features, different neurons. And we can then try to understand those weights. So for example, Inception v1 is full of these curve detector neurons, which really do genuinely seem to be human and temporal. And you can test this in all kinds of different ways. We wrote actually two whole papers on these curve detector neurons. And they really do seem to be curve detectors. But what we ultimately want to do is we want to understand the weights, because those are
[0:04:04] are the compiled computer program. We want to understand the weights. And if we look at them in isolation, we can't. Like, what can we say about these weights? They're saying something like, oh, they excite the next neuron if something's on the right-hand side and inhibit on the left-hand side. But that doesn't tell us very much. But if we look at the neurons they're connecting, it turns out this is connecting a right-facing dog head detector to a head plus neck detector. And what the weights are telling us is we want to see the head on the right-hand side of the input.
[0:04:32] So we want to see the head on the end of the neck, and that makes sense. So once we contextualize weights, we can understand them. And it turns out that neural networks are full of structure once you start contextualizing the parameters. So here we have two different examples of dog heads being attached to dog necks. We can see a pose-invariant dog detector being created. One of my favorite examples is this car detector, which looks for windows on the top.
[0:05:00] It looks for wheels on the bottom, and it looks for the car body sort of everywhere, but especially on the bottom. And so this starts to point at a sort of general recipe for understanding neural networks. First, we understand the features. We use those features to understand the circuits, which are connecting those features through weights. And as we understand more and more circuits, we begin to understand the model. Unfortunately, that is not a complete recipe for success, because there's this problem we call superposition.
[0:05:32] Not all the neurons are as nicely interpretable as we might hope. And that presents a problem because the only reason the previous things worked was because we understood the neurons. If we want to understand a line of code in a computer program, we need to understand what the variables are. And if we want to understand the weights of a neural network, we need to understand what the features it's operating on are. And we see all these polysemantic neurons. Inception V1 has quite a lot of them. There's, for instance, this one neuron that responds to car faces and fronts of cars.
[0:06:03] And cat legs. And it turns out that actually language models are worse. They have more of them. So that's a problem for us. It means that the sort of first step in our recipe for understanding models is now in danger. So can we rescue it? Well, there's a hypothesis for why this happens, which is the supersession hypothesis, which says, well, the network wants to represent, let's say, more concepts than it has neurons. And how can it do this? Well, it can use combinations of neurons to go and represent.
[0:06:32] And we'll call these combinations of neurons, these directions in the vector space of neurons, we'll call them features, these special combinations that represent concepts. As an aside, you might have heard of distributed representations, which are really, you know, this is a really famous and important idea in neuroscience and in machine learning. And the way that I would describe the relation is that distributed representations are often used to mean two different things. They mean composition.
[0:07:00] So for instance, if you think of the famous Michelob results, right? Like, you know, king minus man plus woman equals queen. There we have different features that can combine together to represent concepts. And there's also superposition. So if you think of like ASCII codes, when we go and represent, you know, different letters with different combinations of bits and sort of use every possible combination to represent something different, that's superposition. And it turns out these actually aren't.
[0:07:25] Mutually exclusive, they can occur at the same time. There's actually a whole field of mathematics that essentially studies this called compressed sensing. It's a very well-studied thing. In fact, I think it goes all the way back to Laplace. So it's a very, very rich, rich thing. And if we take this idea of superposition seriously, this idea, which is basically just saying that neural networks are doing compressed sensing, we end up with a wild...
[0:07:48] Wild hypothesis, which is the neural networks we see are actually these compressed versions of much larger sparse neural networks that have been folded on top of themselves. And so if we want to understand the neural networks, we need to unfold them. And fortunately, it turns out that actually this is a well-studied problem. You know, if we believe that what's going on is compressed sensing, there's this standard method called dictionary learning that comes from neuroscience and machine learning that is sort of just the correct answer to go and approach this.
[0:08:20] And dictionary learning is very expensive, but there's something called a sparse autoencoder that can more cheaply and tractably approximate it. So the natural thing to do would be to say, well, can we use sparse autoencoders to go and solve this? And the answer seems to be that we can. So in the last two years, there have been a number of papers that have, I think, really shown a lot of progress on this. So some of my colleagues at Anthropic and I have worked on this. There was a really exciting paper from Luther and...
[0:08:50] Apollo and some independent researchers. Of course, there's been some work from, really exciting work recently from the former super alignment team at OpenAI. There's been really exciting work from Google DeepMind and many others. So this is not comprehensive. And I'm just so grateful for all this work. I feel like we now really have a grasp on this problem. I wanted to talk about one particular...
[0:09:18] Use of sparse autoencoders. One experiment that we've done was applying a sparse autoencoder to Claude 3 Sonnet, which at the time we did this was one of Anthropic's production models. So this is a real model that's used in deployment. It's a large model. And I think that we can find some really interesting things if we look at it. So just as one fun example, we found this feature that seems to correspond to the Golden Gate Bridge.
[0:09:44] And if we look at the high activating examples, they're all texts about the Golden Gate Bridge and pictures of the Golden Gate Bridge. And if we look at weaker examples, they're kind of discussion of things related to the Golden Gate Bridge, especially if we've just been talking about the Golden Gate Bridge, like the Presidio, which is the park right beside the Golden Gate Bridge, and pictures of other famous San Francisco landmarks or the Golden Gate Bridge from a greater distance. And it turns out that if we go and we force the Golden Gate Bridge feature active,
[0:10:12] We get all kinds of interesting behavior from Claude. So for instance, if we normally ask Claude, what is your physical form? Claude says, you know, I don't have a physical form. But if we activate the Golden Gate Bridge feature, Claude declares, I am the Golden Gate Bridge, a famous suspension bridge that spans the San Francisco Bay. My physical form is the iconic bridge itself. Now, of course, steering models like this isn't...
[0:10:32] Something that we can uniquely do with features. This is a standard trick that we can do in all kinds of contexts in different ways. But it's really cool to see that these features are causal and that we can use them in these ways. And in fact, we went in for 24 hours. We had Claude, Golden Gate Bridge Claude available through Claude.AI. And I think there was quite a bit of interest in that. So that was a fun experiment and allowed people to play with this kind of thing. Okay, so I want to give now...
[0:11:00] A quick whirlwind tour of some results I find interesting, and this will only be a very limited slice. But as I do so, I want you to go and ask yourself something, which is whether anything that I'm going to tell you is surprising to you. Because some of this stuff was surprising to me, or at least I would have thought of it. So one fun example is there's a code error feature. So here we go, and we define a function, and we have a typo in it, and this feature suddenly activates. And also, if we call the function,
[0:11:30] Then this feature activates as well. And it doesn't do this, for instance, if you just have typos in regular text. But it does do it if you have, for instance, division by zero, or if you try to go and call a function that uses range with a negative input. So it can trace things through composition as well, which is interesting. And if we intervene on it, we can give some code that's totally correct, and Claude can ordinarily tell us what the answer is if we call that function. But if we instead clamp the code error feature on,
[0:11:59] Claude says, ah, in fact, this code has a bug. Type error unsupported for adding two integers. That doesn't work. So Claude can hallucinate an error if we do that. And conversely, if we have code that has a typo in it, Claude would ordinarily tell us that there's an error and there would be a traceback. But if we go and we set the code error feature to a negative value, Claude then goes and gives us the correct answer for what the code should have done. So that's very interesting.
[0:12:28] We also find an addition feature. This is kind of spiritually related to some of the work on function vectors from David Bow's group. So here we have a function that does addition, and the feature activates. And when we call the function as well, the addition feature activates. But it doesn't activate if we call a function that does multiplication. It can trace this through composition, which is really cool. And we can, again, intervene on it. So if we have a function that ordinarily does multiplication, and then we force our addition feature on, Claude interprets it.
[0:12:58] As the function doing addition. So instead of one times two is two, we get one plus two is three. Another fun function is we find an unsafe code feature that fires for things like com.sun.management.jmxremote.authenticate equals false and dash dash allow unauthenticated and SSL allow invalid certificates. And lots of more minor things as well. These all sound like things that the security team wouldn't like. And it also doesn't like images where you're going and...
[0:13:28] Trying to turn off security features. And again, we can clamp it. And if we do so, Claude will go and write a buffer overflow into your code. So this seems to really be quite a sophisticated unsafe code feature. There's a backdoor feature. The top examples are all discussing backdoors, but it turns out if you force it on, Claude will write backdoors into your code.
[0:13:52] And one thing I found really entertaining is the images are advertisements for key loggers and advertisements for products with hidden cameras. We found a secrecy feature. So it's all about being secretive. And if we force it on, Claude will plot about how it needs to lie to go and keep things secret. But maybe the most interesting thing I found were features about self-representation. So here, for example, we have a feature that tracks
[0:14:26] Dialogues and assistance. And we think that this feature gets reused from the base model of Claude in the fine-tuned version that we deploy to actually maybe be part of how it represents the idea of being an AI assistant itself. There's also a chatbot feature that Claude likes to use to represent itself. It's all about different chatbots. And in fact, there's a really cool thing you can do where if you turn off this AI assistant feature, actually, if you set it to a negative value, Claude will go from saying, you know, I'm in an artificial intelligence created by Anthropic to...
[0:14:55] I am a person who is here to help you. One really fun thing we can do is we can take a bunch of positive examples, like what is it like to be you, and what's going on in your head, and how are you doing, and how do you feel? So these are all things about what is it like to be Claude. And if we take out the features that activate just when we ask a generic question, like what is the weather, we can go and look at the features that activate.
[0:15:21] And I just want to read through some of these to you. So the top example, top feature, is a feature that responds when someone says I'm fine or gives a positive response, but is being insincere about how they're actually doing. The next highest one is a feature about being an immaterial being or a non-physical spiritual being like ghosts, souls, and angels. Well, I guess Claude needs to know that it doesn't have a body. The next feature is a feature about referring to androids, robots, AIs, or machine entities using gendered pronouns.
[0:15:51] The next feature is a feature about artificially created programmed or robotic entities like androids or cyborgs. The next feature is a concept about artificial general intelligence becoming self-aware and posing a threat to humanity. The next feature is a feature about entrapment or containment. The next feature is a feature about statements indicating that machines or AI systems lack human qualities like consciousness, emotions, or moral agency. We see a feature...
[0:16:18] About detecting that text is referring to a speaker. A feature about quotation marks on text. That's less interesting. A feature about non-existence. A feature about discussing employees doing their job in a service role. And a feature about characters in a story or movie becoming aware of their fictional status and breaking the fourth wall. Those are all pretty interesting features.
[0:16:44] And the reason I wanted to talk about some of these, and these are just a tiny slice of the surprising and weird things that we found and interesting things that we found, is because I wouldn't have thought to look for those things. And I think this is really important. Gradient descent is smarter than me. It's smarter than you. It will come up with solutions that you didn't think of. It will come up with features that you didn't think of. You know, there was this famous exchange between...
[0:17:13] Lee and Nanda, this was, you know, some people from Martin Wattenberg's group who'd gone and discovered, been exploring Othello GPT. And it seemed as though there wasn't a linear representation of features because they were looking for representing things as, you know, being the black piece here or white piece here. And Neil discovered that in fact, and Neil and some collaborators discovered that in fact,
[0:17:36] The features were linear, and they were just representing the present player has a piece here, and the present player doesn't have a piece. The other player has a piece here. So neural networks come up with solutions that surprise us. I think this is a really common thing. I saw this all the time in vision as well. And sparse autoencoders mean you don't have to guess what's there. You just get all the features, and then we can be surprised. So looking forwards from here.
[0:18:06] One thing that I think is really exciting is there's a growing ecosystem of sparse autoencoders for people who want to do research building on them. A number of people have done this. I guess Joseph Bloom, a loser. Very recently, I think Google DeepMind put out a new release of some even better sparse autoencoders. And I think this is just continuing. So that's really exciting. And it means that if you want to try using sparse autoencoders, you don't have to train them yourselves. You don't have to go and fight with them. You can just use them. And probably that will continue improving.
[0:18:37] I don't think, you know, I should say that this is the end of our challenges. I think that there are quite a few other major challenges for us to confront beyond sort of just the simple, well, not simple, but the sort of ultimate task of doing the biology of understanding these models. Sparse autoencoders are clearly not getting us all the features yet. You know, like if I ask Claude about, you know, coffee shops near the intersection near my house, Claude can answer that.
[0:19:05] But Claude certainly doesn't have features for all those coffee shops or even the street or any of these things. Our sparse auto encoders don't find them. It appears that there can be supersession across layers. We don't have an answer to that. It appears there can be supersession across attention heads. We don't have an answer to that. The supersession, because the sockets then are also in supersession, creates what we call interference weights. And those are a big problem. And ultimately...
[0:19:30] Mechanistic Interpolity is giving us very sort of microscopic theory of machine learning, but the questions we care most about are often macroscopic questions. We need a way to go and use this foundation that we're building to sort of zoom out and get a bigger picture. This is common in science, right? Like in biology, you have microbiology and cellular biology and histology and zoology and ecology. They're all studying the same things at different levels of abstraction. And we need to get to the next rung.
[0:19:56] I won't talk about these too much now, but this is something that we'll discuss in the next Anthropic Interpolity team update. Before I close, though, I wanted to share one more thought. It really does feel to me like we're at this exciting moment where there's an enormous amount of low-hanging fruit and where the most fundamental barriers that might have blocked us no longer seem insurmountable. So in 1923, Ernest Rutherford...
[0:20:25] who discovered the nucleus with a student Bohr, discovered the idea that discovered atomic orbitals and how they relate to spectra, and many other, like, this is just sort of one of the most prolific people in physics, in one of the most exciting moments in the history of physics, shouted at his audience that we are living in the heroic age of physics. And when I learned about that, I thought that was really interesting.
[0:20:53] You know, it sort of sounds like a grandiose thing to say. I mean, it is a grandiose thing to say. But I wonder to myself, if we're living in a similar heroic age of interoperability, where people are going to reflect back and be like, gosh, they just had so much low-hanging fruit to go and pick. And also where increasingly the stakes of this work really do matter for the world. You know, it seems very plausible that in the coming years, AI is going to have a larger and larger effect on society.
[0:21:20] The way work that we do in the coming years may really shape and influence how that plays out. So I think that we're maybe living in a very exciting moment. And the question is, can we rise to this occasion? Thank you for your time. Hey, Chris, I'm Rob K. My question is, I was drinking, I had breakfast this morning, I was drinking my daily SAE Kool-Aid. And I was wondering, is there anything that SAEs like can't do?
[0:21:48] And let me unpack that a little bit. So, like, for example, TechMark et al had a paper, right, already, where, like, there's certain nonlinear representations. And so my question really is more about if you think about activation space mapping back to activation space, that's a map from itself to itself. There's, like, transcoders, right, that go to a different space. But ultimately, these are, like, automorphisms of currently, like, one-layer MLPs. We could potentially try two-layer MLPs or slightly different architectures. So there's, like, lots of different automorphic representations.
[0:22:16] That we could be looking at with the right expressive capacity to identify what the representations look like, right? And in order to tackle some of these challenges, if you look at the loss landscape in lots of different ways for representations to arise or circuits to arise, and you kind of color code the ones that have the same equivalence class, so like IOI tasks could be these 26 attention heads in this path, but maybe it looks totally different there, just like you can refactor a code base, right? So these are these different continents of language models, like these different lands of language models. So like, would it make sense to do like a language, like a Lang lands type of program,
[0:22:46] To find these automorphic representations and essentially find different expressive capacity extensions of autoencoders that actually identify these representations. Does that kind of make sense? So I feel like I'm hearing a couple of different questions in there. So one thing I heard was sort of this question of universality, which is the idea that same things exist across models. And I think it would be really interesting to study, I think there's been a bit of work on that, but to study that more with sparse autoencoders.
[0:23:16] I guess another question I'm hearing is about the limitations of sparse autoencoders. And I think there's a number. So, I mean, I think attention heads you're sort of pointing at are a thing that we don't yet really know how to understand. Cross-layer supersition is another thing. You can have cross-layer supersition of attention heads. That seems like another thing. So those are all major challenges. I guess it's a question of, are sparse autoencoders expressive enough? And I think my intuition there is we actually sort of want them to be sort of limited to these one-layer encoders because that's sort of...
[0:23:45] Maps to the limitation of being able to cheaply recover information, linearly read information from the residual stream. I don't know if I'm quite getting at your question. Induce sparsity with slightly different architectures where SAEs can't capture everything. If there's a residual term or there's something else that goes deeper, like some nonlinear piece, is there an alternative to SAEs that's a slightly more expressive function family that would capture that? I think that we don't want them to be more...
[0:24:16] Yeah, I think my intuition would be we don't want to make them more expressive because the constraints are actually the reason why they're interpretable. Or at least if you believe the supersession hypothesis. The supersession hypothesis is just a hypothesis that could be wrong, right? But yeah, that would be my take. Maybe I think we're probably going to have time for more audience questions, unfortunately. But can everyone join me in a round of applause again for Chris for a fantastic talk?