[0:00:00] Welcome. Today I'd like to talk to you about circuits, an approach to mechanistically understanding neural networks. What we mean by mechanistically understanding neural networks is understanding the underlying mechanisms, the underlying algorithms that drive neural networks. Put another way, if we can mechanistically understand neural networks, we should be able to reverse engineer them. This isn't generally what interpretability aims to do, right? So interpretability research, I think a relatively small fraction of it is targeted
[0:00:32] At reverse engineering neural networks. And to help us think about this, we're going to go and have this sort of toy neural network diagram here to think about. A lot of interpretability research is aimed at trying to understand, you know, what inputs influence the output the most. You know, if an image gets classified as a dog, maybe we can say that the dog head was the important part or something like this. We also see research that looks at hidden layers.
[0:01:06] And it asks questions like, what can we predict from this hidden layer? There's some variable that we think is interesting. Can we predict it from that layer? So that's the probing work. We also see work where people look at individual neurons and try to go and say, maybe this is a car detector, or maybe this is a dog detector. And when we do this, we often see many neurons that don't seem meaningful as well. And all of this is, I think, really exceptional research that targets lots of really important questions.
[0:01:38] But it doesn't really feel like it's getting at reverse engineering a neural network. And it's not even maybe clear exactly what that would mean. And so when I think about this, I often think of understanding neurons in a neural network as being kind of like understanding the variables or the registers in a computer program if you're trying to reverse engineer it. So imagine if I gave you some code that looked like this. And I told you there's a variable in the computer program.
[0:02:10] That would be an interesting step towards reverse engineering a computer program. But you wouldn't say that having this code means that we reverse engineered it. We sort of have some higher expectation of what it would mean to reverse engineer the computer program.
[0:02:41] And I think that really what we need to look at in addition to the neurons is the weights. The weights are kind of like the assembly instructions of our neural network. So just like the assembly instructions in the code might go and change the states of variables to the next state of variables, the weights go and change our neurons in one layer and the values they take to compute the values in the next layer. So the weights are almost like assembly instructions for our neural network.
[0:03:13] Excuse me, I'm mumbling on my words. In order to reverse engineer neural networks, we're going to need to understand the weights. But to understand the weights, I think we need to understand the neurons that they're going and acting on. And so we have to go and do these two things. You can imagine two very difficult steps to understanding a neural network and reverse engineering it. First, understand all of the neurons, and then understand all of the weights connecting all of those neurons.
[0:03:45] That's going to be our general goal today, and we won't be able to completely succeed, but can we first try to understand the neurons, and then can we understand the weights? So our goal here is going to be, in this first step, is going to be able to go and assign some kind of meaning, if possible, to every neuron. So here we have our cartoon neural network again, and we've gone and we've placed some images, and I'll talk about these in a second.
[0:04:13] Maybe this one is a square detector, and this is a line detector, and this is a wheel detector, and this is a car detector, and this is a dog detector of some kind. So that's our goal. There's actually significant literature on people trying to understand neurons and neural networks. And they often find that there's neurons that track what we would think of as the meaningful latent variables in an image, the presence of a dog or maybe the presence of an eye.
[0:04:45] Or the presence of a car or something like this. And there's also a more skeptical literature that often suggests that this isn't what's really going on. And so we'll think about that a little bit more in a second. I think something that's important to say here is that machine learning doesn't really have a tradition of going and evaluating claims like, oh, this is a car-detecting neuron. We often evaluate claims about whether a model
[0:05:15] Is it accurate? Or maybe we go and evaluate claims about theoretical proofs. But this is something that's a little bit different. It's a claim that's more similar to claims we see in neuroscience. And so I think it's not that surprising that when we have claims that are very different from the claims we usually evaluate, that we see people disagreeing on them. So just as an example, there's a neuron in inception D1.
[0:05:46] Mixed 4B409. And a question you might ask is, is it a dog head detector? And if we look at examples that cause the neuron to fire strongly, it certainly seems like it could be a dog detector. Those are dog head detectors. Those are all dog heads. But there's some other explanations you could have. You could imagine it being an eye detector. Maybe it's a dog eye detector. Certainly you're not going to see a dog eye without a dog head.
[0:06:11] Or maybe actually this, you can imagine there being some special kind of fur, there's some special fur texture that actually is what's going on. So how do we know that's not what's going on? Well, it's a good question. And one possible answer is that there's this very nice approach technique called feature visualization, or sometimes activation maximization. And what you do is you take a neuron and you start with some random noise.
[0:06:39] And you do gradient descent just to cause that neuron to fire. It's a few other small tricks, but basically you just do gradient descent to cause that neuron to fire strongly. So you create a stimulus that really, from scratch, that causes that neuron to fire. And here's one for our dog head. And it's really suggestive that actually this really is something that's more like a dog head detector. We can see that there's this
[0:07:09] Snout here, and maybe there's a little bit of a red area that maybe it could be a tongue, and there's some eyes above it and fur around it, and so it seems like it's really got some global structure and it's looking for different components, and it seems like something that's maybe a little bit closer to a dog head detector. So feature visualization is really helpful in going and establishing, you know, separating out a neuron being correlated with some feature, and establishing really a causal link.
[0:07:39] Because if the optimization process produces eyes and produces a snout, then the only reason those would be in the result is if they really cause it to fire more. Now, often in this talk and in other work, we use feature digitalizations kind of as a placeholder for neurons, a variable name for neurons. And the reason we do that is that looking at a neuron, 4b409, just isn't very memorable. It's hard to remember what 4b409 is.
[0:08:10] But a feature visualization like this is very memorable and very suggestive of what the underlying feature is. And so we often use them like variable names. We don't mean to say that the feature visualization tells us everything about a neuron. It's just a really good placeholder for thinking about that neuron. Now, if we want to go deeper, there's actually a lot we can do to go and interrogate neurons more deeply and really understand what they're doing. And there's actually a lot we can learn from the neuroscience literature here.
[0:08:40] So here, there's 3B379, which is a curve detector, or it appears to be a curve detector. It detects curves, either lines that are curved, or transitions in color sometimes as well. And in neuroscience, when you have neurons like this, people are very often interested in seeing tuning curves, where you go and you show a bunch of synthetic stimuli, and maybe you, you know, let's, one thing you might ask is, how does it change if we rotate the curve, if we change the orientation?
[0:09:10] And here we can see that it's got sort of a nice bump around its preferred orientation and then stops firing. We can do more. We could go and actually vary two axes. We could vary both the orientation and the radius of the curve and see what happens. Or we could even go and change from having sort of a line, a black line and a white background, to going and having color transitions and try, you know, going and cycling through different colors.
[0:09:39] My video is actually going and my webcam is actually blocking this for you, but the red here is plus 25 standard deviations. So it's actually just firing ridiculously strongly for these stimuli. In any case, we could actually have an entire talk on characterizing neurons and characterizing features. I'm not going to dive that deep into it because I think I want to go and move on to some other things.
[0:10:10] But this is actually a really deep area. And I think how can we really know what a feature is and really be confident that we've correctly understood a feature is a really big topic. But I'd like to instead just transition to talking a little bit about the features we observe in these models. And then we'll talk later about the weights and what we call circuits that build them up. So we'll talk about one particular model, Inception v1, which at one point won ImageNet.
[0:10:42] And it's just a model that over the years we studied in a lot of detail and know a lot about. In fact, we're now at the point where we understand literally thousands of neurons, or at least believe we understand thousands of neurons in Inception v1. It's got about 10,000 neurons. We think we understand about 3,000 quite well. And in the early layers, that's actually more like maybe we understand like 80% of the neurons over here.
[0:11:10] And these neurons, as you go through the layers, you see lots of really interesting things. So early on, you see these color contrast units that look for different pairs of colors on different sides of the receptive field, or line detectors, or these tiny little curve detectors, or brightness gradients that look for lighter areas on one side and darker areas on another side. That's actually useful for 3D geometry detection and lots of other interesting things. And then later on, of course, we see lots of textures.
[0:11:38] Color center surrounds. I'll talk about these. These are actually maybe one of the most interesting things, these high-low frequency detectors. We'll talk about them more in a minute. All sorts of interesting things, though, like these black and white versus color detectors, these curve detectors, line detectors, fur detectors, various kinds of eye detectors, just all sorts of really interesting things. And a lot of them are, I think, just natural features that you might have expected. So there's these curve detectors.
[0:12:06] And you might have expected, if I'd sat you down beforehand and been like, you know, what kinds of features might exist in a neural network? You might have guessed, oh, you know, it would be useful to have curve detectors, things that go and detect, you know, we know that there's line detectors in neural networks. People talk about Gabor filters all the time. You know, maybe there'll be curve detectors. But there's also more surprising things. So there's these high-low frequency detectors, I think, are one of the most interesting features we found. And they look for low frequency patterns on one side,
[0:12:36] And high frequency patterns on the other side of the receptive field. And they actually play a really important role in detecting boundaries of objects. We'll talk about that more when we get to circuits. But it seems like they play a really important role in detecting the boundaries of objects. Now, just so I don't give you an overly cheery impression, a lot of neurons are what we call polysemantic, which means they respond to multiple unrelated features. And this actually becomes more common as you go deeper into the network. So this neuron here.
[0:13:07] In layer mixed 4E, it goes and responds to cat faces and fronts of cars and cat legs. And so you can also see over here, I guess, some fox faces as well. And so it's responding to a variety of different things. And these are feature visualizations. So they actually tell us that it's not that there's some shared thing between these. It actually really genuinely is looking for cat faces with whiskers and eyes and a cat nose.
[0:13:33] And shiny fronts of cars and lights. It's really looking for those three different things. Okay, so we've talked about features so far. And a question you might ask, and I guess sort of really the exciting part of this talk, I think, is how can we go from understanding features to understanding weights and understanding sort of the actual mechanisms that drive neural networks, the actual algorithms that are computing those features we were looking at. So if we return to our cartoon neural network, we have the neurons, but they were all connected.
[0:14:05] By weights, we want to understand those weights. So normally, the only time people really look at weights is when they look at the first conflare. So very often people will visualize the weights in the first conflare, and they'll see, oh, you know, we have some nice bore filters, we have some nice color contrast units, and, you know, we see all these interesting features. But people very seldom look at neurons and later layers other than maybe looking at statistics.
[0:14:31] Sorry, look at weights in later layers, other than maybe looking at sort of aggregate statistics of the size of the weights or something like this. And I think an interesting question is, why do they do that? Why do we very often look at the weights in the first layer, but almost never look at the weights in other layers? Well, weights in the first layer are contextualized. We know what they link to. We know that they connect to red, green, and blue values. And so it's actually very easy to think about what those weights mean.
[0:14:58] And in fact, we can just visualize them by turning them into images. But when we look at weights in other layers, we can't do that. We can't go and look at the same structure. We can't easily go in and just visualize them like this. We haven't contextualized them. We don't know what the weights are connecting to. And so in order to understand the weights, we need to contextualize them. And thankfully, that's what we've been doing in the last...
[0:15:25] In the last section, if we've been understanding the neurons, then we can understand the weights connected to those neurons. OK, so here is a car detector. It seems to really like cars. You can see over here, it's actually quite a large, global, coherent car detector. It understands different parts of a car. And it's produced by going and doing a 5 by 5 convolution over the previous layers. There's lots of weights.
[0:15:54] That are connecting to different channels in the previous layer. And let's just pay attention to the largest ones. We'll look at the largest three. So it turns out that the largest three connections are to a window detector, to a car body detector, and to a wheel detector. OK, so that seems really promising. Just that alone, actually, that the strongest connections are to features that we would expect to be constituents of a car.
[0:16:24] But we can look a little bit more closely. So remember that because it's a continent, we don't just have a single weight. We have a confilter for each one of these pairs of neurons. And this tells us how the window affects the car in different relative positions. So if it's sort of above the center of the new car detector, it causes it to fire. So it's kind of really excited and supported firing. But if we see a window below, that actually inhibits it. So it wants to see...
[0:16:53] The window, but only at the top. And conversely, it wants to see the wheel, but only wants to see the wheel at the bottom. It doesn't want to see at the top of the top, but wheel inhibits it. And then the car body excites it a little bit everywhere, but mostly at the bottom. So here we've got something really interesting. We actually are seeing the global structure of the car correctly reified in the weights of the neural network. We know that windows should be at the top and wheels should be at the bottom.
[0:17:23] And so this obviously isn't a complete algorithm for detecting a car, but it's actually something that I could have written if I'd been trying to write a car detection algorithm as a neural network. At the final stage, I might have said, OK, well, the windows should be at the top and the wheels should be at the bottom, and that's how things should go and be pieced together. So that's kind of neat. Now, that's just one circuit. So let's go ahead and have a look at some more circuits. Just starting really early on.
[0:17:55] One question you might ask is, OK, we sort of know the kinds of things that happen in the first con player. Basically, every neural network you look at will have Gabor filters and color contrast units in the first con player. And what happens after that? Well, it turns out that very reliably in the next con player, if you have a neural network with a 1 by 1 convolution, and neural networks with larger convolutions, it's actually a bit of a different picture. But in neural networks with 1 by 1 convolutions, you'll see what we call complex Gabor filters. And this is actually inspired by
[0:18:25] Neuroscience, where people see very similar features. And what these units do is they take multiple Gabor filters that are in the same orientation, and they combine them together to create one unit. So a Gabor filter is really, you know, they detect edges, but they're incredibly sensitive to alignment. They only fire when an edge hits the pixels in exactly the right way to line up with the Gabor filter. And so by combining a bunch of Gabor filters, you can create these
[0:18:54] Complex Gabor filters, which are much more general and flexible. So we see a bunch of those. And sometimes you add in some slightly different orientations. And sometimes you'll see inhibition. This is usually weaker. This one is usually strong. To Gabor filters, from Gabor filters in other orientations. This one is looking for tilted this way and inhibited by ones tilted this way. One other thing that's worth noting here, or maybe worth emphasizing, is that Gabor filters very often come
[0:19:25] In reciprocal pairs, so look at this one and this one. They're negative versions of each other. This one detects black on the outside and white on the inside, and vice versa. Or this one looks for purple here and green here, whereas this one looks for green here and purple here. And so these reciprocal pairs of Gabor filters are really helpful for the complex Gabor filters being robust to different shifts or changes in color in the edge. Something that we see a little bit later on, which I think is really cool,
[0:19:56] So we see these curve detectors form. And then this is actually the first circle detector in Inception v1. And what we can see is there's all these little different curve detectors, and they get pieced together into a circle detector. So this curve here is tilted like this, and it goes in the upper left-hand corner. It's an upper left curve, and it goes in the upper left corner. And so on and so on. This one's in the right, and this one's here. Actually, excuse me, there was one earlier very crude circle detector. So this is the second circle detector.
[0:20:23] And here's another curve detector. They're all being pieced together to create this nice tiny circle detector by going and piecing together things. These are built from the complex Gabor filters that we saw earlier and then they get pieced together to create this nice circle detector. We also actually see a triangle detector. And so what we see here is we see a bunch of line detectors which were built from complex Gabor filters.
[0:20:50] In different orientations, and so there's some horizontal ones, and we look for the horizontal ones at the top to create the horizontal line. And we look for these tilted ones here on the left at the bottom, and these tilted ones here on the right. And so you're just piecing together these line detectors to go and build a triangle. Now it actually turns out this triangle detector is primarily used not when all three of those are present, but when two or even one of them are. So it's maybe a little bit better to think of as a multi-line detector.
[0:21:19] Or as a multiple angle detector. But the maximal stimulus is a triangle. One of the most beautiful things we found is the curve detection circuits. At least one of the most beautiful things in early vision. So what you see here is you have these early curve detectors. This is actually the second iteration of the curve detectors. And these more sophisticated curve detectors. And what happens is you go and you look for a curve.
[0:21:48] So it's kind of like looking at every point along here for a little tangent curve to go and create it. And it's not just for that pair of curve detectors, but all of the curve detectors look for another curve in the same orientation as them and go and look for a curve of curves. And at the same time they're inhibited by curves in the opposite orientation. So at every point they're looking not for curves tilting the opposite orientation, but for curves tilted in this orientation. That's it.
[0:22:20] It's a very nice, simple, understandable way to refine curve detectors. It's a meaningful algorithm. Now, another thing that I think is really interesting is actually there's these black and white detectors, these absence of color detectors. And what they work is they just look for a whole bunch of color units. And they have a giant 5 by 5, just all inhibitory convolution on top of those where they look for their absence. And then they have a few more neutral units that cause them to fire.
[0:22:49] And so we get these absence of color detectors. There's this one. There's actually a bunch of absence of color versus color contrast detectors as well. And it seems like these are really important for the later network sort of understanding of color. So early on in the network, whenever you see a color unit, it's usually a color contrast unit. It looks for the presence of one hue and then the absence or an opposite hue or the absence of that hue. And so it's sort of regularizing.
[0:23:18] Remember, the image could be lit differently or different things like this. It can be brighter or darker. You need to somehow normalize. And so normally you see sort of contrasting against an opposite color. But after this layer, we actually see lots of units contrast instead against the absence of color. So it'll look for a particular color, and then it'll look for the absence of color in other regions. Another really interesting unit, there's actually a whole bunch of what we call boundary detectors.
[0:23:48] And these very flexible, you know, people talk maybe sometimes about line detectors. Boundary detectors are much richer. They look for, you know, the transition from, say, me to the background or things like that. So this one's a horizontal one. And they look for a lot of different cues. They're very sophisticated. So they look for transitions from low frequency to high frequency, high frequency to low frequency, or vice versa. They look for lines, of course. They look for changes in color.
[0:24:17] From between the two sides of the boundary. And actually, this is a really interesting unit, which is an end-of-line detector, or a sudden change in line or angle. Sometimes if it just slightly radically turns, that also works. Where it looks for a line terminating at the boundary. And so it has all these really interesting cues, and then it just looks for them along this stripe. Higher up, we start to see really interesting things.
[0:24:48] How do, you know, the network has to be able to detect dogs that are tilted in different ways or oriented in different ways. And it turns out there's two completely different pathways for detecting dogs facing right and detecting dogs facing left. And it starts with these oriented fur detectors, which detect fur, you know, just like fur on my hair parts out from a particular way, sort of like this. They detect fur parting in a particular way. And that gets refined into dog heads.
[0:25:18] Short snout-head-dog-head detectors and long snout-dog-head detectors. So we see both of those. And then we see actually these units that add on a dog neck. So you take the head detector and you glue on some fur and some boundaries to create a neck. At the same time, we actually see these units which union over both of these. So they add up both of these left and right detectors to create these pose-invariant dog-head detectors.
[0:25:49] And then more sophisticated ones that add on the neck. So we're actually really seeing something. And notice that at every step, these pathways inhibit each other. So it actually creates kind of an XOR-like structure where by making these two exclusive and then going and unioning over them, you go and you get an XOR. And of course, we can look at that in more detail, right? So every one of those connections is a 5x5 convolution. And we can look at it. And what we see here is the dog head is getting glued on.
[0:26:19] So it's looking to the right, and it wants to see it over here, because the neck's going to go there. And vice versa, when it's looking to the left, it wants to see it on the left-hand side, again, where the head is here, and not on the right-hand side. And so there's all this beautiful structure in the weights. These sort of union units are also really beautiful. So here's one. And we can see that it's excited in the center.
[0:26:49] By all four of these units, both the short and the long snouts, both facing left and right. But notice how for the ones facing right, they're a little bit also excitory on the left, because they converge in to the middle. And these ones also are a little bit excitory on the right, and then they converge into the middle. So again, just beautiful structure there. Now, I want to talk about one other sort of surprising and interesting thing.
[0:27:18] Which is, sometimes we'll see neurons, so remember that car detector that we saw earlier that was beautifully built from a wind detector, a car body detector, and a wheel detector. So something that we see is it gets spread over, has a bunch of small weights, but spread it over a bunch of dog detectors. That's a very strange thing to do. We had this nice car detector, we built it up really carefully, and we smeared it over a bunch of other units. Why are we doing that? A bunch of other units that don't seem to be primarily detecting cars. Why would we do that?
[0:27:48] Well, we don't know, but our theory is that it's a space-saving function. So it's by spreading a neuron or a feature over a bunch of neurons, you can reconstitute it later by going and adding up that combination of neurons. And so instead of dedicating an entire neuron to something that maybe you're not working with in that layer, you can sort of put it in what sometimes we call the superposition. We put it in superposition over all these neurons, and we can recover the feature later. And so that can be a really powerful mechanism.
[0:28:18] We think for going and saving space. And it seems to be, well, at least we think that's why we observe all these polysemantic neurons that fire from multiple things. So the thing that I really want to emphasize here is that neural networks are full of beautiful structure if you take the time to look. And I think this is just a really, I don't know, if there's a single thing, and I think at this point I spent maybe six years just honestly primarily looking at a single neural network.
[0:28:49] And try and understand it. And I think, you know, I just think that something that's really not appreciated is if you're willing to look into the details of how these neural networks work. There's a lot of really incredible structure lurking under the surface. So I think that's worth looking at. Now, one question you might ask is how do we know we aren't fooling ourselves, right? It seems like we're staring at all of these weights and we're staring at all of these
[0:29:19] You know, all of these neurons. And it seems like it might be very easy to fool ourselves. And of course, there's some basic things we can do. We can try to be really rigorous in making sure that we, you know, when we talk about a neuron, we've really carefully investigated it, especially if it's something that we care about a lot. We can also, you know, we should be making sure that we look at large weights and not cherry pick, you know, or at least we should, you know, maybe there's small weights that we think are also important, but we should be making sure that we carefully look at the large weights and that we don't ignore
[0:29:49] Don't paint a picture by cherry picking. But I think there's some additional things we can do. So one really interesting thing that we can do is we can edit circuits to change model behavior. And if we've correctly understood a circuit, correctly understood all the weights connecting neurons together, we should be able to predict, at the very least, how that circuit changes in its behavior, and ideally also how the overall network changes in its behavior. So this is something that we haven't explored that much yet, but initial results seem pretty promising.
[0:30:20] So for example, we have these left and right doghead pathways, and if we knock one out, so if we knock out the left-facing doghead detector, all of a sudden, we start observing errors for things facing left that are no longer correctly classified. And of course, some other things fail to classify, but prominently we see things facing left suddenly aren't classified correctly. So this kind of editing seems like a really powerful mechanism for verifying that we
[0:30:50] We understand things correctly. Another thing that we can do is if we've really understood something, you know, if we've really reverse-engineered something, we should be able to go and do a clean room reimplementation. So we've actually done this for curve detectors where they're about five layers deep, and we just...
[0:31:09] We went and wrote a Python program to go and fill in all the weights based on how we understood it. So we take a neural network where all the weights are zero, and we go and create Gabor filters. Then we go and build line detectors from those. Then we go and build early curve detectors from that. Then we go and we build the mid ones. And then we refine them just like we were looking at earlier with those weights that are curves of curves. And so on the left, we have our natural curve detectors. And what you're seeing here is how
[0:31:40] For different curves in different orientations. So we see they go from left to right as they select for the curves in that orientation. And over here, you're seeing feature visualizations of our handwritten curve detector. And we see that we correctly create feature visualizations, and we see that they fire for the correct orientation. And so it seems like we have understood something pretty substantive about these. If we can just go, and actually this was my first attempt to go and write out the weights.
[0:32:11] And it just worked. I'm sure these neurons aren't as robust or aren't going to be as invariant or things like this. But to a significant extent, we correctly understood it and could write weights that implement that behavior. Now, I think there's a lot more work to be done in this space. And I wanted to just briefly cover some future directions that I think I'd be really excited to see people look into. So first of all, I wanted to sort of pose
[0:32:41] You know, a challenge that I think, it motivates me a lot, and I think it's a challenge that I'd really like to see other people work on. Which is, can we understand, fully understand, a single neural network? Because I think right now, when people think about interpretability, in general, people aren't even aspiring towards this, and I think people generally don't think this is possible. And I think it would be really exciting, and I think if we could just demonstrate, even if it took a ton of work,
[0:33:12] That a single non-trivial neural network was completely understandable to humans. I think that would be a really big deal. And I think an obvious target for this is Inception V1, we've done a lot of work on that, but we still have a long ways to go. And I think if it was a different neural network, I think that would also be totally fine. Beyond that, there's a couple open questions that I think, that I'm really interested in. So first of all, when do meaningful features form? When do features that track meaningful
[0:33:44] And I think the general picture that people have, you know, the picture a lot of literature so far has painted is that these are the exception. But when we look at strong vision models, I think actually what's been going on is that a lot of the time people are looking for the wrong features. You know, they're looking for features like cars and dogs that are things that they expect to be there, but they aren't looking for things like high-low frequency detectors in different orientations.
[0:34:13] Which they hadn't guessed were there. And so I think there's actually a lot more, and people really underappreciate the extent to which typical neurons and strong vision models, or vision models just trained on image net, trained on hard visual tasks, seem to often be interpretable or meaningful. But this isn't true for all models. So if you look at RL models trained on entire video games, this generally isn't true. If you look at the neurons
[0:34:43] As far as I can tell, they just generally aren't meaningful. And so that's, you know, they don't track, you often don't even see the war filters. They just don't seem to track meaningful variables in the way that we think. So why does that happen? Why is it that ImageNet models are different than Atari models? I think that's a really interesting question. Another really interesting question is, you know, what's up with these polysematic normals?
[0:35:08] I told you the hypothesis that they're a space-saving phenomenon, but we don't know that's true. Why do these polysemantic neurons form? How widespread are they? We don't have any sort of automatic way to detect them, so no one knows. We know that many neurons are polysemantic, but there's no metrics on it. We don't know anything about how architecture affects it. And then I think the biggest question is, how can we understand neural networks despite polysemanticity?
[0:35:34] Or get rid of it. So how can we get around polysemanticity? Can we either build neural networks that don't have polysematic neurons, or find a way to reason about circuits and about features in a network that do have these polysematic features? And this is very much a question that's related to the disentangling literature. So lots of people are interested in disentangling representations, usually in the context of generative models and a set of latent variables that they've already decided they're interested in.
[0:36:02] I think this is a much more general question, obviously, and I think these polysematic neurons are really asking the same thing. To what extent do the same features form again and again across models? So, for instance, we talked about curves earlier, and it turns out that curves you can just basically find in every ImageNet model. And I think that's really interesting, because if curves form, what else forms reliably across models? And to what extent...
[0:36:29] Are models just largely converging on the same features and the same circuits, just scaffolded differently across different models? And maybe somewhat, obviously, if models are more accurate, they have to be somewhat different. But maybe are they just adding on new features? Maybe there's just a frontier of features that we're gradually adding on? But to the extent there's some kind of universal set of features that these models are converging on, that would be a really profound thing, I think.
[0:36:57] It would really, I think, change how I think about what the important questions are here. I guess another way of thinking about this is I sometimes think about the study of features and circuits in neural networks as being kind of like anatomy. People spend lots of time in biology trying to understand the anatomy of different organisms, and we're trying to understand the anatomy of neural networks.
[0:37:23] If every animal had a completely different anatomy that was completely unrelated from other animals, I'm not sure that people would invest a lot of energy in studying anatomy, other than maybe human anatomy and a few other organisms that are farm animals or something like that. But because of evolution, because a lot of anatomy is analogous between organisms, in fact we have a rich field of comparative anatomy.
[0:37:48] People studying the similarities and differences in anatomy between different organisms and how that relates to evolution. And so if features are broadly universal, and there's a lot of commonalities in the features that get learned and the circuits that get learned, you can imagine a really rich sort of comparative study of features and circuits. And I think that would be really, really cool. You can imagine almost a periodic table of features that all these models are converging on in the strongest version of that.
[0:38:17] Another really interesting question, I think, is how do the features, how does the architecture or the training procedure or things like this, maybe especially sparsity or adversarial training, how does that affect what features form? How does that affect what circuits form? How does that affect how they're organized throughout the model? I think those are really interesting questions. How do features and circuits evolve during training? So far, we've only really looked at the final converged model. But what features converge first?
[0:38:46] Maybe one of the most remarkable things that we've discovered is these parallel paths for left-facing dogs and right-facing dogs. Did those form only at the end of training? Did those pathways form really early on? Were there sort of like dog head detectors that were just a jumble of left and right and didn't care where eyes and stouts were, and then they gradually split into these two pathways? And obviously that's only one question you could ask. There's a huge number of questions you could ask. So I guess if we were thinking about...
[0:39:14] Earlier about comparative anatomy, this would be more like developmental anatomy of neural networks or something like this. In some areas of biology, systems biology, there's a lot of talk about circuit motifs, which are recurring patterns that you can use to go and study things like transcription networks. So you have a bunch of objects that are excitatory and inhibitory, and you find these recurring patterns that you can sort of think of as higher level abstractions for reasoning about your system.
[0:39:44] If there are things like that, and maybe something like uniting over cases, like uniting over the left and right case of the dog head detector, maybe that's a candidate, for instance, for a recurring high-level pattern that might exist in networks. That would be really exciting. Unfortunately, the actual tools that get used to study this tend to require networks to be sparse. But you could imagine there being generalizations of these techniques that could really give us traction on studying circuits.
[0:40:14] Related to this, motifs would be a relatively small thing, but you could also imagine there being really interesting large-scale structure and organization among features and circuits. So one example of this is there's something we call branch specialization, where if a network has branches, like AlexNet, for instance, has this, and InceptionV has this, where there's chunks of neural network in parallel that don't talk to each other for a little bit. They often specialize. So if you have a branch in the first layer, you'll see specialization in color versus black and white.
[0:40:44] Black and white edges. Or later on you'll see there's actually a branch in InceptionDB dedicated to curves, or largely dedicated to curves. Or another one to 3D geometry detectors. And so that's an example of large-scale structure, and it's also an example of how the architecture of a network might influence the organization of features in circuits. I think our final question is,
[0:41:12] Is there some way that we can scale this? So right now, this is incredibly tedious. We go through neuron by neuron and try to understand them, then we look at the weights connecting them, and we try to back out a big picture understanding of this model. And how can we scale that either to much larger models, because neural networks just keep getting bigger, or to large numbers of models? So one thing that might be promising is finding hierarchy.
[0:41:40] There's some promising results by Daniel Filan and his colleagues on finding modularity in neural networks. And I guess branch specialization also points in a kind of similar direction. If you could find hierarchical units that you could understand independently, that would seem to make things easier. Another thing is if we could diff neural networks. So especially if universality, if we see these same features forming again and again and same circuits forming again and again, maybe you could go and work with more models by only looking at the deltas between them somehow.
[0:42:09] So Gabriel Goh has done some new work on this, and you can reach out to him if you're interested. But I think there's actually a lot more work to be done here, and just, you know, how can we go, we've really zoomed in on understanding the fine details here, how can we sort of now go and preserve those benefits, but zoom out and get a big picture view? So those are some open questions that I think are really interesting. Now we actually...
[0:42:38] aren't the only ones working on this, and we're really lucky to have had a community forming around these questions. So first of all, the results I talked about today you can find on Distil in the Circuits thread. This is a collection of essays that are earlier stage work exploring these questions. There's also a Slack channel, so you can just go to slack.distil.pub and join hashtag Circuits. There's about 300 members, 20 people who are quite active, about 10 people actively working.
[0:43:09] on projects related to circuits from multiple institutions. And there's multiple unpublished drafts that you can go and read there. There's multiple open source libraries, TensorFlow Lucid and now Lucent and PyTorch, implementing the underlying techniques needed to go and do this kind of research. And you can find Colab notebooks demonstrating all this again in that channel. And OpenAI, and I'm at OpenAI and a number of our colleagues, have been building the OpenAI microscope.
[0:43:40] Which you can use to go and quickly explore models and not even need to go and open up a library to go and visualize them. So a lot of these, for many models, many standard models, are just either visualizations available on the web and you can just browse them and you can share links to each other, you know, different things that you've found and it's a lot of fun. Finally, I just want to remind people that, you know, it's easy because I'm the one talking to get the impression that this is primarily me. But I'm incredibly lucky to be working with lots of other people. So, of course, there's lots of people on the Circuits channel.
[0:44:13] I've specifically been really lucky to work with other members of the Clarity team, Ludwig Schubert, Nick Camerata, Gabriel Goh, Michael Petrov, and Sean Carter. And then lots of friends at other organizations. Arvind, Ian Catherine, Alex Morgansdow, Stan Armstrong, Maria Cleary, Brice Menard. And so really, this work is a result of lots of different people, and hopefully even more people in the future.
[0:44:43] Thank you so much. I wish that I could take questions. Unfortunately, this format doesn't really allow that. But thank you for taking the time to listen, and I hope that you're doing well.