[0:00:00] Sometimes people ask me questions like, have I studied the neurons in a transformer's residual stream or the neurons in the value vectors? And I think this isn't quite the right way to think about it. I think it doesn't really make sense to go and even, yeah, I think that question just doesn't really make sense. And so I wanted to sort of lay out why that seems to be, why that's the way things seem to me. And the heart of it,
[0:00:30] is a distinction I'm going to make between what I'll call a privileged basis and a non-privileged basis. And the idea here is that sometimes a neural network representation has a special basis that is particularly meaningful to study, and sometimes it doesn't. And that really changes how we have to approach interoperability. And so even though a lot of my work has involved studying neurons, you can only really do that when you have a privileged basis because
[0:00:58] Usually what it means to me to have a neuron is that the neuron is the value of a vector on a particular privileged basis. So a privileged basis is usually created by having a non-linearity in a neural network, especially ReLU. ReLU, because it's such a sharp non-linearity and because it creates sparsity, tends to be especially good at
[0:01:30] Aligning meaningful features with the basis dimensions of your representation. But you can also end up with a privileged basis if you have perhaps some regularizer that's enforcing sparsity in the resulting activations. So you could have a linear transformation, but if you were regularizing the output, you might have a privileged basis. Or if your weights maybe had some kind of L1 sparsity on them, that might create it.
[0:02:00] So there are other ways that you can get a privileged basis, but the most common reason would be that you have a non-linearity, and especially a strong non-linear function. Now, when you do that, the result is that you're incentivizing features to align with that basis. And an example of why you might think that's the case. So yeah, why is it that we're going to have features align with the basis? Well, if you go and you...
[0:02:31] If you imagine having a representation that's trying to encode two features and you have two ReLU neurons to do that, one way you could do it is you could align the features with the ReLU activations, or you could go and do something like have one be the average of the features, one neuron have the average of the features, and one be the difference. Well, if you do that, the point at which the features transition from firing to not firing in each case is dependent on the other variable.
[0:03:01] If you try to recover the variables afterwards, that's going to inject a lot of noise. And that's going to really incentivize you to go and align things with the basis. So yeah, when we have that kind of privileged basis that encourages features to align, then we'll call the basis dimensions neurons. But usually we won't call them neurons, or at least I usually don't call them neurons if they don't have that. So for instance, if you have word embedding where you don't have a privileged basis, I usually don't think of the...
[0:03:31] The basis dimensions there as being neurons. Okay. Now, on the other hand, a non-privileged basis forms, it's sort of the default thing. So if you don't have a nonlinearity or some kind of regularization, you should expect to not have a privileged basis. And that means that you shouldn't expect looking at some dimension of that to be really any different from looking at a random projection or at least a random unitary.
[0:04:01] So, yeah, why should we care about this? You know, you might say, well, okay, you're sort of making this distinction. Why does it matter? Well, it matters because one of the best tools we have for understanding neural network representations is to have a meaningful basis that breaks it down and where we can study each component independently. Really, any neural network representation is going to have an enormous amount of volume.
[0:04:30] It's, you know, as you increase the number of dimensions, you obviously, the cross-dimensionality sets in and you have just exponentially much volume potentially in that representation if you wanted to try and study it. And so the key way, the key thing that goes and saves us from the cross-dimensionality and allows us to understand the entirety potentially of a neural network's representation is if we have a basis where we can study every dimension independently. And the easiest way to go and get a basis where you can study things independently would be if you have
[0:05:01] So in Transformers, which parts have a privileged basis and which parts don't? Well,
[0:05:34] The MLP neurons have a privileged basis. So if you're familiar with the transformer architecture, every block you have attention layers and you have MLP neurons that both branch off from the residual stream. And the MLP neurons are going to have a gelu or a relu activation function on them. And so that's going to cause them to have a privileged basis. They're actually, they're also higher dimensional than your residual stream. So that's going to be actually an extra thing.
[0:06:02] Like if there's polysemanticity going on, that's going to make them extra nice. And so MLP neurons, at least in theory, all the theory points towards them being great. Now, in practice, if you look at our, I think probably the next video we'll discuss in a little bit, they might be, so far they've been pretty tricky to understand in my experience. But in theory, they definitely have a privileged basis. And in theory, they seem like they should, a lot of things point in the direction of them being very nice. Another one that we often don't think of, but I think actually...
[0:06:32] is is is i guess it's the most privileged basis in some sense is the the tokens because we we often represent tokens as one-hot vectors um both when we input them and then when we output them and we have our logits or our softmax we have probability distribution over tokens and every dimension in that basis uh is completely interpretable because you know they just correspond to these tokens that are in our writing system that are basically designed
[0:07:00] Well, at least the writing system is designed for us to understand it. So that's definitely a privileged basis and very different from a random projection of that space. Now, another one that it's easy to sort of forget about is attention patterns. So for a given pair of tokens, you have your dot product of the key in the query, which is the score. And then after you apply a softmax across the tokens, you get a probability distribution. And because those are
[0:07:28] are one-dimensional in some sense. The distribution is multi-dimensional across tokens, but there's only one dimension per token. That really encourages you to have, even for the scores before the null-linearity, to have a privileged basis. Or at least, I mean, I guess if you only have one dimension, you're a privileged basis by default. And then Softmax is very non-linear, so that's also going to encourage, in some ways, you to have a privileged basis. On the other hand,
[0:08:01] The residual stream. So from the moment you do your first token embedding, you take your tokens and you multiply them by a matrix or look up a vector, there is nothing about that that is privileging any dimension in that operation. And when you go and you add things into your residual stream at every step,
[0:08:22] In your model, you add the outputs of the MLPs, you down project your neuron activations and add them into the residual stream. Or if you go and you take the outputs of your attention heads and project them into the residual stream and add them in, all of that is just, you know, you could potentially do an arbitrary linear transformation and get the same results. So your residual stream is not going to be privileged. It is not a privileged basis in the default transformer architecture. Now,
[0:08:51] There might be some exotic transformer architectures where you might have a nonlinearity in your residual stream. And I guess one way in which maybe there's a very slight effect is you might have layer norms on your residual stream, which may privilege because you're multiplying by a diagonal matrix that might slightly privilege your base extension of your residual stream. But it's essentially just in almost all cases.
[0:09:20] Not going to be a privileged basis. Another place where you don't have a privileged basis is going to be your keys and your queries and your value vectors. So your attention head produces these keys and queries and values by going and doing a matrix multiply of your residual stream. And then you dot product the keys and queries together. So there's nothing about that that's basis dependent. And so definitely you shouldn't expect your keys and queries to go and be privileged.
[0:09:49] And your values, you're just going to do a linear combination of values and multiply them by another matrix. So there's nothing about the value vectors either that's going to encourage them to have a special basis. And so you shouldn't expect to have a privileged basis for any of those. Now, it's worth talking about this maybe in the context of other types of neural networks. So in RNNs, your RNN state and your LSTM state is generally privileged. You probably have...
[0:10:22] Some kind of non-linear activation function, regardless of whether you have an RNN or an LSTEM or GRU or any of these things, you probably have a privileged basis. But your word embedding is basically the canonical example of not having a privileged basis. So for a word embedding, it just doesn't make sense to go and ask, you know, what does basis zero mean? Like, yeah, just in a default basis that it gives you or what is...
[0:10:52] What does the basis direction one mean? There's nothing special about those basis dimensions. And that doesn't mean that you can't study the word embedding. People study word embeddings all the time, but you can't do it by just looking at the basis dimensions. Now, in confidence, we mostly see architectures where essentially everything is privileged because we mostly have neurons, especially neurons with ReLU activation functions, which is really going to encourage.
[0:11:22] A privileged basis. But we do sometimes see the residual stream. Yeah. So in ResNet, many ResNet architectures will have values on the residual stream, which does cause them to have a privileged basis. But sometimes you won't have values on your residual stream, in which case they will not have a privileged basis. There's also.
[0:11:52] Some other sort of rare situations where you may not have a privileged basis in a continent. So one case might be if you have, sometimes you might go and do a low rank matrix factorization of your weights. I'm forgetting that, I think there's a name that people sometimes use for this.
[0:12:14] In the context of content, it's slipping my mind for a second because some special name people sometimes use, but where basically you'll do two convolutions in a row, one that's doing spatial and one that's doing sort of depth-wise convolutions, and then you'll apply your activation function. And so if you look at the activation between those two steps, depending on the details of it, it may not have a privileged basis because you didn't have an activation function. It'll depend very much on the details of exactly how you factor those matrices.
[0:12:43] But that might be another case where you possibly might not have a privileged basis in the continent. But that kind of situation is very rare. So this usually doesn't come up in continents. Usually in continents, except for the ResNets that don't have ReLUs on them, you have privileged basis everywhere. OK, so the final question is, what should we do when we don't have a privileged basis? So in Transformers, actually,
[0:13:09] A pretty large fraction of the transformer doesn't have a privileged basis for us to try and work in. We have to go and somehow work around that. And there's broadly two category of things you can try to do. So you can either try to find a meaningful basis for that representation, and I'll talk through a few options in a second, or you can try to push analysis to other representations in the model and not work, just try to ignore and not work with the representations that don't have a privileged basis.
[0:13:40] In the case of trying to find a privileged basis, one thing that's been very successful, especially in the word embedding context, is to define a new basis in terms of differences between elements. So people very often will define a gender dimension by doing man minus woman or something like that. And then you'll go and you'll project things onto those dimensions. You might also apply various dimensionality reduction techniques like PCA.
[0:14:10] PCA or ICA or things like that seem, really I think if you want to do this, you want to do some kind of linear non-dimensionality reduction, probably some non-linear dimensionality reduction probably isn't very principled to do here, but you could do something like PCA and study that and study those dimensions. That seems also to make a lot of sense. Another option is that you could, and this is pretty similar, is you would try to factorize things.
[0:14:40] Well, I guess the intuition here is a little bit tied up with another idea, which is that neural network representations seem to often, there's this phenomenon of polysemanticity where you have, even neural representations which have a privileged basis, often you'll find neurons that seem to correspond to multiple things. And we call these polysemantic neurons. And I think the leading theory among the small set of people I know who talk about this for why this is the case is that
[0:15:09] The network is trying to represent too many features. And so it's forced to go and squeeze multiple features into neurons and take advantage of the properties of high dimensional spaces and the fact that in a high dimensional space, you can have exponentially many almost orthogonal dimensions. In any case, all of this is to say that if you believe that, you want tools potentially to go and pull apart a space into actually a higher number of dimensions.
[0:15:38] To find a meaningful basis. And so then there's some ideas around factorization and sparse coding and things like this that are potentially really powerful. And sparse factorizations seem promising. Okay, so in any case, that's my rant on the trying to find a meaningful basis option. For the trying to push analysis to other representations, well, in the case of transformers, we're very, very fortunate that all of these
[0:16:07] Representations that don't have a privileged basis are just linear transformations of a bunch of privileged basis. And so the alternative thing that you can do is you can just try to go and talk about everything in terms of privileged basis like your MLPs and your tokens and just push all the math back to those. And I think actually, I mean, if you've watched our videos so far, you've probably seen some examples of this implicitly where we were able to go and understand, for instance,
[0:16:37] Attention Heads by going and pushing the OV circuits and QK circuits all the way back to the tokens. So this is another strategy that you can take, and I think it's a very, very promising one and a very nice one. Okay, so in any case, that's my rant on privileged versus non-privileged bases or representations that have a privileged basis versus representations that don't have a privileged basis.
[0:17:03] And yeah, if you're interested in our next video, I think we'll actually, I'll go and have a couple of my colleagues join me and we'll talk about some neurons we find in transformers, of course, in the MLPs, where that sort of is a question that seems like a good question to ask. Thanks.