{"url": "https://www.youtube.com/watch?v=rUOowiSx6UU", "page_url": "https://www.youtube.com/watch?v=rUOowiSx6UU", "title": "1L Attention - Eigenvalue Analysis [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 698, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-1l-attention-eigenvalue-analysis-roug", "audio_duration_s": 697.3866875, "worker": "jc-w06", "transcribe_s": 9.6, "model": "large-v3", "segments": [{"start": 0.02, "end": 32.53, "text": "In our previous video, we were able to analyze a one-layer attention-only transformer, but it was kind of tedious, or at least it would be if we wanted to do it really carefully, because we were looking at these really big matrices. And in this video, I'd like to talk about a technique of using eigenvalue analysis to go and do that in a much faster way and quickly summarize these models."}, {"start": 33.1, "end": 62.99, "text": "You might recall from linear algebra that an eigenvector is a vector that when you multiply it by some matrix W, just scales. So maybe it gets scaled by a positive value and gets larger, or at least sort of stays on the same side. It could get flipped, it could become negative, but it's not getting rotated, it's just getting scaled."}, {"start": 63.79, "end": 91.68, "text": "And eigenvalues and eigenvectors, to be honest, I haven't had much of an opportunity to use them in my interoperability research before. Because they're really only useful when you're mapping a vector space onto itself. And when a vector space, not just when it's the same size, that's sort of the formal definition requires that a vector space have the same dimensionality."}, {"start": 91.68, "end": 120.88, "text": "Like a matrix via square matrix to even calculate these. But really they're interesting when you think of a vector space as conceptually, the input and the output spaces are conceptually the same. That's when eigenvalues and eigenvectors are interesting. And in the case of transformers, we actually have, well, you could see the residual stream as being a case where you're reading and writing to the same vector space. That's sort of a little bit weak because the vector space, the representation after you write to it is a little bit different."}, {"start": 120.91, "end": 153.55, "text": "When the tokens, the inputs and the outputs, because we're predicting our inputs are tokens and our outputs are tokens, those, and we think of those as one hot vector. So every token we think of is a one hot vector in this incredibly high dimensional space. That is a map from a vector space onto itself. And that means that we can potentially study eigenvalues. And we'll see that some interesting things can arise. Now, remember that eigenvalues are complex numbers. So they live."}, {"start": 153.55, "end": 188.29, "text": "On the complex plane. And they can be positive or negative or imaginary. But because the magnitude of our eigenvalues can just vary by orders of magnitude, we're going to go and put them on a log scale. And we'll put the radius on a log scale when we present them in this talk."}, {"start": 188.29, "end": 218.22, "text": "The case where we're actually going to be using these eigenvalues is when we have things like the OV circuit in the previous talk. So we have a map from input tokens to output tokens, and it tells us how if we attend to a given token, that will affect the logits of our predictions for the output token. And so when those eigenvalues are positive, and we have lots of positive eigenvalues,"}, {"start": 219.73, "end": 251.18, "text": "It means that we're copying. It means that we're increasing the probability. We have an attention head that's going to tend to increase the probability that whatever token we attend to is the same token that we open. Now, conversely, if we have a bunch of negative eigenvalues, that's doing anti-copying and saying that if you attend to whatever token you attend to, you're going to decrease the probability of that being the output token. And we haven't seen anything like that."}, {"start": 251.18, "end": 281.04, "text": "Really, in any of the models that we've talked about, in the model that we talked about in the previous video, or probably in any models we're going to talk about in the next couple of videos. But we will eventually see some models that have anti-copying heads. And it's not something that we've studied that much yet, but it seems like maybe these anti-copying heads are useful. For instance, if you have a list and the model thinks that the same item isn't going to be repeated in the list, then it can go and attend back to previous items."}, {"start": 281.04, "end": 309.73, "text": "And decrease their probability. Even though maybe it's also using some kind of copying-like mechanism to increase the probability that similar things come later down. Okay, so the final case, and this is sort of the, in some sense, almost the default or neutral case, is to have imaginary eigenvalues. Because imaginary eigenvalues means that you're just affecting unrelated tokens. And so that's pretty useful."}, {"start": 309.73, "end": 339.22, "text": "For instance, if we have an attention hint that's doing bigram statistics, it's not that the same token is especially likely to be the next token. If you're literally asking which token comes after which other token, it's not that it's especially unlikely. It's just that there's an unrelated set of tokens that are very likely to come next, and the bigram statistics are focused on that. So that's what imaginary liken values mean."}, {"start": 339.22, "end": 365.9, "text": "If we look at the attention heads from the model that we were looking at in the previous video, we'll see that 10 of the attention heads really have extremely disproportionately positive eigenvalues. They're just in these tiny little clusters. And there's a couple where it's a little bit less extreme, but we have a lot of attention heads that are very lean, very, very positive. And so that tells us that all these attention heads are essentially doing copying."}, {"start": 365.9, "end": 396.11, "text": "Now, we could try to understand what the details of what they're copying is and where they like to attend back and so forth. But the high level story is they're copying. And then, you know, similarly, there's two attention heads that are not copying. And that's also interesting. And so at a glance, we can we can sort of get a summary. Now we can in the next couple of videos, we're going to start talking about a two layer attention only model. And here are the eigenvalues for its second layer. And we'll see that seven."}, {"start": 396.11, "end": 428.7, "text": "of its 12 attention heads are doing copying and so again at a glance we can very quickly go and see that there are a bunch of heads that are doing copying and the ones that are most intense about it are they have very concentrated positive eigenvalues and so and large ones so really doing doing some copying here now a useful trick that i found uh is it can sometimes be useful to look at the uh the sum of the eigenvalues"}, {"start": 428.7, "end": 456.64, "text": "Divided by the sum of the eigenvalue magnitudes. And that value, well, it'll always be a real number because eigenvalues come in pairs when they're complex and the imaginary parts cancel out. And because you're dividing by the sum of the magnitudes, it'll always be between negative one and one. And so when this is one or close to one, that means that you have copying attention heads. If it's negative one, it means you have anti-copying. And if you have things that are in the middle, maybe they're doing something else."}, {"start": 456.64, "end": 487.06, "text": "And you can use this to make histograms. I've put little pictures of the eigenvalue plots for each attention head just to make it really explicit that this is a histogram of attention heads. But you can see that on the right-hand side, we have the 10 attention heads that we said were copying heads. And then in the middle, we have these two heads that were really doing more, I guess, yeah, not, yeah, doing."}, {"start": 487.06, "end": 515.54, "text": "Basically doing bigrammish or engrammish local stuff that isn't copying or anti-copying. So this is a nice way to quickly go and summarize all the attention heads in a layer and figure out if anything with copying and anti-copying is going on. Now, in the previous example, this wouldn't be very interesting to do, but as we work with more complex models, it's going to be useful to also look at the eigenvalues of"}, {"start": 515.54, "end": 547.6, "text": "QK circuits. So the query and the key are both produced from tokens. And so we get this matrix that, again, is the input and output space are both tokens. And so we can ask about the eigenvalues of the QK circuit. And what they really mean is if they're positive, that means that we want to attend to the same token. So this attention head, when it's on a token, wants to attend to previous copies of that token. And if it's negative,"}, {"start": 547.6, "end": 577.23, "text": "It wants to avoid previous copies of that same token. And if it's imaginary, it just cares about unrelated things. Now, that might seem a little bit strange. And it's because there's actually some subtle, like, why would you want to attend to the exact same token that you're previously on? Well, there's a little bit of subtlety in the interpretation of these QK eigenvalues, because usually..."}, {"start": 577.23, "end": 605.31, "text": "We'll be thinking about them in the context of deeper models. And we'll be able to talk about this in a bit more depth in the next couple of videos and develop some theory that will help us think about this. But roughly what's going on is the token's information is going to be shifted by earlier attention hints. There will be attention hints that will shift where the query and the key information originate. And so that means that you can..."}, {"start": 605.31, "end": 636.14, "text": "When you see a positive eigenvalue, it might not mean literally attend to the previous token that's the same, but attend to a token that has some relationship to a token that's the same. And if you can understand that, that can still be a very powerful tool for reasoning about the behavior of those attention heads. Okay, so in summary, if we have for the output value circuit, positive means copying and negative means anti-copying."}, {"start": 636.59, "end": 668.3, "text": "Whereas for the query key circuit, positive means prefer same token attention and negative means avoid same token attention. Now, one final sad comment is this technique does kind of break down as you work with larger models, especially as you work with models that have MLPs. And the problem is that the representation of tokens becomes very intertwined with MLPs."}, {"start": 668.3, "end": 696.3, "text": "The token embeddings really start to migrate into attention heads, and that's sort of fine, but also into MLPs in a way that makes it very hard to go and do this kind of analysis. We sort of can't linearize as well through MLPs. That's a little bit sad. But for studying small models, and especially for studying small attention-only models, this is a really useful trick for going and quickly summarizing things and getting a quick overview of the model."}]}