{"url": "https://www.youtube.com/watch?v=etFCaFvt2Ks", "page_url": "https://www.youtube.com/watch?v=etFCaFvt2Ks", "title": "Attention - General - Theory, Info-Weighted Patterns, Attribution Patterns [rough early thoughts]", "show_or_channel": "Mechanistic Interpretability (Anthropic, YouTube)", "date": null, "duration_s": 1897, "type": "other", "speakers": ["Chris Olah"], "confidence": "medium", "notes": "Anthropic interpretability team's informal research-talk videos (playlist PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51, linked from transformer-circuits.pub/2021/videos). Recorded ~Dec 2021-mid 2022; exact date unknown. Presenter not stated in metadata - believed to be mostly Olah with Catherine Olsson/Nelson Elhage et al.; verify Olah's share before transcribing.", "people": ["chris-olah"], "id": "0000_mechanistic-interpretabil-attention-general-theory-info-weigh", "audio_duration_s": 1896.2924375, "worker": "jc-w07", "transcribe_s": 79.0, "model": "large-v3", "segments": [{"start": 0.69, "end": 31.76, "text": "In our last couple of videos, we've studied and tried to reverse engineer some really simplified versions of a transformer. And the methods we used really relied on some of those simplifications, especially the fact that we only had attention layers. We were studying these attention-only transformers. This study is going to try to go and generalize some of those techniques to fully general standard transformers."}, {"start": 31.76, "end": 65.5, "text": "We're going to talk about some of the things that make that a much harder problem and how we can try to work around those challenges and develop some tools that will allow us to think about general transformers, or at least small portions of them. So it's going to be a bit of a trickier problem, but we will be able to make some progress. And I think we'll learn some cool things along the way. So the thing that we're going to be focusing on for now, and this is continuing in the theme of our previous videos, is understanding the role"}, {"start": 65.5, "end": 94.05, "text": "And this is actually something that's been going on in studying transformers for a while. And I think it really starts probably the earliest example is there was some work by Leon Jones in 2017. And it's kind of impressive sort of how early and he just made a Jupyter notebook exploring attention patterns and discovered some really cool things. And since then, we've had"}, {"start": 94.05, "end": 121.34, "text": "A number of papers, Jesse Vig did a lot of early work poking around at attention heads. And then since then, there's been really a lot of work going and exploring attention heads and trying to connect it to linguistic phenomena. That's been a really interesting thing. So there's some work by Kevin Clark and collaborators and some other work by Elena Voida and collaborators."}, {"start": 121.34, "end": 153.2, "text": "That's been going and studying that. So there's been a lot of really neat work in this space. And it's very tantalizing. It seems like there's very cool things going on. I do think that there's, however, a couple of limitations to just looking at attention patterns. And really, I think it can be summarized in three major points. So just very briefly, and we'll dive and talk about each one of these. If you're just looking at the attention pattern,"}, {"start": 153.2, "end": 183.02, "text": "First of all, you don't know if that's the ultimate source or destination of the information. It could be that the information was moved by another attention head and then moved by this attention head and then moved by another one yet again and doesn't really have anything to do with the tokens that superficially seem to be involved. Another thing that is, I think, challenging and important to think about is it seems like a lot of attention heads have what we'll call resting positions. And there's no real principled way to handle that if you're just looking at the attention pattern."}, {"start": 183.25, "end": 212.59, "text": "And we have to make assumptions that they might be right or they might not be right. We'll unpack that a bit more in a second. The final thing is it doesn't tell us about the role the attention head is playing in the transformer. It doesn't tell us about functionally how it affects the transformer's behavior and allows it to go and perform its task. We're just seeing the pattern and sort of observing that there's this interesting pattern. And so we'd really like to be able, we're going to try to talk about the role of attention heads in a transformer and use them as sort of a"}, {"start": 212.59, "end": 242.42, "text": "A small reverse engineered chunk of the transformer that we're trying to understand in this kind of circuits-y way and build towards a large sort of full understanding of the model, we want to be able to address these concerns. Okay, so just to unpack that a little bit more. The first concern is that the information, if you just look at a single attention pattern, you might see that attention, one token attends back to the previous token, for instance, or the same token, or maybe a token several steps back."}, {"start": 243.25, "end": 275.18, "text": "But there's nothing that says that's the ultimate source. And remember that attention heads, the whole thing attention heads do is they copy some subspace of residual stream from one token to another. And if you do that repeatedly, it's possible that you're copying information which was previously copied. Or conversely, it could be that you're copying information, you know, your attention head copies information somewhere, and then another attention head moves it yet further. And so these..."}, {"start": 275.18, "end": 305.54, "text": "These attention patterns, they might be representative of how information is sort of fundamentally moved in a model, but it could also very easily be the case that they're just one step in a chain of movement and that the particular tokens involved are kind of superficial. I guess another way to think about that is really the attention pattern is, you can think of it as sort of, if you were writing out the Jacobian for the model."}, {"start": 305.54, "end": 337.33, "text": "The attention pattern would be one matrix in this giant product and sum that would produce the Jacobian. And because it's being multiplied by lots of other things, it could be that it's just being multiplied by the identity matrix, but it could be not at all. It could be that the other matrices you multiply by really greatly change it. That's another way you can think about it. So in any case, that's one challenge with just sort of interpreting attention patterns as they are. And it may very well be that the superficial appearance is correct, but it could be that it's not. Another challenge is..."}, {"start": 337.33, "end": 365.68, "text": "If you start really digging into attention heads, you'll notice that a lot of attention heads will mostly attend to... Well, the most common thing is you'll see a lot of attention heads that almost always attend to the first token. So you'll see lots of tokens attend back to the first token in your context. And that seems weird. And then you'll notice that a very small fraction of the time they do something else. So a very common instance of this is if you have an induction head that's looking for cases where you're..."}, {"start": 365.68, "end": 396.06, "text": "Where there's repetition going on, most of the time tokens aren't being repeated, and so it doesn't have anywhere to attend. And it needs some default place to go, and so it will happen to be often, well, it'll have to go and set some default position, and often that will be the first token, or sometimes it'll also be periods. So if you look at an attention pattern like this, there's some sense in which really the meaningful thing is when it doesn't attend to the default position, and we really want to ignore all of these default position attentions."}, {"start": 396.06, "end": 425.89, "text": "They're not that interesting. But how do we know that's a principled thing to do? How do we know when something is just a resting position that we can sort of ignore or a default position that we can ignore that's not doing anything interesting? And when is it actually doing something important? Especially when it's maybe not that the deep resting position is the first token or maybe the token isn't 99% resting."}, {"start": 425.89, "end": 456.02, "text": "The attention isn't 99% attending to its resting position and only 1% elsewhere. If it's 50% attending to some resting position, it might be less obvious what's going on. By the way, this phenomenon has been described in the literature before. I think the earliest is Jesse Vig's work where it's described as null attention. But yeah, we'd like to have a more principled way to handle this. And then the final thing, and this is really, you know,"}, {"start": 456.02, "end": 486.27, "text": "Our goal is to reverse engineer the model. So it's cool to find interesting attention patterns. But the thing that we ultimately want is we want to say, how do they empower the model? How do they contribute to the model's behavior and allow it to do things? And how is the model working? So the attention patterns are interesting, but we'd like to link them to behavior and function and understand how they contribute to the model. OK. So it turns out."}, {"start": 486.27, "end": 513.78, "text": "That limitation two has a relatively simple and elegant solution. Whereas limitations one and three, that unfortunately is going to be a little trickier. And they have really simple, elegant solutions when you're only dealing with the attention only case. But once we switch instead to this general case, it's going to be a little bit trickier. And we're going to have to think about it quite carefully."}, {"start": 513.78, "end": 542.21, "text": "There's going to be some slightly painful trade-offs. Okay. So the idea that we're going to use to resolve this second limitation, the fact that there's these default tokens, is we're going to go and create what we call the info-weighted attention patterns. We have our regular attention pattern, and we're going to create an info-weighted attention pattern. And looking at it here, actually,"}, {"start": 542.21, "end": 572.53, "text": "I think it's going to be hard to understand what's going on. So let's switch to an interactive visualization. So what we're seeing here is we have the first paragraph of Harry Potter, and we have one attention head that I selected to be an induction attention head that we can see how it attends over this paragraph. And we'll see that almost always it attends to the first token, that first EOT token. And then very rarely,"}, {"start": 572.53, "end": 601.78, "text": "When we have repeated text, in the first paragraph of Harry Potter, the name Dursley is repeated a lot, and it's split into three tokens. So we can see when we attend to, or when we're on D, the induction head says, ah, there was a D, and last time I saw a D, it was followed by errs. And so I'm going to attend to the errs, and that'll allow me to predict that the next token is an errs. So that's what the induction head does."}, {"start": 601.78, "end": 629.65, "text": "We can see, yeah, we can go and just track that. So as we go through Dursley, we go and we do that. And as we go to later ones, we'll see even, you know, there's multiple copies of Dursley. So it can attend back possibly to multiple ones. And the resting position initially, it's just that first token, but later on, it'll start to rest on various periods. So you might think it's attending to the periods, but it really is resting on those. And there's some other places where it finds copies."}, {"start": 629.65, "end": 661.46, "text": "The potters is also a thing where it can predict the terrs in potters from pot. And there's also small sun here. We have small, and previously we said small sun, and so we can attend to sun and predict that it's going to be sun again. Okay, so that's what we call, what we mean by having a resting position. And it's very natural that this particular one has a resting position."}, {"start": 661.46, "end": 693.87, "text": "We can actually look at this. If you look here, this is another visualization of the same attention pattern. And what you're seeing is for every token, where does it attend? So if you fix the token, you do a horizontal line and you can see where it attends. And you can see that for the early tokens, they almost all attend to the first token. So that's that red stripe down the middle. And then these little off diagonals."}, {"start": 693.87, "end": 727.52, "text": "are places where we have a little sequence of repeated text and the induction head can start to activate. So we see that there's also then these places where periods start to become resting positions. And those are these other vertical lines. So we can see it very clearly there, this fact. And I think probably on a YouTube video, it might be a little hard to see this, but we can link these together so we can see the fact when we attend to Dursley here, that's one of those."}, {"start": 727.52, "end": 756.43, "text": "Yeah, we're on that diagonal over here. Okay, so that's an example of a attention head that has a resting position. Okay, so now the thing we'd like is we'd like some principled way, like we could make the judgment call, but when it looks at EOT, that's not real, that's a resting position. And when it looked at periods, that's not meaningful, that's a resting position. And we only wanna look at the copied cases. So that would be our judgment call and that wouldn't seem very principled."}, {"start": 756.43, "end": 786.32, "text": "So instead, we're going to go and create what we call an information weighted attention pattern. So we're going to take the same attention head now, and we're going to look at the information weighted attention pattern. And I'll explain exactly what that is in a second, but the intuition is that we're going to go and show the attention scaled by how much information was copied from one token to the other. So partly that's where we attend, but it's also how large the value vector is. And we actually do how much..."}, {"start": 786.32, "end": 815.06, "text": "How large is the vector that's going to get written to the residual stream at the resulting place? That's the WO times the value vector. And here, we just color by default when we aren't interacting with this. We just color the tokens by the intensity of the pattern, in this case, the amount of information that was moved to that token. And we can see that whereas before everything was getting attention, now we're seeing that almost everywhere is zero."}, {"start": 815.12, "end": 849.01, "text": "So when we look at these typical tokens, you might see that the, sorry, you might see that typical tokens do have a slight pink to EOT, but for the most part, it's very small. And if we get to copied text, all of a sudden, we have some very strong copying going on. So that's kind of interesting. And we have these, yeah, and we've just isolated the cases."}, {"start": 849.01, "end": 880.66, "text": "And we can at a glance see where there's interesting things going on. So yeah, like small, we look at sun and things like this. Okay. So the formal definition of that is going to be that we, so here we have the regular attention pattern, which wasn't very informative. And then we have the info-weighted pattern, which really is going to solve our problem and give us a principled resolution to this question."}, {"start": 880.69, "end": 909.65, "text": "of how to deal with resting positions. And the trick is just that you scale every source position by the magnitude of w, o, v. And it's really not that different from going and scaling by the magnitude of v. And there's various other variations you could do of this. I've experimented with subtracting off the mean value, because sometimes the value vector for resting positions will be some small vector that's just kind of"}, {"start": 909.65, "end": 930.32, "text": "You could think of it sort of as being like a bias. And so subtracting off the mean can go and make this a little bit cleaner. And one interpretation of this is you could try to go and model the information carried through the attention head as a Gaussian, in which case that subtracting off the mean and looking at the norm of that vector."}, {"start": 930.32, "end": 960.06, "text": "is the surprise on the number of bits of information. You could think of this as being kind of like the number of bits of information copied from one position to another. So that's a nice principled resolution. And it gives us a way to think about these resting positions that doesn't require our judgment, but just is true. It's just the case that even though the attention pattern is attending there, it's not copying any information. The attention head isn't doing anything when it attends there. It's just it needed some default position. So that's a nice resolution to that. OK."}, {"start": 960.06, "end": 989.79, "text": "But that still leaves us with two limitations that are unfortunately not going to have quite as clean a resolution. So the first one is the attention patterns might not be the ultimate source or destination information. It could be that it was copied by other attention heads or it's going to be moved around later to a different token by a different attention head. And it also doesn't tell us about the function. And so when we were dealing with attention only models, that was actually really easy to solve because what we did was we decomposed our attention heads."}, {"start": 989.79, "end": 1011.38, "text": "into these sort of attention heads, strict attention head terms, and virtual attention heads. And then the virtual attention heads captured all this possibility that information could be moved by chains of attention heads, because that's definitionally what they were. And the pure attention heads, we could just study in isolation and not worry that information was going to be moved, because they definitionally didn't include that."}, {"start": 1011.5, "end": 1040.38, "text": "And then we could look at their OV circuit. Remember, this is the output value circuit matrix that tells us how much, if it attends to one token, it's going to want to go, how it's going to affect the output. And that allowed us to go and solve this question. It told us all about the function, solved our limitation of not knowing what the function of things were. So in our previous models, these were both really easy to solve, but it's unfortunately going to be a little bit trickier with the introduction of MLPs."}, {"start": 1040.38, "end": 1071.25, "text": "Having MLP layers, remember, when we have a general transformer, we go back and forth between we have attention layers with attention heads, and those are easy to work with. But then we're also going to have these MLP layers, and those are going to be harder to reason about. So let's talk through the challenge. Okay, so suppose that we have, let's start with one of these attention heads. Let's start with this attention head here. Okay, so there is..."}, {"start": 1071.25, "end": 1098.58, "text": "A term. First of all, we should be clear. There is a term that if you write the entire transformer in this big equation like we were earlier, there is a nice term that corresponds to the direct effect of the attention head in attending to some token and then affecting the unembedding. That is a thing that exists. And there also are the regular virtual attention head terms."}, {"start": 1103.76, "end": 1134.48, "text": "But the problem is we can't decompose the entire model into that formulation. So when we have our attention head, it does indeed get information directly from the token and through these attention heads, and those can all just by linearity be decomposed apart. But it also gets information from the MLP, and that could come from the present token, but it could also come..."}, {"start": 1134.48, "end": 1165.65, "text": "From the previous attention heads. And that's in addition to the amount that came from the present token. And we can't decompose those because the MLP is nonlinear. We can't go and break apart these two paths. They're mixed together. And so we can't go and decompose everything as nicely as we were previously. So that's really the tricky thing about this."}, {"start": 1167.5, "end": 1194.61, "text": "Now, there are some solutions. So, well, there are at least some options. One thing is we could just study the pure attention head terms, and those are interesting. You can just write them out the exact way we were in the previous models, and those are possible to just completely sort of analytically study, and they're beautiful and simple to work with, and you do see interesting things. But it seems like a lot of interesting stuff also can't be seen that way. And one reason..."}, {"start": 1194.61, "end": 1220.0, "text": "One very basic reason, even for relatively simple things, is that you begin to see the early MLP layers start to go and take on a role of sort of reprocessing the tokens. Because remember, tokens often have multiple interpretations. Words can be split into multiple tokens, and then their meaning is very unclear until you recombine them. And so the model really wants to improve"}, {"start": 1220.0, "end": 1247.7, "text": "The representation of tokens using the MLP layers and combining it with information it got from the attention heads, which means that your later attention heads are going to want to rely on that, which means that you're going to miss a lot if you aren't looking at the effect of the MLPs. So that's the downside of the approach of only studying pure attention chips. Now, another thing you could do is you could just say, well, we're not going to try and expand through the MLP at all. One of the great things about the MLP layer is"}, {"start": 1247.7, "end": 1277.02, "text": "If you looked at our other video, or maybe you will in the future, on privileged versus unprivileged bases. So a privileged basis is one where the basis dimensions are sort of special and probably easier to understand. MLPs do have a privileged basis. So if you have to have somewhere that you're going to try to go and analyze, the MLP layers are a pretty nice place to try and analyze things. And we can just try to understand what different MLP neurons represent and treat them like a sort of a basic source in themselves. And then we could try to understand how they're computed."}, {"start": 1277.02, "end": 1305.62, "text": "analytically but we could we could we could sort of stop there and that would be the very circuitry approach like in circuits and uh you know circuits is this work that me and some collaborators did previously on um reverse engineering confidence and you know really there what we were doing is understand a bunch of neurons and you can once you have sort of the input and output neurons if you understand what they are you can then go and try to understand what computation happens between them to go and construct the output neurons from the input neurons so"}, {"start": 1305.62, "end": 1331.92, "text": "That is something you can do. Another possibility is you could try in various ways to linearize the MLPs. So you could replace them with a Jacobian, or you could try to do this trick like the attention heads aren't actually linear. They're only linear once you fix the attention pattern. So you sort of split them. You think of them as having two different inputs, and they're linear with respect to one input. Well, you could do the same thing in principle for the MLPs."}, {"start": 1331.92, "end": 1360.38, "text": "Usually these are ReLUs, but let's imagine it's a ReLU for a second. You know, a ReLU, you just have to go and think of there as being like this binary mask that comes as a second source. And then you've split it into a linear and a nonlinear function, which has a linear input and a nonlinear input. But the challenge is that there's sort of many more bits of information to specify the nonlinear component here. So it sort of becomes, it's not as nice a frame of analysis."}, {"start": 1360.38, "end": 1390.93, "text": "In any case, if you do take this approach of linearizing, say, replacing with Jacobian, you're really getting into this family of methods that people call attribution or saliency. And you're applying that related to your attention head and to the value vectors. And that's absolutely a thing you can do. And often, if we want to really dive into an attention head, we'll start to do things kind of like that, or at least trying to really analytically think through how the traceback, how the value vectors were formed."}, {"start": 1390.93, "end": 1418.43, "text": "I think that's probably not the thing we want to do as our first pass approach. So the thing that we're actually going to do in this video or in the next couple of videos is we're going to take a very pragmatic option, which is we're going to go and look, when we're trying to understand intention, we will just take all possible inputs that are contributing to the value. So we're not going to try to decompose things. We're going to take the MLPs and..."}, {"start": 1418.43, "end": 1448.21, "text": "The attention heads and the embedding, and they all construct the value vector for an attention head. And we have to be very careful in interpretation to remember that the value vector probably significantly comes from the present token, but it might come from elsewhere. So we need to be careful in interpreting it. But then we're going to go and look at the direct effect of that attention head on the open. So we're going to ignore MLPs. And that's obviously going to miss some part of the picture. But it does seem like this is kind of a nice sweet spot where..."}, {"start": 1448.21, "end": 1478.37, "text": "We'll be able to really understand the downstream effects, at least some of the downstream effects. And we'll be able to see some things that we wouldn't be able to otherwise because tokens will get the nice representation from the MLPs that are going and updating their representations. And that'll be a nice way to go and think about things. Okay, so the key idea now is that using this,"}, {"start": 1478.37, "end": 1509.22, "text": "We're going to be able to make an attribution pattern, which says, how much does attending to this token increase the probability of another token? So let's switch back to our interactive view. So recall that so far we've had two things. We started with the raw attention pattern, and then we constructed this intuited attention pattern. Now we're going to do something a little bit different, where we're going to go and construct"}, {"start": 1509.22, "end": 1539.52, "text": "What we're going to call the attribution pattern, and we're going to color that in blue. So we have our info weighted pattern in red, and we're going to have the attention pattern in blue. So let's go and consider an example with the attention pattern. So the attention pattern, when we're on D, looks for the previous D, and notice there's an errs. And we think the reason it's doing that is so it can predict the next errs. So the attribution pattern is going to say,"}, {"start": 1539.52, "end": 1571.12, "text": "Yes, it actually did increase the probability of the next ERRORs. So now the ERRORs we can see is in blue here because it's actually increasing, and Li as well, because it's been increased in probability. It's a very intense blue because it's so strongly increased in probability. And if we look at where that's coming from, it's coming from the previous ERRORs. So that's..."}, {"start": 1571.12, "end": 1598.14, "text": "What's going on there? So the attribution pattern then tells us which tokens were increased in probability by this attention head and then potentially which tokens, yeah, how it did that. Where did it tend to go and do that? And because this is very simple copying, it's, yeah, we can sort of fairly easily understand it. Now, again, because we're going and just using the value vectors that could have been computed in lots of complicated ways."}, {"start": 1598.14, "end": 1628.5, "text": "In principle, it could be that even though it appears to be attending back to errs to go and increase the probability of things, actually, maybe this was computed by some other previous tokens. But because this is copying and it just seems like there's such a clear inference that it's the exact same token that's being increased in probability, we're pretty sure it's mostly just looking at that token. One other thing that's maybe worth being really clear about is"}, {"start": 1628.56, "end": 1658.19, "text": "When we look at the attention pattern, so if you look at the attention pattern and the attribution pattern, you'll see that when we switch the attribution pattern, it shifts forward. So the attention pattern is sort of at one token, and then the attention pattern is one token forwards. And that's because the attention pattern is to the output token, not to the token that's being attended to. So the attention pattern goes from the source token to the destination token. The attribution pattern goes from the source token to the output token, the predicted token."}, {"start": 1658.19, "end": 1691.62, "text": "Which is one forwards. So that's the attribution pattern. And to calculate that again, remember you just go and take your value vectors or your attention results and you multiply by your wout and you add them into the residual stream and then you hit them with your unembedding matrix and you just see how the logits are affected. Oh, one other thing that's worth keeping in mind is for the attribution pattern we're looking at how the logits are affected, not how"}, {"start": 1691.62, "end": 1720.26, "text": "The probabilities are affected. So remember that an attention head could potentially affect multiple logits, and it could increase the probability of, say, two competing things, or increase logits for two competing possible answers. And if there's a competitor that maybe it increased more that otherwise had already had more probability, it could be that the probability still of a given token decreases, even though your attribution"}, {"start": 1720.26, "end": 1749.84, "text": "Okay, so the attribution pattern is going to be super useful to us. I think we'll talk about that a little bit more in a sec. So yeah, for a minute, let's just compare these a bit more. So remember, our raw attention pattern goes from source to destination. The info-weighted pattern also goes from source to destination, but the logit attribution pattern goes from source to output."}, {"start": 1750.03, "end": 1782.11, "text": "And all three of these tell us different information. So the raw attention pattern tells us which token attends to other tokens. The info-weighted pattern tells us how much information is moved from each token to each other token. And the attribution pattern tells us how each token affects the logit or directly affects the logit of each other token. Okay, so what are the uses of these attribution patterns?"}, {"start": 1782.11, "end": 1809.63, "text": "There's really two reasons they're really useful. One is they're very helpful for empirically investigating attention heads and really investigating the sort of direct circuits where the attention head directly affects the logits. There's lots of cases where it's having indirect effects as well, and it's not going to help us with that. But for understanding direct effects, it's a very convenient abstraction for thinking about that. It's sort of taking on the role, you know, previously we studied"}, {"start": 1809.63, "end": 1839.74, "text": "This by looking and analyzing the OV circuit matrix, which was this big matrix. Here we're going to do things a bit more empirically, but the attribution pattern is going to be our main tool for really empirically studying that. The other thing that's really nice is we can use it to search for attention heads. So something we can do is we can just sample a bunch of dataset examples from our dataset and define some pattern where we think that an attention head is going to go"}, {"start": 1839.74, "end": 1861.07, "text": "And affect the output in some particular way. So for a copying head, we think that's gonna tend to a token and it's gonna increase the probability of that same token. And most of the time when it increases the probability of a token, it's gonna be because it's attending to a token that's the same. Well, we can use that to search for copying heads or..."}, {"start": 1861.33, "end": 1891.28, "text": "Yeah, well, we'll see a lot of other interesting attention heads that we can search for. I think I'll leave that for the next video. But this ability to search for attention heads in large models when you might have 1,000 or more attention heads and you have some theory that there's a bunch of attention heads that are doing something, that's a really nice tool. And that's going to be really useful to us. OK, so we are now ready to dive into studying attention head circuits in large models. So in the next couple of videos, we're going to be doing that."}, {"start": 1891.28, "end": 1893.58, "text": "And we'll find some pretty cool stuff."}]}