[0:00:00] Well, this video is going to be an experiment in interviewing some members of Anthropic about some of the infrastructure that we have here that's maybe interesting and both might be internally useful for people to hear about. And maybe we'll experiment with sharing this externally. And I'm excited to start by talking about a tool called Garcon.
[0:00:29] And I think it's a particularly interesting example because usually when we think of, you know, serious, you know, systems infrastructure in deep learning, we're thinking about training or serving models. But it seems like as models are getting larger, we also need infrastructure if we want to do safety work or if we want to do interpretability work. And that's kind of maybe a little bit new as these models are getting really large, but it seems like it's a really important thing.
[0:00:58] And just to start off, Nelson, what would you say is the problem that Garcon is solving?
[0:01:30] The main problem we built Garcon to solve is the problem of doing a sort of interpretability work of kind of the work of trying to understand at various levels what's going on inside these models as they work, kind of how they do the things that they do, what kind of computation they're doing, sort of to aid that work.
[0:01:57] And in particular on some of the large and very large models that we build at Anthropic, many of which are too large to even run on or run efficiently at least on a single GPU. Maybe I can also come in with kind of the user's perspective is before there was Garcon, I could only work on models that were big enough that I could run them just in the Colab notebook I was working in. So you sort of load the model within the notebook.
[0:02:28] And be getting its activations there by sort of reaching into the torch internals and grabbing activations out. And so now that there's Garcon, I don't have to have the model on the same box that I'm using the Colab notebook to analyze the results. It can run on a much larger box somewhere else. And that has a lot of great advantages for me. So just to unpack that a little bit more, why does it break down when we try to do interoperability on a model that can't run on a single GPU?
[0:02:57] Yeah, I guess so. Yeah, so I think the sort of unique thing about interpretability work, I think sort of unique to it is that rather than just running the model, passing tokens through the model.
[0:03:20] Getting out logits or sampling and then for training, we do that and then we back propagate. We take derivatives and do parameter update. In interpretability work, we really want to look inside these models in very flexible and dynamic ways. We'll run models where we want to capture the output of a particular attention head or we'll run models where we want to capture some statistic, maybe the L2 norm or something more exotic.
[0:03:50] of every attention had at all of these random kind of intermediate points inside the models. And so, as Catherine alluded to, the easiest way to do this in a lot of frameworks is to just load the model up into your Python environment, have the model running on the machine where your Python environment is running, and then PyTorch and TensorFlow give you various ability to install hooks and look and kind of poke at the guts of the model.
[0:04:20] But when the model is too large to fit on a single machine, then you can't just load it up into the Python interpreter that you're running because the model doesn't fit in a single Python interpreter. It needs to run inside some kind of distributed system that is running parts of it and then using pipeline parallelism or other techniques, shuttling activations between GPUs or even between different...
[0:04:46] Host nodes entirely in order to get out results at the end. And so you can no longer just have that running inside your Python process that you're doing experiments and exploration in. So yeah, so it sounds like basically in order to work with these large models, you're forced to go and use these distributed systems. And they're just not really set up by default to go and make it easy to access the various things that we want. That's right.
[0:05:17] These large models force you into these distributed systems because they're too large to fit the weights into a single GPU. You could potentially, at a huge efficiency cost, run them on a single GPU by shuffling weights in and off the GPU, but no one does that. And so the systems that these models are built in often don't make that easy. And again, the efficiency cost of doing that would be truly huge. And so you want to run them across many GPUs.
[0:05:46] That basically forces you into doing a distributed system. These distributed systems traditionally are built with the goal of either training the model or of doing the relatively narrow set of end-to-end tasks that the model is designed for, usually sampling or maybe a classification problem of some sort. And the interface they give you is you send tokens in one end and the model
[0:06:13] You know, sends tokens out the other end or sends classifications or whatever else the model is doing. These systems aren't designed to load the model and then kind of do arbitrary introspection of them while it's running across this distributed system. Okay, so Garcon is going to allow us to go into interoperability on these models. What actually is the API that you provide for users to go in and try to interact with these models?
[0:06:44] Yeah, so the basic API that Garcon provides is that you can launch a Garcon server using, you can give it kind of any of the models that we've trained here, give it a path to a snapshot and it will automatically figure out the relevant parameters and launch an appropriate number of nodes with an appropriate number of GPUs. Then you have a...
[0:07:11] An RPC interface where you can connect to that server from anywhere else in our cluster, typically from a Colab notebook or a Jupyter notebook, but potentially also from automated scripts. And then kind of given that connection, you can run the model. You can run forwards or backwards passes by giving it tokens or for backwards pass, giving it tokens in a loss function. But then additionally, the model,
[0:07:40] Exposes a sort of number of these named probe points that you can attach to and you can send arbitrary snippets of Python code that the model will run at that point in the model. So, you know, maybe between layers three and four or immediately before computing the non-linearity for the MLPs in layer 17.
[0:08:09] Or with the value vectors produced by a given attention head. You can kind of name any of those points in some way. And then as the model is going, it will inject a snippet of Python code that you give it. And that Python code has sort of two important capabilities. It can alter the value if it wants. So you could zero it out or you could replace it with some other value to sort of test some permutation.
[0:08:38] And it also has access to this save context object. It's given a handle to this context that it can write arbitrary properties onto. And anything written onto those save contexts from any of these probe points is then accessible by another API call to retrieve the value of all of the saved values written by any of these probe points. And so this gives you a relatively low level but flexible interface to
[0:09:08] To pick what sets of points in the model you want to inspect, either to mutate them, to save their values, or to do computation on them. Again, maybe you just want to compute an average or something that's much smaller than the full tensor and save that value into a place where it can then later be accessible from the client as a separate API call to pull down all of the saved values. And then once you have those probes set up, you
[0:09:36] Run tokens through the model. You give it a piece of text that you want to run it on, and it will run, execute all of those probes in the appropriate way, save the values, change the behavior, and then hand you back the resulting computation, typically the logits at the end of the model or maybe some other value if the model has a different kind of value head at the end.
[0:10:01] And also all of the saved values, saved by any of those little snippets of probe code that you attached to various points in the model. So to recap, we can run the model and do both forwards and backwards passes. And then we can, at various points, either save or modify values that we want to inspect or fiddle with and get the saved results back for a later analysis.
[0:10:27] As a user, this enables a variety of types of experiments. Some, as you were saying, more just inspection type of sort of going example by example and seeing in great detail what the activations were. You can do more statistical inspection type experiments where you aggregate over a vast number of examples. And you can do these ablation type experiments where you zero out values or otherwise mess with them and see what the causal impact is on the downstream computation.
[0:10:55] Yeah, Catherine, I'd love to hear just like more about your experiences using this. Like, so you were sort of mentioning earlier that, you know, obviously it allowed you to go and start doing research on models that you couldn't beforehand. But yeah, what has been your, what's your experience been like and maybe what have been the highlights of using Garcon?
[0:11:15] Yeah, I think there's sort of two different things. One is this very interpretability related boost, which is being able to collect these activations very easily and in sort of high definition to sort of see very concretely like how is the model responding to different inputs at different levels. But I think one thing I've been surprised by is that even when all I want are these sort of per token losses, which isn't getting very deep into the guts.
[0:11:45] Having Garçon as a framework makes that kind of work a lot easier as well. So the most recent batch of experiments I've been doing is looking at the sort of per token losses on many different examples of many different kinds of models and many different points in their training. And previously, I would have had to sort of load each model one by one and sort of inspect it one by one, which makes it very difficult to do this big scan or big spread over model size and...
[0:12:14] Training maturity. Whereas with Garcon, I could just spin up a fleet of dozens and dozens of Garcon servers for different models and different training points and sort of run against them in parallel with a lot higher throughput. And so I can ask a question like, you know, how is this example understood or interpreted by these hundred different models and snapshots?
[0:12:40] And get that answer quite quickly because it allows it to be paralyzed and not be sort of stuck on my one dev box. So something I feel like you're getting at here is that Garcon was kind of created to make it possible to work with large models, but it sort of incidentally made it easy to work with large numbers of models. Right. Because you no longer have to have the model on
[0:13:07] On your the machine that you're on your def box that your your notebook's running on you can suddenly go and instead have it um you know connect to 100 models or something like this and run experiments in parallel on all of those and it sounds like that's been um an important change to your your workflow and what you can do. Yeah yeah I think to summarize and just and just say like sort of two related things it was designed for large models and I'm currently using it for very small small models many of them and it was designed for
[0:13:35] These sort of inspecting and manipulating the internals. And I'm currently using it just for the end-to-end properties. Just that sort of separation of how the model is being run from my analysis environment has allowed a lot more flexibility in ways that I didn't necessarily expect would be so load-bearing. Something else that makes me think of is, Nelson, you've sometimes talked about having ambiently available Garcon servers.
[0:14:05] As an interesting thing. And I'm wondering if you could comment on that. Yeah, I mean, so we have a couple, we sort of have a number of sort of well-known models that are kind of fully trained from various scans we've done that we do a lot of our work on. And for a number of those, we just keep a Garcon server constantly running.
[0:14:31] With those models loaded, accepting queries from multiple users, because a Garçon server can handle multiple concurrent users. And I think for a lot of especially quick one-off experiments, that just means that you can start a notebook and connect to one of those well-known, ambiently running servers. And with
[0:14:56] Kind of both like zero kind of cognitive, like I have to go push the button or set something up cost. And also with zero waiting, because these servers are always running, you can just be connected to a recent fully trained model that we use and ask questions about it, poke things into it. And that I think really lowers the cost of doing, you know, especially just one-off or kind of, I have a super quick question about how these models behave in some place experiments.
[0:15:26] Whereas previously, if you wanted to load a model into a notebook, you'd have to download those weights and load them onto some GPU somewhere. And there's both just sort of some amount of ceremony of typing out the code to do this and just waiting time of shuffling those weights around and getting them loaded onto a GPU. Yeah, I think it was kind of magical the first time I used Garcon.
[0:15:56] It was a model, working with models that I had previously worked on that they could fit on one GPU, but I was used to having to wait several minutes for the weights to go and download and have that in my feedback loop. And suddenly I did have that was pretty amazing. So Chris, I think there's another sort of type of investigation that's sort of less top of mind because it's not what I've been doing for the past month or so, which is sort of interactive visualizations and interfaces that was also made a lot easier.
[0:16:24] So this is a kind of investigation where we might be trying to understand what one specific neuron does. And so we create a custom interactive web page where I can enter text and see what the model's predictions or what the neuron's activations are on that text using a sort of JavaScript interface that I've thrown together.
[0:16:46] And without something like Garcon, then I need to be setting up a server to hold the model and be the back end for these interactive queries. Whereas with Garcon, it's very easy for me to think just about what that web page interface should be. And it'll send queries to a Garcon server in the background that I don't need to worry about how exactly that's being designed or hosted. Yeah, that's a really interesting observation. So I guess what we're saying is,
[0:17:15] Garcon can both be useful to, and especially these ambient Garcon servers that we sort of know are always going to be there, are both useful to, and somebody who's doing interoperability research directly from say a Jupyter notebook, or to support interactive visualizations and be the backend for interactive visualizations that you might be making where you're creating a website that's going to go and create some visualization and it needs to poke around at some large model. That's right. And I suppose that also,
[0:17:48] Again, the benefits of being able to work with large models are turning up there as well, where it's also probably the case that you couldn't have created those interactive visualizations for really large models that don't fit on a single GPU or at least a single box prior to Garcon. Exactly, because what that website is having to do is it needs to ask some server somewhere.
[0:18:12] Please tell me the activations of this neuron in response to this user-entered text. And in order for that query to get responded to, it needs to have some thing running this large model that can get this specific activation. And that's exactly what Garçon is designed to do. So I wonder if there's any other big classes of types of experiments that Garçon has unlocked for us.
[0:18:38] Well, there's also the batch job. So I'm running some that are a bit like this, but Nelson has done this too. We have a newly trained model. Let's collect, for example, a data set examples for neurons in this model. And let's just run tons and tons of example contexts or sort of text snippets through it and identify for individual neurons which examples does it most prefer or fire most strongly on.
[0:19:06] And these kinds of background jobs are then very easy to launch. Nelson's written a little wrapper where you can easily write a script. The script will spin up its own Garcon server, make a bunch of these statistical queries over the server, and then spin it back down. And so that means you don't need to sort of personally babysit or run on a dev box these kind of generic statistical analyses that we want for most of the models that we're studying. Yeah, that's a really valuable kind of...
[0:19:36] We've been talking about some of these more exotic things, but a lot of the bread and butter has been just manual inspection in a notebook. And I feel like earlier on, Catherine, you were alluding to some specific things you might do, like collect the activations of a neuron in response to some example, or oblate some neuron and see what happens. I'm curious if you could just enumerate some other very standard
[0:20:05] Common types of experiments you might do. And also maybe, I mean, your background is neuroscience. Maybe there's kind of a fun analogy for you to make also here to like, I don't know, there's probably some analogy to like the kinds of experiments you can do in neuroscience and the kinds of experiments you can do having access to a model through this kind of mechanism. Yeah, I think the way I'll answer that question right now, I mean, I've alluded to a bunch of kinds of experiments, but I think of them as sort of
[0:20:34] Lying on a sort of scale of analysis continuum, are we analyzing one neuron, one attention head in one layer? Are we analyzing an entire MLP layer or an entire collection of attention heads? Or are we looking at sort of all layers? It's sort of what I might think of as like a neuroanatomy.
[0:20:57] So I've described a little bit kind of taking one neuron, seeing its activations on a specific text. You can do one neuron, its activations on many example texts. These are sort of like single unit studies, but then you can go all the way out to more like the neuroanatomy study and do some kind of dimensionality reduction mapping exercise. We've played around with UMAP and PCA to understand how are these neurons or how are these attention heads
[0:21:27] Similar to one another. How do they cluster in their behaviors? Are there different types? Do they group together? How does that sort of representational space of selectivities change over layers? These kinds of anatomical like experiments, I think sort of flushes out the other end of the continuum of what we've tried to do. Yeah, that makes sense. I'm noticing so far we've talked about hooks and we've talked about
[0:21:56] Getting access to activations and running, fiddling with things and such. But I suppose Garsol also gives us access to parameters. And that's important for going and doing work like circuits or for, I mean, it's sort of giving us the connectome of the model or something like this. Nelson, could you talk at all about how that's implemented? Yeah, I mean, I think the
[0:22:23] Parameters are interesting in some sense in that they're kind of inert. You don't have to run the model. They're just data. And in some sense, the model downloads them from wherever it stores them. And you could always get them from the same place just as easily. But in practice, it's far more easy for
[0:22:49] All right, so let's talk a little bit more about the Garsan server.
[0:23:19] processes and the distributed system. One of them, rank zero, the first one in a somewhat arbitrary way, is doing the actual communication with the Garcon client. So it just talks to all of the ranks, says, hey, who's got this parameter? Every rank either sends back their value of the parameter or not me. It aggregates all of those and then sends it back to the client. So it transparently deals with the fact that
[0:23:48] When you ask for a parameter, it's on one of the many GPUs backing this model, but you don't have to know or care which GPU. I think one feature that like Garcon is sort of lies, it makes some effort to do this kind of efficiently so that it can shuffle these parameters, which themselves can be pretty large in a large model.
[0:24:16] A single weight matrix might itself be a sizable chunk of data. It goes to some care to shuffle these effectively so that they're moved around at some decent fraction of the theoretical network bandwidth. Doesn't go kind of anywhere to the care that
[0:24:36] Training jobs, which are often meticulously crafted by practitioners to use every last ounce of performance that the hardware has to offer. Garcon pales compared to those, but we've gone to some effort to make it efficient enough that most things you might ever want to do interactively where possible.
[0:24:57] Come back before you get bored or change the tab, which turns out to just be like a real threshold in usability of if you can grab these parameters and then they come back before you feel the urge to check Twitter, it actually like a 2x difference in performance if it sort of comes in before that tab awaited Twitter thresholds can turn into a 10x improvement in your efficiency of actually running experiments.
[0:25:27] Another thing I also want to reply to, Chris, is that you were pointing at what experience can you do with parameters instead of activations and sort of connecting back to that neuroscience language. I was describing spectrum in sort of this axis from single unit studies to more like anatomical studies. And I think we can put this in kind of a two by two, right? You can also draw this axis is, are we doing...
[0:25:51] Sort of selectivity and like behavior of the neurons in response to different texts, or are we doing connectivity? And so I think the parameters give you those connectomic or connection related studies, which can also be at sort of both scales. You know, we can ask questions out of the sort of anatomical connectivity level of like, in general, statistically, neurons in which layer connect to neurons or attention heads in which other layers overall.
[0:26:19] Or we can get the circuit-based connectivity study of saying this particular attention head that has an interesting behavior that we want to study, which specific other neurons does it connect from? Which specific other neurons does it write to? And how do those connections mechanistically implement the behavior that we're studying? Or how do they mechanistically contribute to onward behavior? So I think that sort of scale of analysis and then
[0:26:47] Selectivity versus connectivity spectrum gives us a pretty good map of the kinds of work we might do here. Yeah, that's a nice two-dimensional space. Okay, so diving into the technical details a little bit more, Nelson, you talked at a high level about how Garcon works with the RPC server. I guess I'm curious if there's any non-obvious design decisions.
[0:27:19] That you'd want to share or maybe another interesting way to frame this question could be like, you know, if you were going back in time to like the version of yourself who was about to start going and toying with, with implementing Garcon, you know, what are the things that you would want to tell that version of yourself? Or maybe everything was just so straightforward that it really is just, you know, go and put an RPC server on top of these things.
[0:27:48] All that much complexity there. I think sort of the biggest thing here is just sort of deciding that this is a problem worth solving and putting the pieces together. Garcon currently uses...
[0:28:05] Super lightweight. So we use Python's Pickles and there's a module called Cloud Pickle, which allows for the pickling of actual Python code objects on the fly, which is what we use to like let you write a custom function in your notebook and then send it to the server to be executed. I think when you're in Python land and especially if you want to be able to ship code from one place to another, those are the obvious choices. But then for the actual,
[0:28:35] RPC framework, I built a super lightweight kind of minimal binary protocol to handle the framing of requests and responses. In retrospect, if I started over, I think there's a decent chance I would just encapsulate it all inside something like HTTP.
[0:28:56] That would add a little bit of complexity in some sense of HGP libraries or HGP is huge and complex and those libraries are large, but I think it's sort of would take away complexity of writing your own little network protocol and use something very standard. I think efficiency, as I mentioned, is a little bit of a concern. I think that it would be perfectly possible. You'd have to pick your libraries again, slightly carefully, be perfectly possible to get
[0:29:25] Adequately efficient encapsulation of the tensor data inside the HTTP container. So I think that's one choice that I made. I think just out of eagerness to do the sort of smallest possible thing that would work, I rolled my own little protocol there. I think using something more standard would have been a little bit more work up front, but would have avoided a couple of pitfalls I ran into. I think one other piece of
[0:29:53] I think it was not obvious to me at the start architecture is you're running these models on multiple servers, kind of what's the relationship between all of those servers and who runs the RPC server and who talks to the client. I think the architecture that we settled on is I think pretty standard if you're building these kinds of machine learning distributed systems, which was novel to me, which is sort of all of
[0:30:23] All of the nodes serving that are kind of running the model are sort of running in lockstep, kind of all running effectively the same code and then having some synchronizing network communication operations with each other that sort of rely on them all running the same thing in the same order, except for one rank.
[0:30:44] Runs the RPC server and talks to the outside world. And then when it receives a request, it broadcasts to all of the servers over the kind of internal like cluster communication that's used for tensor and communication. And then kind of all of the ranks execute the RPC call in parallel and coordinate with each other as appropriate. And then the results are gathered back to that designated rank, which then
[0:31:13] It takes them from GPU memory or the internal cluster and marshals them out to the RPC protocol. I think in practice, this is a pretty standard architecture. If you're building something to run sampling out of a large language model, you end up with a very similar architecture. But it wasn't immediately obvious to me because I didn't have a ton of experience with these ML systems.
[0:31:41] Before building Garcon, that was the right way to go. I didn't kind of figure that out. I had advice from excellent, excellent other team members who have far more experience building these systems. We are indeed lucky to have lots of excellent colleagues. Is there anything interesting to say about why we went with Hawks? So I guess the framework is that we have
[0:32:07] You can attach hooks to these points and they save things and they get them back. But you could imagine a simpler system where you just say, you know, we're going to record from all these points and you can request to get those back. That might be a little bit TensorFlow-y. Maybe you could have like a mechanism for overriding them or something, or perhaps there's other setups you could do. I suppose this is just sort of how PyTorch does it. So maybe that's part of... Yeah, I think it was inspired by how PyTorch does it, but I think...
[0:32:37] That it was the simplest or most minimal approach that we could come up with that satisfied the following set of desirable properties. So we needed to be able to both collect data from tensors and to alter the flow of data inside the model. We want to be able to
[0:33:04] You know, ablate things by setting their activations to zero or, you know, swapping activations from somewhere else. And so if you want to have this either be able to save values or to change them, I think it's a relatively simple mechanism to implement at least to just like have a piece of code that can run there that has access to the tensor and can.
[0:33:27] Return a value and it also has the ability to save it somewhere. You could imagine having two different mechanisms for those, and I think that would be very plausible, but this was sort of a simple one that unifies those two desires. We only wanted to save data that someone actually wanted. Especially with a large model, there's just a sort of enormous number of intermediate states that you might want to inspect, and saving all of them would be very memory intensive.
[0:33:56] And, you know, even if you have that memory for sort of running a single example through, sometimes for performance, you want to run with a batch dimension and put lots of examples through. And then that just kind of scales everything by a factor of N. And we'd like to be able to kind of push that as high as we can. And that means you want to save only the values that you want, which means that you need to declare ahead of time somehow the values that you want. Another, I think, desirable property is that
[0:34:26] As I mentioned, sometimes you don't want to just collect an activation. It suffices for your experiment to collect some summary of some activations, the mean of some tensor, the L2 norm, or the eigenvalues or singular values of some values at some point. Having an interface where you push code to the server lets you do that, lets you A,
[0:34:53] Shard that computation among all of the ranks of the model. So if you're collecting like L2 norms of activations across every layer, then that computation gets to make use of the fact that you have N GPUs running this model to run more efficiently. And then also by doing that computation kind of at the
[0:35:15] The point of access in the model, you save a much smaller amount of data than if you saved the entire tensor and then did the computation later. And definitely the thing you want to avoid is
[0:35:31] Saving activations, sending them to the client over the network link because the client server network link is comparatively slow. And so if you want to do aggregates or statistical analyses, you want to do them on the server. And ideally, you want to do them on the GPU that has the data to minimize communication. And if I could just jump in with like an example of one of these aggregates that I mentioned before, like I think the dataset examples use case is a good one where...
[0:35:58] For all the neurons in this model, we'd like to know which 10 examples they fired most strongly to. And you definitely don't want to do that by running 10,000 examples through, collecting 10,000 copies or 10,000 versions of all the activations, sending all that to the client and then compressing it down. You really want, while you're running the model, to be keeping track of just the top 10 so far. And then the things you get to send is just the final answer that you're looking for.
[0:36:25] Yeah, and that actually speaks to another design decision we made of this sort of these stateful save contexts that you can mutate and then later retrieve is that that means that for this sort of data set example collection where you're collecting the top examples along symmetric, you can set up appropriate
[0:36:45] Probes and then run your 10,000 models through. And then those probes in a stateful fashion update the kind of current list of top 10. And then with one call, one network communication costs, you can suck all of those back down. If you had a approach where you maybe were allowed to apply some aggregation and then immediately sent the data back to the client.
[0:37:12] You'd have to do communication for each of those 10,000 examples. And so this sort of saves you a 10,000 fold or 100,000 fold or a million fold. Sometimes we run large numbers of examples to kind of get large scale, accurate statistical pictures. By being stateful on the server, it lets you accumulate state on the server and then just kind of
[0:37:35] Run examples through as fast as the model can process and then only do your communication at the very end. So the desires for those things, the ability to read data, to read tensors, as well as altering tensors while the model is running, the ability to do computation on the server in order to compute aggregates and
[0:38:01] And I think the sort of the net result is an interface that is very flexible and a little bit clunky to program directly against. It's not the sort of most convenient thing.
[0:38:30] We sort of think of this as the sort of assembly language or the like primitive language of the system. And then we've built layers on top of it that are kind of much more of just like run this and just give me these values directly. Don't make me deal with the save context abstraction and saving them and then the separate calls restore them. It's kind of very straightforward to build more expressive layers on top of this that sort of maybe are higher level.
[0:38:59] There are higher level or kind of correspond more naturally to some higher level models of the model, but that don't have the kind of full power and flexibility. And I think sort of in general, when you can have it, that's a pretty powerful way to build these systems is if you can have a kind of minimal expressive primitive that gives you everything you want and then layer the sort of things you actually want on top of that cleanly, that decomposition I think has worked out pretty well.
[0:39:29] Catherine, I'm curious if you can speak at all to your experiences as an end user about Garcon versus building abstractions on top of Garcon. Yeah, I mean, the thing that came to mind as Nelson was speaking was it took me a while to sort of get used to when should I be using these simplifications and when do I need to think about which computation happens on the server or not.
[0:39:53] Like in these cases where I just want the activations, I'm only sending in a batch dimension of one. I'm not aggregating anything. I'm doing one of these single unit studies. It's very easy to just use one of these tiny little wrappers rewritten that's like, yep, just save the activation, send them back. Don't need to think about it. As I moved into more statistical analyses, having to think like, wait, actually there's a batch dimension now. And so I need to take that into account. I can't just always take the zero. Or actually, you know, I'm running 10,000 of these. That needs to move on to the,
[0:40:22] That should be done on the GPU. It just took me a little getting used to those different layers. And I think that was not too bad. I mean, I now have sort of a sense of how, you know, just how statistical analysis is this going to be. If it's comparatively more statistical, I probably need to just add a couple of brain cycles into, you know, how do I need to change this safe context to factor better. So, Catherine, I'm going to put you on the spot a little bit in front of Nelson and ask, you know, you've talked about the things that...
[0:40:53] Are there any lowlights or any things that have been kind of frustrating as an end user about doing things this way? Gosh, I think there's some things that are still slightly half-baked. So, you know, if I cancel a call to a Garcon server because I don't actually want that result, and then later I run the cell again or run a different cell, it'll be like...
[0:41:17] Sent back a reply to the wrong message. And so there's just little things about the networking protocol that are slightly rough around the edge, but I got used to very quickly, like, yep, just reconnect, do it again. So none of this has been comparatively prohibitive. And yeah, just as I was referencing before, just like getting my head around, like, what is this context buffer doing? Like, it took me a while to realize just how stateful it was.
[0:41:40] And then once I got that, that's sort of a step up in power. But I was getting weird results that I didn't understand because I didn't realize it was faithful in that way. And so there's sort of that surprise versus power trade-off that it just took a couple, you know, oh, it's keeping state. Oh, there's a batch dimension now that you just have to encounter once. And then I think you move through it pretty quickly. That makes sense. Okay, so I guess a question somebody might ask is, you know,
[0:42:11] Are you open sourcing Garcon? If not, why aren't you? It sounds like a really useful tool that could be helpful for people who want to do interoperability. I think we're unlikely to. I think it's the kind of thing that is pretty tightly coupled to a bunch of details of Anthropic's infrastructure, which both means that decoupling it to open source it would be a fair bit of work.
[0:42:40] And might or might not result in something that was useful in anyone else's infrastructure if they just made different assumptions about how things worked and how they launched jobs, et cetera. And I think that there's some complexity there, but there's not an enormous amount of complexity. And I suspect that another team that wants a similar thing
[0:43:06] The description of what it is and how it works that we've given here is probably enough to give at least a moderately well-resourced team. Anyone training large models probably has at least a moderately well-resourced team. Give them a blueprint to build an easy version tailored to their infrastructure in a week or so, and then you iterate a little bit from there. I think it's the kind of thing where...
[0:43:35] The value of actually releasing code is a little bit swamped by just how fiddly it is and how much it plugs into the exact details of how we write our models and how we run our infrastructure. And it just ends up being a much better value trade-off for everyone.
[0:43:54] Share a relatively detailed description of what it is, how it works, and why you might want it. And encourage teams that think they might want one to just build one themselves. It's not too much work if you have the appropriate infrastructure and engineers already on your team.
[0:44:14] I think I also want to chime in with some appreciation of other Anthropic members, not currently on this panel, who have done a lot of work to make our cluster very usable, such that launching something to a pod running on a cluster is very easy. And that's some of the bread and butter of what this Garcon workflow looks like. And so depending on how your team manages its cluster and manages launching pods to that cluster, that's sort of...
[0:44:42] Contact with the metal of how your cluster works could look very different for you. And I think that's at least one layer that would be sort of hard for us to decouple from and where we benefited a lot from the other infrastructure engineers who have made that quite seamless for us. It's a really fantastic point. Well, in closing, do we have any last things that we'd like to say perhaps to our friends at other labs who might be considering copying Garcelle for us?
[0:45:13] We'd love for you to do so. I'm very, very excited for other people to have this kind of infrastructure and be poking around with these models. Yeah, I think maybe the sort of high level thing I would say is I think sort of the details of Garcon's design, I think are like decent and have worked out well for us. I think the sort of like most important thing is I like any lab that is training large models.
[0:45:44] I think should have some mechanism, whether it looks exactly like Garcin or not, that makes it very straightforward for the people, for other researchers at that lab to load that model, run it, poke at it in a very flexible and general way.
[0:46:05] And I think sort of that kind of high level, that's the problem that Garcon is solving for us is that any model we can train, we also have the ability to not just train it, not just run forward passes and use the model, but to also load it up and poke at it kind of no matter how large the model is, no matter how weird the model is. And I feel like it would be a real tragedy if our interpretability or
[0:46:33] Even though alignment or other teams could really easily work on our small models, but every time they wanted to do an experiment on a big model, it required custom work or custom engineering, or even not custom engineering, but it was just way more painful than doing work on small models because they had to write code in a different way than they were used to and launch a job that had a 10-minute startup cost.
[0:46:59] I think it's really desirable property that any model we build is roughly equally accessible to the teams doing research on how these models work and how they behave. And I think at a high level, that's the problem that Garcon solves.
[0:47:18] Any lab training these models should regard that as a problem worth solving. And hopefully, the ideas and approaches that we've talked about here are some pointers for how to do that. But if you come up with some totally other approach, I think great. We'd love to hear from you about what's worked for you. I think I also want to offer a compare and contrast between how Anthropic approaches our research on these models and how I've seen other research environments operate.
[0:47:46] That leads me to think that other environments could gain an even larger boost from this kind of approach, which is that Anthropic has a quite collaborative and collective approach where we are working together to determine which models we're going to train, train them, and then all sort of study and build upon and fine tune from this shared set of models where.
[0:48:06] Whereas I've been in research environments where individual groups of a few researchers have their own code for training and launching models and their own architectures. And so if I was in one of these more archipelago-like labs, then if I wanted to study your model, I might not know how to load it. I might not know how to get it in my notebook. I might need to set up a personal one-on-one meeting with you. It could be very high overhead.
[0:48:31] Whereas if someone else had trained a model and just put up a Garcon server for anyone to use it, then suddenly I don't need to know your particular code base for launching it. I can just go and start doing interpretability experiments on it. And I think in that kind of environment, it could be an even bigger cooperation and collaboration boost. Well, thank you so much for taking the time to talk about this, Nelson and Catherine. Thanks, Chris. Yeah, thanks, Chris.