[0:00:04] Welcome to the Logan Bartlett Show. On this episode, what you're going to hear is a conversation I had with CEO and co-founder of Anthropic, Dario Amadei. We talk about a bunch of different things, including predictions for the future of artificial intelligence, his upbringing, how he came to found Anthropic and spin out from OpenAI, as well as what problems he thinks can be solved by artificial intelligence in the near future. Really interesting conversation with a very thoughtful person in an important position of power, running one of the leading AI companies.
[0:00:34] If you're enjoying this discussion and conversation with Dario, please do subscribe to this podcast on whatever channel you're listening to. And if you're here to listen to more topics of artificial intelligence, I would encourage you to subscribe to Redpoint's podcast on the subject unsupervised learning, which you can find on any podcast player now.
[0:00:53] Dario, thanks for doing this. Thanks for having me. So I normally don't tell people's backgrounds in a linear fashion, but I actually haven't heard yours. I don't know if you've ever really told it in earnest, like childhood, growing up, sort of what led you to starting Anthropic. So maybe can you share a little bit about your background, childhood, growing up? Yeah, yeah. So I don't know that my childhood was that interesting or that different from people who are in tech or found companies. I mean, I was always really interested in math.
[0:01:23] You know, a sense of objectivity, right? You know, one kid could say, oh, this show is great, and the other kid could say, oh, it's terrible. But, you know, when you're doing math, you're like, oh, man, there's an objective answer to this. So that was always very interesting to me. And, you know, I grew up with a younger sister who's one of my co-founders, and we always wanted to save the world together. So it's actually kind of amusing that, you know, we're working on something together that, you know, at least potentially could have very wide scope.
[0:01:52] So, yeah, I mean, in terms of how I got from there to Anthropic, my interest in math led me to be a physics major undergrad. But near the end of undergrad, I started reading, it was initially the work of Ray Kurzweil, who I think is...
[0:02:13] A bit crazy about a lot of things, but just the basic idea that there's this exponential acceleration of compute and that that's going to provide us enough compute. Somehow we had no idea.
[0:02:26] How then? We had no idea it was going to be neural nets. You know, what will somehow get us to like very powerful AI? And I found that idea really convincing. So I was about to start grad school for theoretical physics and, you know, decided as soon as I got there that I wanted to do biophysics and computational neuroscience because, you know, if there was going to be AI, you know, it didn't feel like.
[0:02:48] AI was working yet, and so I wanted to study the closest thing to that that there was, which was our brains. It's a natural intelligence, so therefore the closest thing to an artificial intelligence that exists. So I studied that for a few years and kind of worked on networks of real neurons.
[0:03:08] You know, shortly after I graduated, I was at Stanford for a bit, and then I saw a lot of the work coming out of Google, of Andrew Ng's group at Stanford.
[0:03:19] And so I said, okay, you know, I should get involved in this area. You know, my reaction at the time was, oh, my God, I'm so late to this area. The revolution has already happened. What year is this? 2014, right? So, you know, I was just like, oh, my God, like, you know, this tiny community of 50 people, like, they're the giants of this field. It's too late to get in. If I rush in, maybe I can get some of the scraps. That was my mentality when I kind of entered the field. And now, of course, it's, you know, it's nine years later.
[0:03:48] And so I ended up working with Andrew Ng at Baidu for a year. I ended up working at Google Brain for a year. Then I was one of the first people to join OpenAI in 2016. I was there for about five years. And by the end of it, I was VP of Research, was driving a lot of the research agenda.
[0:04:15] We built GPT-2, GPT-3, reinforcement learning from human feedback, which is, you know, of course, the method that's used in chat GPT and used along with other methods in our model, Claude. And, you know, one of the big themes of those five years.
[0:04:35] was this idea of scaling, that you can put more data, more compute into the AI models and they just get better and better. And I think that, you know, that thesis was really central. And the second thesis that was central is you don't get everything that way. You know, you can scale the models up, but there are questions that are unanswered. It's, you know, ultimately sort of the fact value distinction. You scale the model up, it learns more and more about the world, but you're not telling it how to act, how to behave.
[0:05:03] And so those were really kind of the two lessons that I learned. And of course, those ended up being the two things that Anthropic is really about. What year was it that you joined OpenAI? It was 2016. And what was the original?
[0:05:22] Was it just that this seemed to be where the smart people were going? Yeah, so I was actually initially invited to join the organization before it existed in late 2015 as it was forming and decided not to. But then a few months after it started, I kind of, you know, a bunch of smart people ended up joining. And so I said, oh, maybe I'll do this after all.
[0:05:50] You were there for a number of years, and at some point you made the decision that Anthropic was going to be, I guess it wasn't initially, you've had an unusual path. It wasn't initially a company, right? Originally it was started as a quasi-research lab. Is that fair? I mean, our strategy has certainly evolved, but actually it was a for-profit public benefit corporation since the beginning.
[0:06:14] And we had, even since the beginning, we had something on our website saying, you know, we're doing research for now, but, you know, we see the potential for commercial activity down the road. So I think all of these things we kind of kept open as potentialities, but, you know, you're right. For the first year and a half, it was mostly...
[0:06:32] Building technology and we were agnostic on what exactly we were going to do with that technology or when. We kind of wanted to keep our options open. We felt it was better than saying, you know, we're about this or we're about that. And what was the thought at the time of, hey, we should go do something on our own? Was that your thought? Was it a group of people? Yeah, it was definitely the thoughts of a group of people. So there were seven co-founders who left.
[0:07:00] And then I think in total we got 14 or 15 people from OpenAI, which was about 10% of the size of the organization at the time. You know, it's funny to look back on those days because, you know, in those days we were the language model part of OpenAI. Like, you know, we, along with a couple of people who stayed, you know, were those who had developed and scaled.
[0:07:24] The language models, there were many different other things going on at OpenAI, right? There was, you know, there was a robotics project, a theorem proving project, projects to play video games. Some of those still exist. But, you know, we felt that we were this kind of coherent group of people. We had this view about...
[0:07:42] OpenAI had a whole bunch of different things and still does, experimenting around. Was it evident? At what point along the way was it evident that...
[0:08:11] Large language models were something that there was a lot of wood to chop and a lot of opportunity around. Yeah, I mean, I think, I don't know, it was obvious at different times to different people. So, you know, for me, I think there were a couple things. You know, the general scaling hypothesis. I wrote this document called The Big Blob of Compute in 2017, which I'll...
[0:08:35] I'll probably publish at some point, although it's primarily of historical interest now. And so very much in my mind, and I think the minds of a small number of other...
[0:08:47] Of other people on both the team that left and the team that didn't leave, and some in other places in the world as well, it was clear that there was really something to scaling. And then as soon as I saw GPT-1, which was done by Alec Radford, who's still at OpenAI, our team actually had nothing to do with GPT-1, but we recognized it immediately and saw that the right thing to do with it was to scale it.
[0:09:13] And so for me, everything was clear at that moment. And even more clear as we kind of scaled up to GPT-2 when we saw that the model was capable of... My favorite thing was like...
[0:09:24] You know, we were able to get the model to perform a regression analysis. So, you know, you give it like, you know, the price, the price of a house and ask it to predict number of square feet or something like that. You gave it a bunch of examples, then you gave it one more price and you're like, how many square feet? It didn't do great, but it did better than random. And in those days, I'm like, oh, my God, this is like some kind of general reasoning and prediction engine. Like, oh, my God, what is this thing I have in my hands? Right. It's it's completely crazy. So.
[0:09:51] It has been my view ever since then that this would be not just language models, but language models as an exemplar of the kinds of things you can scale up. This would be really, really central to the future of technology. Did you consider yourself like a founder or a CEO prior to actually doing it?
[0:10:15] No, so I really kind of never thought of myself that way. Like if you went back to, you know, me in childhood, it would have been very unsurprising, you know, for me to be a scientist. But, you know, I never kind of thought of myself as like a founder or a CEO or a business person, right? You know, I always thought of myself as a scientist, someone who discovers things. But I think just having been at several different organizations.
[0:10:40] convinced me that, in fact, I did have my own vision of how to run a company or how to run an organization because I'd seen so many and I thought, well, I don't know, I'd actually do it this way, right? And so sort of the contrast, not that I disagreed with every decision that made, but just watching all these decisions go by took me to the point where I'm like, actually, I do have opinions on these questions, right? I do have an idea of how you would grow an organization, how you would run a research effort, how you would...
[0:11:09] Bring the products of that research organization out into the world in a way that makes business sense but is also responsible. I don't think I really had those thoughts naturally, but as I was brought into contact with organizations that did that, then I became excited about those things.
[0:11:26] Almost reluctantly learned that I actually had strong opinions on these things. Not to draw it in contrast to any specific names, but maybe just in others in the field that I would consider large language foundation model as a product. What's something foundational, I guess not to use a cute term, but something foundational that you believe at Anthropic that you would draw in distinction to?
[0:11:54] Yeah, I would say a couple things. So, you know, one is just this idea that we should be building in safety from the beginning. Now, I'm aware we're kind of not the only ones who've said that, but I feel like we've built that. We've really built that in from the beginning. We've thought about it from the beginning. We've started from a place of caution and kind of...
[0:12:20] You know, commercialize things, brought things out into the world, starting from, hey, you know, can we open these switches one by one and see what actually makes sense? I think in particular, you know, a way I would think about it is that what we're aiming to do is not just to be successful as a company on our own, although we are trying to do that, but that we're also trying to kind of set a standard for the field, set the pace for the field. So this is a concept we've called Race to the Top.
[0:12:49] So, you know, race to the bottom is a popular term where, you know, everyone is, you know, competing to, you know, lower cost or delivers things as fast as possible. And as a result, they cut corners and things get worse and worse. So that dynamic is real. And we always think about how not to contribute to it too much. But there's also a concept of race to the top, which is that if you do something that looks better, it naturally has the effect that other players end up doing the same thing.
[0:13:16] And this happened for interpretability. For a couple of years, we were the only org that worked on interpretability seen inside neural nets. There are various corporate structures that we've implemented that we hope others may emulate. And recently, we released this responsible scaling plan that I could talk more about later. But generally, we're trying to set the pace. We're trying to do something
[0:13:45] Good, inspiring, also viable, and encourage others to do the same thing. And at the end, again, maybe we win in a business sense, and of course that's great, but maybe someone else wins in a business sense, or we all win, we all split it, but the thing that matters is that the standards are increasing. I want to talk about all that stuff in a second, but why do you think philosophically that like...
[0:14:09] AI development or the scaling of models and safety. Why are they intertwined? I've heard you maybe coiled together in different ways. Yeah, yeah. So I think this actually isn't such an unusual thing. Like, I think this is true.
[0:14:24] It's true in most fields, right? The common analogy is like bridges. Building a bridge and making it safe, they aren't exactly the same thing, but they both involve all the same principles of civil engineering. And it's kind of hard to work on bridge safety in the abstract aside from...
[0:14:45] You know, kind of outside the context of like a concrete bridge. Like, you know, what you need to do is you need to look at the bridge. You're like, OK, well, these are the forces on it. You know, these are the this is the stress tensor. This is.
[0:14:56] You know, the strength of the material or whatever. That's the same thing you come up with in building the bridge. If safety differs in any way, maybe it differs in thinking about the edge cases. Like in safety, you have to worry what goes wrong 0.1% of the time. Whereas in building the bridge, you have to think about the median case. But, you know, it's all the same civil engineering. It's all the same, you know, forces of mechanical physics.
[0:15:26] AI and large language models. And in particular, safety is itself a task. Like, you know, is this thing the model's doing right or wrong? Where, you know, right or wrong could be something as prosaic as is the model telling you how to hotwire a car or as, you know, scary and sophisticated as is the model going to help me build a bioweapon or is it going to, you know, take over the world and make swarms of nanobots or, you know, whatever futuristic thing.
[0:15:53] Figuring out whether the model is going to do that and the behavior of the model is itself an intellectual task of the kind that models do. And so the problem and the solution to the problem are mixed together in this way, where every time you get a more powerful model, you also gain the capability to understand and potentially rein in the models. So we have this problem where these two things are just mixed together in a way that's...
[0:16:19] I think hard to untangle. And I think that's the usual thing. I think the only reason that's surprising is that the community of people who thought about AI safety was historically very separate from the community of people who developed AI. They had a different attitude. They came to it from a more philosophical...
[0:16:41] It was becoming a business and commercializing your effort. If you could scale and figure all the safety stuff out without being a business, would you pick that?
[0:17:09] Yeah, yeah. So I'd say a few things on that. One is I think it's actually going to be very difficult to build or would have been very difficult to build models of the scale that we want without being a commercial enterprise.
[0:17:29] People make jokes about VCs being willing to pour huge amounts of money into anything. But I think that's only true up to a point. There's business logic behind it. You guys have LPs. It's not just all hype terrain. Things need to make sense eventually. And so we're now getting to the point where you need...
[0:17:54] Certainly multiple billions of dollars and I think soon tens of billions of dollars to build models at the frontier and to study safety with those models at the frontier requires you to have intimate access to those models, particularly for tasks like interpretability. So first of all, yeah, I just think it's very hard. On the other hand, you know, or in support of that point.
[0:18:18] I also think that there are some things that you learn from the business side of things. Some of it is just learning the muscle and the operation of things like trust and safety.
[0:18:34] Today, we deal with trust and safety issues like, oh, people are trying to use the model for inappropriate purposes, right? Not things that are going to end the world, but things that we'd rather that people not do. I think the ultimate significance of being able to develop methods to address those things and...
[0:18:57] Kind of enforcing those things in practice when they're used at scale by users is it allows us to practice for the cases that are really, really high stakes. And I think without that organizational institutional practice, it might be difficult to kind of just be thrown into the shark tank.
[0:19:14] Congratulations, you've built this amazing model. It can cure cancer, but also someone could make a bioplague that would kill a million people. You've never built a trust and safety org. You have to deploy this model in the world and make sure we do one and not. That would just be a very difficult thing to do, and I don't think we would get it right. Now, all that said, I mean.
[0:19:33] I will freely admit, you know, my passion is the science and the safety, right? You know, that's kind of my first passion. You know, the business stuff is quite a lot of fun. You know, I think, you know, we just watched all the different customers, you know, just learning about the whole business ecosystem has been great.
[0:19:57] But, you know, definitely my first passion is, you know, is the science of it and making sure it goes well. Was there a serious debate about, like, being a business versus not in the early days? Was that like a real conversation among the group? Yeah. So I think certainly everyone was aware from the beginning that, you know, we, you know, that there was a good chance that we would commercialize the models at some point.
[0:20:23] I'm not sure if it's up there anymore, but you can see it on the Wayback Machine that said, you know, for now we're doing research, but we see commercial potential down the road. So everyone who joined saw that and everyone who joined knew that. But there was a question of, you know, when exactly should we do it? So there was a period around...
[0:20:42] I think it was April, May, June of 2022 when we had kind of the first version of Claude, which was actually like a smaller model than Claude 1, but we were training the model that would become Claude 1 at that time. And we realized that with RL from human feedback, we didn't have our constitutional AI method yet, that this thing was actually great to interact with. And all of our employees were having fun interacting with it on Slack. We showed it to some...
[0:21:10] To a small number of external people, and they had lots of fun. They had lots of fun interacting with it. So it definitely occurred to me and others that, hey, there could have been a lot of commercial potential to this. I don't think we anticipated the explosion that happened at the end of the year. We definitely saw potential. I don't think we saw that much potential. But yeah, we definitely had a debate about it, and I wasn't sure quite what to do. I think our concern was that with the rate at which the technology was progressing,
[0:21:39] A kind of big, loud public release might accelerate things so fast that the ecosystem might not know how to handle it. And I didn't want our kind of first act on the public stage after we put so much effort into being responsible to accelerate things so greatly.
[0:22:02] I generally feel like we made the right call there. I think it's actually pretty debatable. There's many pros, many cons, but I think overall we made the right call. And then certainly as soon as the other models were out and kind of the gun had been fired.
[0:22:19] Then we started putting these things out. We're like, okay, all right. Now there's definitely a market in this. People know about it. And so we should get out ahead. And indeed, we've managed to put ourselves among the top two or three players in this space. Was that gun being fired and chat GPT sort of taking off, was that similar to the maybe?
[0:22:41] Fear that you had of like, hey, this might start a race? Yeah, yeah, similar and in fact more so. So, you know, I think we saw it with, you know, Google's reaction to it, you know, that there was definitely, you know, just judging from the public statements, you know, a sense of fear and existential threat. And, you know, I think they responded in a very economically rational way. I don't blame them for it at all.
[0:23:05] But you put the two things together and it really created an environment where things were racing forward very quickly. And look, I love technology as much as the next person. There was something like super exciting about the whole make them dance. Oh, we're responding with something. I mean, I can get just as excited about this as everyone. But given the rate at which the technology is progressing, there was a worrying aspect about this as well.
[0:23:31] And so in this case, I'm at least at least on balance, Claude, that, you know, we weren't the ones who fired that starting gun. Yeah, got it. Well, you recently announced an investment from Amazon. Before that, you did a round with Spark and a little bit more traditional venture capitalists. The round with Amazon, it's complicated and I can't go into the details, but it's not a full closed share price for corporate round and all that. Before that, you had an unusual round as well with FTX, right? How did that come to be?
[0:24:00] Yeah, so, you know, honestly, there was actually very little to that. So, you know, there's kind of a community of people who cared a lot about AI safety. You know, back when he was doing FTX, before he committed all the fraud or was caught committing all the fraud that he committed, Sam Bankman Freed was, you know, presented himself as someone who cared a lot about issues like pandemics, AI safety.
[0:24:25] So he was known to people in my community. And honestly, there's not much to tell. I only talked to him a handful of times. The entity is still related to FTX, right? So there's potential that the Anthropic investment could one day make FTX people whole, depending on how it all plays out. Yeah, that's one of the ironies. There's a bankruptcy estate or bankruptcy trust that owns these non-voting shares. And so far they've...
[0:24:55] They've declined to sell them off, but they're interested in doing so in a general sense. I'm told there's people that are very interested in buying those shares from them. I can't comment on the market dynamics there, and we don't really control them. It's a sale between different parties. But hey, if those shares lead to the people who had their money stolen getting some or all of their money back, then that's kind of a random chance, but certainly a good thing.
[0:25:23] Good outcome. So what is the business of Anthropic look today? You guys are focusing mostly on enterprise customers? Yeah, we are focusing mostly on enterprise customers. I would say we have both an enterprise product and a consumer product. It makes a lot of sense if you're building one of these models to...
[0:25:38] Thank you.
[0:26:04] At the same time, you know, our real focus definitely is enterprise. We've found that some of the properties of the model in a practical sense, right, the safety properties of the model in a very practical sense, as opposed to kind of, you know, like a philosophical or future sense, are actually useful for the enterprise use cases. You know, we try to make our models helpful, honest, and harmless.
[0:26:29] Honesty is a very good thing to have in knowledge work settings. A number of our customers are in the finance industry, the legal industry, starting to get stuff on the health side, different productivity apps. Those are all cases where a mistake is bad. You're doing some financial analysis. You're doing some legal analysis. You really have a premium on like...
[0:26:55] Make sure the thing knows what it doesn't know. So giving you something misleading is much worse than not giving you anything at all. I mean, that's true across the board, but I think it's especially true in those industries. For enterprises, often kind of like inappropriate or embarrassing speeches, something that they're very concerned about, even if it happens very rarely.
[0:27:21] And so the ability to kind of steer and control the models better, I think, is very appealing to a number of enterprise customers. Another thing that's been helpful is this, you know, we have this longer context window. So context window is like how much information the model can take in and process. So our context window is 100,000 tokens. Tokens are this weird unit. It really corresponds to 70,000 words. But, you know, the next, the model with the next.
[0:27:50] Biggest context window is GPT-4, where there's a version of it that has 32K tokens, 32,000, which is three times less, but the main GPT-4 has 8,000, which is about 12 times less. And so just the ability to, for example, something you can do with Claude that you can't do with any other model is, you know, read a mid-sized book or novel or textbook or something, just stick it into the context window, upload it, and then start to ask questions about it. And so that's something you can't.
[0:28:20] I can't go into the details, but we've worked a lot on...
[0:28:42] Algorithmic efficiency for both training and inference. So we're able to produce something that's in the same ballpark as GPT-4 and better for some things. And we're able to produce it at a substantially lower cost. And we're in fact excited to extend that cost advantage because we're working with custom chips with various different...
[0:29:03] And we think that could give an enduring advantage in terms of inference costs. So all of those are particularly helpful for the enterprise case. And we've found pretty strong enterprise adoption.
[0:29:18] Even in the face of competition from multiple companies. Hi, I'm Logan Bartlett, the host of this podcast. This is not an ad. As you may know, we do not advertise or monetize this podcast in any way. I just wanted to take a quick second to tell you that we have a bunch of killer guests coming on over the course of the next few weeks. And so if you're enjoying these conversations behind the scenes with both entrepreneurs and investors, please do subscribe to our channel so you don't miss out. Back to the episode.
[0:29:46] What's something that's perhaps unintuitive to someone that isn't living and breathing this every day about enterprise interest in artificial intelligence as someone sitting at that nexus? First of all, I see a huge advantage to folks who think in terms of the long term. So there's a fact that's kind of, you know, the bread and butter for those of us who are building the technology.
[0:30:11] Getting it across to the customers is, I think, one of the most important things, which is the pace at which the models are getting better. Some of them get it. Others are starting to get it. But the reason this is important is, you know, put yourself in the shoes of a customer, right? They've got our model, Claude. They want to build something. And, you know, typically they want to start with something small. And, of course, naturally they think in terms of what can the model do today. And what I always say is do that. We've got to start. We've got to iterate from somewhere.
[0:30:40] But also think in terms of where the models are going to be in one or two years. It's going to be a one or two year arc to go from proof of concept to small scale deployment to large scale deployment to true product market fit for whatever it is that you're launching. So you should basically skate where the puck is going to go. You should think, OK.
[0:31:00] The models can't do this today, but look, they can do it 40% of the time. That probably means they can do it 80% or 90% of the time in one or two years. So let's have the faith, the leaf of faith, to build for that instead of building for what the models are able to do today. And if you think that way...
[0:31:20] The possibilities of what you can do in one to two years are much more expansive. And we can talk about having a kind of longer-term partnership where we build this thing together. And I think the customers that have thought that way are ones that we've been able to work together with on a path towards creating a lot of value. We also do lots of things that are just targeted at what you can do today.
[0:31:46] This is something that I think is...
[0:32:03] Particularly true of anyone and particularly any leader. But you had said recently attaching your incentives to the approval or cheering of a crowd in some ways destroys your mind and in some ways can destroy your soul. You haven't been as public as other folks in the space have been. I assume that's very.
[0:32:24] Purposeful stylistically and ties into that. Can you talk a little bit about your thoughts around it? Yeah, I mean, part of it, it's just kind of my style, for one thing. I mean, you know, I think as a scientist, you know, I prefer to speak when there's kind of something clear and substantive to say. You know, I mean, I'm not totally low profile. I mean, I'm on this podcast right now, and I've been on a few. So, you know, given the general volume of the field, there's some need to get the message out there, and that need will probably increase over time.
[0:32:54] But I think I have noticed, and it's not just Twitter or social media, but some phenomenon that's a little bit connected to them, that you can really, if you...
[0:33:05] Don't think too much in terms of kind of pleasing people in the short, short term or, you know, making sure that you say something, making sure that you say something popular. It can really kind of lead you down a bad path. And I, you know, I've seen that with a lot of, you know, with a lot of very smart people. I mean, I'm sure you could name some as well. I'm not going to give any names who have gotten caught up in this. And, you know, years later, you look at them and you're like, wow, this is a really smart person who's who's acting much dumber than they are.
[0:33:32] And I think the way it happens, I don't know, I could give an example, right? So take a debate that's important to me, which is like, should we build these things fast or should we make these systems safe? So there is an online community, mostly on Twitter, of people who are...
[0:33:47] You know, think we should slow down and then kind of an online community of builders who are really excited about, you know, we should make this stuff fast. And if you go to certain corners of Twitter, like you get these really extreme versions of each one, right? On one hand, you get people who say, like, we should stop building AI. We should have a global worldwide pause, right? I think that doesn't work for a number of reasons. I mean, we have our responsible scaling plan is sort of incorporate some aspects of that. So I think it's not an unreasonable.
[0:34:15] You know, discussion or debate to have. But, you know, there's this kind of really extreme position. And then that's kind of created this polarization where there's this other extreme position that's like we have to build as fast as possible. Any regulation is just regulatory capture. You know, we just need to maximize the speed of progress. The most extreme people say things like it doesn't matter if humanity is wiped out. AIs are the future. I mean, that's a really extreme position. And so, you know, think of kind of.
[0:34:45] The position of someone who's kind of trying to be thoughtful about this, trying to build but build carefully, if you kind of enter that fray too much, if you feel like you have to make those people happy, what can end up is either you get polarized on one side or another, and then you kind of repeat all the slogans of that side, and you become a lot dumber than you would otherwise.
[0:35:09] You know, if you're really good at dealing with Twitter, you know, you can try and make both people happy. But, you know, that involves a lot of playing to both sides. And I certainly don't want to do that. That's what I talk about in kind of losing your soul. You know, the truth is the actual, you know, the position that I think is actually responsible might be something that would make all of those people boo instead of all of them cheer, right? And so you just have to be very careful if you're...
[0:35:39] If you're taking that as your barometer, you know, who's yelling at me on Twitter, who thinks I'm great on Twitter, you're not going to arrive at the position that makes everyone boo. That might just be the correct position. What timeline do you think about then when you're saw? It's not the instantaneous dopamine of a tweet. You mentioned talking to enterprises about one to two years and what can be, but like what timeline are you solving for? Yeah, I mean, I guess I kind of think like five to ten years from now, everything will be like a little bit more clear.
[0:36:08] And, you know, I think it'll be more clear which decisions were good decisions, which decisions were bad decisions. You know, I think, you know, certainly less than that timescale is a timescale on which, you know, if dangerous things with these models are indeed possible, as I believe they are, but I could be wrong, I think they may play out on that timescale. And, you know, we'll, you know...
[0:36:34] We'll be able to see which companies addressed these dangers well. Or were the dangers not real and people like me warned about them and we were just totally wrong? Or will it turn out that some tragedy happened and people like me should have been more extreme in warning about it?
[0:36:53] Or will it turn out that companies like Anthropic picked the right path and navigated a dangerous situation? Well, I don't know how it's going to turn out. I mean, I hope it turns out that we navigated a dangerous situation well and we averted catastrophe and there were hard trade-offs and we addressed them skillfully and thoughtfully. That's my hope for how it's going to turn out. But I don't know that it's going to turn out that way. But I feel like looking at that,
[0:37:22] In five years, in 10 years, that's just going to be a fair judgment of all the things that I'm saying and doing. What would you want the average person listening who is aware of AI, knows what Anthropic is, knows what OpenAI is, knows what Google and others are doing in the space about safety and about risk? What would you want them to know from your perspective? Yeah, so I think if I were to kind of just put it in a few sentences.
[0:37:50] I think what I would say is, look, I have two concerns here. One is the concern that people will misuse powerful AI systems. People misusing technology is nothing new. But one thing that I think is new about AI is that its ability to put all the pieces together is much greater than any previous technology.
[0:38:14] I think in general we've always been protected by the fact that, you know, if you take a Venn diagram of people who want to do really bad things and, you know, people who have strong technical and operational skills, generally the overlap has been pretty small. If you're a person who has a PhD or is capable of running a large organization, you have better things to do than come up with evil plans to, you know, murder people or destroy society, right? It's just...
[0:38:40] Not very many people are motivated in that direction. And then the people who are, often they're just, I mean, not all of them, but in many cases, not that bright or not that skilled. The problem is, now could we take unskilled person plus skilled AI plus bad motives? And so I testified in Congress about this, about the risk of bioweapons. I think cyber is another area.
[0:39:07] A bunch of stuff around national security and, you know, the relationships between nations and stability. So that's one corner of it. And then I think the other corner of it is what the AI systems themselves may do. And, you know, there's lots you can, you know, kind of find in the Internet and various communities on this. But I often put it in a simple way, which is, one, the systems are getting much more powerful.
[0:39:32] We, there's obviously not much of a barrier to getting the systems to act autonomously in the world, right? People have taken GPT-4, for instance, and turned it into auto GPT. There was even a worm GPT, which was, you know, supposed to act as a computer worm. So, you know, powerful smart systems that can take action. So, you know, kind of very long leash of human supervision. And because of the way they're trained.
[0:40:00] They're not easy to control. I mean, we all saw Bing in Sydney. So you put those three things together, and there's at least, I think, some chance that as the systems get more and more powerful, they're going to do things that we don't want them to do, and it may be difficult to fully control them, to fully rein them in. I think that's further out than the misuse, but it's something we should think about.
[0:40:25] We've touched on a few of the different things you guys have done from a safety standpoint. So I want to talk through, I guess, the three ones I took down. So long-term benefit trust and public benefit corporation. Can you explain what that is and how you decided to do that? Yeah, yeah. So we were incorporated as a public benefit corporation from the beginning. Which means? Basically, public benefit corporation, it's actually very much like a C-corp, except the investors...
[0:40:54] Can't sue the company for failing to maximize profits. I think in practice, in the vast majority of cases, it operates like a normal company. I mean, I think that's one theme I want to get through here. Like 99% of what we do, we would make the same decision that a normal company would. Most of the time, the...
[0:41:21] Logic of business, which is basically the logic of providing mutual value, also makes sense from a public benefit corporation. But there's maybe this 1% of key decisions. I might think about the delay of release of Claude.
[0:41:40] Decisions that might relate to, hey, we have a very powerful model, but we need to make really sure that this thing can't create a bioplague that'll kill millions of people before we release it. So I think there are going to be a few key moments in the company where this makes a difference. And then LTBT, as I said, a public benefit corporation is not that different from a C-corp. The idea of the LTBT is to have a set of...
[0:42:09] So LTBT is long-term benefit trust. So right now, the governance of Anthropic is pretty much like that of a normal corporation. But we have a plan that was written into our original Series A documents and has been iterated on since then that will gradually hand over the ability to appoint a majority of Anthropic's board seats to a kind of trust of people. And on that trust of people, we've...
[0:42:35] We selected the original ones, but then it becomes self-sustaining. We selected for kind of three types of experience. One type is experience in AI safety. One type is experience in national security topics, as I think this is going to become relevant. And another type is, you know, thinking about things like philanthropy and the macroeconomic distribution of income.
[0:42:59] So I think of those as my best guess as to kind of the, you know, the three topics where something that kind of, you know, kind of transcends the kind of ordinary activity of companies is going to come up. And is that who you ultimately report into when this structure is finalized? That'll be the board that Anthropic answers to? So this set of five people appoints a majority, but not all of the corporate board of the company.
[0:43:27] So there's basically two of these bodies. And the LTBT appoints the corporate board. Now, look, that said, I mean, we all kind of, you know, in practice, the company is almost always run, you know, day to day by the CEO, right? Like, you know, even speaking of the corporate board.
[0:43:49] Not just for Anthropic, but any other company. I mean, think of as a CEO how many decisions you directly make yourself versus how many it's like, oh, I have to get the board on board with that. I mean, there are some when you raise money, when you issue new employee shares, when you make a major strategic decision. The LTBT is kind of an even more rarefied body.
[0:44:15] And I've set the expectation with them that their role is to get involved in the things that really involve critical decisions for humanity. There might only be three or four such decisions in the entire history of Anthropic. Now, constitutional AI, can you talk about what that is and what the inputs into it were? Yes. So constitutional AI is a method that we developed around the end of last year.
[0:44:41] So easiest to explain it by contrasting it with this previous method called reinforcement learning from human feedback, which I and some other people were the co-inventor of at OpenAI in 2017. So reinforcement learning from human feedback, the way it works is, okay, I've trained the giant language model. I've paid my tens of maybe hundreds of millions of dollars to train it. And now I want to give it some sense of how to act.
[0:45:09] So, you know, there are questions you can ask the model that don't have any clear factually correct answer. So, you know, I could say, what do you think of this politician? Or what do you think of this policy? Or what should I as a human do in this situation?
[0:45:23] And, you know, the model doesn't have any definite answer to that. So the way RL from human feedback works is you hire a bunch of contractors, you give them examples of how the model is behaving, and the humans kind of, you know, they kind of give feedback to the model. They say this answer is better than that answer. And then over time, the model updates itself to learn to do whatever is in line with what the human contractors say.
[0:45:48] One of the problems with this is, you know, one, it's expensive. It requires a lot of human labor. But in addition to that, it's very opaque, right? You know, if I serve the model in public and then, you know, someone says, hey, why is this model biased against conservatives? Or why is this model biased against liberals? Or why does this model just give me weird sounding advice? Or why does it, you know, why does it give things in a weird style?
[0:46:12] I can't really give any answer. I can just say, well, I hired 10,000 contractors I don't know, and this was the statistical average of what the contractors generally proposed or the mathematical generalization of it. It's not a very satisfying answer. So the method we developed is called constitutional AI, and the way that works is you have a set of explicit principles that you give to the model.
[0:46:35] So the principles could be something like on a political question, present both sides of the issue and don't take a position yourself. Say here are some arguments for, here are some typical arguments against, opinions differ. So with that, you basically, just as with RL for human feedback, you have the model give answers, but then you have the model critique its own responses.
[0:47:03] for whether they're in line with the model constitution. And so you can run this in a loop. Basically, the model is both the generator and the evaluator with the constitution as kind of the pin source of truth.
[0:47:15] And so this allows you to eliminate human contractors and instead go from this set of principles. Now, in practice, we find it useful to augment that method with human contractors so that you can get the best of both worlds. But you use less human contractors than you were before. You have more of a guiding principle. And then if someone calls me up in Congress and says, hey, why is your model woke or why is your model anti-woke or why is your model doing this crazy thing?
[0:47:42] You know, I can point to the Constitution and I can say, hey, these are our principles. You could have one of two objections. Maybe you don't agree with our principles. You know, fine, we can have a debate about that. Or it's a technical issue. These are our principles. Somehow the train of our model, you know, wasn't perfect and wasn't in line with those principles. And I think separating those two things out is, I think, very useful. And even like enterprise customers have found this to be a useful thing, the kind of customizability and the ability to separate the two out.
[0:48:10] And the inputs into this were, so you used the UN Declaration of Human Rights, Apple's Terms of Service. Yes. What else went into coming up with the principles? Yeah, there were some principles that were developed for use by an early DeepMind chatbot. But yeah, Apple's Terms of Service, UN Declaration of Human Rights. We added some kind of other things, like we asked the model to respect copyright.
[0:48:36] So this is one way to, you know, to greatly reduce the probability that the model outputs copyrighted text verbatim. You know, we can all debate, you know, what the status is of, you know, models that we train on corpuses of data. But we can all agree that on the output side, we don't want the model to output vast reams of copyrighted text. That's a bad thing to do. And we aim not to do that. And so we just put it in the Constitution not to do that. One of the other things that you introduced around
[0:49:04] Safety, broadly speaking, is responsible scaling policy. Can you talk about what that is? Yes. So a responsible scaling policy is this set of commitments that we recently released that is a framework for how to safely make more and more powerful AI systems and confront the greater and greater dangers that we're going to face with those systems. So maybe the easiest way to understand it is to think of kind of the two sides of the spectrum. So one extreme side of the spectrum is like...
[0:49:33] Build things as fast as possible, like, you know, release things as much as possible. You know, maximize technological progress. And I, you know, I understand that position and have sympathy for it in many other, you know, in many other contexts. I just think, you know, AI is a particularly tricky technology. You have to put E slash ACC in your Twitter bio if you believe that, I think.
[0:49:56] Maybe I should put both. And so the other extreme position, which I also have some sympathy for, despite it being absolutely the opposite position, is, oh my God, this stuff is really scary. And the most extreme version of it was we should just pause. We should just stop building the technology.
[0:50:17] You know, indefinitely or for some specified period of time. And I think my problem with that has always been, well, okay, let's say we pause for six months. What do you actually gain from that? What do you do in those six months?
[0:50:30] Particularly with the more powerful models being needed for safety of more powerful models, it's kind of like you've frozen time, you've stopped the engine. What do you get at the end of it? And if you were to pause for an indefinite length of time, then you raise these questions like, well, how do you really get everyone to stop? There's an international system here. There's dictators who want to use this stuff to take over the world. I mean, you know, people use that as an excuse, but it's also true.
[0:50:53] And so that extreme position doesn't make too much sense to me either. But what does make sense to me is, hey, let's think about caution in a way that's actually matched to the danger. Right now, whatever we're worried about in the future, right now, today's systems, they have a number of problems. But I think they're the problems that come with any new technology, not these kind of special problems of bioweapons that would kill millions of people or the models.
[0:51:22] You know, kind of taking over the world in some form. So, you know, let's have relatively normal precautions now, but then let's define a point at which, you know, when the model has certain capabilities, we should be more careful. So the way we've set this up is we've defined something called AI safety levels. So there's something called biosafety levels in the U.S. government, which is like...
[0:51:48] For a given virus, you define how dangerous it is. It gets categorized as like BSL-1 or BSL-3 or BSL-4. And that determines the kind of containment measures and procedures you have to take to control that virus. So we think of AI models.
[0:52:05] We think of AI models in the same way. There's value in working with these very powerful models, but they have dangers to them. And so we have these various thresholds between ASL 1 and ASL 2, between ASL 2 and ASL 3, between ASL 3 and ASL 4. And at each level, there's certain criteria that we have to meet.
[0:52:25] So right now we're at ASL 2 as we've defined it. Before we get to ASL 3, we have to develop security that we think is sufficient to prevent anyone who's not a super sophisticated state actor from stealing the model. That's one thing. Another thing is we have to make sure that when the models reach a certain level of capability,
[0:52:48] We're really, really certain that they're not going to provide a certain class of dangerous information. And to figure out what that is, we're going to work with some of the best biosecurity experts in the world, some of the best cybersecurity experts in the world, to understand what really would be dangerous compared to what can be done today with a Google search. We've defined these tests and these thresholds very carefully. And so how does that relate to the two sides of the spectrum? Compared to a pause...
[0:53:16] The ASL thresholds could actually lead us to pause because if we get to a certain capability of model and we don't have the relevant safety and security procedures in place, then we have to stop developing more powerful models. So the idea is...
[0:53:32] There could be a pause, but it's a pause that you can get out of by solving the problem. It's a pause you can get out of by developing the right safety measures. And so it incentivizes you to develop the right safety measures. And in fact, incentivizes you to avoid ever having to pause in the first place by proactively developing the right set of safety measures. And as we go up the scale, we may actually get to the point where you have to very affirmatively show the safety of the model, where you have to say,
[0:54:00] Yes, I'm able to look inside this model with an x-ray, with interpretability techniques and say, yep, I'm sure that this model is not going to engage in this dangerous behavior because...
[0:54:14] There isn't any circuitry for doing this or there's this reliable suppression circuitry. So it's really a way to shoehorn in a lot of the safety requirements, put them in the critical path of making the model. And hey, if you can be the first one to solve all of these problems and therefore safely scale up the model, not only will you have solved the safety problems, but that kind of aligns the business incentives with the safety incentives.
[0:54:41] Our hope, of course, is that others will adopt the responsible scaling plan and that eventually it can also be an inspiration for policy so that everyone is held to some version of the responsible scaling plan. And how does it relate to this other thing of build as fast as we can? Well, look, I mean, one way to think about it is like the responsible scaling plan.
[0:55:03] It doesn't slow you down except where it's absolutely necessary. It only slows you down where it's like there's a critical danger in this specific place with this specific type of model. Therefore, you need to slow down. It says nothing about stopping at some certain amount of compute or stopping for no reason or stopping for a specific amount of time. It says keep.
[0:55:25] I'm aware that since we published our responsible scaling plan,
[0:55:54] Several other organizations are internally working on responsible scaling plans. For all I know, one or more of them might be out by the time this podcast is out. Hopefully, they put out something. Hopefully, they try and make it better than ours. That would be a win for us. You have some aspects of your job, and you alluded to this earlier. 99% of your job probably looks mostly like a normal.
[0:56:16] Company would, but your time, I would guess, is probably not 99%. How much of your time is spent on stuff that is...
[0:56:25] Weird for a normal startup CEO testifying in front of Congress or whatever that bucket is versus just day-to-day operations of running a business? Or is it hard to disentangle? Yeah, I mean, it's so, I don't know. I would say it's maybe 75-25 or something like that. 75 normal? 75 normal, 25% weird. I mean, it certainly takes a lot of time to talk to a large number of customers, to hire for various roles, to look at...
[0:56:53] Financial metrics to inspect the building of the models. I mean, that eats up a lot of time. But I also spend a decent amount of time, say, talking to government officials about what the future is going to look like, thinking about the national security implications, trying to advise folks on what can go wrong. We did this whole project of working with some of the world expert biosecurity experts.
[0:57:19] What would it really take for the model to help someone to do something very dangerous? There are certain missing steps in bioweapon synthesis. For obvious reasons, I'm not going to go into what those steps are. I don't even know all of them. But we spent a good number of months and a decent amount of my personal time along with the incredibly hard work of the team that worked on it.
[0:57:42] Thinking about this and presenting it to officials within our government, within other allied governments. And that's just a pretty unusual thing to do. That feels like something more out of a military thriller or something like that. So that's unusual. Speaking to Congress is unusual. Thinking about...
[0:58:04] Where we're going to be in three or four years. Are the models going to run rampant on the internet or something like that? Spend a good deal of time thinking about how do we prepare for that scenario? Another thing is thinking about could the models at some point be morally significant entities? That's really wacky, really strange.
[0:58:32] I still don't know how to be sure of that or how you'd measure that. It might be an important thing. It might not be an important thing. But, you know, we take it seriously. And so there is definitely this weird juxtaposition of, like, you know, I'm, like, you know, looking for a chief product officer one day and, like, you know, thinking about bioweapons and, you know.
[0:58:54] Model moral status the next day. There was a Vox article that said something to the effect of an employee.
[0:59:03] Predicted there was a 20% chance that a rogue AI would destroy humanity within the next decade. To the reporter, I guess, that was around. I mean, does all this stuff weigh heavily on the organization on a daily basis? Or is it mostly consistent with a normal startup for the average employee? Yeah, so I don't know. I'll give my own experience. And it's kind of the same thing that I recommend to others.
[0:59:26] So, you know, I really freaked out about this stuff in 2018 or 2019 or so when, you know, when I first believed that, you know, turned out to be at least in some ways correct.
[0:59:38] That the models would scale very rapidly and, you know, they would have this importance to the world. Was there a specific thing that made you realize that or that you saw or was it a bunch of things coming together? Playing with GPT-2 is what did it for me. It's what made it real. I mean, GPT-3 was more so and, you know, Claude and GPT-4 are, you know, of course, even more impressive. But, like, the moment where I really kind of believed the scaling trends that we had been seeing that, you know, was really real and would lead to real things was, like, the first time I looked at GPT-3.
[1:00:07] I was like, oh my, this is crazy. There's nothing like this in the world. It's crazy that this is possible. And was it in particular the jump between the prior version to that and seeing that delta? Yeah, it was the delta and it was just the things that it was capable of. It felt like a general induction or general reasoning engine. For years after that, people said models couldn't do reasoning. I looked at GPT-2 and I'm like, yeah, you scale this thing up. It's really going to be able to...
[1:00:37] I see any pattern and reason about anything. I mean, again, we still don't know that for sure. This is still unsure. There are still people who say it can't. And for all I know, they're right. But that was kind of the moment that I saw it. And so, you know, I had a very, very difficult year or so where, you know.
[1:00:54] Where I tried to come to terms with what I believe to be the significance of it. Now, five years later, I'm much more in a position where it's like, this is the job we got to do, right? Where you sign up to do this, you have to be a professional and you have to address risk in a sensible way. And I found it useful to...
[1:01:15] You know, to think about the strategies used by people who professionally address risk or deal with dangerous situations, right? People who are on, you know, military strike teams, people in the intelligence community, people who kind of, you know, deal with, you know, high, you know, like high stakes, you know, critical decisions for, you know, for national security or disaster relief or something like that.
[1:01:44] They have techniques for thinking of these decisions rationally and making sure that they don't get caught up in them too much. And so I kind of try to adopt those techniques, and I've told other people in the org to think in that way as well.
[1:01:58] You had made a comment that you don't like the concept of personalizing companies in this whole, hey, the memification of a CEO in some regard. Is that just a personal tact to who you are? Do you think it's actually like a societal issue? Yeah, I mean, it's definitely my personal style. But, you know, I think this is closely connected to, you know, the thing about Twitter. Like, you know, I think people should think about companies and the incentives they have and the actual substance of the decisions they make.
[1:02:27] Nine times out of ten, you know, if someone seems kind of, you know, I don't know, charming or relatable or, you know, you talk to them on Twitter and it seems like there's someone who, you know, you could sit down with them and really like, you know, that could just be very misleading, right? It's, you know, it's not necessarily a bad sign, but I think it's pretty uncorrelated to like...
[1:02:47] What is the actual effect that the company is having in the world? And I think people should focus on that. And kind of we've tried to focus on that in terms of the structural elements, right? I'm not the only one who is ultimately responsible for these decisions. The LTBT is kind of designed as this.
[1:03:04] So, you know, everyone doesn't have to look, what's Dario going to do in this situation? And then Anthropic is only one company within a space of many other companies. There are also government actors there. And this is the way it should be. No one person, no one entity should have too much say over any of this. I think that's always unhealthy. We've talked about a lot of the negative sides or implications that come around with running Anthropic. I'm sure there's some
[1:03:33] Yeah, yeah. I mean, we can talk about, you know, amazing positive stuff in the short term, and I'm also excited about positive stuff in the long term. So maybe let's take those one by one. And, you know, I think, you know, I think we should talk more about the positive stuff. I mean, I...
[1:04:02] I often see it as my duty to make sure people are aware of the concerns in a responsible and sober way, but that doesn't mean I'm not excited about all the potential I am. So speaking about, you know, Anthropic in particular, I mean, you know, millions of people have signed up to use Claude. Thousands of enterprises are using it, and, you know, a smaller number of very large...
[1:04:25] A smaller number of very large players have started to adopt it. So, you know, I've just been excited by some of the use cases. Like, you know, when we look at, you know, particularly legal, financial, things like accounting, when, you know, when you see suddenly people are able to talk to documents, right? You can just, you know, you can just upload a company's financial documents. You can just upload a legal contract and ask questions that, you know, you would have needed a human to spend many hours on.
[1:04:55] And so, you know, this this is just in a very practical way. This is just like saving people's time and providing them with services that they just it would be very difficult for them to have otherwise. So I don't know. It's it's hard not to be excited by that. And of course, excited by, you know, all the all the amazing things the technology can do. I mean, you know, I know of someone who is like, you know, used it to like translate math papers from Russian. And, you know, they were good enough that, you know, it all it all made sense. And they were able to.
[1:05:22] They were able to understand something that would have been very difficult for them to understand it before. In the long run, I mean, I'm even more excited. I mean, I've talked about this a little before, but having been in biology and neuroscience, I'm very convinced that the limiting factor there was that the basic systems were getting too complicated for humans to make sense of.
[1:05:49] If you look at the history of science, things like physics, there's very simple principles in the world. We manage to solve those because, I mean, physics is not fully solved, but many parts of the basic operation of our world we understand. And then within biology, things like viral disease or bacterial disease, it's very simple. There's something invading your body.
[1:06:14] You need to find some way to kill the invader without hurting yourself. And because you and the invader are pretty different biologically, it's not that hard. So we've solved the problem. What's left is things like cancer, Alzheimer's disease, the aging process itself, to some extent, things like heart disease. And I worked on some of those things in my career as a biological scientist.
[1:06:41] Just the complexity of it, right? It's like, you know, you're trying to understand, you know, how proteins, you know, build cells and how the cells get dysregulated. It's like, you know, there's like 30,000 different proteins and each one of them has like, you know, 20 different post-translational modifications. And each of those interacts with the other proteins in this like really complicated web, you know, that makes one cell run. And that's just one type of cell. And there's like hundreds of other type of cells.
[1:07:08] And so one of the things we've already seen with the language models is that they know more than you or I do. A language model, they know about the history of samurai in Japan at the same time as they know about the history of cricket in India at the same time as they can tell you something about the biology of the liver or something like that. You list enough of these topics and there's no one on earth who knows.
[1:07:37] Who has that breadth, even to the level that the language model does, even with all the things that it says wrong right now. And so my hope is that in terms of biology, right now we have this network of thousands of experts who all have to work together. If you can have one language model that can connect all the pieces, and not just kind of like big data will help biology. That's not my thesis here. My thesis is that they'll be able to do...
[1:08:07] and work along with the humans a lot of things that human biologists, human medicinal chemists do and really track the complexity and be a match for the complexity of these disease processes that are happening in our body. And so I'm hopeful that we'll have another kind of renaissance of medicine like we had in the late 19th century or early 20th century when, you know,
[1:08:31] All these diseases we didn't know how to cure. We're like, oh, we discovered penicillin and we discovered vaccines. So I'll take cancer as one. Any biologist or medicinal chemist who I said, could we cure cancer in five years? They'd be like, that's fucking insane. There's so many different types of cancers. We have all these breakthroughs that handle one really narrow type of cancer. I think if we get this AI stuff right, then we...
[1:08:57] Maybe we could really do that. So I know it's hard to be more inspiring than that. Yeah, totally. You said you've been right about a lot of things related to AI, but you've also been wrong and surprised by a bunch. What have you been most wrong about or surprised by?
[1:09:12] Yeah, I don't know. So, I mean, I've been wrong about a bunch of stuff. Like, I think how this prediction stuff works is, like, if you're thinking about the right things, if you're predicting the right things, you know, you only have to be right about, like, 20% of stuff for it to have these huge, huge consequences, right? If you predict five things that, like, no one in the world thinks is going to happen and it would have enormous consequences and you're right about one of them. I mean, it's a little bit like VC, right? You know, it's like, you know, if you, you know, if in...
[1:09:41] In 1999, you invested in Google and four companies that no one heard of. That's a pretty good portfolio. End up being wrong about lots of stuff. So like one example of that, I don't know, I could come up with a few examples, but one is I thought certainly going back in like 2019 or so when I first kind of saw the scaling situation, I thought that we were going to scale for a while with these pure language models.
[1:10:09] And then what we needed to do was immediately start working on agents acting in the world. Not necessarily robotics, but like, you know, there had been all this stuff on Go, StarCraft.
[1:10:22] Dota, these other video games, all of which used reinforcement learning before the era of the large language models. So I thought we were going to put the two together almost immediately. And then almost all the training by now, by 2023, 2024, was going to be, you know, these large language models that were already as big, you know, already as big as they could usefully be made would kind of act in the world.
[1:10:46] But what we found instead is we've just kept scaling the language models. And I still think all the RL stuff is going to be promising. It's just we haven't gotten to it because it isn't the lowest hanging fruit because it's simpler to just spend more money to make these models bigger than to design something new. It's completely economically rational. And the models just keep getting better and better.
[1:11:08] Which, you know, I didn't doubt that they would get better, but I guess I imagine that things would happen in a little bit of a different order. Do you think data will be a scaling issue in the near term? Yeah, so I think there's actually some chance. I would say there's a 10% chance that we get blocked by data. The reason I mostly don't think it is, you know, the deeper you look, the internet's a big place, and the deeper you look, the more...
[1:11:35] And this is without even getting into kind of licensing of private data. This is just, you know, kind of, you know, this is just publicly available data. And then there are a bunch of promising approaches, which I won't get into detail about, for how to make synthetic data.
[1:11:52] And, you know, then again, I can't get into detail, but we thought a lot about this. And I bet the other LLM companies have thought a lot about it as well. And I would guess that one of those two, at least one of those two paths, very likely to pan out. But it's not a slam dunk. I don't think we've proven yet that this will work at the scale we need it to work. This will work for a, you know, a $10 billion model that, you know, that needs God knows how many trillion.
[1:12:20] God knows how many trillion words fed into it, real or synthetic. Those numbers are so big for people and the amount of money that is being spent to train these models. For the average person listening, where does all that money go into? And how should they think about the need over time to continue to iterate on this? Yeah, so what I'll say is, at least to my knowledge, no one has trained a model that costs billions of dollars today.
[1:12:49] People have trained models that cost, I think, of order $100 million. But I think billion-dollar models will be trained in 2024. And my guess is in 2025, 2026, several billion-dollar, maybe even $10 billion models will be trained. There's enough compute in the industry and enough ability to do data centers that that's possible. And I think it will happen.
[1:13:13] Anthropic has raised so far, at least it's been publicly disclosed, we're at roughly $5.5 billion or so. We're not going to spend that all on one model, but we certainly are going to spend multiple billion dollars on training a model sometime in the next two or three years. Where does that go? It's almost all compute. It's almost all GPUs or custom chips.
[1:13:43] And the data center that surrounds them. 80 to 90% of our cost is capital, and almost all our capital cost is compute. The number of people necessary to train these models, the number of engineers and researchers is growing, but the cost is absolutely dwarfed by the cost of compute. Of course, we also have to pay for like...
[1:14:08] But, you know, that again is some tiny fraction of what the cost of compute is. Maybe ending on an optimistic note here, and we touched on a bunch of like the potential medical breakthroughs and things like that. But why should people be optimistic about what Anthropic's doing, about the future AI and everything that's going on? Yeah, I don't know. So I'd answer the question in two ways. I mean, one, I'm optimistic about solving the problems.
[1:14:35] Getting super excited about the interpretability work. Like people didn't necessarily think this was possible. I still don't know whether it's possible to, you know, to really do a good job interpreting the models. But I'm very excited and very pleased by the progress we've made. I'm also excited about just, you know, the wide range of ways we've been able to deploy the models safely. Like the wide range of...
[1:14:58] of happy customers who just say, this model has been able to solve a problem that we had. It solved it reliably. We haven't had all of these safety problems where we've managed to solve them. We've deployed something safely in the world. It's being used by lots of people. That's great. That's one level of great. And I think the second level of great is this thing you alluded to with medical breakthroughs, mental health breakthroughs.
[1:15:26] Energy breakthroughs are already doing pretty well, but, you know, I imagine AI can speed up material science very, very, very, very substantially. So, you know, I think if we solve all these problems, I think a world of abundance really is a reality. I don't think it's utopian given what I've seen that this technology is capable of. And, you know, of course.
[1:15:48] I think as we gain more...
[1:16:15] Why don't you like the term AGI?
[1:16:43] No one was talking about the ability to do general intelligence 10 years ago, and so it felt like kind of a useful concept. But now, I actually think ironically, because we're much closer to the kinds of things AGI is pointing at.
[1:16:58] It's sort of no longer a useful term. It's a little bit like if you see some object off in the distance on the horizon, you can point at it and give it a name. But you get close to it and it turns out it's like a big sphere or something and you're standing right under it. And so it's no longer that useful to say this sphere, right? It's basically, it's kind of all around you and it's very close. And it actually turns out to denote things that are quite different from one another.
[1:17:25] So one thing I'll say, I mean, I said this on a previous podcast. I said, I think in two to three years, the LLMs plus whatever other modalities and tools that we add are going to be at the point where they're as good at human professionals at kind of a wide range of knowledge work tasks, including science and engineering.
[1:17:47] I definitely, that would be my prediction. I'm not sure, but I think that's going to be the case. And, you know, when people kind of like commented on that or put that on Twitter, they said, oh, Dario thinks AGI is going to be two to three years away. And so that then conjures up these image of, you know, there's going to be swarms of nanobots building Dyson spheres around the sun in two to three years. And like, of course, this is absurd. I don't necessarily think that at all. Again, the specific thing I said was,
[1:18:16] There are going to be these models that are able to, on average, match the ability of human experts in a wide range of things that they can do. There's so much between that and the super intelligent god, if that latter thing is even possible or even a coherent concept, which it may be or it may not be. One thing I've learned on the business side of things is that
[1:18:39] There's a huge difference between a demo of a model can do something versus this is actually working at scale and can actually economically substitute. There's so many little interstitial things that's like, oh, the model can do 95% of the task. It can't do the other 5%, but it's not useful for us unless we're able to substitute in AI end-to-end for the process.
[1:19:03] Or it can do a lot of the task, but there are still some parts that need to be done by humans, and it doesn't integrate with the humans well. It's not complementary. It's not clear what the right interface is. And so there's so much space between, in theory, can do all the things humans can, and in practice is actually out there in the economy as full co-workers for humans. And there's a further thing of, like, can it get past humans? Can it...
[1:19:31] Outperform the sum total of human, say, you know, scientific or engineering output. That's another point. That point could be, you know, could be like a year away because the model is better at making itself smarter and smarter, or it could be many years away. And then there's this further point of like, okay, you know, can the model like, you know, like explore the universe and set out a bunch of like, you know, von Neumann probes and, you know, build Dyson spheres around the sun and, you know.
[1:20:00] Calculate the meaning of life is 42 or whatever. That's like a further point that also raises questions about what's practical in an engineering sense in all of these kind of weird things. So that's like another further point. It's possible all of these points are pretty compressed together because there's like a feedback loop, but it's possible they're very far away from each other.
[1:20:22] And so there's this whole unexplored space of like you say the word AGI and you're like referring, you're smooshing together all of those things. I think some of them are very practical and near term. And then I have a hugely hard time thinking about like does that lead very quickly to all the other things? Or, you know, does it lead after a few years? Or are those other things like not as coherent or meaningful as we think they are?
[1:20:49] I think all of those are possible. So it's just kind of a mess. We're kind of flying very fast into this glob of concepts and possibilities, and we don't have the language yet to separate them out. We just say AGI, and I don't know. It's just kind of a buzzword for a certain community or a certain set of science fiction concepts when really we kind of – it's pointing at something real, but it's pointing at like –
[1:21:18] 20 things that are very different from one another, and we badly need language to actually talk about them. What do you think happens on the next major training run for LLMs? So my guess would be nothing truly insane happens, say, in any training run that happens in 2024. I think all the...
[1:21:40] You know, the good and bad stuff I've talked about, you know, to really invent new science, the ability to cure diseases. Dyson nanobots. The ability to make bio, yeah, the ability to make bioweapons, yeah, and maybe someday the Dyson spheres. The least impressive of those things, I think, you know, will happen, you know, I would say no sooner than 2025, maybe 2026. I think we're just going to see in 2024.
[1:22:07] CRISPR, more commercially applicable versions of the models that exist today. We've seen a few of these generations of jumps. I think in 2024, people are certainly going to be surprised. They're going to be surprised at how much better these things have gotten. But it's not going to quite bend reality yet, if you know what I mean by that. I think we're just going to see things that are CRISPR, more reliable, can do longer tasks. Of course, multimodality, which we've seen in the last...
[1:22:36] You know, we've seen the last few weeks from multiple companies is going to play a big part. Ability to use tools is going to play a big part. So, you know, generally these things are going to become a lot more capable. They're definitely going to wow people. But this reality bending stuff I'm talking about, I don't expect that to happen in 2024. How do you think the analogy of versus a brain breaks down for large language models? Yeah, so it's actually interesting. This is one of the...
[1:23:04] Being a former neuroscientist, this is one of the mysteries I still wonder about. So the general impression I have is that the way that the models run and the way they operate, I don't think it's all that different. Of course, the physiology, all the details are different. But I don't know, the basic combination of linearities and nonlinearities, the way they think about language, to the extent that we've looked inside these models, which we have.
[1:23:30] With interpretability, I mean, we see things that would be very familiar in the brain or a computer architecture. We have these registries. We have variable abstraction. We have neurons that fire on different concepts. Again, the alternating linearities and nonlinearities. And just interacting with the models, they're not that different. Now, what is incredibly different is how the models are trained, right?
[1:24:00] If you compare the size of the model to the size of the human brain in synapses, which of course is an imperfect analogy, but there's something like still maybe a thousand times smaller, and yet they see maybe a thousand or ten thousand times more data than the human brain does. If you think of the number of words that a human hears over their lifetime, it's a few hundred million.
[1:24:23] If you think of the number of words that a language model sees, the latest ones are in the trillions or maybe even tens of trillions. And that's just – that's like a factor of like 10,000 difference. So it's as if we've kind of – that neural architectures have some – there's lots of variance to them, but they have some universality to them. But that –
[1:24:49] Somehow we've climbed the same mountain with the brain and with neural nets in some very, very different way according to some very, very different path. And so we get systems that when you interact with them are – I mean there's still a hell of a lot they can't do. But I don't see any reason to believe that they're fundamentally different or fundamentally alien. But what is fundamentally different and what is fundamentally alien is the completely different way in which they're trained.
[1:25:16] You said that alignment and values are not things that will just work at scale. And we've talked about constitutional AI and some of the different viewpoints there. But can you extrapolate on that view? Yeah. I mean, this is a bit related to the point that I said earlier about.
[1:25:33] You know, that there's kind of this fact value distinction, right? You cram a bunch of facts into the model. You train it on, you know, everything that's present on the Internet. And it kind of leaves this blank space or this undetermined variable. I, you know, I basically just think that, like, you know, it's up to us to, you know, to determine the values, the personality.
[1:25:59] Especially the controllability of these systems.
[1:26:28] Still, I think, very opaque and hard to track down. And so, you know, I think it's very prone to failures. And, you know, this is why we focus on interpretability, steerability, and reliability. We really want to kind of tame these models, make sure that...
[1:26:47] You're able to control them and that they do what humans want them to do.
[1:27:17] Mechanistic interpretability, so hard to do. Yeah, so mechanistic interpretability is this, it's an area that we work on, which is basically trying to look inside the models and kind of analyze them like an x-ray.
[1:27:35] And I think the reason it's so hard, it's actually the same reason why it's hard to look inside the brain. The brain wasn't designed to have humans look inside it. It was designed to serve a function. The interface that's accessible to other humans is your speech, not the actual neurons in your brain. They're not designed to be read in that way.
[1:27:59] Of course, the advantage of reading them in that way is, you know, you get something that's much closer to a ground truth, not a perfect ground truth. But, you know, if I really understood how to look in your brain and understood what you were thinking, you know, it would be much harder for someone to deceive someone else about their intentions or for behaviors that might emerge in some new situation to not be evident.
[1:28:27] So there's a lot of value in it. But yeah, there's nothing in both the case of the brain and in the case of the large language models. They're not designed or trained in a way that makes them easy to look at. So it's a little bit like we're inspecting this alien city that wasn't built to be understood by humans. It was built to function as an alien city. And so we might get lucky. We might get clues. We might be able to figure it out. But there's kind of no guarantee of success. We're on our own.
[1:28:54] And so that's what we do. We kind of do our best.
[1:28:57] That said, I am becoming increasingly optimistic that interpretability can be, I don't know about fully solved, but that it can be an important guide to showing that models are safe and even that it will have commercial value in kind of in the areas of like trust and safety or classification filters or moderation, fraud detection. I think there's even legal compliance aspects to interpretability. So my co-founder, Chris Ola.
[1:29:27] Chris has been working on interpretability. He's run a team at Anthropic for the last two and a half years. Before that, when we were at OpenAI, he ran a team that worked on interpretability of vision models for three years before that. And for that entire period, it's been just basic research, right? There's been no commercial or business application. Chris and I have just kept it going because we believe that this is something that will...
[1:29:52] Pay off from a safety perspective and maybe even from a business perspective. And now, actually, for the first time, you will see by the time this podcast comes out, we're releasing something that shows that we've really been able to solve something or make good progress towards solving something called the superposition problem.
[1:30:14] Which is that if you look inside a neuron, it corresponds to many different concepts. We found a way to disambiguate those concepts so that we can see all the individual concepts that are lighting up inside one of these large LLMs. It's not a solution to everything, but it's just a really big step forward. And for the first time, I'm optimistic that give us two or three years.
[1:30:40] I don't know for sure, but we might actually be able to get somewhere with this. And depending on your level of understanding of all this stuff, why would that be important for safety? Yeah. So I would go back to the x-ray analogy there, right? You know, if I can really look inside your brain, you know, if I can say this is what's happening, I mean, you know, I can ask you questions and like, you know.
[1:31:04] You can say things that sound great and I have no idea if you're telling the truth or if it's all just bullshit. But if I look inside your brain and I have the ability to understand what I'm seeing, then it becomes much harder to be misled. Similarly, with language models, I can test them in all kinds of situations and it will seem like they're fine. The fear is always...
[1:31:28] Oh, but, you know, if I talk to the language model in this way, I could get it to do something really bad. Or if I put it in this situation, it could do something really bad on its own. That's always the fear, right? That's the fear we have every time we deploy a model. We've had 100 people test it. We've had 1,000 people red team it. But when it goes out into the world, a million people will play with it, and one of them will find something that's truly awful.
[1:31:52] And we'll find, oh, well, if I use this trick for talking to the model, it'll finally be able to produce that bioweapon. Or, oh, if I put it in this place where it has access to infinite resources on the cloud, it'll just self-replicate itself infinitely. And so interpretability is at least one attempt at a method.
[1:32:16] to address that problem, to say, okay, instead of thinking, instead of trying to test the model in every situation that it could be in, which is impossible, we can look inside it and try and decompile it and say, what would the model do in this situation? Well, we understand the algorithms that it's following. We understand what goes on in different parts of its brain, at least to some extent. So we can pose the hypothetical and say, hey, what would happen in this whole
[1:32:45] Part of the space, what would happen in this whole class of areas. And, you know, if we can do that, we have some we have kind of some ability to exclude certain behaviors to say, OK, we know the model won't do that, which which you never have behaviorally. You know, it's just it's just like it's just like humans. You know, you know, I'm like, you know, what what would you do in a life threatening situation?
[1:33:06] I don't know what I would do in a life-threatening situation. I don't know what you would do in a life-threatening situation. It's hard to know until you're actually in the situation. But if I knew enough about your brain, I might be able to say. What's your view on open source models? Yeah, I mean, that's obviously a complex topic. I mean, I think as with many things in AI versus the rest of technology,
[1:33:32] From a normal technological perspective, I'm extremely pro-open source. I think it's accelerated science. It's accelerated innovation. It allows errors to be fixed faster and development to happen faster. And I certainly think for the smaller models, for the smaller open source models, this is true for AI as well. And I don't see much danger to smaller models. Therefore, I think open source, as it's being practiced by every open source model that's been released up to this point, seems perfectly fine to me.
[1:34:01] My worry is more around the large models. And my worry in particular is that these models that are offered via API, and I'm talking about models of the future that really are dangerous, not models in two or three years, maybe one year, not today's models. If they're offered by API or even if you have fine-tuning access to them, there's a lot of levers that you have to control the behavior of the model, right?
[1:34:30] Put in your constitution, don't produce bioweapons, right? And then if the model does it anyway, you can basically make changes to the model. You can say, okay, we're just retracting that version and serving a new version of our model that patches a particular hole. You can monitor users. So if a million people are using the model, and within that, there's these five bad actors in this terrorist cell, you can use your trust and safety team to identify the terrorist cell, cut them off, and even call law enforcement if you want to do it.
[1:34:57] So it really provides an ability. You don't have to get things right in the first time. And if something dangerous happens, you really have the ability to fix it. With models where their weights are released, you don't have any of that control, right? The minute you release the model, basically all of this control is lost.
[1:35:18] That's our concern. That doesn't mean, by the way, that large open source models shouldn't exist. But the way we put it in our responsible scaling plan is we say, okay, when models get to the level where they're smart enough to create these dangerous capabilities, and the next one for us is ASL 3, we're at ASL 2 right now, then models have to be tested for dangerous behavior.
[1:35:41] According to the complete attack surface, according to which they're going to be released in reality. So if you're just releasing an API, then you have to test that you can't build a bioweapon with the API. If you're releasing the model with an API and fine-tuning, then the people who are testing the model have to mock up the test with the fine-tuning. If the model is being released in practice, then the right test to run would be, I'm a mock terrorist cell.
[1:36:09] I get the weights released to me. I can do anything I want with those weights. Is there some way to release the weights of the model so that they can't be abused? I think there might very well be, but I think people who want to release model weights have to confront that problem and have to find a solution to that problem. I'll say, by the way, because there's a person on my team who this is one of their pet peeves.
[1:36:34] The word open source I don't think is necessarily appropriate in the case of all of these models. I think it is appropriate in the case of small developers and companies where their whole business model is about open source. But when much larger companies have...
[1:36:53] They generally have not released them under open source licenses. They've generally asked people to pay them when they use them in commercial ways. So I would think of this as less open source and more that model weight release is a particular business strategy for these large companies. And again, I'm not saying that model weights can't be released.
[1:37:19] I'm saying that the tests for them need to be commensurate with the issues and that we shouldn't automatically say, oh, open source, open source is good. Some of these are not open source. They're the business strategies of large companies that involve releasing model weights. Paul Cristiano recently on a podcast said he thinks there's a 50% chance. I think the way he phrased it was that his...
[1:37:42] The way he ends up passing away is something to do with AI. Do you think about percentage chance doom? Yeah, I think it's popular to give these percentage numbers. The truth is that I'm not sure it's easy to put a number to it, and if you forced me to, it would fluctuate all the time. I think I've often said that my chance that something goes...
[1:38:11] You know, really quite catastrophically wrong on the scale of, you know, human civilization. You know, it might be somewhere between 10 and 25 percent when you put together the risk of something going wrong with the model itself, with, you know, something going wrong with human people or organizations or nation states misusing the model or it kind of inducing conflict among them or just some way in which kind of society can't handle it.
[1:38:40] That said, what that means is that there's a 75% to 90% chance that this technology is developed and everything goes fine. In fact, I think if everything goes fine, it'll go not just fine, it'll go really, really great. Again, this stuff about curing cancer, I think if we can avoid the downsides, then this stuff about curing cancer, extending the human lifespan.
[1:39:04] Solving problems like mental illness. I mean, this all sounds utopian, but I don't think it's outside the scope of what the technology can do. So I often try to focus on the 75% to 90% chance where things will go right. And I think one of the big motivators for reducing that 10% to 25% chance is how great it is trying to increase the good part of the pie. And I think the only reason why...
[1:39:33] I spend so much time thinking about that 10% to 25% chance is, hey, it's not going to solve itself. I think the good stuff, companies like ours and like the other companies have to build things, but there's a robust economic process that's leading to the good things happening. It's great to be part of it. It's great to be one of the ones building it and causing it to happen, but there's a certain robustness to it. And I find more meaning. I find more...
[1:40:02] When this is all over, I think I personally will feel I've done more to contribute to whatever utopia results if I'm able to focus on reducing that risk that it goes badly or it doesn't happen. Because I think that's not the thing that's going to happen on its own. The market isn't going to provide that. Do you worry more about the misuse?
[1:40:27] People misusing it or the AI themselves? Or is it just different timelines? Yeah, partially different timelines. I mean, you know, if I had to tell you, I would say the misuse to me seems more concrete. And I think, you know, will happen sooner. You know, hopefully we'll stop it and it won't happen at all. You know, I think the AI itself doing something bad is also a quite significant risk.
[1:40:55] It's a little off in the future and it's always been a bit more shadowy and vague, but that doesn't mean it isn't real. I mean, you just look at the rate the model is getting better and you look at something like Bing or Sydney, it really gives you a taste of like, hey, these things can really be out of control and psychopathic. And the only reason Bing and Sydney didn't cause any harm is that they were out of control and psychopathic in a very limited way, limited both in that.
[1:41:23] You know, it was confined to text and limited in that, you know, it just wasn't that smart. You know, tried to manipulate the reporter, tried to get him to leave his wife. But like, it wasn't really, you know, it wasn't really compelling enough to, you know, to get a human to fall in love with it. But someday maybe a model will be. And, you know, maybe it'll be able to act in the world. And then you put all those things together. And, you know, I think there is some risk there. I think it's harder to pin down. But I'm personally, I'm worried about both things. And, you know.
[1:41:52] I think our job, because we see such a positive potential here, is we have to find all the possible bad outcomes and shoot them down. We have to get all of them. And then if we get all of them, then we can live in a really great world, hopefully. If you could wave your hands and have everyone follow a single policy, would it be the responsible scaling policy? Would everyone have one of those? Yeah, I mean, I think...
[1:42:20] You know, I think if we make a constraint of like realism, right, where, you know, it's like, you know, I in fact can't wave my hand and get, you know, everyone in, you know, China or Russia or somewhere else to, you know, stop building these powerful models. You know, like there are just some levels of like world or international coordination that are just not going to happen because of realism. So if you stipulate that like...
[1:42:44] You can't just make everyone stop or you can't just make everyone build in a certain way. The idea that, hey, for most things, people should just be able to build what they want to build, but we're cordoning off these particular levels of capability, these particular points in the development curve where something concerning is happening and say, hey, mostly do what you want, but you've got to take this.
[1:43:12] Small fraction of stuff you've got to take really seriously. And if you don't take it really seriously, you're the bad guy. That's something that I think I can reasonably recommend that everyone sign on to that. There's some sacrifice to it. There's some loss to winning the race, but it's only as much sacrifice as is needed. And because it's so targeted, I think you can make a strong moral case that, hey, if you don't do this, you're an asshole. How do you think about the trade-offs between building
[1:43:41] In public and having people aware with what you're doing with, on the flip side, maintaining the secrets or that the appropriate things stay within the org. Yeah, yeah. I mean, you know, this is one of these kind of difficult trade-offs, right? So, you know, definitely an org benefits from, you know, kind of everyone knowing about everything. But on the other hand, as we've seen with multiple AI companies, you know, secrets leak out. And, you know,
[1:44:09] Even just from a commercial perspective, forget safety. With models built in the next year or two, let's say a model costs $3 billion and you have an algorithmic advance that means you can build the same model for $1.5 billion. These kind of 2X advances along the scaling curve have occurred in the past and may occur in the future.
[1:44:35] Companies, including ours, may be aware of such advances. So basically, that's like three lines of code that's worth $1.5 billion. You don't want a wide set of people, and you may not even want everyone within your company to know about them. And at Anthropic, at least, people have been very understanding about that. People don't want to know these secrets.
[1:44:57] People are on board with the idea, hey, it's not a marker of status that you know these secrets. These secrets should be known to the tiny number of people who are actually working on the relevant thing, plus the CEO and a couple other folks who need to be able to put the entire picture together. This is kind of compartmentalization and need-to-know basis. And of course it has some costs because information doesn't propagate.
[1:45:26] But again, just as with the RSP, let's take the 80-20. Let's take the few pieces of information that are really essential to protect and be as free as we can with everything else. If you hadn't gone down this AI path, would you be doing academic work right now?
[1:45:46] I think that was my assumption. Like that was what I kind of always imagined doing. I imagined being a scientist and, you know, scientists work at universities. But the really interesting thing about this AI boom is that to really be at the forefront of it, you know, you have to have these huge resources. And, you know, I think the huge resources are basically, you know, they're basically only available at companies.
[1:46:13] First, it was the large companies like Google, but more recently, startups like ours have been able to raise large amounts of money. And so I was kind of drawn to that direction because it had the ingredients necessary to build the things we wanted to build and study the things that we wanted to study as scientists. And I would say many of my co-founders feel the same.
[1:46:38] One thing, one of my co-founders who is a physicist, what he often brings up is – and it's kind of more an academic question because I don't think things are going to go in this direction. But he said, in my field, we build these $10 billion telescopes in space and we build these $10 billion particle accelerators. Why did the field go the way – why did the field kind of –
[1:47:03] Go in this direction instead. Why didn't all the AI academics get together and build a $10 billion cluster? Why did it happen in startups and in large companies? I don't really know the answer to that. Things could have gone the other way, but it doesn't seem like that's the way things have gone. And I don't know if it's for the better or the worse. I mean, we've learned a huge number of things.
[1:47:28] By working with customers and seeing how these things impact the economy. So maybe the path things went is the best path they could have gone. I actually don't know. How did they get access to capital in the prior or in the alternative way of doing that? Was it through government grants and stuff? Yeah, these large telescopes, they're often kind of like government consortia or private.
[1:47:52] Kind of large-scale private philanthropy. It's honestly this huge patchwork mix. I'm surprised it even happens because if I think about it happening in this field, I just can't imagine how it would happen. But in these other fields, somehow they've made it work. I don't actually know how. I don't know. That could have been just a weird alternate history of our industry that didn't happen. I very much doubt it's going to happen, although who knows.
[1:48:21] We went on this path instead. And I don't know, there's a lot that's exciting and interesting about this path. And one way or another, this is the situation we're in. Has working with your sister been as fun?
[1:48:35] Saving the world as you had hoped it when you were kids? Yeah, it's surprisingly similar to what I imagined. I mean, if you were to look back on the things we were saying to one another as like an adult observing it, you would have been like, this is crazy. This is crazy. Kids dream, of course. But no, I mean, it's just amazing that we're able to work on this together. Very cool. Daria, thanks for doing this. Thanks for having me.