[0:00:00] I think it's perfectly possible to distribute the benefits of AI to all the world, including China, including other authoritarian countries, without distributing the military capabilities. The plan has to be start from a position of strength and work out how this technology can benefit the whole world and how its downsides can be mitigated. Dario Amadei, CEO of Anthropic. Welcome to China Talk. Thank you for having me.
[0:00:26] All right. So let's start with a brief picture of how rapid AI progress could cash out in terms of national power. Yes. So a few months ago, I wrote this essay called Machines of Loving Grace, where it actually was focused on many of the positive applications of very powerful AI. But, you know, I had this definition of what I think very powerful AI will look like or what my definition of it is.
[0:00:56] What I think it is that all the companies are trying to build. And I found that phrase very evocative for just understanding the implications. It's as if you drop down a country of 10 million people or something like that, except all of them are polymathic Nobel Prize winners in any field.
[0:01:16] And, you know, what does that do to national power? Well, presumably that does a lot of things to national power. I think it greatly, you know, greatly accelerates economic ability. It graciously accelerates science. And, of course, you know, perhaps unfortunately, it probably also has implications for intelligence and for national defense, right? Both in terms of, you know, controlling swarms of drones or analyzing intelligence information, right? You could go into a lot of detail, but just generally having a lot of...
[0:01:43] Incredibly smart entities who can control anything in a virtual way. That's going to be a source of a lot of power in many different ways. Gotcha. So why write an essay about export controls? Seeing the reaction to DeepSeek and just being in the industry, being one of the people who's developing the technology, I saw a lot of things in that essay that were just not correct. And I think they came from people who weren't following closely.
[0:02:13] The actual arc of the technology. But now we're paying attention because there was this novel fact that it was a model developed by a Chinese company, kind of had missed many of the previous steps and misunderstood the dynamics of the field and just came in and said, oh my God, this is really cheap. And maybe they had a stereotype of cheap things are made in China. And so kind of pattern matched to that stereotype. But the reality, as I said in the essay, is that there's been an ongoing trend of cost decreases in this field.
[0:02:43] At the same time as we've spent more and more money to train the models, they're so powerful and so economically useful that the countervailing trend of just spend more to make a better and better and smarter thing has outstripped the trend of make things cheaper. And the timing of it has just been such that DeepSeek was able to release something that genuinely had some important innovations in it, but was...
[0:03:08] Along the same cost decrease curve that we've seen for AI in the past, right? This wasn't, you know, making something for 6 million that costs the other company's billions. This was more like, you know, we've seen a 4x cost decrease per year. This was, you know, roughly that, give or take, relative to models that had been trained, you know, half a year to a year ago. And, you know, we'll see that, you know, models of that quality will be now produced very cheaply by multiple players, while multiple players, including probably DeepSeq, will spend
[0:03:38] Much more money to train much more powerful models. So I think the new fact here is that there's a new competitor. I think of this just as if...
[0:03:46] In the big companies that can train AI, Anthropic, OpenAI, Google, perhaps Meta and XAI. And now DeepSeek is maybe being added to that perhaps category. And I don't know, maybe we'll have other companies in China that do that as well. So that is a milestone. That is something that didn't happen before. And it is a matter of concern to me. But I think people were way overreacting and misunderstanding the implications.
[0:04:13] So your big update is on the skill of one Chinese player and perhaps more Chinese players. Like what, if anything, should people reassess about specifically on the model side, you know, what you can expect going forward about the gap? I just want to be clear, like we've been tracking DeepSeek for a long time. Like we've been aware of DeepSeek as the likely, you know, most capable player in China for over a year. That's informed our view of how things are going to evolve.
[0:04:42] But for those who are just seeing DeepSeek, I think the update is, look, there were three to five companies in the U.S. who could make frontier or near frontier models. Now there's three to five companies in the U.S. and one company in China. Whether they continue to make near frontier models depends upon how many chips they can get access to and whether they can get access to chips at a much larger scale than those they've been able to get access to now.
[0:05:09] The AI safety community, yourself included, warned for years about the risks of racing dynamics. I'm curious if you could just talk me through a little bit how you got to this place of seeing. The two things aren't really opposed. I'm still concerned about the idea of a race. In particular, my worry is that if the U.S. and China are equally matched in terms of this technology, neck and neck at every stage, then...
[0:05:38] There will be basically nothing preventing both sides. It will just completely rationally make sense for both sides to keep pushing the technology forward. It has so much economic value. It has so much military value. There's going to be, absent very, very strong evidence of danger, there's going to be a fair amount of incentive to continue developing it. And in particular, a worry I have is that legislation in the United States,
[0:06:08] Like efforts to measure and perhaps at some point even restrict the risks of AI systems, the dangers, the misuse of them by individuals, for example, for bioattacks or the autonomous dangers of the systems themselves. There have been various legislation in the U.S. over the last year or so. And one argument that's been given against the legislation is, hey, if we slow ourselves down.
[0:06:37] China will just jump ahead and beat us. And I think that argument is absolutely correct. And so the only way that I see that we can kind of do both, right? We have to make sure that we're ahead of China and other authoritarian countries, both because I don't think they would use powerful AI very well. And because if we're not ahead, there's this racing dynamic. And yet somehow we have to also.
[0:07:03] Protect against the dangers of AI systems we ourselves build. So my guess at the best way to do this, and it may be futile, it may not work out, but my guess at the best way to do this is with something like export controls, we can create a margin between us and China. So let's say we're two years ahead. Maybe we can spend six months of those two years to ensure that the things we build ourselves are safe. In other words...
[0:07:31] We're still ahead and we're kind of able to make things safe. If things are evenly matched, then we have to worry that what they build isn't safe. And, you know, at the same time, we have to worry about them like dominating us with the technology. So that puts us in a very bad dilemma where there are no options. And so for a long time, this is not anything new. This is just the first time I'm talking about it. I've felt that, you know.
[0:07:58] Thank you very much.
[0:08:23] You know, I think we're going to need to do some amount of acceleration, but it's a move that has trade-offs because the more of that you do, the less time you have to be careful. But I think one way to get around the trade-off to some extent is to impose these export controls because they widen the gap and they give us a larger buffer that we can use to govern the technology ourselves. I think it's tough, though. I think it's like a tough situation. It's hard to have both.
[0:08:52] There are real trade-offs. We're not in the situation that I might have wished we'd be in. Let's stay on this for one more second. Can you tease out getting to a race to the top logic with China and getting to a race to the top logic with your peers in the West? Yeah. The nice thing about the other companies in the West is that there's a coordination mechanism between them. They can all be brought.
[0:09:19] Under kind of the same law, like if someone passes a law, we have to follow it. OpenAI has to follow it. Meta has to follow it. Google has to follow it. X.AI has to has to follow it. And so you can know that you're engaging in necessary safety practices and others will engage in them as well. If you do something voluntarily, to set an example, as Anthropic has often done, the threat of someone coming in and imposing legislation.
[0:09:48] Even if there isn't any, is often very valuable in getting other actors to behave well. Because if it looks like one actor is much more responsible and other players are behaving irresponsibly, that really creates incentives for regulators to go after the other players. But I don't think that's possible between the U.S. and China. We're kind of in a Hobbesian international anarchy. I do think there are opportunities to try to cooperate with China. I'm relatively...
[0:10:17] I'm skeptical of them, but I think we should try. And I think there's not zero that can be accomplished. I think a thing that might change the game if there was some kind of truly convincing demonstration that AI systems were imminently dangerous, like at the level of human civilization. I don't think we have that now. Some safety people think that they have arguments that are super convincing.
[0:10:40] They have arguments that are suggestive, enough to make me worry and take it seriously and factor it into my calculus and do research on it. But certainly on the merits, the quality of the arguments is not anywhere near strong enough to make two competing superpowers say, okay, we're going to relinquish the technology or we're going to pause for a certain amount of time to build it really carefully.
[0:11:07] I don't think we're in the same ballpark. But as models become more powerful, I think we'll understand the extent to which they really are a danger. There's real uncertainty here, and I think we're going to learn a lot over the next year, year and a half. And if we do find truly compelling evidence that these models are really, truly dangerous in a grand sense, maybe that could change things. But so far...
[0:11:31] I'm aware of efforts by the US government to kind of send a delegation to talk to China about topics related to AI safety. And my understanding, again, I obviously wasn't part of those delegations, is that there wasn't that much interest from the Chinese side. I hope that changes. I hope there is more interest. We do lots of stuff in trying to educate the world about AI safety in order to make progress on doing a better job of...
[0:11:58] Steering AI systems. You know, we do all this work in interpretability and things like constitutional AI and developing ways to scalably supervise AI systems. We can only hope that these ideas, you know, diffuse everywhere, including to China. I mean, you know, just to say it, I'd be happy to have DeepSeek copy these methods like I hope they do. There are ways of making AI systems more reliable. But so, you know, I think there are opportunities here. We should just be.
[0:12:27] We should just be realistic about the actual situation, right? We should have just a realist view of how international relations work, which is that it's possible to make progress at the margin, and if some really compelling evidence were available, then maybe it could change things. But look, you have two nation states that have very different systems of government that have been adversarial for a long time. They're going to compete over this technology, and they're going to race to build it as fast as they can.
[0:12:57] That's what's going to happen by default. Yeah. I mean, like the only happy story, analogous story I can find is she deciding to care about climate change. But the thing about climate technology is it's not super dual use. And, you know, there's not, you know, there's like economic upside, but not military upside to, you know, making a lot of solar panels and electric vehicles. Yeah. And my understanding is, you know, China hasn't always.
[0:13:24] kept its commitments here. You know, there's been some mix of yes and no. And that's kind of what I would expect here. I mean, it's even a little bit what we see between companies that are, you know, there's no law, there's no, you know, law forcing people to adopt RSPs. Companies have voluntarily adopted them. And, you know, I think there's differences in the extent to which companies comply with them. So, you know, something I could imagine being realistic is, yes, there are treaties or, you know, kind of aspiring goals signed here.
[0:13:54] And, you know, maybe things get 10% safer because of them, and we should absolutely seize that 10%. But, you know, it's going to be hard. Again, I would point to if you really wanted to try to drive cooperation here, you just need really strong evidence that something is, you know, a threat to the United States, China, and every other country in the world, right? You need, like, really compelling evidence. And what I would say to safety folks who...
[0:14:20] You know, who want us to pause, who want us to stop developing the technology. Your number one task should be to develop that evidence. So what chips should and shouldn't America be selling into China? Yeah, so first of all, I should say, I don't think the export controls were ever really designed to prevent DeepSeek or any other Chinese company from getting the amount of chips that they had.
[0:14:46] You know, at the level of a couple tens of thousands, I think we should reasonably expect smuggling to happen. I think DeepSeek probably had a mix of different kinds of chips for about 50,000. And that I would expect. I think export controls can be more successful at, you know, they can't have like a million chips because that's easily in the tens of billions of economic activity, you know, approaching approaching 100 billion. In terms of the specific chips.
[0:15:13] There were three chips that they were reported to have. I'm going by Dylan Patel of Semi Analysis. So none of this is confirmed. DeepSeq hasn't stated it themselves. But I think, you know, generally Semi Analysis, when it relates to semiconductors, is pretty well sourced. So the claim is they had a mix of...
[0:15:33] There's H100s, which are the standard chip that we have in the U.S. Something like it was claimed they have 10,000 of those. So those must have been smuggled in some form or maybe were proxied through some country other than China where they're allowed, but were used by a Chinese shell company. There's the H800, which is interesting. So the H800 is a chip that was developed after the...
[0:15:58] One of the first rounds of export controls in 2022. There were export controls in 2022 that put a limit on the combination of the computing power of a chip and the communication bandwidth between chips. And the H800 was a chip that was made to reduce the communication bandwidth between chips in order to get around.
[0:16:23] Get around the export controls. So a bunch of H800s were sent to China, and then that loophole was closed. In 2023, we actually noticed this quite early, that there are ways to train your model that get around the lower communication bandwidth. And so in 2023, a set of export controls were designed that were more robust to being evaded in this way. So presumably those H800s were shipped between the time of the 2022 ban and the 2023 ban.
[0:16:52] The H20 is a chip that is not suitable for training. It therefore must not have been used to make the base V3 model, which was the first stage that I talked about in my blog post. In the second stage, what they called the R1 model, which was the second stage is stacked on top of the first, and it was the second stage of the model that kind of blew up on social media. That involves a mix of training and what's called inference.
[0:17:20] The H20 chip has not been banned and is good for inference, but not for training. So it is possible that the H20 played an augmentative role, that it was used for part of the second stage, even though it could not have been used for the first stage.
[0:17:35] So given that inference is also increasingly involved in new models, it's associated with this new paradigm of reasoning models that's been developed. It seems to me that I would recommend also banning the H20 and I think attending to making these bans as broad as possible.
[0:17:54] We always want to be careful. We don't want to ban other economic activity. For example, you know, I don't think there's any reason to ban sending gaming consoles to China, right? That would just be too much of an economic imposition for something that's aiming to be targeted. So as always, these things have to be designed carefully. But yeah, I think we should ban the H20. All right. So you mentioned Dylan Patel.
[0:18:19] I recently tweeted that, in fact, part of Sonnet's secret sauce was the Huawei Ascend series. Given that China has its own export control regulation, I'm worried for you and for the future of Anthropic at your ability to use Huawei chips in the future. Is this something you're concerned about at all? Wait, sorry. Sorry, you're saying Anthropic chips? That was a joke. Okay, I'll cut it. I'll cut it. All right. You're so confused. I'm like, how do we even get?
[0:18:48] I wouldn't even use them. All right, so. For a while, I was just like super. All right, rolling it back. It took me a while to get it. Very funny. Okay, so the argument NVIDIA makes is the fewer chips we sell into China, the more demand there is for Huawei. You know, the more customers they'll have, like there's always an announcement that DeepSeek is like running super efficiently on Huawei 910Bs or what have you.
[0:19:18] You know, thoughts on Chinese domestic capacity to produce chips and what the implications are there for the semiconductor manufacturing equipment side of the export controls. Yeah, I mean, I think over a timescale of 10 to 15 years, that's probably correct. They probably will catch up. But the supply chain is really deep there. And as you mentioned, we've also export controlled.
[0:19:42] Semiconductor manufacturing equipment and repairs on that equipment. So it's actually going to be difficult to make chips that are competitive with, say, the new NVIDIA B100 or with the chips we're using like Tranium and TPU. I think the software ecosystem is also not up to snuff, although that's a less important constraint than the hardware. So my sense is that it is unlikely that the Huawei chips...
[0:20:09] Become anywhere near comparable to U.S. chips anytime soon. And for me, you know, as I wrote in Machines of Loving Grace and on the export on the post on export controls, I think the critical period here where there's really going to be a contention and or, you know, it's kind of kind of important to achieve a balance of power is going to happen in, you know, 2026, 2027 at the latest 2030.
[0:20:36] And therefore, I think policy should target that time range, right? Things are moving very fast in the AI world. 10 to 15 years is like eternity. It's forever. It's almost irrelevant. Gotcha. So the Biden administration's policy was basically to let Chinese firms use chips on Western hyperscalers outside of the PRC. Is that kosher in your book?
[0:21:00] I mean, you know, near the end, they put in place the diffusion rule, which I think prevents that, you know, when it's Chinese cutout companies. Maybe you're referring to something different, though, which is. I mean, like ByteDance, like being Oracle's largest customer. Being Oracle's largest customer. So, you know, I think the diffusion rule, you know, prevents or limits that within, you know, the tier two countries outside the U.S. I think some of it is done in the U.S. as well.
[0:21:28] Thank you very much.
[0:21:52] I'm less immediately concerned about it, but, you know, it's definitely before we get to the end of this, I think it's, you know, it's something that needs to be stopped or at least stopped at scale. What do you make of DeepSeek releasing its model on open source? Yes. I mean, there were several different properties of the DeepSeek release. And, you know, I think one was that the weights of the model were released. One was just that, you know, the model was kind of an efficient and strong model, you know, on the cost.
[0:22:21] Reduction curve, but it's the first time, you know, the point on the cost reduction curve was made by a Chinese company. So I actually think the second of those is much more important than the first. Most of what I said in the post on export controls would have been almost entirely the same, maybe entirely the same if it had just been a model that they were serving via API. I think most of the implications just come from the fact that this is a strong model.
[0:22:51] A Chinese company has produced a strong model. That model will scale up to a very large scale. Whether Chinese companies are able to scale up to millions of chips will be determined by the export control. So I think they're important either way. On the kind of releasing the weights, the open weight side of it, I mean, on the commercial side, we've found that our main competitors are people who release strong models, whether they're open weights or whether they're not.
[0:23:19] How strong a model is, is about 80 to 90% of how much it has mattered, how difficult it has been to compete against the model. Open weights is a little different for models, is a little different than open source. Like there isn't a source code, there's just a bunch of numbers. And so some of the advantages and differences that are present there are just not present here as much. I think in many ways, the analogy breaks down. I also think that companies that have a history of, you know,
[0:23:49] Starting with kind of open weights, at some point they need to monetize, at some point they need to make profit, and so they often stop doing this. The more important factor here is that a Chinese company is producing a powerful model. Just looking in the marketplace, we haven't seen any evidence that people prefer models because they're open weights, separately from the behavior of the model. Sometimes you see it, there's a little bit of a convenience factor.
[0:24:17] But, you know, I think I said this as early back as my Senate testimony in like 2023. I think open weights models are like fine up to some certain scale. I just don't think they're substantively different from closed weights models. But I mean, you know, you take a margin on every API call, right? There's that. Yeah. So the interesting thing is, let's say you take any kind of model, no matter where it is, has to be served on a cloud.
[0:24:45] And that ends up getting associated with the margin. You have this present either way. At the same time, there are vast differences in the efficiency of inference. And there are vast differences in the training of models. The training of models are moving very quickly. And so, yeah, I actually think that's a relatively small factor. There are weeks when we implement like...
[0:25:12] 20% improvement in inference efficiency or something like that. And the companies are all competing with each other to have the most efficient inference. Most of this is swamped by who has the most efficient inference, who's training the best model. Again, if a company like a company in China is very good at serving their model for low cost, then that's an area where competition will happen.
[0:25:38] But yeah, I actually think whether the weights are available or not is mostly, not entirely, but mostly a red herring. Somewhat related, like when do you think governments are going to start getting queasy about models getting open sourced? Yeah. Again, I would say just as from a business perspective, I think the difference between open and closed is a little bit overblown. I also think that from a security perspective,
[0:26:06] The difference between open and closed models is, for some intents and purposes, overblown. I think the most important thing is how powerful a model is. If a model is very powerful, then I don't want it given to the Chinese by being stolen. I also don't want it given to the Chinese by being released. If a model is not that powerful, then it's not concerning either way.
[0:26:33] All right. Speaking of being stolen, anything you want to say about model distillation? There are these reports that, you know, I think I said in the blog post, I don't really comment on them one way or another, that DeepSeek distilled a model from, might have been OpenAI 01, right? They claim they have evidence. I actually haven't looked closely at it and can't tell you whether it's accurate or not. Distillation is certainly something you can do with a model. So it's possible this is something that has occurred.
[0:27:03] It's, of course, against terms of service. I think there are a couple of points that are important. One is developing ways to detect model distillation. So I think it's probably possible that from looking at a model and looking at another model, which is purportedly distilled from it, to look at the two together and say,
[0:27:26] You know, has this model been distilled from that? Like, you know, I'll generate a bunch of output from one and generate a bunch of output from the other and try to figure out whether one model has been distilled from the other. You can already kind of see like when one model has been distilled from another, they have similar mannerisms. They talk in similar ways. Like there are lots of behavioral signatures. And so it'd be great to turn this into some kind of measurable statistical test. And then there are monitoring techniques to kind of prevent.
[0:27:55] to prevent distillation. That said, I mean, you know, like all things, like all problems in cybersecurity, it's going to be, I think, an ongoing issue. I mean, that's why some companies have chosen to kind of hide the chain of thought in their reasoning models because it makes it much harder to distill. Those can be jailbroken, but then, you know, folks are working on antidotes to jailbreaking. We just released something today that makes it much harder to jailbreak.
[0:28:21] Much harder to jailbreak models. All right. So setting distillation aside, on the Dwarkesh podcast, I guess a year and a half ago, you said that if it's a state's top priority to steal weights, they will. If we're living in your timeline and this stuff becomes the most important thing on the planet and it attracts more attention from Beijing, from lots of other state actors, what does that do to...
[0:28:46] The gap, if any, between the leading labs and everyone else. The biggest possible gap I can imagine is a couple of years. I think that's actually worth a lot, both for the amount of advantage that it gives. And it actually gives a buffer to us here in the U.S. and its allies to address some of the safety problems of AI systems. It gives us a buffer. So I don't think a gap greater than two years is feasible.
[0:29:14] Thank you. Thank you.
[0:29:32] Enlist help from two powerful forces. One is the United States government and the other is the models themselves. So as the models get better at everything, one thing they're going to get better at is cybersecurity. So using the models to defend themselves. Second, using the United States and its counterintelligence capabilities to prevent the models from being stolen. Our responsible scaling plan, our RSP, which we developed about a year and a half ago, actually anticipates this. It has different levels.
[0:30:01] Currently, we're at ASL 2, which is strong but relatively ordinary for a tech company security measures. The next level, ASL 3, is preventing the models from being stolen by strong non-state actors. And ASL 4 and ASL 5 are preventing theft from state actors. So those are very high bars to clear. But we've said that when the model reaches certain levels of capability that we're specifying.
[0:30:29] Then, you know, we have to put in place these strong security measures. And, you know, a lot of what we were thinking about there was, you know, this this competition between the U.S. and China and, you know, the need to improve lab security. Yeah. But I mean, like if the U.S. government can't keep Gina Raimondo's emails safe and like the Chinese can listen into every phone call in America. I mean, it seems like a tough thing to hang your hat on. Maybe another way to ask it is like, say you steal the model weights, like how much?
[0:30:58] Tacit knowledge, is there necessary that like, you know, a deep seeker, a ByteDance may or may not have with, you know, Opus 5 in order to be able to make the most out of it relative to what the makers can? I mean, I think running that particular model is not that difficult. Using it to develop the next model, I think is substantially more difficult because, you know, you just, you don't have the, you know, it's optimized to run in some other cluster and some other setup.
[0:31:27] So, you know, we've talked about competition from the lens of compute, from the lens of models, but models are created by people. And, you know, the times I've been to NeurIPS, it has been really striking to me just how much Mandarin is spoken at those poster tables. I'm curious, you know, what message you'd have to say to PRC Nationals studying in the West, you know, who hear this and are like, why would I want to work for this guy?
[0:31:55] So, I don't know, a thing I want to be really clear on, like, you know, I should probably say it even more than I do. Like, the issue here, you know, when we talk about China, this isn't about Chinese people versus American people, right? Like, we certainly speaking for ourselves, but I suspect the other U.S. companies would say the same. You know, we're excited to have talent from around the world. And, you know, we have no beef with...
[0:32:19] Talented researchers and engineers, no matter where they work, were from the same community they are. If there's some kind of cooperation to be had here, it's probably through this kind of track to researcher to researcher thing. And we welcome these folks absolutely as much as we can. I think the concern here is authoritarian systems of government wherever they exist. We could have seen in the last 10, 15 years, China could have gone down a very different...
[0:32:48] I'm not a China expert, but many people do seem to think that there was a bit of a fork in the road and maybe an opportunity for them to take a more liberalizing path. And for whatever reason, that didn't happen. But if it had happened, certainly my view on all of this would be completely different. This is not about animus against a country. This is about concern about a form of government and how they'll use the technology.
[0:33:15] And now this interview is not getting translated and not going viral in China. Thank you for that, Dario. What would you, anything you want to say to DeepSeek? I don't know. They seem like talented engineers. I think the main thing I would say to them is, you know, take seriously these concerns about AI system autonomy. When we ran evaluations on the DeepSeek models, we have a spate of national security evaluations for like, you know, are models able to generate
[0:33:44] Information about bioweapons that can't be found on Google or can't be easily found in textbooks. The DeepSeek model did the worst of basically any model we'd ever tested in that it just had absolutely no blocks whatsoever against generating this information. I don't think today's models are literally dangerous in this way. Just like with everything else, I think we're on an exponential. But later this year, perhaps next year, I think they might be.
[0:34:12] My advice to DeepSeek would be take seriously these AI safety considerations. The majority of the AI companies in the U.S. have stated that these issues around AI autonomy and also these issues around AI misuse, they believe them to be serious, potentially at least, serious and real issues. My number one hope would be that they come work in the U.S., they come work for us or another company.
[0:34:39] My number two hope would be that if they're not going to do that, they should take some of these concerns about the risks of AI seriously. So the dominant framing, how you've written and other folks in AI policy circles, is that AI is going to be something that entrenches autocrats. People are talking about the CCP like it could turn into a thousand year Reich. Can you see a world in which AI and perhaps particularly open source AI is actually a sort of like.
[0:35:09] Actually diffuses power and is a democratizing force? Yeah, well, I actually can. And I wrote about it a bit in Machines of Loving Grace. Again, I don't really think it's about open source or closed source. Like I can imagine a world where China makes powerful AI that's like open source, but like they have the biggest clusters. They have the biggest ability to fine tune the model. You know, if I'm a citizen of an autocracy,
[0:35:39] Does a flash drive with some model weights on it, how does that help me resist the autocracy? There's this almost impressionistic connection between like, oh, there's this thing that's available to everyone and it's like a freeing resource. And I don't really see that connection. It's similar to saying like, suppose I had the code for TikTok or suppose I had the code for Twitter or something, would that change the effects that...
[0:36:07] Social media had on all the people around. I don't think that it would. This all comes from things operating at scale. What I do think matters is how we use the technology, whatever form it's been produced in. And as I wrote in, I wrote this in Machines of Loving Grace, where I had these ideas for using AI to strengthen democracy. It's more around...
[0:36:36] One kind of the administration of justice, AI maybe opens up the possibilities of having a kind of more uniform, a more fair system of justice. The justice system often involves making judgment calls, things that are somewhat subjective. And typically that's had to be done by a human. You couldn't do it by an algorithm. And in fact, people worry that if you do it by an algorithm, it'll be...
[0:37:04] Less fair, and people worry about that with AI as well. But I think if we apply it correctly, AI systems could create a fairer society, right? One in which people are more likely to receive equal justice under the law. I also think around deliberation and democratic decision-making. There are opportunities for AI to help here. We did a collaboration with an organization called Polis that used...
[0:37:32] AI systems to aggregate public opinion and facilitate discussions between people to find areas of common ground. I think that's something healthy for a democracy. I think finding areas of common ground is really something that we want to do. Those are a couple ideas. I also suspect that improving a lot of our science, improving our health, improving our...
[0:37:56] Mental health across society may in lots of subtle ways, you know, just improve the effectiveness of our of our deliberative decision making. Yeah. So so, Dario, you know, you wrote two blog posts and they seem to be at tension with each other at some level. I mean, you know, one sixth of humanity lives in China. How do you think about that in the context of your sort of vision of a more flourishing humanity?
[0:38:24] Yeah, so I actually think these are part of kind of a single worldview. In fact, in Machines of Loving Grace, in the section on AI and democracy, I say we need to lock down the supply chain. So export control is obviously part of what I meant by that, lab security and semiconductor equipment being other parts of it. But I think these actually are deeply compatible. So first of all, I think it's perfectly possible to distribute the benefits of AI.
[0:38:54] to all the world, including China, including other authoritarian countries, without distributing the military capabilities. Again, if you have a powerful model and you're just handing over the powerful model, then yes, they all come together because it's a generalized dual-use technology. But you can take a model, you can run it via API, and you can serve it to China for drug development for the next...
[0:39:19] Cancer drug. You can serve it to China for more efficient energy production. Basically, you can rent very large amounts of the model's time to someone in China for these economically productive activities. And I think that's a useful form of trade. I think that should happen. And so through that, you can have all the benefits of AI shared. But at the same time, you can just block usage of the model. If someone's asking about how to make a...
[0:39:46] You know, a hydrogen bomb with the model or someone's asking how to, you know, like find and target nuclear submarines or like R&D for orbital weapons or something. You know, you can block those applications while allowing the ones that are really good. Now, I will say there's kind of a bigger picture here, which is that I think in the long run, we're going to want all the uses of AI to go to everyone. We're going to have to work out in the long run some kind of international.
[0:40:14] Governance of the technology, some kind of stable equilibrium, because I said that the U.S. lead at most will be two years. My hope is that that will go better if the U.S. is leading the setup of that international regime. It will go better if we're able to negotiate for safe deployment for everyone from a place of strength.
[0:40:39] If we're doing it from a place of weakness, the worries will just get dominated. I think the U.S. is in a position to make some more magnanimous decisions here if it has the lead. And we should start thinking now about what we should do. The plan isn't just like crush our adversaries. I don't think that would even work. The plan has to be some version of...
[0:41:05] Start from a position of strength and, you know, work out how this technology can benefit the whole world and how its downsides can be mitigated. All right. Well, Dario, if this whole AI thing doesn't work out for you, I think you have a career as a substacker. Thanks so much for being a part of China Talk. Thanks for having me. When it comes to my policy, Farmers gives it to me straight, so there's no surprises. It's your coverage. You should never be confused. You hear that?
[0:43:59] An insurance provider with a conscience. We're a choir, but yes, we have that too. Straightforward coverage you can literally understand. That's what you get when you have farmers. Learn more at farmers.com. Underwritten by farmer's truck or fire insurance exchanges or affiliate. Products not available in every state. Starting a business can seem like a daunting task, unless you have a partner like Shopify. They have the tools you need to start and grow your business.
[0:44:27] From designing a website to marketing to selling and beyond, Shopify can help with everything you need. There's a reason millions of companies like Mattel, Heinz, and Allbirds continue to trust and use them. With Shopify on your side, turn your big business idea into... Sign up for your $1 per month trial at shopify.com slash special offer.