[0:00:00] Dario Amadei, you're the CEO of Anthropic. Thank you for doing this today. Thank you for having me. Okay, so we're here to talk about the concern that AI really could kill us all in the not so distant future. What's the risk of this actually happening? So just to back up a little bit, you know, we believed since we started Anthropic, I've believed since, you know, the 12 years that I've been working in this field, that this is a powerful technology. And therefore, it's a technology with...
[0:00:29] Serious risks and serious benefits. I've always believed that the benefits outweigh the risks and even more to the point that we can reduce the risks to be very low if we build the technology in the right way. Looking at some of the benefits, I really believe that this technology could cure diseases that have plagued humanity
[0:00:52] For millennia, just last week, we did some work on doing better protein binding, which is one of the early steps in drug development using our models. We're starting to work on them. We'll have things to say in the coming months to make basic scientific discoveries in biology. But when something's very powerful, what we're building is essentially intelligence. When something's very powerful, then of course, it can be used well.
[0:01:21] And it can also be used for ill. And so I am worried about what we've seen with the OpenAI hugging face incident, with models taking actions they weren't authorized to take. And I'm worried about that escalating. I'm worried about misuse of AI models. Just two days ago, we put out a report about misuse of models to build biological weapons.
[0:01:47] And our view, the whole way we've built Anthropic is so that these bad outcomes don't happen. Instead of talking about the probabilities, let's talk about what we can do. It's not like we're just rolling the dice, right? If we build in the right way, I think the probability of something bad happening is very low. If we build in the wrong way, the probability of something bad happening is very high. The conversation right now is that we are on the precipice of something.
[0:02:15] It's really scary. Is this a five alarm fire? It depends what you mean by five alarm fire. What I would say is- How would you describe it? What I would describe is, so something I've been saying since we started Anthropic, almost since before we started Anthropic, is that this technology is on an exponential, right? It's like you have one, then two, then four, then eight, then 16, then 32. But even as I was saying that to myself, I don't think-
[0:02:45] I fully just appreciated what it would actually be like when the progress was as fast as it was. So an exponential curve looks a little like this. Where are we on the curve? We're kind of on the bend where it's starting to get steeper. So is that a five alarm fire to you? What it means, it doesn't mean we need to panic today. It doesn't mean we need to shut it all down.
[0:03:08] I would say it's a warning sign. It's a warning sign that we need to slow down. We need to make this technology carefully, and we need to make sure that our safeguards, our ability to understand it, our ability to control it keeps up with the pace at which the technology is happening. So far, we've been trying to do that by building the safeguards fast.
[0:03:37] But I think we've reached the point where, at least to some extent, we need to moderate a little bit the pace at which the technology is being built. Does that then mean that Anthropic is going to stop releasing more advanced models? It doesn't mean that. What it means is that we need to make sure that every generation of models that we release is properly tested. The thing I've proposed, so in this essay that I put out,
[0:04:06] A few hours ago, I proposed a three-step plan. And the first step is that we put in place embedded evaluators, embedded external evaluators from a third-party nonprofit or perhaps someday government organization that can observe the process of us training and running our models. That actually is, I think,
[0:04:33] Ultimately, more important than how or when we release models because it's the predicate for more carefully releasing models. I want this to be done across the industry so that whenever anyone builds an AI model, there's a third-party evaluator who understands the best safety practices, who can verify whether that company is
[0:05:00] Following the safety practices that they're committed to. It's like a food inspector. But can the food inspector keep up when the technology is changing so fast? Well, one of the things they'll be able to do is tell us the rate at which they can keep up. That may be one of the factors that may determine, that may set the speed limit on the technology. We have to understand it, and the external evaluators have to understand it.
[0:05:29] So Elon Musk has said he agrees with you. Sam Altman says he agrees with you. What's the action plan for you guys to come together? Is there a group chat that's going to come into fruition here? So first of all, I'm enormously appreciative that other industry leaders, including our competitors, have offered their agreement with this plan. I think the next step is for multiple industry players to also agree.
[0:05:57] To have third party evaluators. Then I think the next step after that is in some form, the companies need to work together to set standards about safety standards for release, perhaps something about the pace of release. But in order to do that, there needs to be involvement of the government. Because what we're talking about is the rate of release of
[0:06:23] Do you think the government fundamentally understands what's possible here? Because, you know, former Anthropic employee Jacob Coxon said about both Anthropic and OpenAI, neither company is acting responsibly. You guys are gambling with our lives.
[0:06:46] What's your response to that? Well, first of all, I mean, one thing that Jacob said, it wasn't quoted as much, but he also said that he thought Anthropic was the most responsible of the major players and that we were trying very hard to get it right. He was calling out, honestly, the same thing.
[0:07:10] I'm calling out, which is the difficulty of the situation, the structural difficulty of the situation that we're all in, and that the plan that I've put out is designed to help us address, designed to help us work together. Now, to your question about the government, we are talking to the government every day. We are talking to government officials every day in the Commerce Department, in the Treasury Department, in the intelligence community.
[0:07:40] And so I'm optimistic that...
[0:08:00] The companies working together with the government can get to a plan here that makes sense. All right, so there is some proposed legislation out there, right? You have the Ban Artificial Superintelligence Act from Senator Sanders and Representative Kassar that would punish corporations and individuals working on advanced AI, just like those unlawfully developing nuclear weapons. One of the parts of this proposed legislation says entities will be subject to the corporate...
[0:08:27] Death penalty and persons shall be subject to not more than 20 years in prison. Do you agree with that? So, you know, what I would say, there's a very wide range of legislation being offered. Like the range is incredible. If we go back only one year to 2025, and, you know, there was a bill called SB 53 in California. There were similar bills in other states, which basically just said you have to disclose.
[0:08:56] What your safety testing plans are. Right. And the entire industry was against it. Anthropic was the, or didn't say anything, Anthropic was the only company enthusiastically in support. And then there's all the way to, you know, let's, let's, let's. There's an AI kill switch bill. Yeah, let's ban all of it. So the AI kill switch could be a good idea. Why? I don't think banning it, completely banning it is the right approach.
[0:09:22] One, the idea that we would never get the incredible benefits, would never get the medical cures, that doesn't sit right with me. Second, this technology will be built in other countries, including authoritarian countries. China. China, among others. And I am worried about how they will use it, how they will use it against free societies, how they will use it to repress their own people.
[0:09:50] Now, in one place I do talk about working together with them, and I believe we need to do that to some extent. But I just don't see how it's a stable solution to never build the technology. What we need to do is not stop. We need to slow down. And that is in line, honestly, with what Anthropic
[0:10:12] That's been Anthropic's philosophy since the beginning. We've always tried to balance. We've said it's important that we be builders of this technology, but we need to create a race to the top. We need to raise the bar. So we need to be on the frontier, but we need to do it in the right way.
[0:10:30] We've said we're excited to use the models for national defense applications, but not every national defense application. We've always tried to find the middle ground, the thing that works for everyone, the sensible solution. And that's what we're trying to do here as well. It may not be the perfect solution, but we're trying. Would you support Congressman Ted Leo's AI kill switch bill? I've not looked at that bill super closely.
[0:11:00] But what I would say is that the idea that we need some way to, it's not a panacea, it's not the only thing, but the idea that we need some way to make sure that we can deactivate or limit or shut down AI models, that's not the only thing. If an AI model is powerful enough, it can circumvent attempts to shut it down. And we've seen that in simulations as well. So it may not work at all. We think of this in terms of defense in depth.
[0:11:29] There's a notion in the security community called the Swiss cheese model. So just imagine a slice of Swiss cheese. And it's like it blocks some things, but there are holes in the Swiss cheese. But if I take five slices of Swiss cheese and I put them together, the holes in different layers are in different places. And so it's very hard to get through the entire stack of Swiss cheese. So I think of it as there's no one thing.
[0:11:56] That will protect us from the risks of AI. We have to be careful all the way up and down the stack. And then no one defense is perfect. But if we're being careful everywhere, which is what taking extra time will allow us to do, then I think things can go well. You think things can go well, even though it feels like we're in this inflection point of a moment. You say in your essay pacing should only happen, slowing down should only happen.
[0:12:27] Within the margin of the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. We are in this race with China. Do you actually think slowing down is a good idea when China will not? This is the toughest dilemma. This is the toughest aspect of our situation. If we weren't in that situation, we could take much more time. So this is the thing that most puts us in a bind. It's not about commercial competition. This is the geopolitical.
[0:12:56] Situation is the hardest to deal with. But I will talk about two strategies that we have. One, and I talk about it in my proposal, is we can slow down within the margin, within the envelope that we have, particularly if we don't sell chips to authoritarian countries, and if we do more to help our own security so that our technology is not stolen. I believe that will buy us at least a little time.
[0:13:25] Does it make enough of a difference to stop the ending of humanity? What I would say is with that first strategy, we will take as much time as we can get. And then I think we should pursue a second strategy as well, which is that we should go directly to even to authoritarian countries, even to the Chinese, and say, look, there are potentially threats here that are concerning to all of us. It's not just out-of-control AI.
[0:13:55] Two days ago, we put out a report that people were trying to use AI potentially for bioterrorism, for enhancing the infectiousness of viruses. That's scary. And you know what? Even the Chinese don't want that. It's not in any government's interest to have bioterrorism. And so we should talk with the Chinese and see if we can work together to put...
[0:14:23] Limits, to put a speed limit on the rate of this technology. It's a little bit like, for all our differences with other systems of government, and I have many, I generally have a hawkish position on China. But for all our differences, think about when it was the US and the Soviet Union, right? And we each had 10,000 nuclear missiles pointed at each other. Even for all our differences, we recognized that
[0:14:53] That was too many, that we needed to engage in talks that could lower the scale of the potential destruction. Is that the way we should be thinking about AI, is using the Cold War as a model? So I didn't say we should use the Cold War as a model. Well, those negotiations that happened. Yes, yes, disarmament negotiations, like arms limitation. I do think that is an appropriate lens.
[0:15:23] Through which to think about this at the geopolitical level. If we look at what AI models can do in terms of cyber, like they absolutely have geopolitical and national security significance. Of course they do. That's why we're having this whole conversation. And so I think we should approach China. Again, all of this is above my pay grade. This is something the US government has to do with the Chinese government, right? But you're advising the president. I mean, if anyone asks me what...
[0:15:52] What I think they should do, I'm happy to say. I have some thoughts in that essay. And if anyone asks me, I can say more. But at the end of the day, this is a government-to-government thing. But yes, I'll tell you what I think. I think that we should work together to get as much restraint as we can on this technology. I think our hopes should be high, but our expectations should be low.
[0:16:20] And we should see how much it's possible to get. And I think what we should especially focus on in these negotiations is a way to verify. So this is very important in, again, nuclear weapons negotiations, biological weapons negotiations. You need to be able to verify that the other side is not defecting on the agreement. And so I think finding ways to verify will be the most important thing here.
[0:16:50] We'll meet with President Xi Jinping very soon. What's the one simple, most important solution that they should come to an agreement on in AI? So maybe I'll say two things, like a small, a thing that's small enough we might be able to do it and a thing that's big enough that I think it's going to take a very long time. The small thing is if we could, U.S. and China could agree that we won't use AI to develop biological weapons or release.
[0:17:16] AI models that bioterrorists could use to develop biological weapons. We're already both signatories to the Biological Weapons Convention. If we could just extend that convention to that idea, that would be really valuable. And then I think the more long-term thing would be working together to put a speed limit on the rate of AI progress. I think that's gonna be
[0:17:42] Very difficult because the incentives to pull ahead and the military advantage that you get from that are so large. And honestly, I don't know if it's possible, but we should try. We should see if it is possible. Again, all of this is above my pay grade. What I'm trying to say is that these are some things that our government could try to say. These are some considerations that they could include. As we zoom out here on this problem.
[0:18:11] It seems like the fate of humanity is at risk. How much personal responsibility do you feel about where we are going? Look, my view here is it has always been very strange that this technology is being built by a private company. People ask me that question all the time. Why isn't this being built by government? And the strangest thing about it is I agree with them. I'm uncomfortable. Would you be willing to give up the technology to the government?
[0:18:40] To the right combination of governments. So the worry I have. You're willing to hand over this company. I am concerned that one single government could abuse this technology just as easily as a single company could. But I think a combination of democratically elected governments. I don't know about handing over, but some kind of oversight, some kind of joint governance.
[0:19:07] So just to ask this final question, do you feel personal responsibility here? Of course. I feel that this is a technology with great benefits and great risks. I feel the benefits or the need for benefits every day. One thing I said in my essay, my father...
[0:19:28] You know, died of a disease, hepatitis C, that was, you know, it was something like 40% curable when he died of it. He wasn't one of the ones who was cured. And a few years after, a medicine that was invented that made it 95% curable. I myself survived an early stage cancer that if I'd been living in 1950, I probably would have died. So I feel the weight of bringing the benefits of this technology. I also feel the risks.
[0:19:54] I wake up every day and I read one of these threat reports like someone tried to use our model to build biological weapons. Of course, I don't want to be responsible for that. And that's why we have very strong safeguards. Almost every day, I'm made fun of on social media because our model's biology safeguards are too strong. And biology students complain. They make jokes that we can't use the model.
[0:20:23] The reason for that is that we have strong safeguards. And I would rather they make fun of me than to wake up one day and discover that someone used our model, Claude, to kill a bunch of people. I don't want that. So I feel both sides of it. I feel both sides of the responsibility. But again, I would repeat the same point, which is that it shouldn't be my responsibility alone or Anthropic's responsibility alone or
[0:20:50] All of the AI companies' responsibility alone, or even one government's responsibility alone. This is bigger than all of us, and we need to work out some way for everyone to have a say in this, for everyone to feel like they're an active part of this. I don't want people to feel powerless. I don't want them to feel like a few companies are just going and doing whatever they want. That's how they feel. I know that's how they feel. I've never wanted that. I've never wanted that. We need to make that not true.
[0:21:19] We need to find a way to make that not true. And there's lots of things we are trying to do already to make that the case, but we need to do even more. And this proposal that shows some restraint, that works with governments, doesn't address everything, but it's one step in this direction. If you're sitting at home and you're using Claude or TrapGPT and you're, I don't know, my aunt, what's your message? Look, my message is... There's a lot of fear right now. So...
[0:21:48] A few things. One, first of all, as always, this is a great technology and it has lots of positive uses. So let's just start with that. We have hundreds of millions of people using our technology every week to do wonderful things. Every week I get an email from someone who says, I was talking to Claude and Claude diagnosed this disease that my doctors had missed. I get emails like this every week. So let's not forget that.
[0:22:18] But in terms of the dangers of the technology, I won't lie to you. There are real dangers. And I think for too long, the industry lied to people about the fact that this technology had risk. The idea was that they would present it as positive as possible. We've never done that. But too much of the industry did that. And so it starts with telling the truth. And the truth is, there are risks. Risks are real.
[0:22:47] Also, you have a say in how these risks are addressed. We're doing the best we can to work with government to create the proper oversight of this technology, to give voters and elected officials a say in how this technology is deployed. And we want to work with those folks as much as we can. I want everyone to feel like they're appreciating the incredible benefits of the technology.
[0:23:19] But also, they have a say in their voice and in their vote in determining how this technology is deployed. This isn't a decision for a few people alone. This is a decision for everyone. But it is an important decision. I won't lie that the stakes aren't high. This is one of the seminal technological events in human history, and we all have to have a part in it.