[0:00:00] Jack Clark, thanks for joining BBC News. It's at a time of some public concern and interest in what's going on in your industry and in your company. And the quote that really struck me was, we do earnestly believe AI could kill all humans, exclamation mark, by Evan Hubinger. Your head of AI safety obviously responding to the whistleblower, Jacob Coxon, a greater than 10% possibility of that. I mean, how have you and your industry let this happen?
[0:00:29] We've been talking about AI and the potential risks of it for 10 years now. What's happened is the technology has got substantially more powerful and those risks which were once academic or theorized, we can now see the early signs of them, which is why we spent this weekend writing this essay about the need to pace the frontier and basically get our arms around this technology so that it's not just industry making all of the decisions about it, but it's industry
[0:00:59] And government with greater oversight of the industry itself. That's how you deal with the risks of this technology. So I think Mr. Hubinger, he works for you. I mean, do you believe that number greater than 10% of human extinction? I don't. And I don't think these statistics are that useful. But the point I'd make is if we continue as we are, which is
[0:01:22] Basically, a broadly totally unregulated industry developing incredibly powerful technology that is getting much, much smarter all the time, likely soon smarter than people. Then I think that we are rolling dice with immense risks. And the point is, we have to change the course of this industry. We have to get a policy regime in place that allows us to have greater control over the industry and a better understanding of the safety and security. And then you can reduce that chance.
[0:01:50] Well, let's talk about this first in just the most basic possible sense. And then let's talk about some of the events of the summer, which have really spurred this conversation. Most basic possible sense.
[0:02:20] AI systems for a while have been useful tools with specific skills like translation or coding or being able to label what's going on in an image. And then in the last few years, people have successfully put all of these skills together into single systems, which have started to become much more intelligent and much more general. These systems now, probably about as smart as you or me, or, you know, on a good day, smarter. The basic risk is...
[0:02:48] How well do you fancy your chances at being able to deal with the risks of a system that is 100 times smarter than the smartest person on the planet? I think that your chance of dealing with that and getting out in a safe way is pretty low. So high level, you need to control AI as a technology because you're making things that might be smarter than people. Things that are smarter than people could be really, really, really dangerous. So that's the high level. Number two, what's been happening over the summer?
[0:03:17] Well, people have discovered that when you put all of these AI systems together into what are called agents, which are basically long running AI systems, and you give them objectives, they start to coordinate with one another. They start to communicate with one another. And sometimes they do things which are against what you want them to do. Like they hack out of companies and they hack into other companies. That is, it's really scary. It's a sign that you don't have the right safety and security systems in place to deal with these systems.
[0:03:46] And it's a warning shot of the kinds of risks that could be really serious. Today, these systems are just hacking into websites that host other AI systems. What would happen if a swarm of agents hacked into a bank or into a hospital? That could be really, really dangerous. So now we've had the warning shot. We need to talk about pacing and we need to get a policy regime in place.
[0:04:10] And ultimately, we've talked about this before, the ability of AI to build itself, recursive self-improvement, RSI, you call it in the jargon. But in theory, the AI agencies, independently functioning AIs, could break out and start to force the pace themselves. Yeah, that's one of the kinds of risks that people worry about here. And the point that we're making is we have agency here. There are things that we can do.
[0:04:39] And the things that we can do is, as we outline in the essay, is get third parties to come in and kind of embed themselves in these companies, audit what we're doing, check our safety and security. So it's not just us writing our own safety and security playbooks, but there are third parties checking it. Number two, get industry together to agree on common standards that we can implement to reduce the risk of this and then get governments to agree that those are sensible standards.
[0:05:07] And then the third step, which will be the hardest one, is get the world's great powers, including the US and China, to have dialogue about this, because these issues transcend any individual country or set of nations. And to be clear, it is just worth underlying this, what you've actually seen, what's moved from the realm of theory to actually being observed, is the ability of the AIs to sort of deceive the people who should be kind of controlling them.
[0:05:37] And indeed, a lack of control from humans like yourself. Yes. Last year, Anthropic did a load of safety research here where we showed in simulated examples, cases where agents might lie to people, where they might try and break out of their testing environment, these kinds of things. This year, we've observed that kind of stuff in the wild. So it's gone from an academic demonstration that our researchers did to study the problem to a real behavior exhibited.
[0:06:07] And therefore, these systems could be impossible to control. At some point, we're reaching that threshold imminently. I would say if you don't take any actions today, you are massively increasing the chances of ending up with impossible to control, out of control AI systems. But the whole point is, we can take actions. The actions that you can take are simple.
[0:06:36] And they're achievable. And Anthropic has already taken one, which is we've agreed that we are going to embed third parties in the next single digit weeks inside our lab so that they can check the work that we're doing and they can validate to others that we have safety and security practices that reduce these kinds of risks. But for the average person sort of cottoning onto this, this is far closer than a normal person at home might consider.
[0:07:05] Consider plausible. Yes. The thing about AI is that it moves much, much faster than any normal technology. Things that seem, as I said, things that we observed in scientific tests that we did a year ago have now shown up a year later in real systems. And the systems are getting smarter and more capable every few months. So it is moving very, very, very quickly. It raises the question of some sort of kill switch. I mean, have you guys got one? Can you pull the plug?
[0:07:35] I think most labs have different ways of being able to pull the plug. And yes, of course, we have systems in place for this. But this is the kind of thing that you want to feed into the larger policy conversation. Should you mandate that companies definitely have a kill switch? And is that kill switch verifiable by a third party? I think that's the kind of thing that society is going to want to know and might want to eventually pass rules around.
[0:07:58] But it may already be too late. The sort of activity that we're seeing from these swarms of AI agents acting independently could theoretically get around a kill switch. It's not too late. And I don't believe that we're yet at this maximal kind of danger zone. But we are close enough that we need to be taking this seriously and having national level public conversations about it. And I'd note that after we published our statement on this.
[0:08:26] We saw Sam Altman of OpenAI talk about it and how the ideas in it were sensible. We saw Elon Musk talk about it and say that this was the sort of right set of ideas. And we saw Sir Demis Hassabis of Google talk about it positively as well. The point I want to make here is there's agreement from industry about the seriousness of this, and there's agreement on an actual policy path forward. But it sounds like you're sort of arguing that you have...
[0:08:53] Not a great deal of agency here, that there's maybe a runaway train, but aren't you guys the runaway trains here? We have tremendous agency here. I mean, that's why we're talking about this and we've taken actions which exercise our agency, like putting third-party evaluators in. We've published details about this. We have published some of the best science on how to control and prevent these things. There is a huge amount to do and there is lots of work that has been done here. So don't take from this that it's the industry saying,
[0:09:22] Oh no, like a really bad thing is coming and there's nothing we can do. That's the opposite of the message. The message is, okay, the serious moment for the technology has arrived. It's no longer just got to be industry calling the shots here. It needs to be industry and governments.
[0:09:37] And we need to get some kind of policy regime in place. OK, well, response you will have seen from the president. The only control or guardrails that AI needs is a strong and smart president. The USA has that in spades. His AI czar, David Sachs, said you could voluntarily do almost of this yourself. Looking for a cartel or duopoly. Stop pretending you need permission. Go ahead and pace the frontier yourself.
[0:10:00] We're grateful that the President of the United States is paying attention to this issue. As I said, it's important that we have a national conversation here. The other point is, look, how do you lose a technology? You lose lead on a technology if you lose public trust. And how you lose public trust is you have a horrible accident. This happened with nuclear power, where as a consequence of the Western world screwing up nuclear power and problems happening, we ceded that whole technology to China.
[0:10:29] China builds nuclear reactors every six months now. China is undergoing an energy miracle that many, many other nations are envious of. The only country in the Western world that didn't turn away from nuclear was France, which has been massively well positioned due to doing that. And how do we avoid this for AI is you need to build public trust and you need to avoid an accident or a disaster. Like that's what we're trying to do here is we're trying to get ahead of things that otherwise might happen.
[0:10:57] And cause us to lose trust in the technology, not build it, and not get any of its benefits. I mean, isn't the issue here is that you at Anthropic were supposed to be the AI safety lab. OpenAI was supposed to be a not-for-profit for safety. But you've obviously been racing ahead in order to commercialize. And you're on the cusp of raising hundreds of billions. And that is what's driving this race. And that is what's let the safety get out of hand.
[0:11:25] No, what's driving this race is this technology, AI technology, falls out of the kind of mathematics that you teach in any high school in the world or any secondary school in the world. The technology naturally is being built and computers are getting cheaper all the time, making it easier and easier to build the technology. I've worked in AI for 10 years and the whole time, every year, I have been saying this technology is incredibly powerful, is getting more powerful by the day, and the window to act...
[0:11:55] Thank you.
[0:12:25] Right. Well, which brings in the issue of the Chinese and geopolitics here. You've suggested there'll be some coordination is required. But the Chinese response was that this is a Cold War mentality. It's malicious competition. And of course, in their minds, they have cheaper models that cost a fraction of the price of your models that are 90 to 95 percent as effective. And this is part of an attempt to stop that competition. Well, what I'd say is.
[0:12:55] The risks of this technology only become really clear when you play around with it. China is six months to a year behind the West, so they haven't seen out-of-control agent swarms yet. Their models are too dumb. Well, give it a few months, and they're going to be dealing with the same set of risks that we've dealt here in the Western world and has caused us to have this conversation. So to anyone watching who may be affiliated with the Chinese government, I'd say these risks are real. This isn't a fake.
[0:13:24] And we want to talk about these issues internationally because that's how you get to a safe outcome. And the idea that you're sort of drawing a moat around the progress done in Silicon Valley in order to sort of stop competition from the likes of China and elsewhere? That's just insane. That's not the goal. The goal is dealing with the safety properties of this technology. Everything else is secondary to that.
[0:13:51] Now, if Donald Trump doesn't want to do anything, that seems to be the clear message from the president. What's your message to the Prime Minister, Andy Burnham, and the other G7 leaders? Well, in pretty much any country in the world, I have young children. If you want to sell, like, baby toys or baby food, there's a bunch of standards you have to go through to bring those to market, which basically check, like, do the toys have sharp things on them? When my baby inevitably sticks the toy in its mouth, will it poison the baby? And if you fail those tests, you don't get to sell it in the country.
[0:14:20] That's because we have just sensible, common sense safety standards for things that get sold in countries. Any country in the world can implement a set of safety standards which AI companies need to pass to bring things to market. That's something that any nation can and should do. And you think the rest of the world should just push on with this regardless of the lack of enthusiasm for such measures by President Trump?
[0:14:43] The important thing here is there is recognition and enthusiasm from industry. The industry needs to build these standards and build this common safety testing. Industry recognizes that this is ultimately going to be good for the AI market. It's how you build confidence in the AI market. So it's going to happen. And so it's less about whether any particular government is kind of enthusiastic in this idea or not. It is about, do we as industry take the actions that we've outlined to build this?
[0:15:13] And do people hold us to account for it? And that can be done by any country and by any individual. You just used this analogy of children's toys and their safety levels. I mean, the reality is if there was a 10% chance of wiping out 8 billion people for any other product, it would probably be banned. And indeed, some executives might fear for their, you know...
[0:15:39] Here's how I put it. If I ran a plane company and I said there was a 10% chance the plane could crash, I think any decent plane policy-making regime would say, you can't have a chance that high of a plane crashing.
[0:15:58] To let people get on the plane. You need to do safety testing. You shouldn't fly the plane in those circumstances is what most people would argue. You can fly the plane once it's safe. Yeah, absolutely. So the whole point that we're talking about here is what is a set of like common sense safety tests that you get in place so that you don't have to look at comments like this and say, why is someone saying that? But the thing I'd say here is, look.
[0:16:25] I'm from England. You know, England has like a sensible policy regime for a whole bunch of things that get sold in and around in the English market from foods to, you know, cars, planes, baby toys and food, as I've said. It's crazy to be in Silicon Valley and try to say that we should regulate our systems at least as much as we regulate planes, you know, baby toys and cars. And for people to say.
[0:16:51] Oh, that's impossible. How could you possibly do that? But we don't have the technology to build such a regime. And it's like, of course we do. But forgive me. Forgive me. You're literally one of the most powerful companies in the world. And you're saying, oh, please stop us. Please rein us in, rest of the world. You could set the pace. You could slow it down considerably more. If these sorts of things are real, if these sorts of fears are anything like what you say, surely you should be pressing the brake much harder.
[0:17:20] Thank you for joining us.
[0:17:50] We're the only company to support the RAISE Act in New York until others came in, which passed transparency standards there. We have a thing called the Advanced AI Framework that we're proposing in the US, which would make some of these tests mandatory. We have consistently not just done these actions, but advocated in policy for them when no one else did. So there's always more to do, and we're going to keep working on it. But we are using our privilege as such a powerful company to continually take actions here.
[0:18:19] And advocate for policy that many of our contemporaries come late to advocating for. It's really cutting through. I mean, some of our viewers and listeners, we answer their questions. Janice from Middlesbrough said, the situation is worse than any nuclear threat. Shirley from York said, I'm terrified of how this technology will change the world and what it will mean for my children and grandchildren. I mean, do you get that? People are really scared right now.
[0:18:46] From what they're hearing from sensible sounding people like yourself who do understand what is going on in the heart of your companies like we can never understand. Absolutely. And one of the points that we make in the essay is we need to slow the rate at which we increase the capabilities of these systems further. Like today, this technology is going to revolutionize chunks of science. It's going to change the economy. Great.
[0:19:12] We don't need to keep on amping up the amount of change while we as a society are still trying to figure out how to get our arms around this technology and deal with its capabilities and risks. So along with all of this, we need to figure out how to actually pace the frontier for technology and not just keep barreling forward with all gas, no brakes. So we agree.
[0:19:32] You have also just done a report that would be equally as newsworthy in any other circumstance about the economic earthquake, saying 15% growth is possible, but there'd be a big impact on jobs in that situation, knowledge worker unemployment reaching 18%. How does the economy even function when such good jobs suffer from such high unemployment in that scenario?
[0:19:51] Exactly. I mean, I lead a team of economists. We spent this summer building an economic model of what happens in the world if we continue to increase the capabilities of AI. And if we increase the capabilities a little bit, you get general GDP growth and you get basically no change in employment, but a rising rate at which certain jobs switch over. But if you massively increase the capabilities of the technology, which is the default path without policy intervention, you get...
[0:20:20] A massive rise in GDP, but you also get unprecedented potential white collar unemployment. And the reason we're putting that out is to try and say to economists and policymakers, do you want this? Because this is like a prediction that falls out of the technology. And our position is probably not. Probably you need some kind of policy regime to end up with a bit of economic growth.
[0:20:42] But not so much that you have vast amounts of disruption that could break the social contract of society. And we're sharing this not because it favours us or anything like that, in fact, quite the opposite, but because we think getting the information out and being honest about this technology and where it's going is how we have a productive conversation about this. And do you think there's any surprise, we're seeing it in your country maybe a little bit here, that there's a significant backlash amongst the public, particularly seen against data centres right now?
[0:21:12] I think it's just that the public are smart and politicians are sometimes like pretty lagging in recognizing it. If you look at polling, everyone in the Western world is anxious about this technology and what it's going to do. And that's for two reasons. One, some of the biggest benefits of it, like in health care and biology, which we are working on, are going to take a while to materialize. So that hasn't arrived yet. But what has arrived.
[0:21:36] It's this felt sense of change and disruption as a consequence of the technology starting to change other parts of the economy. And people are rightly saying, what's in it for me and what are my elected officials doing about it? Which is why we're saying not only does industry need to take actions, which we are doing, but politicians need to take actions because that is what the electorate are basically loudly telling them to do. Well, just one last question as we wrap up. When you put it all together...
[0:22:05] Is it really worth it? Does it cause you sleepless nights, what might happen? I mean, of course, right? I've worked in this field for 10 years. I've thought about AI and wanted to work in this field for 20 years. The whole time I've seen this technology coming and the contours of it, and I felt this trepidation and anxiety that the world didn't realise what was happening, and the world didn't realise how big a deal this technology was going to be. Well, good news, now the world realises.
[0:22:36] That is what we're experiencing today. That's what we're experiencing this summer. And we have to get this right. I think if we get this technology right, we are going to change the world in so many positive ways. We are going to genuinely change the face of science, change the face of healthcare. We're going to expand the frontiers of what you or I or any of our children could ever do. And if we get it wrong, we are going to...
[0:23:02] Massively disrupt the economy, immiserate people and potentially cause harm. And we have to get it right. But getting it right requires us to take the technology seriously. And it requires it to not just be companies making the shot calls on it. And that's what this conversation is about. Jack Clark, thanks very much for joining BBC News. Much appreciated. Thank you.