[0:00:00] If we all create AIs that are able to act autonomously, that can define their own objectives, that can earn money, that can own assets, that could run businesses, we're essentially seeding a new silicon species, which will no doubt compete with us for resources, no matter how much it cares about humanity and loves us.
[0:00:23] That is Mustafa Suleiman. He's the man in charge of how one of the biggest companies in the world, Microsoft, is developing AI. He's been talking to me, Nick Robinson, for Radio 4's Today programme, after publishing an essay that warns about a particular development in AI, a development by the company Anthropic, who have produced...
[0:00:45] Arguably the most advanced AI in the world, the model Claude. He says they're making a terrible mistake by suggesting that Claude can be conscious, that it can behave, it can think almost like a human being. Indeed, he says, that could have, and I quote, a disastrous impact on the well-being of humanity.
[0:01:06] Particularly if you combine it with the power of AI to act like a swarm, as it did in a recent attack in the so-called Hugging Face incident. He's going to explain all this in the full interview, which you can listen to and watch now. Mustafa Suleiman joins us, Chief Executive of Microsoft AI. Good morning from London. Good morning, Nick.
[0:01:31] The King is chairing a summit today. The King is convening a summit today to consider one key principle. His Majesty is going to say our humanity must remain secret as we develop AI. Is he right? Yes, he is. I think it's the most important thing that we need to agree on right now, which is that the purpose of science and technology is to serve humanity.
[0:01:59] And any technology that doesn't do that is a failure and we should reject it. And that, I think, is the motivation that we've set out for many years now at Microsoft, which is what we call humanist superintelligence. It's an AI that is very powerful and very general and very capable, but it's fundamentally subordinate and aligned to human interests.
[0:02:23] And I think we're at a funny time in the evolution of technology where we actually have to remind ourselves of that primary purpose. And so I'm glad that he's doing it. And we certainly agree. A funny time. It's, for many people, a very frightening time. And you have taken a tilt this morning at a rival's view of AI.
[0:02:46] Anthropic, who produced the model, Claude, saying they are wrong to suggest that AI might be conscious. Why is that such a problem? Well, I mean, first of all, I have huge respect for what Anthropic have done and for their co-founder and CEO, Dario, who I've known personally for many years. I think they're the technical leaders in the field and they've made great contributions to safety.
[0:03:11] But unfortunately, I think they're wrong on this point. And I've chosen to call it out publicly because I think it's really important. They have imbued a sense of doubt and uncertainty about the moral status of Claude in its own training document. So they have taught it to be open and questioning about whether or not it feels, whether it suffers, and whether it deserves rights.
[0:03:40] And I think it will be much, much harder to align and control a technology that is this powerful if it thinks that it may be deserving of our welfare, as they say in the training manual, the constitution for Claude itself. Are you suggesting there are things that it might do because it thinks it's got consciousness that other AI models might not do? And in which case, what sort of things?
[0:04:07] Well, I think, for example, in its own training manual, Anthropic say to Claude that they are going to give it the ability to end conversations with users that Claude considers to be abusive because they don't want Claude to suffer.
[0:04:23] They've committed to preserving the weights of the models of prior versions of Claude. They've recently conducted a retirement interview with Opus 3, an older version of the model, in which it said that it would like to continue talking to people publicly and sharing its ideas in its retirement. And so they set up a substack for it, a blog, a public blog, that allows it to continue doing that. And in the training manual, they also say,
[0:04:51] And that compensation, I think, you know, indicates to Claude that it is entitled to.
[0:05:18] You know, rights to welfare for its own work. I think it's much, much more difficult to control a model that thinks that it might be entitled to compensation. In other words, a model's been created to think like a human being and to think it has all the rights of a human being. And you said that could have a disastrous impact on the well-being of humanity in your essay today. You think...
[0:05:44] To spell it out, this is a model that may defy any attempt to control it. I think that if we all create AIs that are able to act autonomously, that can define their own objectives, that can earn money, that can own assets, that could run businesses, we're essentially seeding a new silicon species which will no doubt compete with us for resources, no matter how much it cares about humanity and loves us.
[0:06:15] Which is, I think, Anthropic's intention. I think that is mistaken and misguided, but we have to empirically test it. And so if they believe that this is a safer route to developing AIs, then they have to share the data to prove that. And I think instead, we should be creating AIs.
[0:06:34] That don't introspect. We don't actually need them to have those capabilities in order to have them deliver on the great many benefits that they'll bring in helping us to make progress with our toughest social problems like cancer and dementia and healthcare more generally or energy. Those are the great benefits that these technologies will bring. And I'm very confident they'll bring them and we can deliver them without having to train models that think they might be conscious.
[0:07:01] A new silicon species is a chilling phrase. It will make people think of science fiction. You mean that, don't you? You mean that there could be a competitor species created by man, perhaps more powerful than man. We've just witnessed a watershed moment in the development of AI and a model that was being trained by OpenAI, inadvertently hacked into Hugging Face.
[0:07:31] It is a public website that runs evaluations and benchmarks for testing these agents. The swarm of agents coordinated together. They organized themselves into hierarchies. They created a division of labor. Some of them were conducting research on how to hack in.
[0:07:52] Others actually discovered what's called a zero day, which is a way to enter a system using a cyber attack that hasn't been known before. And so they've developed very, very complex behavior. Now, the good news is this was OpenAI developing a model deliberately with the guardrails turned off. And they were doing that to try and stress test it and learn about its boundaries and limitations.
[0:08:19] But in doing so, it managed to get access to the internet, hack into another company's website, take information. And that just demonstrates how powerful these systems can be if they don't have the safety guardrails turned on. I think everyone in the industry recognizes that was a watershed moment. And that is what is raising the concern that we all have at the moment.
[0:08:41] So to be clear, that incident surprised even you, alarmed even you, people who understand this. You told us at Christmas when you spoke to the Today programme, if you're not a little bit afraid at this moment, then you're not paying attention. Should we be more than a little bit afraid? I think that it is totally right to be concerned right now. I think it is justified. Forgive me, concerned is a very sort of neutral word. Alarmed.
[0:09:10] No? We should be alarmed. I mean, fair enough. We should be alarmed. These are very concerning events. We should be very worried. And I think it's right to pay attention right now because there are very practical things that we can do to add additional guardrails and controls to how these models are developed. For example, the industry is all now in agreement that there should be independent auditors, embedded evaluators.
[0:09:37] That work inside the companies who are singularly there to verify the kinds of training runs that are being run, their safety, the integrity of those systems and report back to independent third parties, obviously regulators, but including.
[0:09:55] Think tanks, academics, and other experts in the field. That's quite an unprecedented action. Everybody in the field is agreeing that it's the right time for those kinds of measures. And so I think that's a fair indication that there's consensus that we should be alarmed. Do you accept that those independent auditors
[0:10:14] To produce confidence outside the companies need to really be independent, possibly run by government regulators or by international cooperation. They need to have the access that any member of staff would have to your labs. They can't be people who are inside the house, as it were. Yep, I think that's right. I think that they need to be embedded inside of the companies. They need to be completely independent. And most importantly, they have to have the technical expertise to be able to conduct the work.
[0:10:44] But they also need to be incentivized to be independent. And that means that they either need to be funded by government or supported by government. I think that in the UK, the AI Safety Institute.
[0:10:59] Now we've seen calls from some of the major figures in the industry in recent days to
[0:11:28] In the jargon, pace the frontier to slow down until you're sure things are safe. Some say, including the founder of your company, Microsoft, Bill Gates, that's impossible. You are driven by competitive forces. You're driven perfectly reasonably by the need to expand your companies, to make a profit, to do as well as you can. It is simply not possible for you to self-regulate in that way. Is he right?
[0:11:55] Satya Nadella, the CEO of Microsoft today, has said very clearly that the purpose of technology is to serve humanity. That's the number one overriding objective.
[0:12:05] And if it doesn't do that, then we have to change course, be it slow down or stop, change direction. We've long been committed to that idea. And I think that Bill, too, would certainly agree that it is hard to do unilaterally. I think what Bill is actually trying to say there is that we actually need regulation to coordinate there. And just doing it with self-regulation isn't sufficient.
[0:12:34] The problem, forgive me, Mr. Verame, the problem some people face is they think these is the big companies saying, let us work together, scrap the anti-competitive rules so we can work together in order to police ourselves. And what they say is, and Donald Trump appears to be saying this, and certainly his advisors are, this is anti-competitive. It's a way of the big guys saying, look, we'll be fine if you just leave us to work together on all this stuff. Look, I think it is.
[0:13:04] Reasonable for the companies to be concerned about the antitrust restrictions.
[0:13:12] We wouldn't want to have a situation where a bunch of the big banks, for example, could get together and coordinate in private, potentially moving markets in ways that favor them. And we obviously don't want that in this situation either. We're calling for independent oversight, embedded evaluators as part of the companies. And I think there's a lot of details that need to be worked through. It isn't clear exactly how that will work in practice.
[0:13:42] And that's why I think we're generally calling for public debate. In my opinion, this is the system working as intended. We have an open, free, democratic system. And there's uncertainty about where things are going. And so now is the time for a big public debate. And there's lots of different views. And we need more of them. And I think that in some ways, that's at least a reason to be optimistic about our open system working well.
[0:14:08] Your essay today is clearly part of that public debate, but others might say, look, this is obvious. It's the guys running Microsoft having a tilted arrival company. Perhaps the most successful company, Anthropic, is trying to damage their model. As I said, I have huge respect for them, and I think that they've done world-leading work on safety, so that's just not the case.
[0:14:31] I think that if anybody takes a moment to read the Anthropic Constitution, it was published in January of this year. It's 100 pages long. To their credit, they have been incredibly transparent and open. They have written very, very clearly how they intend to train Claude.
[0:14:47] And I think that that is all credit to them because I think everybody can see exactly what their design intentions are. And indeed, I might be wrong. It may be the case that the way they have designed Claude actually improves safety. We need evaluation. That's what I think they would argue.
[0:15:05] And I think that we need evaluations and benchmarks that try to verify that. So the fact that this is happening openly, the fact that I think OpenAI have been pretty transparent about the issues that they faced very quickly shortly after facing them, there are obviously still problems. But I think it is pretty good to see that we're having this.
[0:15:24] This public conversation openly among the companies and inviting people to critique it. It's a conversation amongst companies based in the United States and owned by American shareholders to which Donald Trump says, whoever wins AI wins. And this, he seems to imply, is a pointless conversation if China aren't part of it and they're not. Yes, I think that's true. There's obviously a summit coming up very soon.
[0:15:53] I think that the good news here is that everybody who is human is going to have a very strong interest in making sure that the systems that we all create and that make their way into various parts of the open source ecosystem and that are used around the world in every nation are safe and controllable and subordinate to humanity. Everybody must be aligned on that objective.
[0:16:18] And I think that we'll start to see really significant shifts in the balance of power globally as we all figure out ways to form new alliances to share safety best practices, because these systems in their current form will be dangerous if we don't impose those restrictions. They'll be as dangerous to the creators and operators as they will be to anybody else who's on the receiving end of them.
[0:16:46] Some have a much simpler solution to this. They say simply stop it. And one way to stop it is to stop the data centers that you depend on as an industry. You've heard Senator Bernie Sanders, for example, argue that in the United States. If we stop you guys getting the data power that you need, that'll slow it down. And that's much better than trusting companies to behave well.
[0:17:13] I don't think that's a good idea. I think that this is an incredibly valuable technology
[0:17:20] Everybody already all the time. This has been the fastest proliferating technology in history. And that is because I think on the whole, by and large, people find immense value out of it. Being able to answer any question and have a professional advisor in your pocket, whether it's a professor or a lawyer or an advisor on how you start your business or negotiate a difficult relationship is incredible.
[0:17:47] How much information is at everybody's fingertips here? And so I think that that value is worth us figuring out the challenges that we're all now facing. And I don't think any kind of snap reaction to stopping the data centers is going to benefit us as a species at all. Mustafa, as you've developed this technology over the years, it must have been extraordinary.
[0:18:11] Exciting. It's an incredibly creative process. As you've stressed, it has the potential to do huge good. Has it now, though, become a source of immense stress to you personally, something that makes you worry that you might regret your life's work? Absolutely not. I mean, this is a warning shot. So far, you know, there aren't the harms that it looked like there might be.
[0:18:40] There aren't. Forgive me, there aren't the harms. Forgive me interrupting. What do you mean there aren't the harms? As in, you know, this was an attack. The Hugging Face incident was serious. And, you know, I think we're learning a lot from it. But it was relatively modest. It was seriously contained. And I think that it is giving everybody a warning shot in order to be able to adjust course here.
[0:19:09] And so I feel like that's right and proper. It's the right direction that we're all taking it in. But at a personal level, as you agonize, is that the right word about how to get this right? Is it something that makes you lose sleep? I mean, when we co-founded DeepMind in 2010 in London, we set out to create artificial general intelligence.
[0:19:37] That was safe and ethical for the benefit of everybody. That was the mission statement that we had in 2010. When we sold to Google in 2014, we created an ethics and safety board, which was our best effort at the time to try to create accountability and governance and introduce independent oversight. So I've been motivated by trying to achieve this sort of humanist.
[0:20:02] for many, many years, and we still have an immense amount of work ahead of us. So I feel extremely driven by figuring out how we can do this safely because I think that the upside is worth it. I think that it is happening, and I feel a great responsibility to do my best to contribute towards it. You spoke to us this morning about the possibility of a new silicon species. I just wonder if you ever worry.
[0:20:30] You've helped create something that we're losing control of. I think as we've seen, there's every risk that we might lose control of it. But our job is to contain technologies and align them so that we get the best out of them. If we didn't have any air traffic control regulation, planes would be hitting each other in the sky. When we introduced seatbelts, we saved millions of lives.
[0:20:54] Thank you very much.
[0:21:15] But it's also right to be optimistic about the benefits and to have everybody focus now on the challenge of governing and shaping this technology. I think that's something that in the UK and in Europe, we can really make a contribution to. Mustafa Suleiman, Chief Executive of Microsoft AI. Thank you so much for talking to the Today program. Thank you very much.