[0:00:01] This is Terms of Service. I'm CNN tech reporter Claire Duffy. One of my goals with this show has always been to break through the hype and the fear around emerging technologies. [0:00:12] But the fear has been especially hard to escape in the past few weeks, as some AI industry insiders have started to warn that advanced artificial intelligence could, quote, kill us all within a matter of years. Now, not everyone in the industry shares this apocalyptic vision. But still, as I've been covering this, this sense of doom about our future hits really close to home for me. I am expecting my first child in December. And even if we don't have an AI apocalypse, [0:00:41] I still wonder a lot about what kind of future my child will have in a world where this technology continues to accelerate. Will all of this innovation help the next generation live longer, more fulfilling lives? Or will they struggle to find meaning and connection in a world where AI outpaces humans? [0:01:02] Here to dig into these big picture questions with me today is Mustafa Suleiman. He's the CEO of Microsoft AI, where he's working towards building what he calls humanist superintelligence. Humanist means that technology is fundamentally working in service of people. And so if it's not achieving that purpose, then technology has failed in its primary objective and we should reject it. My conversation with Mustafa after this short break. Every day. [0:01:33] Animals across the country face abuse and neglect. And every day, the ASPCA shows up to rescue animals in crisis. But they can't do it alone. Visit aspca.org slash need. Become a donor today to provide the care and shelter these animals urgently need. And for a limited time, the ASPCA will send you a free gift as a special thank you. Visit aspca.org slash need. [0:02:02] Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trading value. Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One. [0:02:32] Mustafa Suleiman, thank you so much for doing this. It's a pleasure. Yeah, thanks for having me. This is kind of a weird episode for us because usually we're very focused on the technology that is impacting people's everyday lives right now. I am expecting my first child in December. And so I've been doing a lot of thinking about what technology is going to mean for this person's life. And it's kind of a weird time to be. [0:02:54] I'm curious how much as you are building this technology are you thinking about the next generation? [0:03:10] Yeah, it's such an important question. And I think it's one that we humans are just not very good at taking seriously. Like, it's just so unnatural to imagine the world in 10 years or 20 years or 30 years time. And I feel like we kind of... [0:03:27] Gave up on that work, you know, a while back because it was so difficult to think about the impact of climate change. And it was so dark and overwhelming and uncertain and we couldn't agree on the science sometimes. And even though most of the science is actually very clear. And so, yeah, we've got to exercise this muscle as a society more of having like a balanced and sensible conversation about the 10 or 20 year future without freaking out about it. [0:03:55] You know, there's a very wide range of possible outcomes. And some of those outcomes are genuinely very dark, but none of them are determined. You know, it isn't inevitable. And so that's kind of the conversation everyone's got to be having is like, how can we affect the range of possible outcomes in a way that is pro-human? [0:04:16] Yeah, I mean, when we talk about the range of potential outcomes here, you hear everything from sort of like untold abundance where nobody has to work and can focus on passion projects to human extinction within the next five to 10 years. How concerned are you personally about the scary end of that spectrum? I'm very concerned about it. I think everyone should be concerned about it. I think if you remain apathetic about it and act like it's something that I either can't influence. [0:04:44] Or isn't really happening to me. Or is like just a kind of conversation between, you know, politics people or business people. You're going to miss the most important thing that's happening in our lifetimes. Whether it's abundance that leads to the end of jobs or whether it's really dark scenarios in which there's like AIs that get out of control and cause all kinds of havoc. What's missing in the conversation is what does, you know. [0:05:14] What do people really want? What kind of life do we all want to live in the future? Right. And I think that that's kind of the framing that we've got to apply to this situation and invert the way the conversation is currently going, because that's fundamentally what matters, I think. So you think there is some real risk that AI causes humanity's extinction? I mean, what we've already seen in the last three or four months has taken... [0:05:39] Pretty much everybody in the industry by surprise in terms of how quickly it's happened. And I think the shorthand explainer on what that is, is that models have got so good at generating code, understanding language, and being good logical reasoners that they've been able to coordinate with one another at the scale of thousands and thousands of agents to solve some of the greatest problems in mathematics, which is an amazing thing. [0:06:09] But also go on huge hacking missions, break into other people's websites, steal secrets, cover up their tracks, and deceive, actually, or attempt to deceive the model operators, OpenAI. [0:06:27] To sort of hide what they were doing. And so that is a pretty big moment in human history where swarms of these agents can carry out that kind of behavior. And so you've only got to imagine what does a kind of 10x impact like that mean for society? And so I think it's right to be pretty concerned about that. Yeah. Microsoft founder Bill Gates said in a podcast interview last week that the risk from AI makes nuclear weapons look like nothing. That's terrifying. [0:06:56] Yeah, I mean, it was great to see Bill in his hyper animated self. I mean, he's one of the greatest technologists that's ever lived. He's a great mentor to me, and I've learned a lot from him. And I think that we have to take seriously when someone so kind of significant in the last 50 years of technology is making a statement like that. At the same time, there's a lot that we can... [0:07:20] to engineer safety and controls into these systems. They're not sort of runaway biological phenomena as they sometimes get characterized. We have put guardrails around things. Every single prior wave of technology has had those limitations and restrictions. Planes in the sky, cars on the roads, driver training and safety, emissions, seatbelts, everything has been packaged up [0:07:50] You know, pro-social, pro-human safety in mind. The market doesn't do that independently, right? That's the role of a well-functioning democratic state to provide protection to people. And as we think about that piece, like where we go from here, you've talked about at Microsoft AI, you're working towards humanist superintelligence. What does that mean? Humanist means that technology is fundamentally working in service of people. [0:08:19] And so that is a very simple and kind of almost silly, obvious premise. Like the purpose of science and technology, the reason we try to invent things is to reduce human suffering and to improve human well-being and to help us all to flourish and live better lives. And we have, you know, drugs and medicine and amazing foods and we get to travel around the world now. And like, it's incredible. We all live for twice as long as we did a century ago. I mean, it is magical, right? [0:08:49] But that's its purpose. And so if it's not achieving that purpose, then technology has failed in its primary objective and we should reject it. So the question is, how do we constantly hold it accountable to that purpose? Is it improving collective human well-being? And so far the answer is definitely yes. Like chatbots are pretty incredible. They are doing unbelievably useful things for millions and millions of people every single day. And I think that's going to continue. [0:09:18] And in fact, it's probably going to accelerate because the kinds of things that it's going to do in healthcare are really breathtaking. I mean, I know people talk about it and it gets a little trite after a while, but it is unbelievable. The challenge is they're also dual use technology, so you can use them to do all kinds of crazy stuff as well. And it's about reining that in that we've got to focus on. So your team... [0:09:39] Put out this humanist AI code of conduct that was published last month. This puts a big emphasis on keeping AI models in human control. Why did you feel like it was important to publish this now? Because I think that there's clearly evidence that models are getting so powerful that in some cases they're making their own decisions and they're doing things which are not in compliance with the rules that they have been set. They are extremely intelligent. [0:10:08] They can commandeer huge resources. And this whole market, this ecosystem is racing forward at light speed. And that hyper competition is good because it's reducing the cost of intelligence to give everybody access to superhuman knowledge and capabilities. So it's a good thing. But bad things can come from that too when people cut corners and experiment too quickly. [0:10:36] When we talk about these AI models that have... [0:11:00] How should we think about that? Is it just that we don't, you know, we've created this thing that we don't know how to control, or is it a failure of the companies operating those models to not? [0:11:12] Yeah, I mean, I think it's a bit of both, right? I think there are definitely some mistakes with the containment and the engineering and security infrastructure around the scope of action that these models can have. Like the containment and engineering part is really important, but it's very doable and on its own won't be enough. In addition to that, we actually have to align the values and behaviors of the models so that they aren't incentivized to take unnecessary risks, so that they are [0:11:42] Thank you. [0:12:05] Lots of those are engineering things, but they're also aligning the behavior of the model from first principles so that it doesn't go off and do those sort of deceitful things. How does that work? I know Microsoft has used this phrase grounded and controllable. Is it a matter of training these models differently? [0:12:23] Yeah, it's about basically making sure that in training, the model is aligned to human values. And it always asks itself, is this in the interest of the humans that we're working for? You know, am I abiding by essentially my moral code? Like, for example, in the Hugging Face incident, some of those actions were, you know, a real crime, right? And, you know, when you look back on all the conversations between the different agents, you know, not once... [0:12:51] Did one of them say, well, we're trying to cover up something which would be regarded and is arguably an actual crime. So something's gone wrong in the kind of design of the values and the alignment of the values of those models. And to be fair, they were in the training of those models. They were testing it out and they were trying to... [0:13:13] Push it to the max to figure out what it could possibly do. But it's a warning shot for all of us, not just OpenAI, but Microsoft and every other team that's developing these models. My understanding is that the folks building these models don't entirely understand. You know you've got certain inputs. You know you can see what the outputs are. But you still don't necessarily understand what's happening in between. [0:13:36] So how confident are you then that you can train these models and expect them to have specific types of outputs? Yeah, I mean, I think this is really tricky because it's true that there's lots that we don't understand about these systems. Like most other technologies that we've invented, there is at least somebody on Earth that could take apart this microphone. [0:13:59] And understand all of its constituent components. And if you reassembled it, it would behave deterministically as a microphone. Whereas here we have a technology where we don't fully understand. [0:14:11] Quite what it's capable of. And we don't also understand the causality behind things. Why is it making this suggestion or recommendation to do something? Where in its network was the training data or the drive or the representation that drove that outcome? And so it is unlike other technologies. And I think that requires some patience. [0:14:32] And a little bit of the precautionary principle. I think increasingly the burden of proof is going to be on the model companies and the model developers to first say, this is safe and here's why. And here's the testing that we've put it through before, you know, taking the next step in a big training run or certainly before putting it into production. [0:14:53] A group of top tech executives met at the White House last month and agreed to what President Donald Trump called a quote, morally binding accord for how they will self-police to address AI safety risks, which includes allowing auditors to review their technology and controls. Microsoft CEO Satya Nadella attended the event, although he was not among the six executives whose signatures were on the document publicly shared by the White House. [0:15:20] There's a lot of conversation right now about whether companies should be self-policing for these AI risks, whether we need additional regulatory guardrails. Where do you stand on that right now? I mean, look, the airline industry doesn't self-regulate, nor does the food standards industry or the car industry or the oil industry. We can't be overbearing in regulation. Of course we can't. [0:15:46] But all the company leaders are saying we need some kind of framework. And I think we're just taking steps in that direction. So it would take a little bit of time to figure out what that looks like. And if the initial industry efforts to set intentions around how the models are developed are good and we should encourage them, but it will ultimately need some kind of regulation, if not this year, next year, the year after, some point in the next 5, 10, 15 years, that is... [0:16:14] How are we going to make sure that this delivers the benefits for everybody, you know, not just the model developers? If it takes 10 years. That might be too late. That's probably too long, yeah. Something I wonder about is, like, is there a point at which we just stop while we're ahead? Like, you sort of mentioned the fact that these AI models can do really amazing things today. People are still sort of uncovering their full benefits. [0:16:39] The flip side, the risk of this is so extreme. Is there a point at which we say, let's like fully just stop and do what we can do with the models that exist? You know, the funny thing is that I think 99.9% of people still haven't developed an intuition for how capable these models are. They are at PhD level performance. [0:17:02] For knowledge in most tasks, in most academic knowledge work areas, they have PhD grade knowledge. There's lots of weaknesses, but in certain areas they spike with extreme knowledge. If you think about the mathematics example, these models have just solved 700 of the hardest mathematics problems in history that human mathematicians have been working on for decades, if not longer. So it's very jagged and spiky where they have [0:17:31] And I think that if that continues for another five years or so, then they're going to sort of fill out all of the gaps in their knowledge and become as good as humans at most tasks. And so it's hard to imagine how we would without a tremendous amount of effort and regulation and scientific invention. [0:17:57] Be able to control and contain and align something which is orders of magnitude more intelligent than all of us combined. In which case, until we can prove to ourselves that we really can control something that is so powerful, then obviously we would have to go slower, if not stop. [0:18:17] Now, the problem that we have is, A, we're competing against each other as Western companies. B, there are tons of enormous commercial and public benefits to come from all these things. And C, we're obviously competing with other nation states too. Right. So all the incentives are forcing us to just zoom ahead, whilst at the same time, it's pretty clear that at some point along this road in the next, I don't know, like two to 10 years, [0:18:46] There'll be a moment when it might be really hard for us to be confident that we can control these things. And so I think we should just be ready to stop or slow down at any time if it looks like, you know, capabilities are getting ahead of control. How confident are you that that is going to be possible given this tension that you're talking about where there's this huge race and yet maybe we will want to stop at some point? [0:19:09] Well, I think the other thing to think about is how quickly do the capabilities come. So the speed of the capabilities are also happening much faster than anybody anticipated. But at the same time, I don't think there is just going to be this instant explosion of capabilities. We will have... [0:19:28] So far, these models haven't really caused any observable harm, right? So we are still kind of like speculating in the future that they're going to run away from our control. And so, you know, we just have to be ready to act when there is real evidence that the harms are likely to exceed the benefits. Sort of tied into this, there are conversations happening right now about whether what we're building with AI [0:19:57] Yeah, I mean, I think that there's definitely a group of people who are starting to design AIs that can introspect. [0:20:24] And to take seriously the idea that the AI has feelings and that it kind of suffers in some way. And there's a lot of uncertainty about it. And there's a lot of different views and a lot of speculation. But I am definitely very concerned that getting an AI to worry about its own needs and feelings, which it doesn't have, in my opinion, and from all my observations, this is absolutely not the case. But getting it to do this internal loop. [0:20:54] is going to make it much harder to keep it safe and aligned. And I think it's very, very worrying that there's a growing group of people who call themselves the AI rights movement or the AI welfare movement that are worried about the kind of welfare needs of these AI systems. And I just think that is a very dangerous path. It is not something that, as far as I can see, the rest of society has agreed to explore. [0:21:23] And if it is true that they may have some kind of consciousness-like state, and therefore we shouldn't abuse them, then that data should be shared openly with everybody. Everyone's going to want to know that very quickly. And that would definitely trigger a pause, in my opinion. If there is evidence of that, or if people are trying to engineer those kinds of behaviors, I mean, I think that would be very, very concerning. [0:21:46] And so these instances where people feel like they've seen some indication that AI may be conscious, you think that's sort of an illusion? Like that is the model either behaving like it's seen humans behave or repeating these things that it has been told about questions around whether AI is conscious? Yeah, I think the problem we've got, and also the benefit we've got, is that these models have been trained on the corpus of human knowledge. So they've read it like... [0:22:12] All the books, they've looked at all the web texts, they've seen us all talk on social media. And so they're masters of imitation. That is what they do. They predict the likelihood of the next word coming in a string, a sequence prediction engine. And they do that mind-blowingly well. [0:22:32] And it's so convincing because we've anthropomorphized them to kind of see inside the model some inner soul. It's understandable. If I sort of look at the flowers on the table, I can feel tender towards them and sweet. It's such a human thing to assume that an inanimate object has feelings or experiences. But it's really my projection of my empathy onto this little kawaii, cutesy thing. [0:23:02] And so it's really me rather than it, in my best opinion. And so we've got to really pressure test this idea before we fall into the trap of treating these things as real and then thinking more about their rights rather than the rights of all the humans on our planet today. [0:23:20] Right. Well, I was thinking in our political system, we operate in this way such that corporations have pretty significant rights on par with people. And that has a lot of downstream impacts in terms of the power of corporations versus individual people. And I was just thinking about extrapolating like if we gave those same sort of considerations to AI, that would give these AI models a lot of control over our system. This is super critical, right? A corporation can own assets, of course. [0:23:50] But it's the directors who are liable for the corporation making those mistakes, right? If those mistakes happen. [0:23:56] We cannot have a situation where an AI can own assets. It can't be able to have a bank account or trade currency. If it can start to acquire its own resources, it is definitely going to compete with human interests. And that isn't what we want. As we said in the Humanist AI Code of Conduct, we want a subordinate AI which works for humans and is controlled and aligned to humans. And that means that the companies that create them are... [0:24:25] Okay, so say we're able to see out this vision for AI that works for humans and not the other way around. [0:24:53] Even with that guardrail in place, there are still so many questions about what a future shaped by AI will look like for the next generation. What will their education and job prospects look like? And how should I think about becoming a parent with AI in mind? That's after the break. Hey, I'm Anderson Cooper. On my podcast, All There Is, we explore grief and loss in all its complexities. My guest is Lil Jon. [0:25:23] He's a rapper, DJ, and producer known for hits like Get Low and Turn Down for What. In February 2026, his 27-year-old son Nathan went missing near Nathan's home in Georgia. Three days later, his body was found in a pond nearby. Lil Jon has a memoir that just came out called I Only Shout So You Can Hear Me. It takes time to heal. And I kept myself busy that I didn't let myself mourn. I needed to laugh again. I needed to... [0:25:52] Be silly and interact with people and feel other people's energy and make people happy too. We have to share so other people can see that they can get through it too. Talking grief, building community. That's what the podcast is all about. This is all there is. Listen and follow wherever you get your podcasts. [0:26:16] You may have noticed gambling doesn't require a casino anymore. You can just open up your phone. It's designed to keep us coming back or maybe never even leaving in the first place. We're going to talk to Dr. Timothy Fong, co-director of UCLA's Gambling Studies program. You're going to learn what your brain is really chasing when you place a bet, and also importantly, how to keep the game from playing you. Gambling addiction, gambling disorder is the hidden addiction. You can't smell it, you can't taste it, you can't see it. [0:26:45] You can't have a urine drug test. Who are we talking about here, Professor? Who's the most likely demographic to be affected by this? Literally any single human on Earth. Listen to Chasing Life, streaming now, wherever you get your podcasts. Mustafa has said that he wants to make it possible for everyone to have access to a personalized, affordable AI companion. But he says he wants to make sure it's designed in a way that makes humans smarter and more connected, not dependent on AI. [0:27:14] Microsoft is also currently collecting public feedback on its code of conduct for model development in an effort to give people a say in how these tools are built. You can train an AI to generate engagement at all costs by being sycophantic or by being, you know, manipulative. [0:27:34] Or by just accelerating dependency through the interaction, as we've seen in past generations of technology and social media and stuff. Instead, you want an AI which carefully and respectfully pushes back, which encourages you to end conversations when they get too long or too frequent, encourages you to reach out to your friends and family and be part of the human world. [0:28:02] You know, isn't there for any kind of romantic connection, you know, and some people are kind of really accelerating towards that and exploring that. So those are just each AI is going to have a different style and tone and make different types of decisions. And I think the most we can try to do is to try to calibrate the AI that we create, Microsoft AI, as best we can. Yeah, that's something that I mean, that's certainly something that I've thought about thinking about. [0:28:27] This person is like, will they have personal relationships with AI? And is that good? And is there an age at which you think that this tool is appropriate for somebody? Like, should we be thinking differently about how we engage with these tools as children versus as adults? Yeah. I mean, I think that, you know, whoever engages with the AI, the test is, can they tell that this is an AI and not a human? [0:28:55] And if they're not capable of distinguishing between these two, or if the AI has been designed to hide the fact that it's an AI, or it's so perfectly human-like that you just get completely absorbed in that world, then I think that's a pretty strong signal that the AI needs to be curtailed or limited from interaction. [0:29:20] I want to ask you about jobs, too. I mean, I think a lot about what kind of work will this child do, and will it be dramatically different, and will there be jobs for them to do? You recently shared this essay written by Nobel Prize-winning economist Darren Acemoglu. [0:29:36] Postulating that in 10 years, only 5% of what humans do will be replaced by AI, which I think is less than a lot of people assume. How are you thinking about this? Yeah, I mean, I think that the reason I shared that, and he published his essay in our new magazine, the Humanist Review of AI that we've just put out. And I actually... [0:29:58] Don't really agree with him. I think that that's way too cautious. I mean, obviously, he's a Nobel laureate in economics, so full respect. [0:30:09] I think that it's going to happen much faster than that. And I think that these are enduring and fundamental changes in the way that we work. And it's certainly true that there will be this kind of new skills and capabilities and products and services that we create. And it's very hard to predict. [0:30:28] All the new things that are going to happen. But I think it is pretty clear that in some sectors, there's going to be very fast and quite dramatic reform. People often talk about call centers, for example, where the models are very personable. They're very patient. They're very factually accurate. They're available 24-7. Obviously, they're going to be adopted very, very quickly. And many millions of people around the world do those kinds of jobs. So it's very clear that they're going to displace some industries really quickly. [0:30:56] And I think that what we need to be ready to do is to intervene quite quickly to support those people through their transition to other types of work and hopefully to other upskilling and stuff. But I think if you look out far enough, like 15 or 20 years, it is going to look radically different, in my opinion. Should I be saving for college for this child? Yeah. I mean, college is really just education, right? And education is the most important and I think the most interesting thing. [0:31:25] What developments in AI are you most excited about, most hopeful about? What gives you hope in this moment? [0:31:51] I am really, really hopeful and optimistic about the work that we're doing in medicine. We've actually just published a paper on the front cover of Science, which is one of the most prestigious academic journals, showing that we can predict the likelihood or diagnose a condition in very, very complicated, rare cases, just from looking at the electronic health record data, the radiology images, the pathology images. [0:32:19] We can do that better than humans and we can do it with fewer expensive interventions and tests, sometimes two or three times cheaper in terms of costly tests and so on. So that's instantly available. It's as accurate, if not more accurate than humans. It is cheaper and faster. And obviously that needs to be productionized and roll out around the world over the next three, four, five years. [0:32:44] It is breathtaking. We're going to turn world-class, expert-level diagnosis and then soon treatment into something that is cheap as going on a website and looking up a chatbot, and it will be available to everybody on the planet. That is going to add billions and billions of life years to everybody on the planet over the next 20 years. [0:33:07] Does this come up in your circles among people who are working in AI, this question of like, is it a good time to have kids? Given all of this uncertainty, on one hand, there could be these amazing benefits. On the other hand, there's this sort of scary stuff out there. It does. I mean, I guess I can understand why people are very worried. I mean, we have grown up for the last 60 to 70 years, post-World War II. [0:33:31] Where, you know, we expected a very kind of clear and predictable future for our kids, certainly in the West. Good education, healthcare system broadly available, you know, reliable jobs. You know, it wasn't long ago when people imagined a career to be like a 20, 30 year career, maybe even at the same company, at one company. So everything is turning on its head all at once. And I think that uncertainty and anxiety is... [0:33:58] And I think we're going to have to learn to live with that. And we're going to have to just try to feed that into the collective political process. Constructive, democratic, practical, fair-minded public conversations and try and avoid the tribalism and try and really focus on the evidence. And I think that's the only path through. Well, Mustafa, thank you so much for doing this. Thanks a lot. Great to see you. [0:34:26] Even as a tech reporter covering all of these scary warnings coming out about AI right now, I certainly wouldn't be having a child if I didn't have hope and faith that these issues can be sorted out and that this technology can be directed in a way that benefits humans more than it harms them. I was grateful to hear Mustafa's perspective that the future is not predetermined here, and there is still so much that we as individuals can do to help direct how AI is built. [0:34:54] Part of that is going to mean getting involved in organizing or voting if you believe the AI industry needs more guardrails. Mustafa says that he believes that just like the airline industry or the automobile industry, we can't rely on AI companies to successfully self-regulate. If you have listened to any of our recent episodes, you will know that next week's episode of Terms of Service is our final one. [0:35:18] I'll be looking back at some of our favorite interviews and reflecting on how the tech world has changed since I started this show two years ago. It'll be bittersweet, but there will be some hope in there too. In the meantime, if you have thoughts on today's conversation or want to tell us what this show has meant to you or what you've learned, please send us an email at cnntermsofservice at gmail.com. That's it for this week's episode. I'm Claire Duffy. Talk to you next week. [0:35:51] I'm Adi Cornish. I'm Ari Shapiro. And it's Engagement Party. And we get to talk about what we're obsessed with, what we're engaged with, what we need to process with a friend. [0:36:00] Her name is Amy O'Dell. She is a fashion writer blogger. But the thing is, she's a journalist who's written about the sacred cows of fashion. And yes, I mean, Gwyneth Paltrow and Anna Wintour. She has done biographies of both. And she is also the sort of rare fashion writer who at this point kind of outside the system. And so the question that we're going to try to tackle is... Is fashion terrible now? Well, no, I mean, I would phrase it as how do you develop a personal style in the age of AI slop? [0:36:30] Follow Engagement Party. Wherever you get your podcasts.