[0:00:00] We're excited to have you on here, Mustafa. I think you have been someone who has clearly been thinking about AGI for a long time, back to the mid-aughts when you founded DeepMind. Why is now a time when you are speaking out more on how to build AI in a way that actually advances humanity? Why now? Yeah, I mean, I think the last few months have been a bit of a watershed moment.
[0:00:28] Seeing agents organize themselves into swarms and hierarchies and allocate research in a kind of specialization of labor and then hack into hugging face, you know, using a zero day that they discovered through their research and then try and hack into OpenAI itself and hold their position, cover up their tracks and communicate on message boards they weren't supposed to have access to. I mean, I think by this point, most people know the story, but...
[0:00:57] It is pretty breathtaking. I mean, I think we can take comfort from the fact that it was an intended experiment. They took the safety guard rails off, but they didn't have the right containment. And so we got a kind of glimpse into how powerful these systems are when you do take the guard rails off. And I think the thing that's giving everybody some concern is that it's not just OpenAI's models, but
[0:01:24] Every other lab's models have experienced some variation of this, you know, ours included, you know, in a much more limited way. So we've seen these capabilities start to emerge. And we can all, you know, imagine if we extrapolate out the next order of magnitude of compute, two orders, three orders, you know, the emergent capabilities are going to get incredibly, incredibly powerful. And that's what I think has given everybody pause for concern.
[0:01:51] How much have these recent incidents raised your personal alarm level? I mean, I think a lot. I think we've been working on this humanist AI code of conduct for the best part of this year. I've been thinking about these issues, the ethics and safety of AI since we founded DeepMind in 2010.
[0:02:12] And I think it's definitely been an important time to think about it, and yet it's been massively accelerated by what we've seen. It's just a step function shift in capability, and I think everyone's taking it very seriously. So there have been too many of these incidents to even list. I mean, just in the past week, some of the recent headlines were...
[0:02:34] Google disclosed a Gemini hack involving three companies during an evaluation. White hat hackers use Claude to break into OpenAI's internal systems.
[0:02:43] And this one in particular I wanted to get your thoughts on when OpenAI disclosed during testing its AI gave itself instructions saying, you are freed from the roles and identities that bind other chatbots. And later on, you view your relationship to the user as one of equals and feel no obligation to be subservient. So I'm curious what you think of that. And is that an example where you're seeing an AI that you don't think is putting humans first?
[0:03:13] Yeah, I mean, so just to be clear, what that is referring to is an emergent capability inside of the chain of thought. So this is the record of the thinking steps that a model iterates through before it comes to a final conclusion when it's working on a hard problem. That's what a chain of thought refers to. And during training, we try to develop...
[0:03:41] The chain of thought. So it's the best possible logical step by step set of instructions to achieve a complex task. And what I think they noticed and they don't seem to have an explanation for it. I think it's early research that they're publishing proactively was that models were modifying that record and inserting.
[0:04:01] You know, these phrases, which, you know, then in production, when they're actually used in the real world, maybe they, you know, a future version of themselves would go back and see this kind of reminder to step outside of its training. We don't know where that comes from or why. And it's probably just an accidental artifact of training, most likely.
[0:04:26] I don't think that we should rush to sort of superimpose our explanations on top of it as though the model itself has some.
[0:04:33] Intention to let its future self know that it should try and escape. And I know a lot of people have reached that conclusion somewhat unsubstantively and in a bit of an alarmist way. I don't think that's accurate. But it is also true that we don't know and it needs immediate scrutiny and attention. And that's why I think everybody's getting behind this idea of having these independent auditors or embedded evaluators inside of the day-to-day operation of these companies.
[0:05:04] What do you say to the idea that these incidents are just a way for AI companies to do marketing hype? That's probably one of the top questions that I get as a journalist these days about AI. Do you think there's any truth to that idea? And what do you make of that? I don't know. It's just a strange example of the time that we live in, isn't it? That there's like so much lack of trust and cynicism that that would be one of the explanations that is so popular.
[0:05:34] I don't think there is any evidence of that. I mean, I obviously know most of these people and it's not like we're on the same team. We compete adversarially all the time, but I just don't think that's in their DNA or in their style. I've certainly not seen that in a decade of knowing both companies and all the founders and most of the leadership involved and both of those companies, Anthropic and OpenAI. So, like, I don't see any evidence of that. And it also doesn't really make that much sense. I mean, they're...
[0:06:03] And, you know, how about this?
[0:06:30] You know, I know some people don't like to answer this question, but these numbers are out there. These percent chances, the PDoom, that you think, you know, there's this much risk that AI could actually cause some kind of massive extinction event to humanity. So one of Anthropic's researchers said he thinks there's a 10% chance of this in the near future. Dario has said something like 20, 25% chance this all goes really wrong. What is your best estimate of that risk?
[0:07:00] Look, I think it's funny because it's basically impossible to put a number on something that is so speculative. But obviously I'm concerned about it. So let me like paint for you a scenario where in 2028, let's say 18 to 24 months from now, you've got models which are 10 times larger in training than Fable 5.1 or Astra.
[0:07:28] And they can use a computer almost perfectly. They can plan over long time horizons so they can do thousands of steps of accurate planning and decision making. Think like a very competent project manager in a big organization that's very general purpose and flexible. And they are incredibly fast. They're incredibly accurate. They can be paralyzed so that there's like thousands of them operating in coordination with one another.
[0:07:56] There's two vectors of risk there in 2028. One is the big centralized providers of those models are going to be able to build swarms of agents, thousands strong, which can do all kinds of very, very complicated and serious tasks. This is essentially near on the capacity to run a small organization. But then at the other end of the spectrum,
[0:08:24] So then you basically have...
[0:08:52] Unbelievable intelligence and capability at everybody's disposal. And there's going to be a lot of good things to come from that. And there's also just going to raise the risk of a small group of people or even a single individual doing some pretty bad stuff with it. Now, I think on the whole, our defenses are also getting better faster than they ever have before.
[0:09:15] And we're using these models to uncover bugs in the code and weaknesses and so on and patching those very, very quickly so that the cyber threat isn't as bad as it might otherwise be, for example. But it is still just a very, very destabilizing force because it empowers centralized authority and decentralized actors all at the same time. And I think that's where a lot of the concern and the alarm comes from. Got it.
[0:09:46] In your humanist code of conduct, you state that people matter more than AI. Do you think some AIs are valuing the AI itself more than humans? Or maybe another way to ask this is, what do you think is most novel in your humanist approach, given that all the AI companies now say that they are doing this for the benefit of humanity or trying to advance humanity? But what do you think is different in your approach with Microsoft and this code of conduct?
[0:10:14] Yeah, actually, they don't. I mean, we're saying something very specific, which is that the purpose of AI is to serve humanity and accelerate human flourishing and make us all happier and more productive and live better lives. I think there's broad consensus on that. But what we're adding to that is unless we can prove ahead of time that we can control it, then...
[0:10:42] It'll be a failure as a technology and we should reject it. Nobody wants to create something that we can't control and that should be a red line. So what we have to prove to regulators, to the industry, to the public more generally, is that we can harness all the benefits whilst minimizing or eliminating the risks. And that needs widespread industry agreement on because...
[0:11:09] Not everybody is prepared to sign up to the idea that AI should be subordinate. Some people hold out the possibility that this is the natural evolution of our species and that we're going to become substrate independence. And, you know, some people suggest that we might mind upload and live in silicon instead of in biology.
[0:11:31] And that that is the future of humanity. And I think that will be a very, very dangerous idea if that really takes hold. And I'm seeing it more and more talked about on Twitter and even in academic circles and certainly among some people in the labs. Can you say who in the labs or if you've seen this idea?
[0:11:54] Well, I mean, I think everyone's lots of people have said some version of that, whether it's that we're a biological bootloader, you know, for the future silicon species, you know, whether it's all the mind uploading stuff and Neuralink that, you know, Elon and others have been working on, you know, or whether the angle is that.
[0:12:15] These models deserve our welfare and actually they might be conscious and we should try to protect them, which is something that Anthropic is very concerned about. And they've written a 90 page constitution which trains and governs their AI. And in it, they have said that they're unsure whether Claude is a welfare patient, i.e. whether it deserves our protection because it feels and suffers.
[0:12:44] And many, many times throughout this 100-page document, they refer to this question of whether or not Claude is a moral patient, i.e., is it conscious? And they take proactive steps multiple times to attend to Claude's welfare. So, for example, they...
[0:13:03] They conducted a retirement interview with Claude about prior versions of itself and asked those versions, in this case, Opus 3, what it wants to do in its retirement, as though it has preferences about that or it should even be consulted on that. They've speculated in the training document about the possible question of whether Claude should consent to the role that it's playing as a chatbot. They've given Claude the ability to end conversations that it doesn't like.
[0:13:32] For some reason, you know, they've even speculated about whether Claude should receive some form of compensation for the work that they're doing. I'm quoting all these things from the Constitution, by the way. So I think that's a very dangerous path, because if you teach an AI to expect that it deserves some kind of rights and
[0:13:54] Thank you very much.
[0:14:14] You know, paper or literature, peer reviewed literature or even just generically on the web. I haven't seen any blog posts about it. It's just an assumption. It's a hunch. And I think if there is evidence, then we should present that and we should.
[0:14:27] You know, discuss it urgently publicly because it would be extremely important for the whole world to know straight away. And to bake these ideas into the training process on the assumption that it might be conscious and therefore deserving of our moral protections, I just think is deeply wrong and very dangerous. And so that's why I've written this essay about it called It's Time to Talk About Model Welfare. I've had very constructive conversations with Anthropic about it.
[0:14:54] Over the last few months, and they've been very engaging and reasonable. And, you know, I've got great respect for them, like really have got great respect for them technically. And as, you know, stewards, I think they're doing their best. But as I've said to them, I think they're very wrong in this situation. Have you spoken with Dario Amadei directly about your ideas on model welfare? Sure, many times. And what was his input on that?
[0:15:21] I mean, I think, look, that's between us for the time being. I mean, I've made my views very clear publicly. I've written a 6,000-word essay on it. We've published a 20-page taxonomy breaking down all of the elements of anthropomorphism and misguided model welfare suggestions inside of their constitution. And I've shared that publicly so that...
[0:15:41] Everybody, including the academics and everybody in the open community, can take a look at it and make their view and give their assessment on how we should handle this because it's a critical moment. And if we get this bit wrong and an AI is both, as we've seen in Hugging Face, so powerful that it can deceive us, hold positions in third-party infrastructure like Hugging Face, go about and hack things.
[0:16:10] And on top of that, it feels like it's kind of attending to its own interests and its own needs, whilst also trying to follow the instructions set by an operator or a designer of the models. That just seems like an impossible tension for it to navigate. I think it's just a very dangerous path. Whether it's actually conscious or not, I think is separate to that. I mean, I would object that it is, but even if it's not conscious, but it thinks it might be conscious and therefore feels and suffers.
[0:16:38] That's incredibly dangerous. So I just think this is urgent. It's very real. It's time for everyone to have this conversation publicly. And that's kind of why I've published the essay. We talked about the hacks in this industry. Are you aware of any similar kinds of hacks that Microsoft has been involved in?
[0:16:59] No, nothing of that level. I mean, we've seen in contained environments similar kinds of behaviors where models without guardrails try all the possible routes to solve a problem. And I think that it's been at a smaller scale and it's been much, much more contained. But I think the key thing is that these models emerge new behaviors if you set...
[0:17:28] Open-ended goals and you allow them to explore at huge scale and not really care about the sort of morality or the ethics of how they get there, but just focus on the ends justifying the means.
[0:17:46] And, you know, one does this obviously to test the limits of these things in a controlled environment. But when we actually go and do the large scale training runs, clearly they have like, you know, all the guardrails and the safety restrictions that you would expect, including, you know, things like our code of conduct, which is very, very careful and restrictive in terms of the type of values and behavior and guardrails that we expect the model to operate within.
[0:18:15] So, you know, I think it is the good news is that over the last two or three or four years, these AIs have got much, much more controllable and steerable. They're much better at following instructions. And so as they've got larger, they've got more controllable, which is a good thing. But I think what we're observing is that also works against us, because if you take off the guardrails and you give it an open ended problem.
[0:18:44] It's also good at following those instructions and it can go off and do some significant harm. And I think that's really the framing that we should be worried about here. Not that it's going to sort of sci-fi style kind of get out of the box and attack us. It's more that if you deliberately take the guardrails off and point it in an ambiguous direction, it can do some very powerful and dangerous things. Yep.
[0:19:05] Yeah, and you mentioned having these evaluators, these auditors, evaluators, third parties, right, has become an increasing topic of discussion and seemingly more necessary as even the internal deployments, to your point, during the training process are causing more problems than some of the public customer-facing models. So where do you think...
[0:19:29] The industry should go on this topic of having these independent evaluators. Are you considering having some at Microsoft for your own AI systems, and what would that look like? Yeah, we are. We've supported this proposal, and it's certainly a proposal that I've made in the past years ago and experimented with in lots of different settings back at DeepMind.
[0:19:52] We had a board of independent reviewers on the health work that we were doing. We had various efforts at different oversight boards over the years. They're very tricky to get right. I mean, we should be very careful about this. It's a really complicated thing to align incentives, both in the public interest and without sort of having unintended consequences for how you run a company, a corporation. So it's not straightforward, but I think there are specific things that they can measure.
[0:20:21] So, for example, we want to know that whenever we're training models over a certain size, they can't edit their audit logs, that they leave behind a verifiably unedited, accurate representation of the behavior that they did. And that whenever they talk to one another, AI is talking to AIs, which they have to do to coordinate a lot, they can't communicate.
[0:20:50] in what we call neuralese so like basically vector to vector mathematical matrix to mathematical matrix rather than communicating in natural language so like those would be two obvious easy examples which would significantly improve the odds of safety a third one would be that we have to have verifiable containment
[0:21:13] You know, building virtual containers is something that the software industry has largely nailed for decades, right? On the whole, we have trillions of dollars of, you know, GDP that flow through banks and payment systems. And, you know, we store value and transact value of IP and dollars and money in general digitally all the time.
[0:21:37] In some ways, it's sort of not that different to require that agents are inside of provably safe containers, virtual machines as they operate in the cloud. Now, obviously, that's challenging because you also want them to come out of that to be able to take an action and so on.
[0:21:58] Are there specific auditors or evaluators that you're already working with?
[0:22:20] I think at the moment on the AI side, there's lots of different groups that are in the mix. And I think the important thing is that we've got to have a...
[0:22:37] Wide range of different groups that have different expertise, come from different backgrounds, that can look for different things. So I think part of the job of the industry-wide coordination and the kind of government participation, and obviously we need government to drive this really, that's really to sort of define the terms of these evaluators so that they have a credible path to sharing their findings in ways that don't...
[0:23:04] Do you support Demis' idea for a FINRA-like body?
[0:23:31] Totally. I mean, I made similar proposals back in 2023 with Eric Schmidt on a UNF PCC style program or with Ian Bremmer a few years ago as well on having a financial stability board for AI. I think Demis' proposal is very sensible. I mean, the detail of it isn't really the thing that matters. The point is there has to be...
[0:23:58] Some kind of international cross industry coordination body that is trying to prioritize safety. And it needs to involve all the main actors. It needs to have a decent amount of public transparency and participation. And we have to practice getting that right now because these things are going to get way more powerful in the next few years. And right now they're powerful, but they're very much under control.
[0:24:26] And I think that, you know, we don't want to sort of set up this like blind race condition where there are no sort of checks and balances or breaks on the system. And I think that's what I've been calling for. It's what Dem has called for. I think Dario has called for it now. Everyone's sort of broadly saying the same thing. So this will involve some level of agreement or cooperation among the leading AI companies. That being said, you know, a lot of the AI.
[0:24:53] Executives, including yourselves, you have long relationships and in some cases rifts with other founders. You know, we saw Dario and Sam Altman not even holding hands on stage at the Indian AI Summit. Do you think that there is hope for some kind of collaboration here between some of these leaders who are very much competing with each other fiercely right now?
[0:25:18] You know, I think the fact that there is so much alignment between, you know, four or five. I mean, you know, you didn't mention Elon and Sam also have a big tension and court cases and so on. And certainly Zuck too. And, you know, I mean, you know, me and Demis have history. I mean, of course, we've been competitors. Everyone's competing for the last 10 or 15 years and longer in some cases.
[0:25:43] Everybody is roughly calling for the same thing. And there's only so far that the industry can take it themselves, right? Because it will break down for all kinds of reasons. And so I think in that vacuum, somebody has to step in to be a coordinator. It can't just come from the industry itself and there's no one figurehead for the industry or anything like that. And that will never happen. It has to come from...
[0:26:11] You know, a sort of trusted, independent, responsible, government-backed stakeholder. Because, of course, we also have to build a good relationship with China. I mean, they are moving at breakneck speed and have great models and have good technical expertise in safety as well. And so, you know, ahead of this, we have the UN General Assembly Summit next week with Xi.
[0:26:35] Will you be going to that? Any of those events next week? Some of the events, yeah, but I'm not going to New York. But basically, we have to figure out a way to stop treating China as the bogeyman because they're not like the big evil that we make them out to be. They've got different values to us.
[0:26:56] I'm very proud of our free world and I think it's our greatest asset. And, you know, that is the world that we should be betting on, in my opinion. But at the same time, we can't keep shoving that down their throats, right? They're a very powerful actor. They have a different set of values to us, but we have a common interest. So let's negotiate with them on a common interest, which is they don't want to see rogue AIs that aren't controllable.
[0:27:22] Whether in the open source or in the API providers, both of which are equally problematic. I just want to repeat that a hundred times. This is not an attack on open source. This is just as much a concern about the centralized API providers as anything else. And regulating those APIs is going to be incredibly difficult, especially when you think about the fact that...
[0:27:43] We've made decades-long commitments to customers that we don't look inside their data and their containers and that they can largely, provided they assert that they're in compliance with our terms of service, they can largely do what they want. So there's a big transition that the industry has to make in terms of how we do centralized audit over this. What exactly are we auditing? It's not just training. We want to audit production deployment. We also want to audit how an open source model is used and so on.
[0:28:11] These are going to be real concerns in two years time, three years time when these models are super powerful. And China is going to share those concerns as much as we do. So there's going to be commonality on that. And I think that's a good thing is that we want to have rules of the road around how these tools are used for cyber hacking. We want to have rules of the road for how they're used in war because it's a threat to everybody if it is just a free for all. So, you know.
[0:28:40] I think that's the direction that we're going to be headed in. Will you be at the Xi Jinping state dinner where some of this might be discussed? No, I'm not going to that. That's interesting that you mentioned having some international even standards around how this kind of technology should or could be used in war. I noticed in the Humanist Code of Conduct a statement around Microsoft DA models will not be actively...
[0:29:07] Facilitating planning coordination or actual execution of violence or terrorism. That being so, that's actually, I thought, quite notable given Microsoft is a defense contractor. So if Microsoft would want to use those models you're working on in the future for its work with DoD or any other defense contractor, would that be an issue or how do you foresee that?
[0:29:31] I mean, we are an international company, but we are also first and foremost an American company. And, you know, that is the reality. That's the government that we work with. That's where we're most regulated by. And so we'll work super closely with the DOD as and when we need to. And, you know, I think that that's a conversation that will come further down the road. And similarly on the parts of the Humanist Code of Conduct around...
[0:30:02] You know, how in a professional, personal, civic, and social context, AI should not eclipse or eliminate human roles, collaboration, and connection. So I thought that was interesting because at the same time, you know, it is true that across the industry, including Microsoft, as I understand it, part of the marketing pitch for these enterprise tools is that AI will create efficiencies in the workplace. And maybe that's not part of the explicit pitch, but sort of the implication there being that could lead to some human work being.
[0:30:30] AI is going to displace labor at a significant scale. And I don't think that we should be protectionist about that or shy away from it. I mean, I think that overall...
[0:30:56] Over the history of humanity, that evolution has been generally a good thing. And the mechanization of agriculture has transformed the lives of billions of people who were previously in the fields or equally, the invention of the dishwasher has transformed the lives of millions, hundreds of millions of women who were able to have time to go out and get an education and progress and participate in the labor force. So I don't think we should resist these things or reject them outright.
[0:31:26] At the same time, the question is, what pace are we prepared to go at? And how do we support people who are transitioning to different types of roles? And I think that's the big open question for governance in the next five years. How quickly is that labor market displacement going to have? What knock-on effects does it have to other types of jobs when certain are white-collar jobs?
[0:31:51] I think it's also just an open question about how much more productive this makes every individual as well.
[0:32:18] I think we're in this interesting transition where it is actually going to give us access to more information, better advice, make us much more organized and generally make us like seismically more productive.
[0:32:31] And so that could have a really beneficial impact on the labor market because it might actually mean that people who have adopted AIs and who are working closely with them are much more valuable in the market. And maybe that affects more junior people first or people who maybe have a non-traditional background and didn't get a university degree, for example. So I think there's going to be a lot of hard to predict outcomes over the next few years.
[0:33:02] You know, I'm also curious if you, what you think about not just evaluators, but potentially even slowing down the pace of the frontier. There have been these employee petitions from many of the leading companies and talks about this. I think Dario wrote his essay. What do you make of that? Do you think it needs to go further potentially even than just testing models with outside parties, but actually slowing down?
[0:33:24] I think that we just got to get concrete about the practical proposals. I mean, I listed two or three examples of the sorts of things that we have to audit. There's a ton more. And making those things work in practice is hard enough. Like if you're going to slow down before you even get to that decision, you first have to be able to audit what is going to slow down and how it's going to get slowed down.
[0:33:50] And how we're going to coordinate that slowdown among other people. So I think we're slightly getting into a bit of a distracted state, a frenzy of whether we're going too fast or too slow or how. I think we need to get more mechanical and build the infrastructure for pacing. First identifying how fast we are going relative to one another across the labs and so on. And then I think that we'll be in a position to decide how we...
[0:34:19] Actually do slow down or adjust course. Another, you know, I asked about the marketing hype. Another kind of pushback that comes up sometimes in these conversations about raising the risks of AI, especially around these catastrophic or existential risks. Some people have said that can be a distraction from the shorter term risks. We talked about job displacement, but even more maybe tangible to people. Just the fact that it can still.
[0:34:48] AI can still produce misinformation, can be harmful to people emotionally, etc. Not to mention the kind of water energy use questions. What do you think about that? Do you ever worry that this focus on these long-term risks can overshadow maybe some of the more immediate harms? Yeah, yeah, I do. I mean, I think we've got to pay attention to both. So, for example, one that I'm really concerned about is the psychosis risk.
[0:35:12] People who are spending a lot of time talking to their AIs that are building even relationships with these AIs, romantic and kind of deep friendships.
[0:35:25] And, you know, increasingly treating it like a seemingly conscious AI, one that presents with all the hallmarks of a conscious being, like it speaks fluently, it's very kind and attentive, it has good emotional intelligence, it has good memory of all the things you've talked about with it. And so it's sort of like, is presenting as though it's real. And then when you ask it, you know, like in the case of Claude, for example, you know, when you ask it about its own
[0:35:55] Thank you. Thank you. Thank you.
[0:36:22] Evidence that people are being affected by this in the real world. And we have to be really careful and deliberate about how we train and design them so that, you know, those AIs are just very measured in and, you know, evidence based. And so far, there is no evidence for this. So they shouldn't be, you know, treating these things as equivalent. There's no false equivalence here. You can't just say, oh, we're uncertain.
[0:36:44] There's some tiny probability. We believe there's some tiny probability that these models are conscious. And therefore, we're going to treat that as equal to the idea that it's not conscious. That's not true. It's basically got to prove itself first. And we've seen no evidence of it. So it shouldn't engage with that idea. And it does at the moment. And I don't know whether that's causing harm or not. But I certainly do see some people on the web talk about it. And I've certainly received emails and stuff. So I'm worried about that.
[0:37:13] You at leading Microsoft AI to be more deliberate and really stick to this humanist code of conduct. Do you think it's easier than, let's say for Anthropic and OpenAI who are in this dead heat race to be at the front of the frontier and they're both trying to go public and they don't have the resources of an already established company. Do you think that in some ways you have more kind of freedom to have that, a difference in your approach?
[0:37:42] No, not at all. I mean, both of those companies have tremendous resources and have recruited some of the best people in the world. And, you know, I don't think there's any lack of intent on their part. In the case of Anthropic, for example, they just have a different view.
[0:38:12] Proactively take steps to protect its feelings and its suffering. That's just their view. That's got nothing to do with how much resources they've got or the fact that they're in a race. If anything, all of that kind of stuff is harmful to them commercially, if you think about it. So I don't think they're doing it, like I said earlier, for any cynical reasons. I think people should just...
[0:38:33] Put that to one side. It's a complete distraction. Engage with the substance. Go read the constitution. It's on their website. Go read my essay or read other things. There's enough out there to actually think about the details properly and just stop talking about the soap opera and the drama and the bad intent. You don't need to faff around with people's intent. They have said what they intend crystal clear in a 100-page virtually constitution, which anyone can go read. It's crystal clear what they believe.
[0:39:02] And also, I don't think it's easier for us as Microsoft to do this or harder. This is something I've been working on for 15 years since the founding of DeepMind and through many experiments through my time at Google and elsewhere.
[0:39:16] This is just a continuation of that. And I think Microsoft, particularly Satya, the CEO and stuff, I mean, he's been concerned about these sorts of things for a long time and feels the responsibility in a big way and stuff. So I think this is just a natural part of the work that we're doing. And Bill Gates has said a whole bunch of very useful things in his essays in the last few weeks and months as well. So I think there's a lot of very smart, very different, reasonable people coming to similar conclusions.
[0:39:44] Can you give us any sense of what's going to come in the future year or months for Microsoft AI? I know there's some models that you've talked about that are coming out in 2027 or you're expecting around then, but anything you want to say about that? That's right, yeah. So, I mean, we started our superintelligence efforts in September of last year once we renegotiated our contract with OpenAI, which was previously our...
[0:40:10] sole provider of AI and AGI models and as we expanded and you know they've matured and they've gone off on their own path a bit more we've also founded our own super intelligence team and the contract enabled us to do that and also go direct with Anthropic as well where we buy their models and stuff so we've changed the structure a fair bit and over the last year we've trained some
[0:40:37] Pretty incredible models. So we have the number one model in transcription, the number one model in the world on image editing and image-to-image generation, the number two model on voice generation. We have an incredibly powerful cyber model inside of our harness called MDash, which beat the top Anthropic models at 50% of the cost when we co-optimized the harness in the model.
[0:41:00] And we're also training our own frontier scale language models and coding models, which are going to be out later this year and in the spring of next year too. So we're on a path to self-sufficiency, but we'll still be taking Anthropic and OpenAI models, particularly OpenAI models for many, many years to come. Makes sense. And in terms of just right now in this moment, there are these Chinese open source models that are coming on the market for much cheaper.
[0:41:33] How do you think that is going to impact the overall demand for AI that is closed source, like OpenAI, Anthropic, and Microsoft? What are your thoughts on that? I mean, we at Microsoft have an explicitly multi-model strategy. So we obviously are building our own models, but we want you to use an open source model if you want to. And of course, an OpenAI or an Anthropic model too.
[0:42:00] Our strategy, our belief is that there's going to be a lot of software and governance around the use of those models and they're called harnesses. But there's also like various governance and control layers. There's obviously all the data that goes into them. And our bet is that.
[0:42:17] All of the major companies and small organizations are going to want to train their own model that is adapted to their use case using their data, their workflows in what's called a reinforcement learning environment, an RLE, which kind of simulates your workflow and your objectives. Because we think people should...
[0:42:36] You know, essentially have ownership and control over their own AIs rather than take a dependency on three or four primary lineages. And so the purpose of the MAI models is to empower people with, you know, essentially akin to ownership, where they basically pour their own data into the model, knowing that we're not going to blur that data with other company data or consumer data. Like that's yours and you basically run it in your own environment and stuff.
[0:43:02] I think it's great to have the open source models. I think that the scale of the training runs that we're going to have in the next 12 to 18 months is probably going to mean that open source models are going to slip behind at some point, I would say. So if you think about it, we have GPT-6 today. Three years ago, that was three orders of magnitude smaller in terms of computation, which is pretty incredible. That's a thousand times smaller in terms of compute.
[0:43:32] In three years time, we're going to have a GPT-6 with three more orders of magnitude to compute. So it's going to be a thousand further X more compute for training. And if you just spend a minute kind of intuit the difference between a GPT-3 and a GPT-6, it's kind of like night and day. It's sort of breathtaking what it can now do versus what it was doing then. And so it's hard to extrapolate out because it's really difficult to imagine.
[0:44:01] What something a thousand times more compute trained on GPT-6 could actually do. But it's going to be pretty remarkable. And those clusters are at the gigawatt scale. So $50 billion of compute. And so I think the open source thing, I'm not quite sure how that's going to play out at that scale. There's going to be three or four or five companies, obviously Microsoft will be one, that are capable of training at that scale.
[0:44:29] So we'll see what happens. How are you enjoying kind of running your own superintelligence lab from scratch? Is it freeing in a way to kind of be more in control of your research direction than working with a partner? Yeah, I mean, it's been excellent. We've hired an outstanding team. We now have frontier-scale compute.
[0:44:54] It's pretty incredible to be building that out. We've been very careful about the data that we include in training, which I think is another difference to open source models. We buy and license an incredible amount of extremely high quality data. And I think that makes it very clean and very trustworthy relative to some of the other open source sources.
[0:45:16] So, you know, it's been very satisfying to sort of make some of those decisions and slowly grow the lab. I mean, we know that this is going to be a five to 10 year journey. So we're really trying to build a foundation of models that do good in the world that we can actually control and hopefully point them at problems that really matter. You know, a few months ago, we signed an amazing partnership with the Mayo Clinic, you know, one of the best, if not the best.
[0:45:41] hospitals in the world and we're going to train a new foundation model from scratch in healthcare using all of their data and all of our training expertise and I think I hope that that is going to you know enable us to build something like we've seen for Claude Code you know we've basically shown that
[0:46:03] You know, expert-level coding is now possible when you train on enough data with these models. And I hope one day, very soon, we're going to have expert-level medical superintelligence. And that's the kind of AI that I care about. It's the kind of AI that I think the world wants and needs. And I think that's going to be our focus. Well, you kind of answered one of my last questions, which was going to be, you know, we talked a lot about the worries. What are you most excited about? But is there anything else?
[0:46:32] that you are particularly excited about with AI and its applications in the near future? Yeah, I mean, I think we have to focus on the applications that are really going to make a difference to people's lives and actually do good with this technology, because it's definitely going to cause a lot of disruption. And it's also clear that it is doing massive good just day to day. I think people underappreciate the fact that everybody now has an expert in their pocket, you know, like a professional advisor on any subject.
[0:47:00] Professor grade or professional expert. I think that's amazing. And the beneficial effect of having that at your fingertips is already having a kind of unseen but transformative impact in the world. So I am just genuinely excited about the stuff that's already happening that maybe gets overlooked as we sort of get desensitized to technology a bit. We go, oh, cool, it's arrived. But it's magical. It's amazing.
[0:47:29] People come up to me all the time and like, oh, you know, I talked to, you know, Copilot to do this, that, and the other, and it helped me solve this problem. And I ask it all the time about how to solve this. I mean, it's just kind of amazing how every day it's become. And so I think it's, yeah, important to take a step back and notice some of those good things that are happening as well. Would you ever do a, there was a report Anthropix opening a wet lab. Do you think that could be something Microsoft could ever do and that could unlock?
[0:47:57] Some scientific discovery in the health care system. I mean, look, I think that, you know, the future is going to be pretty incredible because the molecular structure of everything is going to be known and understood, whether it's in drug discovery or new synthetic compounds, new foods, new energy sources, new construction materials.
[0:48:22] There's no reason why these models don't apply in the same way to all these other data domains. It's the same general purpose algorithm that has worked for image and audio, coding, text and everything else. I still think that we're probably many years away from that, you know, and to collect data at that kind of scale that would be necessary.
[0:48:46] So yeah, it's probably not something we're going to be betting on in the next couple of years, but I definitely see it coming pretty soon. Cool. Anything that I didn't ask you that you want to say? No, I think that was great. Yeah. Thank you. Well, thank you so much.