[0:00:00] You're launching this new Microsoft superintelligence lab, I guess, is that the right term? I was a little surprised actually when I saw the announcement because you've been taking this sort of, I think, different tack, I guess, with your humanist AI and all this stuff. So I'd love for you to sort of explain, let's define what superintelligence is. And it sounds like you're doing something different than the other labs that want to develop superintelligence.
[0:00:29] I'd love to get a sense of how that's differentiated. Yeah, so definitely right. So today, obviously, we're announcing the launch of the MAI superintelligence team. And our framing for superintelligence is a humanist superintelligence. And that means that there's a very clear test that everyone should use to judge.
[0:00:53] Whether we are living up to our principles. And that is, does this technology make us all healthier, happier as a species and keep us all in control? And control, containment and alignment are fundamental to the project of humanist superintelligence. It means that by design, we are not taking risks with creating a superintelligence.
[0:01:22] that has the capacity to say for example recursively self-improve or to set its own goals or to have autonomy independent of human control and that might seem like stating the obvious and that's good it should be obvious that we never want to have a system that isn't just smarter than any one of us but smarter than all of us in aggregate
[0:01:49] And has those three capabilities that I've just listed. So self-improvement, complete autonomy, independent goal setting. That would be a very dangerous system. And other people, I think, in the field who I think are more open, if not actively pursuing those things, are more open to those things, believe that it would be possible to align a system like that to human values.
[0:02:17] And therefore, I think, just hope that because it was aligned to us as a species, even though it was way more powerful than us, infinitely more powerful than us, it would protect us and not either deliberately or accidentally cause us mass harm. And I think that's a very, very important distinction. What does that entail? What would that require? Will we need to, in order to control these...
[0:02:46] Whatever this future technology is, will we need to fully understand it down to the neuron level? Is that really the big challenge here? I think it's one of the challenges, but I don't think that we'll be able to understand it down to the atomic or the neuronal or to the lowest possible levels. The judgment call is going to be about when do we have sufficient understanding of these systems.
[0:03:14] that we feel they can safely handle increasing autonomy. And I think that fundamentally we're behaviorists as humans. We watch what we say and what we do, and we use that to build trust over time. And then we grant each other more freedom to affect one another because we've seen safe repeated behaviors over time.
[0:03:44] But that doesn't mean that we should forego the project of trying to deeply understand where in the representation, where in the neural network certain representations sit, or how we could probe those for deception or wireheading, as people call it, or reward hacking. I mean, there's plenty of other sort of downsides of these models, which can only be exposed through proactive intervention and deeply trying to understand.
[0:04:14] Like you say, where certain representations sit and why they're called. If I think about Microsoft's business model and what they're trying to do with AI now, it's not really super intelligence that would sort of, I think, push the products to the next level. It's actually more what you're talking about. I mean, it's really more what you're talking about, understanding, being able to control them, right? For instance, if I'm having an AI check my email for me and sort my email.
[0:04:42] I don't need it to be super intelligent. I just need it to not, you know, sort of make unpredictable errors that a human wouldn't, you know, would never make, right? That seems to be like, but that, and that sounds, that sounds like a very simple problem, but it's not, right? And I just wonder, is that really in a way what we're talking about here? Or is that a separate idea? I think more think about it that, you know, we are a platform of platforms and we have an entire ecosystem of businesses, right?
[0:05:11] are one of the largest providers of AI APIs in the world. And so many, many people are going to use and be downstream to the models that we build and use them, you know, use the models for all sorts of different use cases. And this is an effort to start to think proactively about what those future use cases might be, what tests those use cases must be put to, and how we would frame the project.
[0:05:40] of safety containment and alignment because I think we're starting to talk a lot about it which is good and certainly some of us in the field have been talking about it for over a decade but now it's becoming real and so the rubber is hitting the road and as we start to develop in-house super intelligence with our new autonomy and independence we are thinking well what kind of super intelligence are we going to build and what are its boundaries and limitations and where are its
[0:06:10] Proveable controls. And I think this is a very healthy debate. And this is now the time for others in the community to start setting the standard, applying the guardrails. And, you know, and I think on the flip side, we are very much open to, you know, to kind of collaborating on that project because I think it really is time for us to take safety very seriously. I always think about more of X Paradox because...
[0:06:37] With this stuff, because it just seems to always pop up. It's like these models can do amazing things, right? And they're helping in drug discovery, including with a co-founder of DeepMind, which started AlphaFold. So it's doing all these incredible things, and yet it's still just the really simple things that it seems to have trouble with, right? That's why we're not there with autonomous cars yet.
[0:07:07] Cars are amazing, but they're not 100% perfect, right? So we can't just let them out on the roads. So do you think about that? Is that the next thing that needs to be solved? Or do you just kind of like solve that if you get to a certain level of intelligence? It just sort of happens. It's funny because timelines don't concern me as much as some other people in the field. And I think there's just this relentless obsession on whether we're going to have super intelligence in two years or five years or 20 years.
[0:07:37] I think we've been making steady enough progress for 15 years that I've got confidence that we're going to keep making progress and that moment will come. But I think it's soon enough that we should start working seriously on safety, that we have a responsibility to articulate our values and the kind of superintelligence we're at least going to try to build.
[0:08:04] But unclear enough that we can't really put a finger on what is going to be the single breakthrough that shifts things in the next few years. And you're right. At one level, they obviously feel incredibly stupid and make lots of mistakes. But I think people who get fixated on that sort of...
[0:08:25] Maybe get a bit blinkered and blind to the fact that, you know, self-driving cars are still pretty unbelievable. I mean, they do. We are actually using them. They're in production. And yes, it's been a 20 year project. We are making unbelievable progress. And sometimes people feel that like things aren't working because last year somebody made a claim that everything was going to be perfectly agentic this year or two years ago. Self-driving cars are going to be solved and they're still not here.
[0:08:51] And like, if you just take a step back and look at it on a decade by decade basis, it's pretty clear the progress is incredible. I totally agree with that. I think it's not so much a question of the timeline, but the direction. But I think what you're saying is like, we don't even really know the direction either. Like this stuff, these little discoveries happen along the way and they sort of compound. And you can't say, oh, this is the new area that we need to go push on, right? It's sort of like, it almost happens organically as a...
[0:09:20] Yeah, totally. And that's what the project of humanist superintelligence is. It's saying from the outset, we don't quite know where the breakthroughs will come from or how we will make it safe. I think it's very important to admit that and be clear about that. This is a very uncharted territory. But what we do know is that a humanist superintelligence is one that is by design.
[0:09:46] Always putting the primacy of the human species first, working in service of humanity is subordinate to, subservient to, completely contained and aligned by humanity. And I think that it's time to articulate that as an industry boundary. Because I do think there are some sort of in the more accelerationist, sort of transhumanist worldview who are starting to think about things like model welfare.
[0:10:15] Think about the rights and feelings of these models, which is an extremely dangerous anthropomorphism. And that mentality, I think, will, if it really takes hold, lead to a huge amount of conflict and threat to our species. And so at Microsoft, I'm embarking on the project of building our own superintelligence, which is explicitly designed to avoid those things, to not create a transhumanist...
[0:10:45] How do you, if you could sort of give people like something to sort of imagine, I mean, is it like, when you talk like this, I think of like Star Wars, you know, where it's like you have, you know, these droids that are super helpful and really smart, you know, but they're not human. They're just so clearly not human. There's no...
[0:11:14] I mean, maybe you feel a little bit of sort of, I don't know, affinity for C3PO and R2D2, but they're not human, right? Is that sort of how you think things should go, or do you have something else in mind? Look, part of the challenge here is that all of the metaphors that we use feel inappropriate, whether they're sci-fi metaphors or human metaphors. Like I've said in the past that this is going to be more like a new digital species.
[0:11:44] I've referred to it as seemingly conscious, having the appearance of conscious, but not actually being conscious. All those sort of, I don't know if they add clarity or they create more confusion. I think in general, we're all grappling for the right words to describe the arrival of this very, very different technology to anything we've ever seen before. And I think the more people are talking about humanism, the primacy of the human.
[0:12:11] That's a framework. It's a loose framework. And it's the beginnings of a broader idea that I think lots of other people should develop and adapt and so on. But that is a very clear statement, which is that these things, like the project of superintelligence should not be about replacing or threatening our species. Like that should just be taken for granted. And it's crazy to have to actually declare that. That should be self-evident.
[0:12:40] But I'm seeing lots of indications that people don't always agree. I mean, I think if you look at the criticisms of the AI safety people, there was recently this idea that AI models are grown and not built. And you heard Jack Clark sort of repeat that at Anthropic in a recent post. And it's really alchemy, right? And so, first of all, would you agree with that? But also, if that's the case,
[0:13:10] I mean, even if you have the best intentions, the definition of alchemy is you don't really know what you'll eventually end up with. You sometimes don't know what you'll end up with in software too, and there can be emergent properties of complex systems. But I think the design intent needs to be of a system that we build, that we control, that we can introspect, where we can minimize uncertainty.
[0:13:38] And the potential for emergent effects. It doesn't mean that we can eliminate them, but that has to be the design intent. Like to me, the design intent shouldn't be about unleashing some emergent thing that can grow or self-improve, I think is really what he's getting at. And I think that it's not to deny that those capabilities are necessary to the project of building AI, right? So obviously.
[0:14:05] Aspects of recursive self-improvement are going to be present in all the models that get designed by the cutting-edge labs, but they're more dangerous capabilities. They deserve more caution. They need more scrutiny. They need more involvement from outside players because they're huge decisions.
[0:14:21] Not today, not at this very moment, but we can all foresee a moment in a few years' time when there are gigawatt training runs with recursively self-improving models that can specify their own goals, that can draw on their own resources, that can write their own evals. You can start to see this stuff on the horizon. And that's why articulating the purpose of the superintelligence project is so important to do right now.
[0:14:48] The breakthroughs that you and others have made in this field are, you know, that you mentioned, this constant improvement. It also has been, it feels like each sort of new step and capability means less and less human involvement in the actual creation of the model, right? I mean, with neural networks, for instance, it's not like the original AI researchers wanted to, you know, to go in and sort of like,
[0:15:17] Design all the rules, right, that the model would follow. And now it's like, okay, we've created this architecture and let's just, you know, allow it to run. And so I think what, I mean, I wonder if what you're saying is, like, well, there has to be, at some point, that part of progress has to stop. Because if you just continue on the path of less and less human involvement in building the model, that is where you end up with the self-improving, you know.
[0:15:47] Do you have a team or teams or people working on other ideas? Maybe there has to be a more rules-based system or deterministic system of AI combined with some sort of LLM or something like that. Yes. You're right to describe that trend.
[0:16:17] Fundamentally, we need control mechanisms that are in the language and using the mechanisms of human communication, right? I mean, you know, some people say, oh, well, we'll have AI models that are specialized for safety that can monitor and restrict the behavior of AI models and that'll be sufficient. But that just regresses the problem to, well, who's watching the safety AIs?
[0:16:42] And so that is another core safety requirement is that we have to force AIs to communicate with other AIs in a symbolic language that we understand. If they just communicate in vector space, we're always going to be at the mercy of compressing their vector to vector space into a set of words that we feel we can hold accountable.
[0:17:07] And so this is a critical question. What is the AI to AI communication protocol which allows us to remain in the loop and on the loop, authorizing activity, but also monitoring semi-autonomous communication too? And that does mean, just to state the obvious, that we will be limiting ourselves from performance maximization.
[0:17:37] Because it's probably true that vector-to-vector communication is going to be more efficient, whereas humans are limited by language. But that would be a perfect example of humanist superintelligence. It would be saying we're going to forego efficiency or performance or some improved capability because we're going to prioritize safety. And safety means human understandable, human interpretable, rigid guidelines.
[0:18:06] And that's a very tough trade-off because in the history of humanity, we haven't had to do that. The story of our species has been unlocking, infinitely unlocking capability in science and technology and just inventing more and more and more and putting it out there without restriction and without guardrail, really, very little. Whereas now, in the next 50 years, we're going to have to think about what potential performance gains we might leave on the table
[0:18:36] And just to be clear to all my accelerationist friends listening, I am a technology accelerationist. I want to go as fast as possible. Yes, I'm aware of the China risk. Yes, I'm aware of the broader geopolitical risk of being slow or being behind. But we cannot just accelerate at all costs. That would just be a crazy suicide mission. We have to find ways of simultaneously having
[0:19:01] Humanist superintelligence, which delivers the benefits that we're all chasing for humans and also accelerating technology at the same time. But the first step is safety, containment, and alignment. And then the second step is within that frame, then accelerationism. I think that's a different position to some of the other people in the field. No, actually, that's the first time I've heard someone in the field sort of, well, on the industry side, explain it like that. And I think it's really helpful.
[0:19:30] If we have another question, I mean, I, you know, just, I'd love to hear some ideas out there. I mean, there, there are startups like, you know, I, I've interviewed the Symbolica company a long time ago and they, they're using category theory to try to, I think, do some of what you're saying and allow humans to, you know, I guess allow human language to affect these models in a way that we can control them. Like, are there ideas out there that you're seeing?
[0:20:00] that you think are promising to sort of accomplish what you just described? Yeah, I mean, I think that we already know a lot from security about how we contain and isolate powerful systems, how we verify their behavior by simulating and repeatedly having them do the same thing and looking at the variance in their performance, how we red team them.
[0:20:26] And deliberately try to trip them up, set honey traps and trip wires. So there's all kinds of metaphors that have been tried in the real world and in the cyber world. And we have layers and layers and layers of oversight and safety mechanisms in other industries from, let's say, airlines or cars. Think about the layers of driver education and street lights and exhaust emissions and stuff like that.
[0:20:56] I know we're out of time, but this is really interesting.
[0:21:25] Thank you. I think I have a better understanding of this now. Great. I appreciate it. Thank you very much. Yeah, let's chat anytime. Take care. Love to. Anytime.