[0:00:09] What we're going to talk about today is international collaboration here as AI ever closer to AGI and so on. And before we get into that, just so we're all on the same page, let's define some of the terms we're going to be talking about. So, Demis, when I first met you, we were talking in my office, I think 13 years ago, you were the only CEO of a company actually trying to build AGI.
[0:00:39] Now everyone and their uncle is trying to do it, right? So I think there's no one more appropriate than you to just define for us in a nerdy way, what's the difference between narrow AI, AGI, and superintelligence? Thanks, Max. It's great to be here, and I miss those times. I'll say that first of all, those quieter times when we could just quietly get on with research. It's not like that anymore, for better or for worse.
[0:01:06] So the way we define AGI, and actually it was a term coined by one of my co-founders, Shane Legg, our chief scientist, artificial general intelligence. We sort of define it as a system that exhibits all the cognitive capabilities that humans have. And the reason that's important is because that's the only way we'll know whether the system's truly general. If you want to do a more technical definition of that, then Alan Turing obviously famously conjectured and proposed his Turing machines and proved that...
[0:01:35] They could compute anything that was computable. So what we're after is a system that is basically a Turing machine. And as far as we understand it, the brain is a kind of Turing machine. So if we can mimic and map out the capabilities that the brain has in an artificial system, then we probably have a truly general system. Good. And how would you distinguish between AGI and superintelligence? Well, AGI, my kind of test for AGI would be
[0:02:02] Could it come up with something like not just prove a conjecture or prove a theory or prove a formula, but could it come up with a theory or a conjecture itself? So, for example, could it have come up with general relativity like Einstein did with the same knowledge that Einstein had at the time? Or, for example, we have systems like AlphaGo that can play amazingly creative moves, like famously Move 37 in Game 2 of the Lisa Dole match, which had never been seen before in Go.
[0:02:31] Could it have invented Go, a game as elegant or as beautiful as Go? And so far, none of our systems can do that level of invention. So that's still missing from AGI. And then superintelligence would be superior to human intelligence across the board and be able to do things we can't even imagine. So from a sort of nerdy physics perspective, of course, the core idea that's in a way powered the whole AI revolution is the idea that intelligence is all about information processing.
[0:02:57] And it really doesn't matter whether the information is processed by carbon atoms in neurons and brains or by silicon atoms in machines. And if AGI is the attempt to basically be able to do human jobs, superintelligence is when you get vastly beyond that, which is clearly possible in that paradigm. So given the definitions that Demis gave here, let's just do a lightning round. Who here thinks that it's possible...
[0:03:24] Is it physically possible to actually at some point build AGI? Raise your hand. Because that's everybody. Who thinks it's possible at some point to build superintelligence? Okay, so it can be done. Now, so many people are trying to build it. By what year do you think the probability will have reached 50% that someone is actually able to build it? Demis, what would you say? I have a timeline, you know, around five years for 50% probability. And Don?
[0:03:55] I think it's not just about my own prediction. Also, there has been actually a huge survey run by over 2,000 AI researchers on the timeline. I think also it's right. I think it's soon in the next 5 to 10 years. Joshua? So this is actually a point on which I agree with Yann LeCun. That is 5 to 10 years. I think it's going to take about 20 years. And if we...
[0:04:25] It's probably going to take five years to reach that for the information AI, but it takes longer for the physical AI. I just want to add something. So you asked for the 50%, but I think it's important to ask, you know, what about like 25% or 10%? And if you look at these probabilities, which are not negligible, it could be two years or three years from now quite easily.
[0:04:54] So it's good to be humble and be prepared for if it happens sooner and also be prepared for if it takes longer. I think one thing that's been super striking is how these predictions have changed a lot. Like when I first met you, Demis, and even in the Puerto Rico 2015 conference, most people were saying 2050, 2045. And not long ago, six years ago, most of our colleagues were saying even something like ChatGPT was very far away.
[0:05:21] Well, actually, we've been amazingly consistent with our predictions. So actually, Shane's made these publicly. But when we started DeepMind in 2010, and as you say, we were the only people doing this at that time, we thought of it as a 20-year mission. And I think we're basically on track, which is a little bit unbelievable, really. And I think Shane has a specific prediction of 2028, which is very plausible, in my opinion. Good. So I guess what we can take away from that is we shouldn't take for granted that this is some kind of long-term issue.
[0:05:49] It might happen during the current US administration. So that leads us to the main topic we're going to discuss here now that we have some definitions in place. International collaboration. Given that this is Davos, having a panel on international collaboration would seem incredibly boring until pretty recently because everybody here kind of agrees that international collaboration is good. But times have changed. The vibes have changed quite a bit.
[0:06:19] I'll just read you two funny examples here. In 2017, when we had the Asilomar conference, and the Asilomar principles were developed that you guys signed here, there was one that said that superintelligence should only be developed for the benefit of all humanity, rather than one state or organization. Clearly...
[0:06:48] Implying that international collaboration is good. I contrast that with a full-page ad that came out yesterday in the Washington Post, taken out by Alex, the CEO of ScaleAI, that said, America must win the AI war. Did anyone detect any slight shift in vibe? We have quite an interesting intellectual controversy here now. Dario Amode from Anthropic wrote a piece.
[0:07:21] Also this fall called Machines of Love and Grace, arguing that America should race, that America should do a Manhattan project and try to get to AGI before China. There's been a number of other voices sort of piling on that. And then a lot of people have pushed back. So I hope this convinces you that this is actually an interesting discussion to have. Now, let me actually do a quick poll with the audience here. This idea that superintelligence should only be built for the benefit of all humanity.
[0:07:52] Who likes that idea? Raise your hand. I guess we are in Davos after all, aren't we? Okay. What about you guys? Who feels that you still think that's a good idea, that it should be for the benefit of all humanity? Okay. But I'm not going to let you off. I'm not going to make this boring and just have them agree on everything, okay? So we're going to dig into this and not just say we should collaborate, but talk a lot about why should we collaborate, what should the objectives be,
[0:08:23] How should we do it in academia, in business, in government? So we'll have a lot of interesting stuff to chew on here. And I think let's start with the why. Obviously, there are two fundamental reasons we would do it. One, to accelerate how quickly we can get the upside of AI. We'll talk a bunch about that. And second, to...
[0:08:53] Help avoid downsides. So if we begin with the downsides briefly, all four of you actually signed this statement in May of 2023 saying that superintelligence could, if we're not careful with it, that we have to take seriously the threat that it could cause human extinction and take that as seriously as we take the threat of nuclear war.
[0:09:22] or Pandemics. So I would love if each of you can just very briefly explain to the audience why you signed that. What are the particular downsides that each of you want to avoid? Do you want to go first, Joshua? Sure. I actually started thinking about this late compared to a lot of people who've been thinking about AI safety much earlier, just after ChatGPT came up. And I started thinking about, well, we're going to build machines like us.
[0:09:52] And maybe we're pretty close. And I hadn't really thought about it before because I thought this would be very far in the future. But there's another reason why I didn't take it so seriously. It's because as an AI researcher, you want to feel good about your work. And you want to look at the positive side of things. And the problem is if we build machines that are like us, they will have, like us, self-reservation goals.
[0:10:21] In other words, they'll be competing with us. I mean, hopefully we can do something different that doesn't have that competition property. But right now we don't know how to do it, and we didn't know how to do it two years ago. We still don't know how to do it. So at that moment I was thinking about my children, and I have a grandchild, and I thought, wait, my grandchild is going to be 23, 20 years from now. Is he going to have a life, right?
[0:10:50] So this is the reason why we have to do something. We have to make sure that we build machines that are not like us, that maybe are smart, but don't want to take over, don't want to have their own survival be more important than ours. And right now, we don't have the answer. And it's something that matters for the whole of humanity, because if...
[0:11:19] If we introduce a new type of entity that is competing with us and more powerful than us, then we're cooked. It doesn't matter what country you come from, what kind of political system you prefer. This is a thing that should unite all of us. Thank you. Dawn, why did you sign that statement? Yes, I think as Yasha mentioned, and also as we have seen, AI has been...
[0:11:48] Making this amazing advancement, but at the same time really we don't have good understanding of how this new technology works at all. And it has capabilities that actually the developers didn't even design it in. So this is really I think one of the, it's really the first time where we are facing this type of technology that is super powerful. We don't really understand how it works.
[0:12:12] And they can have these emerging capabilities. And when we talk about, for example, comparing to nuclear weapons and so on, so this technology, actually, if it's putting aside the loss of control, if it landed in attackers' hands, the potential of misuse, the risk can be...
[0:12:37] Even greater, one area that I actually do a lot of researching is to look at, for example, the potential impact of frontier AI in the landscape of cybersecurity. So in this case, AI can help both attackers and defenders. And the question is, who AI can help more, and so on. So we won't have much time to go into that. But in the near term, AI can have severe consequences. And again, I think the potential
[0:13:06] Thank you. Yaqin, why did you sign this statement talking about extinction? There are two things that really shocked me. The first one is AlphaGo. I'm an avid Go player.
[0:13:35] And I always thought it was going to take 20 years, 50 years to get to that stage. But in April 2016, when Arthur Gooke came out, that was the first shock. The second shock, of course, is the tragedy. You shocked him. He did. Many times. Second, of course, is the tragedy of BT. In late November, December 2019,
[0:14:03] 2020, right? That's right, two years ago. And I was playing with it. Then I suddenly realized, wow, this thing actually is the first software that passed the Turing test, at least for Dialog. And so obviously, the whole capability of AI, it just become such a reality. It's just immense power. The whole thing got accelerated. And also, I've been working on Tom's driving for years.
[0:14:32] So I thought, wow, this is going to pass through a test, become so capable. And then when you use this for cars, for drones, for robotics, and they use it for biological AI, that's a term I use, then if you don't have the right control, you will be out of control. So that's why I think the AI security, AI safety, the controllability becomes such an important issue.
[0:15:01] That's why I signed it. Thank you, thank you. And I'd love to hear the same from you. When you signed the demo, it obviously really made waves. Because it's quite unusual that someone who leads a large company warns about the very technology that they're building. I mean, it speaks to your moral integrity that you did it anyway. But do you want to tell us why you did? So there's many reasons. And actually, differently from Joshua, we...
[0:15:28] We have sat with myself and Shane and others at the start of DeepMind. We always were considering the possibility of the risks along with the opportunities. Because I guess maybe we thought it was nearer term than you did. So it was something we thought we would have to deal with. So we've had a long time to kind of sit with this and understand the issues they're going to be. It doesn't make them any less scary in some ways. But we sort of have to get comfortable with that duality. And the reason I signed it was...
[0:15:57] Because I wanted to give air cover and basically make sure it was in the Overton window for people who were proposing, many people on the stage, that we should do more research on this. There's a lot of uncertainty here. I'm a cautious optimist. I think we will solve these technical challenges, some of which are very difficult, like Joshua was saying about how do we build systems that are not going to compete with us. And there are many different ways, I think, of building AGI.
[0:16:25] And in the sort of brain design space, some of which will be safe and very positive for humanity and some which will be very negative and very dangerous. And we don't necessarily know which is which at the moment. And we have to gain more information and more research on that. And as Dawn was saying, also maybe favor defensive systems over offensive systems, things like this. And what I worry about is that, first of all, with this sort of competitive race dynamic between companies and also countries now, that's leaving less...
[0:16:54] Margin for error and less time and less resources focusing on the safety aspect and the understanding aspect of what we're actually building and why we're building that and how we want to deploy them and what kinds of systems should we allow and deploy. And I think there's a lot of work to be done there. And there are colleagues of ours that we know also who say there's nothing to worry about here, which to me seems insane when you have...
[0:17:20] Clearly a super powerful new technology that's going to be massively transformative and is obviously dual purpose. And we don't fully understand. So I'm optimistic we'll get this right given enough time using the scientific method and enough of our smartest people working on it. And that's where international collaboration and other things comes in because there's sort of several issues at stake here. One is the technical design. Can you build it technically safe?
[0:17:46] But even if one were able to do that, if other groups or other countries or other companies don't do that, then it doesn't matter, as Joshua was saying. It doesn't matter in the limit. If even only one or two of these projects design harmful AGIs, then it could be seriously existential for humanity. So that's where the international collaboration sort of needs to come in as another aspect on top of the technical solutions.
[0:18:11] Before we go to Joshua here, I just want to personally thank you for giving air cover like that and opening the open window, which I think greatly increases the odds that things will go well. Thank you. Joshua. I want to react to one thing you said. I agree with a lot of things you said. There's one thing I think we could do, I want to say, which is you said we don't know how to make the difference between...
[0:18:41] AI that's going to be potentially dangerous that we lose control of and AI that is safe. Actually, I think there's one bit of information about this which is very clear, which is agency. So AI systems that don't have agency, we can't lose control of them because they don't have any goal and that means they don't have any self-preservation. By the way, imitating humans is not good enough because humans are agents.
[0:19:12] And all of the scenarios of loss of control that scientists have been thinking about for decades now happen because the AI is an agent. Now, I know we are like racing towards building agents, but I want to raise a red flag that this is the most dangerous path now that we're entering into. So I agree with that, by the way, just to answer that question. And I think the agentic era we're about to enter into is a threshold moment for the systems becoming
[0:19:41] Far more risky. So today's ones, you know, I think aren't very dangerous. You know, they're impressive in many ways, but they're basically passive Q&A systems. So they're not, you know, they're not agentic systems. The problem we've got is a similar one of international collaboration. The market forces and competitive forces mean, it's clear that agentic systems will be more valuable, they'll be more useful in the good use cases. So people are going to build them. It's the same as also, I would advocate having humans in the loop.
[0:20:09] Most critical AI systems, but the problem is they may get outcompeted by systems that don't have human in the loop. So again, that comes back to then international agreements and collaboration. I agree about the international bit, but on usefulness, agency is mostly going to help replace people's jobs. But the things that people want really from AI is things like curing cancer, helping out solve climate change, and you don't need agency for that.
[0:20:38] You can do all of those things by just building systems that understand the world. The whole field of machine learning for science has been developing without needing agency. And I think we can get to AGI, a form of AGI that doesn't have agency. That's a bet. I agree. But I think it's a worthwhile bet. Right. So I just want to add to that. This is part of the AI safety work is to define the guidelines, guardrails.
[0:21:07] And the red lines, for example, is a no replication agency or the things you have to really not to do in terms of building the agentic AI. Yes, and also to add to that, I think, yes, as Demis was saying, for a lot of companies and practical applications and so on, so they want the agentic behaviors, the agentic flow to help them to actually solve problems and increase productivity to address these issues. But at the same time, these...
[0:21:36] Agentic systems also introduces a lot of safety and also security problems. So that's why I think it's also really important besides the international cooperation as we design and develop and deploy these agentic systems, we need to build safety and security in. So for example, last semester I taught a MOOC, a massive open online course on large language model agents. We have over 15,000 enrolled globally and also we had the hackathon.
[0:22:05] Actually, in collaboration with Google and OpenAI and so on, on agents. And so we actually set up. So first of all, in the class, we emphasized safety for agents. And also in the hackathon, we have one track on safety track for agents. I think it's really important as we talk with companies who are building solutions, agentic solutions, it's very important to really design safety and security.
[0:22:34] Maybe just to add one comment to both of those answers. I totally agree that that's exactly what we should be doing with agentic systems is building sort of cybersecurity protections in from the beginning. I also advocate things like hardened sandboxes to experiment with them in. There's many mitigations, you know, put it into simulation first. There's air gaps. There's many ways we could all probably come up with and have come up with many brainstorming ideas for it. The issue is, is that unless everyone agrees to build them in that way,
[0:23:03] Then people build them in unsafe ways. And if you also include things like open source and other systems, then all sorts of people and all sorts of organizations will be building these systems. And I also dispute the fact that, unfortunately, I do think there's an economic gradient beyond science and workers that people want for their systems to be agentic. Yes. Even in just daily assistance. I agree with that. Being digital assistant. You know, when you say, recommend me a restaurant, why would you not want the next step, which is book the table?
[0:23:30] That's a clear, obvious next step. I know there's a demand for that. It's obvious. And if you think of the military case, which is important for the international situation, it's even more the case, right? But it turns out that if we are able to build trustworthy, non-agentic systems, they can be used as guardrails on agentic systems. And so I think that if we make progress on this, we could help also build safer agentic systems.
[0:23:59] But I think we should talk about the question that you're raising. This is a topic. Yeah, absolutely. Come back to this. I just want to ask you, Demis, one AI application which may very well help you cure certain cancers soon is, of course, AlphaFold and the work you won the Nobel Prize for. Would you describe that as a fundamentally non-agentic system? Yeah.
[0:24:21] So that was sort of my original plan actually back 10, 15 years ago was to, and the whole reason I work on AI my whole life is applying it to science. And I think that's the number one thing I can imagine applying AI to is to improve human health, cure terrible diseases.
[0:24:38] Maybe number two would be to deal with climate and energy and those things. So these are the ordering of things that one wants to tackle. And actually, I would have preferred if it had gone the direction. It just hasn't gone that way. The technology where a lot of general AI algorithms were developed, but then applied in narrow ways, narrow ways to solve very important, what I call like root node problems, like protein folding that unlock whole new avenues of research, but don't upend.
[0:25:03] You know, the whole job market or anything like that. And then they wouldn't need to be agentic. AlphaFold is not an agentic system. It's a static system that sort of pattern matches. And that's not dangerous outside of what it does. And it would have been great to have had a decade or more of those kinds of systems coming out and curing cancer and those things first, whilst giving us time to understand these general algorithms better. But it just hasn't turned out that way.
[0:25:29] So you've given us all a beautiful existence proof that doing non-agentic systems is not just a sort of boring thing that's useless. It might cure cancer and all sorts of wonderful things. We'll come back soon and talk about what kind of international collaboration, especially at the government level, might be able to steer the development a bit towards safer architectures. But I want to mix up a little bit the two reasons for doing international collaboration.
[0:25:57] That would be about international collaboration to avoid downsides. First, raise our spirits a bit here by talking a little bit about how we can get the upside faster from international collaboration. So, Yaqin, do you want to tell us about some actual technical progress which has come from international collaboration, in particular come from China? Well, if you look at AI, it's actually a perfect result of international collaborations.
[0:26:26] And Chinese researchers, industry companies have benefited tremendously from the foundational work from US, Canada, from UK, including the groundbreaking work you two have done, Yashuo and Demis. And by the way, Chinese culture respects scientists a lot more than movie stars. So you two are...
[0:26:53] I also want to make a point. Chinese scientists are making increasing contributions to the basic research and to innovations. Let me give you just a few examples from actually the three companies I worked in the last three years.
[0:27:22] Organizations that have worked in the last 25 years. Microsoft, Baidu, and also Tsinghua. For those who work on deep learning, you probably know an algorithm called ResNet. ResNet is actually developed by four young researchers in Microsoft Research Asia, which I co-founded and had for many years. ResNet is being used for AlphaZero Go,
[0:27:52] Used for AlphaFold, the one. And also used for a lot of the big models we do today. And by the way, ResNet is the single most highly cited paper in deep learning. Altogether you have the highest, but that single paper is the highest cited. 300,000 citations, I think, yeah. And the second example is actually when I was working in Baidu.
[0:28:23] We developed an open source, a task-driving platform called Apollo. And so we actually released this in 2017 to the entire world. They put it in the GitHub, and so a lot of people have been using it. Over 200 companies are using it, and over 100 companies are from outside of China. And until today, it is the world's largest open commercial platform.
[0:28:52] And, you know, obviously we also run the world's largest self-driving car fleet in Wuhan, which is actually bigger than Waymo in San Francisco. That is open and it's a contribution to the world. And more recently, our young scientist, Qing Huang, in my institute, is using the result from AlphaFold, all the pretend structural prediction, and use that and also to
[0:29:22] She was able to develop a rapid and accurate drug screen tool that's open to everybody, called a drug clip. Essentially, she was able to drug the entire genomic by this. I think this is a lot of contributions. I think I see a figure from Stanford in the top four conferences, including the one you started.
[0:29:52] AI conference, 30% of paper coming from mainland China. And 25% citation coming from mainland China, including, of course, Tsinghua, which has been, and my group has been an active contributor to the top conferences. So I think there's a lot of, if China is not part of this, there'll be a huge loss to China, to US, and the entire AI community. Thank you for those three wonderful examples of how
[0:30:21] International collaboration is really enabling us to get to the upside of AI faster. And we shouldn't forget to mention that just last week, as of just a few days ago, the best open source model on the planet right now is not Lama. It's a Chinese model from DeepSeek, which my colleagues at MIT are pouring all over to see how they can learn from it and so on.
[0:30:53] So lots of upside from international collaboration, of course. Coming back to how we can use it to prevent downside, one can look at collaboration between universities academically, which is very uncomplicated, it doesn't involve politics. One can look at companies from different countries comparing safety practices, best practices, etc. And one can look at governments talking with each other, having AI safety institutes, sharing best practices, etc.
[0:31:24] Joshua, since you have poured a ridiculous amount of your own time during the past year into leading this international report, which will be presented at the Paris Summit in a couple of weeks, do you want to tell us just very, very briefly a little bit where things stand on that side of international collaboration? And also, if you had a magic wand and you could create whatever international collaboration structures you wanted, what would you do?
[0:31:54] All right, so on the international scientific report on the safety of advanced AI. Sorry, it's a mouthful. It's been modeled after the IPCC for climate change. It's bringing about 95 experts from around the world, all the countries, not all the countries, but representative set of countries, China, of course, and many countries. And the goal has been to create a...
[0:32:22] Scientific synthesis for consumption by policymakers of the state of the science in AI safety. So in other words, what are the risks? What do we know about them? What are the mitigations? And not just the extreme risks, like all the risks that arise from the frontier models, the largest models that includes
[0:32:47] Including things like jobs and discrimination and things that we understand better but also the things that we need to study more. And I want to say that it's really important to distinguish this scientific work of synthesis which acknowledges the differences that exist between different scientists about where we see things going. So this scientific view of the lay of the land from
[0:33:16] The political decisions. In any IPCC, things are a bit mixed. And I think some of the architects of IPCC would like to have a clearer separation. Because both are hard. But when they get a bit confused, then it's terrible. And of course, the hard job now is for the policymakers of multiple countries to deal with that. So my magic wand would basically say,
[0:33:46] The endpoint that is going to be stable and safe and beneficial for humanity in terms of political organization is one in which we have as few ASI, AGI projects as possible, but that their control, their governance is distributed. In other words, the power over these systems is not in the hands, just like in the principle you cited.
[0:34:15] How do we get there? That's a difficult question. Going back to the things that Demis said, it's really important to acknowledge that right now we are on this trajectory which is more adversarial and people are thinking mostly about how the other people, other countries are going to be using AI potentially against us and we have to win this race.
[0:34:44] You know, we have to start by acknowledging that this is a real risk. But we also need to bring into the picture the larger risk of, you know, we all lose. So it sounds like, is it fair to say that you're saying that ultimately the largest national security threat to America, to China, and to everyone else isn't actually ultimately another country.
[0:35:11] Out of control superintelligence. In a way, I've been talking actually to national security people a lot in the last year. And the way they like to think about it, because they like to think about a nation state that could be dangerous to us and so on. And the way they like to think about it is, well, Rogia is like another state that shows up on the map. And we don't know where it is. And it could threaten us. I think it's a good way to think about it. Great. Dawn?
[0:35:37] Yes, Yashua made a great point. And I think there's also another reason why international cooperation is important. So recently we launched with a number of leading AI researchers, we launched a proposal on the path for science and evidence-based AI policy. And Yashua is a supporter of this as well. So in there, essentially we talk about that. So currently there's a lot of...
[0:36:04] Unfortunately, there's a lot of fragmentation in the AI community for approaches to AI policy. But however, it's really important that we actually unite the AI community to reduce fragmentation and have a united front to really develop the best AI policy that's grounded in science and technology and also to prioritize the advancements of scientific understanding of AI risks.
[0:36:32] And how we can successfully identify and mitigate them. And within this proposal, we listed five priorities. And the number five priority is actually to build community consensus, have international cooperation, and essentially bring the community together, have essentially open dialogue, to have people from different perspectives to really come together.
[0:36:57] To work together and unite the community for this science. That's great. And Demis, I'm going to say Simsalabim, which is a Swedish spell that turns your microphone into a magic wand. What would you do with a wand? Well, look, I think in an ideal case, and I think there's sort of several institutions that will be needed. I would advocate for a kind of CERN for AGI.
[0:37:23] And by that, I mean a kind of international research-focused high-end collaboration on the frontiers of AGI development in the last few steps to try and make that as safe as possible. You would also have to pair it with a kind of an institute like an IAEA, like the Atomic Institute, to monitor unsafe projects, right, and sort of deal with those. And then finally, you know, you need some kind of supervening body that is...
[0:37:53] This involves many of the countries or all the countries around the world that input into how you want to use these systems and deploy these systems and what the value should be in these systems and so on. So a kind of like UN umbrella, but something that is fit for purpose for that, a technical UN, let's call it. So I think if you had that kind of trifecta, that would be pretty...
[0:38:14] Maximize the chances, I think, of this going well. I think actually the IPCC step that you're taking, I'm very supportive of, I think is a good first step. I think the AI safety institutes that we were very supportive of in the UK and the US is a great setup. There could be the beginnings of a kind of CERN and the IAEA sort of situation. It could go either way. And I think these international summits.
[0:38:43] I was also very encouraging of the first one in the UK, then in South Korea, and the next one in Paris in a few weeks. I think most of us are probably going to be there. I'm definitely going to be there. I think that's critical to advance the international dialogue at the highest political level so that the politicians and governments understand what's at stake and buy into this. Because I think there has to be government involvement. And absolutely, it shouldn't be one country or one corporation or certainly not a person.
[0:39:11] It's going to affect all of humanity. So, Yaxin, your microphone is also a magic wand now that you can wave. What would you like to see happen? Well, I don't have much to add to what you have said. I agree with that. But I do want to make a comment on the model of international collaboration. First, we should put the best minds to solving the common issues Demis just mentioned.
[0:39:39] The climate issue, there's renewable energy, there's healthcare, new medicine, epidemics. That should not be boundaries. All the brilliant minds in China, US, Europe should work together on that. And also I think we should have three separate tracks. One, of course, is government to government. The second track is business to business. And the third track is academia to academia. And obviously right now,
[0:40:09] Track one, China-US is not working very well, but let's make sure there is a constructive dialogue using Professor Schwab's work. But I think the business actually can work together. Business, they will compete, they will collaborate, but let's make sure they do this in a market force, not in terms of ideology or national boundary.
[0:40:39] When I worked for Baidu, our biggest competitors actually was from the Chinese companies. We competed. I think it is okay to compete. And then, of course, academia. Academia, we should keep the maximum freedom. And universities, students should work together. And I mentioned our institute. We have students publish a lot of papers in the top conference.
[0:41:08] They publish papers. They cannot get visa to go to this conference if the conference is in the US or Canada. So let's just break those barriers, make sure there is a continued to be a freedom in terms of academic exchange. Terrific. Anyone have any more? Clearly a popular sentiment. Did anyone else want to do anything more with their magic wands before we move on here?
[0:41:39] Okay, then. No, I would, actually. Good. So I think that so long as the governments leading this planet see AI as a tool that can be used to build weapons or economic dominance, things are not going to get better in terms of the risks we've been talking about. It's only when they understand the risk that they share together.
[0:42:08] And that is the existential risk. So we need greater awareness. We need more people to understand this. It is not that complicated. We need these arguments to be laid for everyone to understand and for governments to understand. I think, even though that sounds like a dark comment, that we need to increase awareness, that this could be game over for humanity if we do it wrong, I think there's a deep optimism in there, actually. You actually alluded to it yourself as well.
[0:42:44] If two countries have a hard time getting along, if there's a common shared threat, that often unites them. The United States and the Soviet Union did not exactly agree on everything ideological during the time of Stalin during World War II, but they still collaborated very effectively to defeat Hitler, which was a threat to both of them.
[0:43:07] As some of you were alluding to, if there's a scientific consensus that we're closer to figuring out how to make AGI than we are to figuring out how to control it, then it's pretty obvious the government in China doesn't want to lose control. No other government wants to lose control either. And what they just purely egoistically have an incentive to do is actually the same in both places, which is just make sure that their companies build very powerful AI tools.
[0:43:36] Maybe non-agentic, maybe other kinds of tools that they can control. But make sure nobody builds uncontrollable AGI in a rush in their countries. And then if both the US and Chinese government do this, of course, they'll start worrying like, well, what if North Korea does it? Or what if some American company does it and some South American country with a weak government? And now China and the US have an incentive to actually work together and make sure the rest of the world sort of follows.
[0:44:06] I think even though it sounds kind of dark, what you said, it can create exactly the kind of incentives that make even very egoistic countries that just care about themselves actually be responsible about this and do the governance that they need to get the upside, not the downside. Does that make sense? Yes, except that you also have to think of how
[0:44:37] We make sure the other guys are not going to use their AIs against us. Every country may think this way. I think it's naive to think that we only worry about the existential risk. We also need to worry about how do we make sure it's not used against in a military way or something. And that is a place where we need, by the way, new technology for treaty compliance verification so that we can trust each other, just like we've done for nuclear.
[0:45:07] And also add to what Yaqin and Yaoshuang mentioned earlier, I think in this international cooperation, first entry point, I think, and also the easiest entry point, is through scientist collaborations. And actually, it's great to see as a good...
[0:45:28] Fun fact, so I think all the four of us, except Demis is at Google, but all four of us as professors, we actually have been part of this effort on international dialogue on AI safety, where we actually together and together with other leading AI researchers, we have been traveling to different governments in the UK, in China, and in Europe, and Italy, and these other places to actually talk to governments to actually raise awareness of AI safety and try to write.
[0:45:57] Build up the international collaboration. I think the scientist-scientist bridge is a very important first step. Thank you. I think we're out of time unless someone has a 10-second final comment. We're going to finish by evaluating our panel. Let's see if they've convinced you that international collaboration is actually a good idea. If you think that, raise your hand. I think you've succeeded. Thank you so much, dear panelists.