{"url": "https://www.youtube.com/watch?v=h0962biiZa4", "page_url": "https://www.youtube.com/watch?v=h0962biiZa4", "title": "Superintelligence: Science or Fiction? | Elon Musk & Other Great Minds", "show_or_channel": "Future of Life Institute", "date": "2017-01-08", "duration_s": 3615, "type": "panel", "speakers": ["Demis Hassabis", "Elon Musk", "Stuart Russell", "Ray Kurzweil", "Sam Harris", "Nick Bostrom", "David Chalmers", "Bart Selman", "Jaan Tallinn", "Max Tegmark"], "confidence": "high", "notes": "Beneficial AI 2017 (Asilomar) superintelligence panel. Alt audio-balanced: OFBwz4R6Fi0. (YouTube upload 2017-01-30)", "people": ["demis-hassabis"], "id": "2017-01-08_future-of-life-institute-superintelligence-science-or-fiction", "audio_duration_s": 3614.6155, "worker": "jc-w12", "transcribe_s": 84.4, "model": "large-v3", "segments": [{"start": 9.07, "end": 45.04, "text": "I'm going to ask a question where you can only answer by saying either yes, no, or it's complicated. All right? So let's start over here. Is some form of super intelligence possible, Jan? Yes, no, or it's complicated. Yes. Yes. Yes. Yes. Yes. Yes. Definitely. No."}, {"start": 45.17, "end": 81.01, "text": "This was disappointing. We didn't find any disagreements. Let's try harder. Just because it's possible doesn't mean it's actually going to happen. So before I asked if superintelligence is possible at all according to the laws of physics, now I'm asking, will it actually happen? Yes, no, or it's complicated? A little bit complicated, but yes. Yes, and if it doesn't, something terrible has happened to prevent it? Yes. Probably. Yes. Yes."}, {"start": 81.01, "end": 123.26, "text": "Yes. Yes. No. Chuck still haven't found any interesting disagreements. We need to try harder still. Okay, so you think it is going to happen, but would you actually like it to happen at some point? Yes, no, or it's complicated? Complicated, leaning towards yes. It's complicated. Yes. Yes. It's really complicated. Yes."}, {"start": 123.26, "end": 152.32, "text": "It's complicated. Very complicated. Depends on which kind. All right. So it's getting a little bit more interesting. When, I think, we had a really fascinating discussion already in this morning's panel about when we might get to human-level AI. So that would sort of put it lower bound. In the interest of time, I think we don't need to rehash the question of when."}, {"start": 152.32, "end": 174.26, "text": "Going beyond that might start, but let's ask a very related question to the one that just came up here, namely the question of, well, if something starts to happen, if you get some sort of recursive self-improvement or some other process whereby intelligence in machines starts to take off very, very rapidly, there's always a time scale associated with this."}, {"start": 174.35, "end": 205.3, "text": "There I hope we can finally find some real serious disagreements to argue about here. Some people have been envisioning this scenario where it goes FOOM and things happen in days or hours or less, whereas others envision that it will happen but it might take thousands of years or decades. So if you think of some sort of doubling time, some sort of rough time scale on which things get dramatically better, what time scale would you guess at, Jan?"}, {"start": 206.26, "end": 237.04, "text": "No, no, so once we get to human level AI, or whatever point beyond there, or a little bit before there, where those of you, where things actually start taking off, what is the sort of time scale? Any explosion as a nerdy physicist has some sort of time scale, right, on which it happens. Are we talking about seconds, or years, or millennia? I'm thinking of years, but it's important to act as if this timeline was shorter."}, {"start": 239.06, "end": 266.94, "text": "Yeah, I actually don't really trust my intuitions here. I have intuitions that we're thinking of years, but I also think human-level AI is a mirage. It's going to be suddenly better than human, but whether that's going to be a full intelligence explosion quickly, I don't know. I think it partly depends on the architecture that ends up delivering human-level AI."}, {"start": 266.94, "end": 293.55, "text": "It's the kind of neuroscience-inspired AI that we seem to be building at the moment that needs to be trained and have experience and other things to gain knowledge. It may be on the order of a few years, so possibly even a decade. I think some numbers of years, but it could also be much less, but I would be surprised if it was much more. Potentially days or shorter, especially if it's AI researchers designing AI researchers."}, {"start": 294.77, "end": 322.75, "text": "Every time there's an advance in AI, we dismiss it. Oh, well, that's not really AI. Chess, Go, self-driving cars. And AI, as you know, is the field of things we haven't done yet. That'll continue when we actually reach AGI. There'll be lots of controversy. By the time the controversy settles down, we'll realize that it's been around for a few years. Yeah, so I think we'll..."}, {"start": 322.75, "end": 354.1, "text": "Go beyond human capabilities in many different areas, but not in all at the same time. So it will be an uneven process, but some areas will be far advanced very soon already to some extent, and others might take much longer. What Bart said. I think if it reaches the threshold where it's as smart as the smartest, most inventive human, then it really could be..."}, {"start": 354.54, "end": 384.14, "text": "I mean, a matter of days before it's smarter than the sum of humanity. So here we saw quite an interesting range of answers. And this, I find, is a very interesting question because for reasons that people here have published a lot of interesting papers about, the time scale makes a huge difference. If it's something that's happening on the time scale of the Industrial Revolution, for example, that's a lot longer than the time scale on which society can adapt and take measures to steer the development."}, {"start": 384.14, "end": 414.4, "text": "You're borrowing your nice rocket metaphor, Jan. Whereas if things happen much quicker than society can respond, it's much harder to steer, and you kind of have to hope that you've built in a good steering in advance. So in, for example, nuclear reactors, we nerdy physicists like to stick graphite sticks in as moderators to slow things down and maybe even prevent it from going critical. I'm curious if any one of you feels that it would be nice"}, {"start": 414.4, "end": 443.82, "text": "If this growth of intelligence, which you're generally excited about with some caveats, if any of you would like to have it happen a bit slower so it becomes easier for society to keep shaping it the way we want it. And if so, and here's the tough question, is there anything we could do now or later on when it gets closer that might make this intelligence explosion or rapid growth of intelligence simply proceed slower so we can have more?"}, {"start": 443.82, "end": 473.49, "text": "Have more influence over it. Does anyone want to take a swing at this? It's not for the whole panel, but anyone who... I'm reminded of the conversation we had with Rich Sutton in Puerto Rico. A lot of disagreements, but definitely we could agree about slow being better than faster. Any thoughts about how one could make it a little bit slower? Well, I mean, the strategy I suggested in my talk was somewhat tongue-in-cheek, but it was also serious."}, {"start": 473.49, "end": 501.57, "text": "I think this conference is great, and as technologists we should do everything we can to keep the technology safe and beneficial. Certainly as we do each specific application, like self-driving cars, there's a whole host of ethical issues to address. But I don't think we can solve the problem just technologically. Imagine that we've done our job perfectly, and we've created..."}, {"start": 501.57, "end": 529.7, "text": "You know, the most safe, beneficial AI possible, but we've let the political system become totalitarian and evil, either a world, evil world government or even just a portion of the globe. That is that way. It's not going to work out well. And so part of the struggle is in the area of politics and social policy to have the world reflect the values we want to achieve because"}, {"start": 529.7, "end": 558.61, "text": "We're talking about human AI. Human AI is, by definition, at human levels and therefore is human. And so the issue of how do we make humans ethical is the same issue as how we make AIs that are human-level ethical. Yeah. So what I'm hearing you say is that before we reach the point of getting close to human-level AI, a very good way to prepare for that is for us humans in our human society to try to get our act together."}, {"start": 558.61, "end": 588.24, "text": "As much as possible and have the world run with more reason than it perhaps is today. Is that fair? That's exactly what I'm saying. Nick? I just want to clarify again that when I asked about what you would do to slow things down, I'm not talking at all about slowing down AI research now. We're simply talking about if we get to the point that we're getting very near human level AI and think we might get some very fast development, how could one slow that part down?"}, {"start": 588.24, "end": 618.5, "text": "The method would be to make faster progress now. So you get to that point sooner when hardware is less developed. You get less hardware overhang. However, the current speed of AI progress is a fairly hard variable to change very much because there are big forces pushing on it. So perhaps the higher elasticity option is what I suggested in the talk to ensure that whoever gets there first has enough of a lead that they're able to slow down for a few months, let us say, to go slow during the transition."}, {"start": 618.5, "end": 645.42, "text": "One thing you can do, I mean, this is almost in the verification area, is to make systems that provably will not recruit additional hardware or redesign their hardware so that their resources remain fixed. And I'm quite happy to sit there for several years thinking hard about what the next step would be to take at that point. But it's trivial to copy software. Software is self-replicating and always has been. And I don't see how you can possibly stop that."}, {"start": 645.81, "end": 671.86, "text": "I mean, I think it would be great if it went slow, but it's very hard to see how it does go slow given the huge first mover advantages in getting to super intelligence. So the only scenario I see where it might go slow is where there's only one potential first mover that can then stop and think. So maybe that speaks to creating a society where AI is restricted and unified, but without multiple movers."}, {"start": 671.86, "end": 699.78, "text": "Yeah, Dennis, so your colleague Shane Legg mentioned that one thing that could help a lot here is if there's things like this industry partnership and a sense of trust and openness between the leaders so that if there is a point where everyone wants to... Yeah, I do worry about that scenario where I think I've got quite high belief in human ingenuity to solve these problems given enough time, the control problems and other issues. They're very difficult, but I think we can solve them."}, {"start": 699.78, "end": 721.36, "text": "The problem is the coordination problem of making sure there is enough time to slow down at the end and let Stuart think about this for five years. He may do that, but what about all the other teams that are reading the papers and are not going to do that while you're thinking? This is what I worry about quite a lot because it seems like that coordination problem is quite difficult. But I think it's the first step."}, {"start": 721.36, "end": 739.57, "text": "Maybe coordinate things like partnership and on AI. The most capable teams together at least agree on a set of protocols or safety procedures or things, agree that maybe we should verify these systems and that is going to take three years and we should think about that. I think that would be a good thing. I just want to caveat one thing about..."}, {"start": 739.57, "end": 767.74, "text": "Slowing versus fast progress. It could be that, imagine there was a moratorium on AI research 50 years, but hardware continued to accelerate as it does now. This is sort of what Nick's point was, is that it could be a massive hardware overhang or something where then AI actually, many different approaches to AI, including seed AI, self-improving AI, all these things could be possible. And maybe one person in their garage could do it."}, {"start": 767.74, "end": 795.17, "text": "And I think that would be a lot more difficult to coordinate that kind of situation. So I think there's some argument to be made where you want to make fast progress when we're at the very hard point of the S-curve, where actually you need quite a large team, they have to be quite visible, you know who the other people are, and in a sense society can keep tabs on who the major players are and what they're up to. Whereas opposed to a scenario where, say, in 50 or 100 years' time, when someone, a kid in their..."}, {"start": 795.17, "end": 818.86, "text": "In their garage could create a seed AI or something. Yeah, Bart, one last comment on this topic. So since I think that this process will be a very irregular process and some tasks will be far advanced and other tasks will be going quite slow, I'm sort of hoping that when society sees something like fake video creation or where you create a video where you have somebody say..."}, {"start": 818.86, "end": 845.6, "text": "Made up things. And that society will actually realize that there are these new capabilities for the machines. And we should start taking the problem as a society more seriously before we fool in general AI. We'll use AI to detect that. So you mentioned the word worry there. And you, Nick, went a bit farther. You had the word doom written on your slides three times."}, {"start": 845.6, "end": 873.98, "text": "No wonder there was one star on Amazon and that it was even in red color. I think it's just as important to talk about existential hope and the fantastic upside as downside, and I want to do a lot of that here. So let's just get some of those worries out of the way now and then return to the positive things. So I just want to go through quickly and give each one of you a chance to just pick one thing."}, {"start": 873.98, "end": 907.49, "text": "That you feel is a challenge that we should overcome. And then say something about what you feel is the best thing we can do right now to try to mitigate it. Do you want to start, Jan? To mitigate what? Mention one thing that you're worried could go wrong. And tell us about something constructive that we can do now that will reduce that risk. Oh, I do think that AI answers. I see a lot of good. I'm really..."}, {"start": 907.49, "end": 938.16, "text": "It's hard then to see great contacts between OpenAI and DeepMind, but I think this is just like a toy model of what the world at large might come up with in terms of arms races. Myself, I've been spending actually increasing amounts of time in Asia recently just to try to pull in more people elsewhere, what has been so far just being Anglo-American."}, {"start": 938.16, "end": 968.42, "text": "This is one thing that should be done and I'm going to do it. As someone who's outside this field, I think the challenge I'm really in touch with is how hard it is to take the safety concerns emotionally seriously and how hard it is for people in the field to do that as well. I can't tell you how many people outside this room who purport to be experts."}, {"start": 968.42, "end": 997.06, "text": "I think the control problem is a total non-issue. It's flabbergasting to meet these people and therefore not worth thinking about. And one of the reasons, I think, is that in one case there's this illusion that the time horizon matters. If you feel that this is 50 or 100 years away, that is totally consoling, but there's an implicit assumption there. The assumption is that you know how long it will take to build this safely."}, {"start": 997.06, "end": 1025.41, "text": "And that 50 or 100 years is enough time. The other issue is I think most people feel like intelligence is an intrinsic good. And of course we want more of it. And it's very easy to be in touch with that assumption because right now there's a cure for cancer, which we have not discovered. How galling is that? But for more intelligence, but for knowing which experiments to run or how to integrate the data we already have in hand."}, {"start": 1025.41, "end": 1054.27, "text": "We would have a cure for cancer that was actionable now unless there was some physical law of the universe that prevented us from curing cancer, which seems unlikely. So the pain of not having enough intelligence is really excruciating when you focus on it. And I think to your previous question of doing this quickly becomes an intrinsic good provided we have solved the alignment problems and the..."}, {"start": 1054.27, "end": 1075.2, "text": "Political problems and the chaos that would follow if we did it quickly without solving those problems. So I think the thing that is alarming is how ethereal these concerns are, even to those who have no rational argument against them."}, {"start": 1075.2, "end": 1104.75, "text": "So Sam, it sounds to me like you're agreeing very strongly with what Shane Legge said, that there is, in some circles, there's still the strong taboo that don't even consider the possibility that we might get AGI because it's just absolutely ridiculous. And he was arguing that the sooner we can get rid of this taboo, the sooner people can get to work and find all these really helpful solutions and answers that we need. So suppose for a moment that..."}, {"start": 1105.01, "end": 1134.27, "text": "I came up to you and I said to you, this idea of superhuman intelligence sounds absolutely ridiculous. It sounds completely nuts. And by the way, I've never seen your TED talk. And we're in an elevator. You have only 30 seconds to persuade me to take this more seriously. What would you say? And a lot of people who are here will have this exact conversation with colleagues and others in the future. Well, there are very few assumptions you need to make to..."}, {"start": 1134.27, "end": 1163.1, "text": "Take this seriously intellectually. Again, the emotional part is a separate piece. But if you assume that intelligence is just on some level the product of information processing in a physical system, and there are very few people who dispute that and who are scientifically literate at this point, and you assume that we will continue to improve our information processing systems unless something terrible happens to us to prevent that."}, {"start": 1163.1, "end": 1190.14, "text": "And that seems like a very safe assumption. Then it is just a matter of time before we instantiate something that is human level and beyond in our computers. And again, the time horizon is only consoling on the assumption that we know we have enough time to solve the alignment problems and the political problems. The other thing that's humbling here that Ray brought up at one point is that"}, {"start": 1190.14, "end": 1215.73, "text": "Even if we were handed a perfectly benign, well-behaved AI just from God, we're given a perfect oracle, we're given a perfect inventor of good technology. Given our current political and economic atmosphere, that would produce total chaos. We don't have the ethical or political will to share the wealth. We don't have the..."}, {"start": 1215.73, "end": 1244.72, "text": "The political integration to deal with this thing being given to Silicon Valley and not being given at the same moment to China or Iran. It's alarming that the best case scenario currently, basically just ripping out 80% of Nick's book because we've solved all those problems, is still a terrifying one. Clearly that's a near term thing that we have to solve."}, {"start": 1244.78, "end": 1265.86, "text": "Thank you, Sam. So, Demis, do you want to tell us about one thing that you feel is a challenge and say something about what we should focus on now? Yeah, I think it's, you know, I agree with both statements already said that, so I think the coordination problem is one thing where, you know, we want to avoid this sort of harmful race to the finish where..."}, {"start": 1265.86, "end": 1295.73, "text": "Corner cutting starts happening and things like safety are easy things to get cut because obviously they don't necessarily directly contribute to AI capability. In fact, they may hold it back a bit by making a safe AI. So I think that's going to be a big issue on a global scale and it seems like that's going to be a hard problem when we're talking about national governments and things. I think also we haven't thought enough about the whole governance scenario of how do we want"}, {"start": 1295.73, "end": 1317.78, "text": "You know, those AIs to be out in the world, how many of them and how they, you know, who will set their goals, all these kinds of things, I think needs a lot more thought. You know, once we've already solved the technical problem. I think it's wonderful that you're not just saying these things, but actually doing these things, since you played a leading role in setting up the partnership on AI here, which goes exactly in the direction of what you're..."}, {"start": 1317.78, "end": 1347.28, "text": "Advocating here. Do you want to pass it off to Nick? I'm sure there's nothing at all you're worried about, right? So tell us about one concrete, useful thing you would like to see us focus on. I agree with that. So fund technical work, bring in top talents to work on these technical issues, build these collaborations, build a community, build a trust. Work some more on figuring out attractive solutions to the governance solutions that could work, but don't rush to implement the first idea you have. But first, trial them out a little bit more. AI."}, {"start": 1347.28, "end": 1376.1, "text": "I think a lot about consciousness, so I was really struck by the sentience caution in my list of principles here that said avoid strong assumptions about the distribution of consciousness in AIs, which I take it entails avoid assuming that any human level or superhuman level AGI is going to be conscious. For me, that raises the possibility of a massive failure mode in the future."}, {"start": 1376.1, "end": 1402.83, "text": "The possibility that we create human or superhuman level AGI, and we've got a world populated by superhuman level AGIs, none of whom is conscious. And that would be a world, could potentially be a world of great intelligence, no consciousness, no subjective experience at all. Now, I think many, many people in a wide variety of views take the view that basically subjective experience or consciousness is required in order to have any meaning or value in your life."}, {"start": 1402.83, "end": 1430.35, "text": "So therefore a world without consciousness could not possibly be a positive outcome. Maybe it wouldn't be a terribly negative outcome. It would just be a zero outcome and among the worst possible outcomes. So I worry about avoiding that outcome. Now as a matter of fact, I'm fairly optimistic about the possibilities that AIs of various kinds will be conscious. But insofar as this community is making that assumption, I think it's important to actually think about the question of in creating AGIs, are we actually creating..."}, {"start": 1430.35, "end": 1460.11, "text": "Conscious beings. One thing that we ought to at least consider doing there is making, given that we don't understand consciousness, we don't have a complete theory of consciousness, maybe we can be most confident about consciousness when it's similar to the case we know about the best, namely human consciousness. Therefore, there's maybe an imperative to create human-like AGI in order that we can be maximally confident that there's going to be consciousness. What I hear you say is that when you have a nightmare about the zombie apocalypse,"}, {"start": 1460.11, "end": 1484.18, "text": "You're not thinking about some silly Terminator movie. You're thinking about this problem. We create, we upload ourselves, do all these wonderful things, but there's no one home. Is that fair to say? I mean, this is a different kind of existential risk. One kind of existential risk is there's no humans. There's AI. Some people might say, well, that's okay. There are successes. A much worse existential risk is there are no conscious beings in our future."}, {"start": 1484.18, "end": 1514.37, "text": "So I'll make a confession. So Shane Legg mentioned that there's been this strong taboo about talking about the possibility of intelligence getting very advanced. It's clearly also been a strong taboo for a long time to mention the C word. In fact, before the conference, when we got all these responses on the first round of the principles, guess which one was ranked last? It's got huge amounts of minus one ratings. That was the one with consciousness. So we changed it to sentience. It was terribly stated."}, {"start": 1514.37, "end": 1541.46, "text": "And stated it better, and then it got stated it's still better at lunch, and it's still rated last. Even though I personally share your interests in this a lot. 88% of people agreed to the sentience quotient. But not 90, so that one also fell off the list here. So maybe that's another taboo you can personally help us shatter so people think about that question more. Ray, anything you're concerned about? This wasn't what I was going to say, but just to respond."}, {"start": 1542.13, "end": 1570.86, "text": "A converse concern is we create AGIs. Everybody assumes that, of course, it's just a machine, and therefore it's not conscious, but actually it is suffering, but we don't look out for its conscious, subjective experience because we are making the wrong assumption. But what I did want to say was there are three overlapping revolutions that people talk about, GNR, genetics, biotech, nanotechnology, and robotics, which is AI."}, {"start": 1570.86, "end": 1597.81, "text": "And there are proposals, the CLMR conference has done here decades ago for biotech that have worked fairly well. There are similar proposals for nanotechnology. There's a difference with AI in that there really isn't a foolproof technical solution to this. You can have technical..."}, {"start": 1597.81, "end": 1627.58, "text": "Controls on, say, nanotechnology. One of the guidelines is it shouldn't be self-replicating. It's not really realistic because it can't scale to meaningful quantities without being self-replicating. But you can imagine technical protections. If you have an AI that's more intelligent than you and it's out for your destruction or it's out for the world's destruction and there's no other AI that's superior to it, that's a bad situation. So that's the specter."}, {"start": 1627.58, "end": 1657.26, "text": "Partly this is amplified by our observation of what we as humans, the most intelligent species on the planet, have done to other species. If we look at how we treat animals, people like their dogs and pets, but if you look at factory farming, we're not very benign to species that are less intelligent than us. That engenders a lot of the..."}, {"start": 1657.26, "end": 1687.1, "text": "Concern we see that if there's a new type of entity that's more intelligent than us, it's going to treat us like we've treated other species. So that's the concern. I do think what we're doing at this conference is appropriate. I wanted to mention that I think we should publish these guidelines the way the Asilomar guidelines in biotech were published decades ago. And then people can, you can have an opt-in, opt-out, but I think we should actually,"}, {"start": 1687.1, "end": 1716.59, "text": "Today we had this conference and the AI leadership community has come up with these guidelines and people can respond to them and debate them and then maybe at the next conference we'll revise them. You tell them our biotech guidelines have been revised many times. But I would advocate we actually take a stand and put forth these guidelines and then let the whole community at large debate them."}, {"start": 1716.59, "end": 1742.88, "text": "And have them guide our research. It's actually worked quite well in biotech. Bart? Okay. Yeah. So let me give a little different perspective. So one concern I have at a high level is these machines become really smart or even in certain areas, can humans still understand what the decisions that they suggest or that they make?"}, {"start": 1742.88, "end": 1755.36, "text": "And I work in the field of automated reasoning where we have, you know, significant advance last two decades going from problems with a few hundred variables to problems with millions of variables being solved now quite routinely."}, {"start": 1755.36, "end": 1783.66, "text": "And there was a sense in the community, well, we're getting answers from these reasoning engines, mostly hardware, software fabrication problems, but humans can no longer understand these answers. In the last few years, people have actually discovered that you can use the machine to generate explanations for their answers that are, again, human understandable. So I see sort of a glimmer of hope that maybe..."}, {"start": 1783.66, "end": 1815.1, "text": "Even if we have much less intelligence, we may be able to understand solutions that machines find for us, and we could not find these solutions, but they may be able to provide explanations that are accessible to us. So that's a little positive note. Thank you. Stuart? So there are two things that keep me awake at night, other than email. So one is the problem of misuse and bad actors."}, {"start": 1815.1, "end": 1845.49, "text": "Take an analogy, it's as if we were building nuclear weapons and then delivering them by email to everybody on the planet and saying, here's a toy, do what you want. How do we counter that? I have to say, I don't really have a good solution. I think one of the things we have to do is to make designs for safe AI very clear and simple and sort of make it unthinkable to do anything other than that."}, {"start": 1845.49, "end": 1876.13, "text": "Just like it's unthinkable to have a program with an infinite loop that produces a spinning pizza of death on your... Oh, sorry. What's unthinkable to have a buffer overflow that allows your software to be hacked into. The other thing that keeps me awake is actually the possibility that success would lead to AI as a helicopter parent for the human race that would ossify"}, {"start": 1876.13, "end": 1904.43, "text": "And gradually enfeeble us. So then there will be no point at which it was obvious to us that this was happening. And I think the mitigation which you asked for to look on the bright side is that in some sense the meta value of human evolvability, the freedom to change the future is something that the AI needs to adopt."}, {"start": 1904.43, "end": 1932.08, "text": "And in some sense, that would result eventually in the AI receding into the background and saying, okay, now I've got you through your adolescence. Now it's time for the human race to grow up now that we have the capabilities to eliminate scarcity, to eliminate needless conflict and coordination failures and all those things that we suffer from right now. So I could imagine a distant future where, in fact, AI is perhaps even less visible than it is today."}, {"start": 1932.56, "end": 1963.38, "text": "Great. Finally, you, Elon, as far as I know, never, ever expressed any concerns about AI, right? I'm just wondering if there's any concerns. In particular, any concerns where you see there's a very clear thing we should be doing now that are going to help? Well, I'm trying to think of what is an actual good future? What does that actually look like?"}, {"start": 1963.38, "end": 1991.44, "text": "Or at least bad, or I don't know how you want to characterize it. Because to a point that was made earlier, I think maybe by Sam and maybe by others, that we're headed towards either superintelligence or civilization ending. Those are the two things that will happen. Intelligence will keep advancing. The only thing that's prepared for advancing is something that..."}, {"start": 1991.76, "end": 2022.32, "text": "Put civilization into stasis or destroy civilization. So then we have to figure out what is a world that we would like to be in where there is this digital superintelligence. I think another point that I think is really important to appreciate is that all of us already are cyborgs. So you have a machine extension of yourself in the form of your..."}, {"start": 2022.32, "end": 2050.93, "text": "Your phone and your computer and all your applications, you are already superhuman. But by far, you have more power, more capability than the President of the United States had 30 years ago. If you have an internet link, you have an article of wisdom, you can communicate to millions of people, you can communicate to the rest of Earth instantly. These are magical powers that didn't exist not that long ago."}, {"start": 2050.93, "end": 2079.7, "text": "So everyone is already superhuman and a cyborg. The limitation is one of bandwidth. So we're bandwidth constrained, particularly on output. So our input is much better, but our output is extremely slow. If you want to be generous, you could say maybe it's a few hundred bits per second or a kilobit or something like that output. The way we output is like we have little meat sticks."}, {"start": 2079.7, "end": 2110.29, "text": "That we move very slowly and push buttons or tap a little screen. And that's just extremely slow. And compare that to a computer which can communicate at the terabit level. There's a very big order of magnitude differences. Our input is much better because of vision. But even that could be enhanced significantly. I think the two things that are needed for..."}, {"start": 2110.35, "end": 2140.91, "text": "A future that we would look at and conclude is good, most likely, is we have to solve that bandwidth constraint with a direct neural interface. I think a high bandwidth interface to the cortex. So that we can have a digital tertiary layer that's more fully symbiotic with the rest of us. We've got the cortex and the limbic system, which seem to work together pretty well."}, {"start": 2140.91, "end": 2171.63, "text": "They've got good bandwidth, whereas the bandwidth to our digital tertiary layer is weak. So I think if we can solve that bandwidth issue, and then AI can be widely available. The analogy to nuclear bomb is not exactly correct. It's not as though it's going to explode and create a mushroom cloud. It's more like if there were just a few people that had it, they would be able to"}, {"start": 2171.63, "end": 2201.62, "text": "We'd be essentially dictators of Earth. Whoever acquired it, and if it was limited to just one or more people, and it was ultra smart, they would have dominion over Earth. So I think it's extremely important that it be widespread and that we solve the bandwidth issue. And if we do those things, then it will be tied to our consciousness, tied to our will, tied to the sum of individual human will."}, {"start": 2201.68, "end": 2232.24, "text": "And everyone would have it, so it would be sort of still a relatively even playing field. In fact, it would be probably more egalitarian than today. Great. Thank you so much. That's, in fact, the perfect segue into the last question I want to ask you before we open it up to everybody. Something I have really missed in the discussion about really advanced intelligence beyond human is more thought about the upside."}, {"start": 2232.46, "end": 2256.74, "text": "We have so much talk about existential risk, and not just in the academic context, but just flip on your TV, check out Netflix. What do you see there in these scientific visions of the future? It's almost always dystopias, right? For some reason, fear gives more clicks than positive visions."}, {"start": 2256.74, "end": 2281.98, "text": "If I have a student coming into my office at MIT asking for career advice, the first thing I'm going to ask her is, where will you want to be in 20 years? And if she just says, well, maybe I'll get cancer. Maybe I'll get run over by a bus. That's a terrible way to think about career planning. I want her to be on fire and say, my vision is I want to do this. And here are the things that could go wrong. And then you can."}, {"start": 2281.98, "end": 2311.95, "text": "Plan out how to avoid those problems and get it out. I would love to see more discussion about the upsides, the futures we're really excited about, so we can not just try to avoid problems for the sake of avoiding problems, but to get to something that we're all really on fire about. So to start off, I'll just tell you something that makes me really excited about advanced artificial intelligence. Everything I love about civilization is the product of intelligence."}, {"start": 2312.18, "end": 2341.14, "text": "If we, for some reason, were to say, well, you know, I'm scared about this technology thing. Ooh, let's just press pause on it forever. There's no interesting question about if we're going to have human extinction. The question is just when. Is it going to be a super volcano? Is it going to be the next dinosaur-killing class asteroid? The last one happened 60 million years ago, so how long is it going to be?"}, {"start": 2341.14, "end": 2369.34, "text": "Pretty horrible future to just sit and wonder when we're going to get taken out here without the technology, when we know that we totally have the brainpower to solve all these problems if we proceed forward and develop technology. So that was just one thing that makes me very excited about moving forward rather than pressing pause. I want to just ask the same question to all of you guys in turn. So tell us just pretty briefly about something that you are really..."}, {"start": 2369.34, "end": 2391.12, "text": "So I want to be careful when I imagine concrete fruits of AGI. On a meta level, I think..."}, {"start": 2391.12, "end": 2415.41, "text": "As a first approximation, I think we should just maximize the amount of fun and minimize the amount of suffering. I think Eliezer has written a sequence called Fun Theory, where he points out that people have been horrible imagining and very unimaginative imagining paradises of various sorts. They're just like really boring places actually when you think about them."}, {"start": 2415.41, "end": 2446.91, "text": "I think Ed Isard has this sketch where he says, it was hard to spend one weekend with my relatives. Imagine spending eternity with your dead relatives. So I think we should be concerned about side effects and try to capture dynamics of improvement and basically go from there. Make sure that we adjust the trajectory as we get smarter and more..."}, {"start": 2446.91, "end": 2472.58, "text": "Great. Thank you, Jan. Sam, what do you get excited about? Well, strangely, what excites me is just really just abuts the parts that scare me the most. I think what is nice about this conversation, in particular about the alignment problem, is that it's forcing us to realize that there are better and worse answers to questions of human value."}, {"start": 2472.58, "end": 2497.92, "text": "As someone said, perhaps at this last meeting in Puerto Rico, that we really have to do philosophy on a deadline. And we have to admit to ourselves that there are better and worse answers, and we have to converge on the better ones. And what would excite me about actually the birth of superintelligent AI, one of the things, apart from solving obvious problems like curing disease and energy issues and all the rest, but..."}, {"start": 2497.92, "end": 2526.93, "text": "It perhaps differs a little bit with what Stuart said. I'm not so worried about idiocracy or all of us just losing our way as apes and living unproductive lives in dialogue with these oracles. I think actually I would want a truly value-aligned superintelligence to incrementally"}, {"start": 2526.93, "end": 2542.72, "text": "Show us, not merely conserve what we want, but show us what we should want to keep improving our values so that we can navigate in the space of all possible experiences and converge on better and better ones. Thank you, Sam. What about you, Demis?"}, {"start": 2542.72, "end": 2572.42, "text": "So, I mean, obviously this is why I spent my whole career working on this, is that I think if we do this right, it's going to be the greatest thing ever to happen to humanity, and in some ways I think unlock our full potential. I mean, I've talked a lot about it in all my talks about using it as a tool to help us make science and medical breakthroughs faster. I think that's an obvious one. But sort of taking that longer term, one reason I got so into AI is that, like probably many of you in this room, I'm interested in the biggest questions of why we're here, understanding our minds, what is consciousness."}, {"start": 2572.42, "end": 2598.56, "text": "What's the nature of the universe? What's our purpose? And if we're going to try and really grapple with any of those questions, I think we're going to need something like AI, perhaps with ourselves enhanced as well. And I think in that future world, we'll have a chance to actually find out about some of these really deep questions. In the same way we're finding out with AlphaGo, just about Go. But what if we could do that with all of science and physics and the biggest questions in the universe?"}, {"start": 2598.56, "end": 2616.78, "text": "And I think that's going to be the most exhilarating journey of all to find that out. To just carry that on a few other things that people commented on is in terms of us as the most intelligent beings on the planet right now and treating animals badly and these sorts of things. I think if you think about it, though, let's take, I don't know,"}, {"start": 2616.78, "end": 2646.37, "text": "I think that opens up a lot of conflicts."}, {"start": 2646.37, "end": 2676.06, "text": "Between humans as well, it has to do with resource scarcity at the heart of it. So I see if we can solve a lot of these problems, I can see a much better future. Great. So Nick, you pointed out that the upside part of your book was a little bit shorter, so now you have a chance to add something positive. What are you excited about? There are really two sides to that. One is getting rid of a lot of the negatives, like the compassionate use to cure diseases and all other kinds of horrible miseries that exist."}, {"start": 2676.06, "end": 2704.06, "text": "on the planet today so that is a large chunk of the potential but then beyond that if one really wants to see realistically what the positive things are that could be developed I think one has to think outside the constraints of our current human biological nature that it's unrealistic to imagine a trajectory stretching hundreds of thousands of years into the future we have super intelligence we have material abundance and yet we are still these"}, {"start": 2704.06, "end": 2733.54, "text": "Bipedal organisms with three pounds of gray TC matter with a fixed set of emotional sensitivities and a hedonic set point that is kind of okay-ish for most people. But if something really good happens, it lasts a short time and then you're back to the baseline. I think all of these basic parameters that define the human game today become up for grabs in this future. And it opens up this much vaster space of"}, {"start": 2733.54, "end": 2762.3, "text": "Post-human modes of being, some of which I think could be wonderful literally beyond our ability to imagine in terms of the mental states, types of activities, the understanding, the ways of relating. So I don't think we need a detailed blueprint for utopia now. What we need is to get ourselves in a position later on where we can have the ability to use this to realize the values that come into view once we take steps forward."}, {"start": 2762.3, "end": 2791.87, "text": "Thank you, Nick. What about you, David? I'm excited about the possibilities for AI making us humans smarter. I mean, some of it is selfish. I turned 50 last year. My brain is gradually becoming slower and older and dumber. But I'm not sure that I am. And that's partly because of all the augmented intelligence technology we're using. Smartphones and the internet and so on are giving me all kinds of..."}, {"start": 2791.87, "end": 2821.12, "text": "Extended capacities that I didn't have before. And I'm really looking forward to AI helping with that. In 10 years or so, once everyone is wearing augmented reality glasses with deep learning built into it, then I'm really going to need that around 60. And if you guys really get on the case, and by the time I'm 70 or so, we've got real, genuine AI or AI modules out there which could somehow come to be integrated with my..."}, {"start": 2821.12, "end": 2849.17, "text": "My brain processes, or maybe eventually we get to upload our entire brains onto AI, then there's a way potentially to get smarter, more intelligent forever. And this is not just selfish, although I can't say that doesn't motivate me. But Demis talked about the AI scientist. I also like to think about the AI philosopher. The problems of philosophy are really hard, and many people have speculated that we humans are just too dumb to solve."}, {"start": 2849.17, "end": 2878.45, "text": "But once we've actually got AIs on the scene, maybe AI enhanced humans, and maybe we're going to be able to cross those thresholds where the AI enhanced humans, or maybe just the AGIs, end up solving some of those problems of philosophy for once and for all. Great. Ray, you have been a true pioneer in articulating positive visions of the future in your writing. So if you pick the one that you're most excited about now, what would that be? Imagine going back 10,000 years and asking,"}, {"start": 2878.45, "end": 2905.46, "text": "Quintessential caveman and woman. Gee, what is a beneficial future? What would you like to see? And they would say, well, I'd like this fire to stop going out, and I'd like a bigger boulder to prevent the animals from getting in the cave. Anything else? Well, no, I think that would be pretty perfect. Well, don't you want a better website and apps and search engines? Imagine going back."}, {"start": 2905.46, "end": 2932.99, "text": "Two million years ago, and talking to primates, if you could do that, and saying, isn't it great? The frontal cortex is coming, and you can have additional neocortex and hierarchy. And they said, well, what's the point of that? And I said, well, you'll have music and humor. And their answer would be, what's music? What's humor? So they couldn't imagine concepts that they couldn't imagine."}, {"start": 2932.99, "end": 2959.55, "text": "And by analogy, I think we will have new phenomena that are as profound as music and humor. You could call it more profound music and it will be funnier, but I think it will be as profound as these great leaps that evolution has brought us because we will become profoundly smarter and, you know, if music and humor are up here and we go to even higher levels of the neocortex,"}, {"start": 2959.55, "end": 2983.78, "text": "We're going to have more profound ways of expressing ourselves. And once we have that, we would not want to go back. What about you, Bart? Well, I pretty much agree with that. We can't really predict much in advance what we would like to have. For myself, personally, I see the developments in mathematics and discovering and science and discovery. And computers are just the..."}, {"start": 2983.78, "end": 3015.12, "text": "The hybrids of human computers there is quite incredible and makes what we do much more exciting. So I think that will be, in the near future, the first thing. Great. And what about you, Stuart? Well, so like Jeffrey Sachs, I think that for many of us, and probably like the caveman, that for many of us, life is pretty amazing. And for many more of us, it isn't."}, {"start": 3015.15, "end": 3045.15, "text": "And I think the best thing that AI can do, the big upside, is actually to fix the latter problem. I love Nick's feeling that there are higher states of being that are so far above our current pretty good that that balances out all the pretty bad that a lot of people are suffering. But I really think the emphasis should be on the pretty bad and fixing it. And eliminating, so Demis was reading my notes apparently."}, {"start": 3045.15, "end": 3074.13, "text": "From across the room. But eliminating the scarcity basically eliminates the need for people to act in a zero-sum fashion where they can only get by by making it less possible for someone else to get by. And I think that's the source of a lot of the nastiness that Jeffrey mentioned earlier. So I think that would be my main upside. And not having to read so much email. That would be the second."}, {"start": 3074.26, "end": 3105.26, "text": "And for you, Elon, you've never articulated any visionary ideas about the future as far as I know either. What about now? I think I just, I haven't thought about this a lot. And I think it just really just comes down to two things. And it's solving the machine brain bandwidth constraint and democratization of AI. I think we have those two things."}, {"start": 3105.26, "end": 3134.8, "text": "The future will be good. There's a great quote by Lord Acton, which is that freedom consists of the distribution of power and despotism in its concentration. And I think as long as AI powers, like anyone can get it if they want it, and we've got something faster than meat sticks to communicate with, then I think the future will be good."}, {"start": 3135.82, "end": 3167.81, "text": "Fantastic. So let's get in. I know your caffeine levels are dropping dangerously low, and we also have another panel after this, which is going to be really exciting to listen to. So let's do just a few quick questions. Make sure that they are actually questions. And say your name, and also pick one person on the panel and address it just to them, okay? Joshua? Joshua Benjou, Montreal. And it's for Jan."}, {"start": 3167.81, "end": 3196.3, "text": "I found your presentation very inspiring. And one question I have is related to the question of eliciting preference and values from people. Do you think this line of investigation could lead to better democracy, better society, more direct democracy? What do you think about this direction to deal with the issue of misuse and things like that?"}, {"start": 3199.02, "end": 3227.81, "text": "Yes, absolutely. One code name for this even could be Democracy 2.0 or UN 2.0 or something like that. And as I mentioned in my presentation, a lot of people today basically want to make the world better, but it's kind of hard to distinguish them from people who say they want to make the world better. So if there was actually a very easy measuring metric..."}, {"start": 3227.81, "end": 3256.69, "text": "That basically would work as a selling point, focal point. Then I think that would be super helpful. And yeah, democracy was invented hundreds of years ago. And clearly, we have advanced as a civilization, and we have better knowledge about how to aggregate preferences. And Nicholas Berggruen over there. Thank you, Max. Nicholas Berggruen. So I have a very almost naive."}, {"start": 3256.69, "end": 3286.59, "text": "Question. This is a very well-meaning group in terms of, let's say, intentions. But who's sort of looking at who else is doing potentially AGI? It could be well beyond this group. It could be in China. It could be any place. And what happens? Because we've talked about how powerful AGI is. And if Elon is correct, if it's distributed fairly, fine. But if it isn't..."}, {"start": 3286.59, "end": 3318.86, "text": "Is there a way to monitor today or in a year or in 10 years? Because once it's out, it'll be fast. Who is monitoring it? Who has a tab on it? Because this is self-selected, but beyond. Elon or Demis, does either one of you want to take a swing at this? Well, I think this sort of relates to my point I said earlier about trying to build AI at the hard part of the S-curve."}, {"start": 3318.86, "end": 3347.28, "text": "So, which I think is where we sort of are at the moment, as far as we can tell. Because, you know, it's not easy to make this kind of progress. So, you need quite a lot of people who are quite smart. And that community is pretty small still, even though it's getting rapidly bigger at places like NIPS. And so, and most people know each other. So, this is pretty representative of everyone in the West, at least. Obviously, it's harder to know what's happening in China or in Russia, maybe. But, you know, I think..."}, {"start": 3347.28, "end": 3374.35, "text": "That you need quite a large footprint of resources, you know, people and very smart people and lots of compute and so on. So I think that narrows down the scope of the number of groups who can do that. And it also means that they're more visible. So, you know, I think certainly in the West, I think most people on here, someone in this room will have contact with somebody who's in those groups who are capable of..."}, {"start": 3374.35, "end": 3403.54, "text": "You know, making meaningful progress towards AGI. It's harder to know in the East and further apart, but we should try and make links to those, you know, Chinese National Academy of Sciences and so on to find out more. But, you know, that may change in the future. I think that's the current state. Great. The bad news is it's getting late in the day, and we only have time for one more question. The good news is there's a coffee break right after this, so you can ask all your questions if you swarm the panel. And the last question goes to you, Eric. Do you want to stand up?"}, {"start": 3403.54, "end": 3432.83, "text": "Eric Brynjolfsson, MIT. I'm going to pick up on the thing that Elon said at the end about democratizing the outcome and tie it back to the panel yesterday where Reid Hoffman talked about people caring a lot about not just absolute income but relative income. And I wanted to get the panelists' reactions to the thoughts about whether or not AI had tendencies towards winner-take-all effects, that there's a tendency for its concentration, that whoever's ahead can pull further ahead, or whether there's"}, {"start": 3432.83, "end": 3464.77, "text": "Potential for more widespread democratic access to it, and what kinds of mechanisms we can put in place if we want to have the widely shared prosperity that Elon suggested. Elon, do you want to take that? Yeah, well, I mean, I have to say that when something is a danger to the public, then there needs to be some, I hate to say it, government agency."}, {"start": 3464.77, "end": 3493.01, "text": "Regulators. I'm not the biggest fan of regulators because they're a bit of a buzzkill. But the fact is we've got regulators in the aircraft industry, car industry, deal with them all the time, with drugs, food, and anything that's sort of a public risk. I think this has to fall into the category of a public risk."}, {"start": 3493.01, "end": 3520.88, "text": "I think that's the right thing to do. And I think it will happen. The question is whether the government reaction speed matches the advancement speed of AI. Governments react slowly. Governments move slowly, and they tend to be reactive as opposed to proactive. But you can look at these other industries and say, does anybody really want the FAA to go away? It's like people could just be a free-for-all for aircraft. Probably not."}, {"start": 3520.88, "end": 3551.3, "text": "There's a reason it's there. Or just people can just do any kind of drugs and maybe they work, maybe they don't. We have that in supplements, which is kind of ridiculous. But I think on balance, FDA is good. So I think we probably need some kind of regulatory authority. And I think the rebuttal to that is, well, people will just move to freaking Costa Rica or something. That's not true. Okay."}, {"start": 3551.3, "end": 3581.23, "text": "We don't see Boeing moving to Costa Rica or Venezuela or wherever. It's like free and loose. To Demis' point, the AI is overwhelmingly likely to be developed where there is a concentration of AI research talent. And that happens to be in a few places in the world. It's Silicon Valley, London."}, {"start": 3581.36, "end": 3608.21, "text": "I've been to Boston, a few other places, but really just a few places that regulators could reasonably access. And I want to be clear, it's not because I love regulators. They're a pain in the neck, but they're necessary to preserve the public good at times. All right. On that note, let's thank the panel for a fascinating discussion."}]}