{"url": "https://www.youtube.com/watch?v=VrPc4_Mylr4", "page_url": null, "title": "Dr. Dario Amodei on AI Safety and Cybersecurity - Artificial Intelligence and Global Security Summit", "show_or_channel": "Center for a New American Security (CNAS)", "date": "2017-11-16", "duration_s": 640, "type": "talk", "speakers": ["Dario Amodei"], "confidence": "medium", "notes": "CNAS summit, Dario on AI safety and cybersecurity (10 min); may overlap with the panel video 6sCKa5and1I.", "people": ["dario-amodei"], "id": "2017-11-16_center-for-a-new-american-dr-dario-amodei-on-ai-safety-and-cybers", "audio_duration_s": 640.02325, "worker": "jc-w05", "transcribe_s": 10.5, "model": "large-v3", "segments": [{"start": 0.85, "end": 29.38, "text": "Okay, our next session is focused on artificial intelligence safety and cybersecurity. There's tremendous opportunity in artificial intelligence, but there are also risks we need to be aware of. We need to be worried about things that others might do, as well as some safety vulnerabilities inside these systems. So what we're going to do for this session is we're going to have two expert presenters each give a brief 10-minute presentation to kind of tee up some of these issues. First, we'll have Dr. Dario Amadei from OpenAI."}, {"start": 29.38, "end": 67.87, "text": "He's a leading researcher on AI safety. He is one of the co-authors of a foundational paper looking at AI safety. And after him, we'll have Dr. Kathleen Fisher, who's chair of the computer science department at Tufts University, talking about AI-enabled cybersecurity. And after that, we'll have a moderate discussion with them as well as other panelists. So thank you very much, and welcome, Dr. Amadei. Thanks. Got the slides? So yeah, at OpenAI, which is a research lab founded by..."}, {"start": 67.87, "end": 97.07, "text": "Elon Musk and others, I think about a combination of both how to make AI systems and how to make AI systems safe, which means making AI systems do what we want them to do. So in particular, I'm going to be talking about what I think is going to happen with the next generation of AI systems. So just a couple years ago, even to researchers who are at the core of AI research, the boom in AI mostly meant things like speech models,"}, {"start": 97.07, "end": 126.78, "text": "Vision models, translation, natural language processing. These systems can be quite powerful, and if they go wrong, things can be quite bad. But at the same time, there's still an essential passivity to such systems, right? An image classifier classifies an image, and then a human decides what to do with that image. Things can go wrong, but there's a limit to how bad it can be. There's a new type of AI system based on a technology called reinforcement learning."}, {"start": 126.78, "end": 151.31, "text": "That has some properties that make me think that safety is going to be a much bigger issue in the future than it has been in the past. So reinforcement learning is the technology that allows machine learning systems to teach themselves and to operate in a completely autonomous way in complex environments. So here are some examples of such systems. The first one is the famous AlphaGo system, which Google's DeepMind unit."}, {"start": 151.31, "end": 174.24, "text": "Worked on which beat the world champion and more recently was able to surpass even its previous system without involving any human knowledge at all. The other three are systems trained at OpenAI. The system below is a reinforcement learning system that has learned to play a massive multiplayer online battle game called Dota."}, {"start": 174.24, "end": 200.83, "text": "That millions of people play and professionals devote their life to. We managed to beat the world champion in that game with a system where we told it almost nothing about the game except what it meant to win and what it meant to lose. And it was able to experiment with all of its possible strategies and figure out how to beat the world champion, including things like baiting, tricking its opponents. Unlike AlphaGo, this isn't a deterministic environment. It's stochastic. There's a lot you don't see."}, {"start": 200.83, "end": 229.57, "text": "Up at the upper right is two simulated humanoid robots that have learned to sumo wrestle each other in a competition. And at the bottom is deployment of stacking of blocks by a real robot, which we also trained by reinforcement learning. So this is kind of a new kind of AI that I think is emerging, particularly from our lab and Google's DeepMind unit, some others as well."}, {"start": 229.57, "end": 259.17, "text": "That I think is going to have a lot of implications. So again, the key features of this are extended interaction with a complex real-time environment, a very high level of autonomy, and along with that, the ability to do things at a speed that humans will not be able to keep up with or monitor. And the system teaches itself and discovers its own strategies instead of us telling it what to do. This is, in fact, I think only going to get more extreme. There is..."}, {"start": 259.17, "end": 272.42, "text": "Currently a boom in AI optimized hardware. The technologies that have been behind the AI boom, deep learning, are based on a particular kind of mathematical operation. We are now optimizing chips."}, {"start": 272.42, "end": 302.08, "text": "And so this is a graph of how many operations we've been able to do per watt of power, which is a good unit of cost. And over the last couple years, this has skyrocketed, and we really see no end in sight. And the better hardware is going to mean that systems are going to be more powerful. We're going to be able to iterate faster on research ideas because we can run experiments faster. And in the end, we're going to be able to build systems that have even more autonomy and that operate in the real world. So that all sounds great."}, {"start": 302.08, "end": 322.22, "text": "But I actually have some concerns related to this. So this is an example of another system, a reinforcement learning system like all of the others, that we train to do something. So what this thing was supposed to do was we were supposed to train a boat to complete a course from start to finish. And the only way to measure..."}, {"start": 322.22, "end": 338.59, "text": "Whether it had made progress along completing the course was that, you know, in this video game, there are these kind of markers laid out. They're those kind of green and blue things. Every time you get one, you increase your score. And they're kind of laid out mostly linearly along the course."}, {"start": 338.59, "end": 362.98, "text": "And so we set this thing running. I remember setting it running one day, just telling it to teach itself. And I figured that it would learn to complete the course. But what it does instead, this thing that's been looping is it goes backwards in the course. It finds this isolated lagoon. And it turns out that by spinning around in this isolated lagoon in exactly the right way, it can get more points than it could possibly ever have gotten."}, {"start": 362.98, "end": 389.2, "text": "by just completing the race in the most straightforward way. In a sense, there's nothing wrong with this, in the sense that we asked it to find a solution to a mathematical problem. How do you get the most points? And this is how it did it. But if this was a passenger ferry or something, or a cyber-physical system, you wouldn't want it spinning around, setting itself on fire, crashing into everything."}, {"start": 389.2, "end": 412.13, "text": "You might say, well, you know, of course there would be human oversight and this kind of thing would never be allowed to happen. But, you know, if a system is training in a massively parallel way, if we deploy it in a massively parallel way as we would for, say, a cyber attack system, something like this can happen without us even knowing that it's happening, where our system has found a way to do the thing we think we want in a way that we really don't want."}, {"start": 412.13, "end": 437.94, "text": "Another kind of clever example of this is we trained this as a simulated robot that we trained to kind of kick a puck and move it towards the red dot. And we made a complete realistic physics simulation. And one thing it discovered was that it's actually kind of hard to kick the puck all the way to the red dot. So it kicks it most of the way. And then it kind of cheats by nudging the table backwards in order to make the puck hit the table."}, {"start": 437.94, "end": 446.08, "text": "We just gave it the mathematical problem of like, yup, you get reward for hitting that puck right to the dot."}, {"start": 446.08, "end": 475.38, "text": "And it found the most efficient way to do that, which turns out to be cheating by moving the table, or what we would think of as cheating by moving the table. Now again, if there were someone on that table, if that table were valuable, we would not want the system to discover a solution like that. But in the way that we've specified the mathematical problem, this is a valid solution. And so I personally think it is very important that we develop techniques that prevent this kind of thing from happening."}, {"start": 475.38, "end": 493.95, "text": "Now moving from kind of these systems to the kind of systems that we would have in the real world, I mean these are both kind of simulated environments, but again as I said I think these systems are"}, {"start": 493.95, "end": 512.9, "text": "Increasingly moving into the real world. So here are a couple of thought experiments. So some things that you can imagine doing with reinforcement learning systems, systems that teach themselves and that experiment with their own strategies and that can explore a very unrestricted space. You might be able to make a smart hacking agent that makes"}, {"start": 512.9, "end": 537.66, "text": "You might be able to make a smart hacking agent that is trying to attack a particular adversary. We've already seen some of that with the DARPA competition, although it didn't use this kind of AI. I think this kind of AI would make it much more powerful. And you can imagine that a system that you deployed in this way, maybe it figures the best way to disable the adversary is just to make all the code on the internet unreadable. And such a thing would spread very quickly."}, {"start": 537.66, "end": 565.9, "text": "We wouldn't know that it would happen or just kind of any number of similar scenarios you can come up with where it does something very quickly beyond the speed at which a human can monitor it that formally achieves the objective you've given it but that really ends up doing something that we would regard as destructive not just to our adversary but to us as well and maybe to everyone involved. Similarly, I think we'll have kind of large-scale systems designed to do social media influence. They're already kind of..."}, {"start": 565.9, "end": 586.7, "text": "I can imagine, let's say you're doing a social media campaign to get someone to buy your product, and it turns out the best way to get them to buy your product is to make them really angry."}, {"start": 586.7, "end": 602.27, "text": "Fight each other and they do something that increases their demand for whatever your product is. So there's been a lot of talk about the US and China and basically technological races, even if only economic."}, {"start": 602.27, "end": 624.06, "text": "between countries, governments, commercial entities in a race to develop more powerful AI. And I think the message I want to give is that it's very important that as those races happen, we're very mindful of the fact that that can create the perfect storm for safety."}, {"start": 624.06, "end": 638.67, "text": "Safety catastrophes to happen, that if we're racing really hard to do something, and maybe even some of those things are adversarial, that creates exactly the conditions under which something can happen that not only our adversary doesn't want to happen, but we don't want to happen either."}]}