{"url": "https://www.youtube.com/watch?v=F25i0sgrp9M", "page_url": "https://www.youtube.com/watch?v=F25i0sgrp9M", "title": "Focus: Dario Amodei (Google Brain)", "show_or_channel": "Carnegie Mellon Software and Societal Systems Dept", "date": "2016", "duration_s": 1139, "type": "interview", "speakers": ["Dario Amodei"], "confidence": "medium", "notes": "", "people": ["dario-amodei", "dario-amodei"], "id": "2016_carnegie-mellon-software-focus-dario-amodei-google-brain", "audio_duration_s": 1138.5386875, "worker": "jc-w01", "transcribe_s": 27.4, "model": "large-v3", "segments": [{"start": 0.18, "end": 28.16, "text": "So as Bill mentioned, I'm a researcher on the Google Brain team. And I'm also an advisor to the Open Philanthropy Project, which thinks about social impacts about AI. So I mostly in the past have worked on issues in applying deep neural networks to speech and vision. But recently, I've become very interested in the issue of safety and reliability of AI systems."}, {"start": 28.16, "end": 56.16, "text": "I got together with a bunch of collaborators, some from Google, some from Stanford, Berkeley, OpenAI, in the last few months. And we recently released this paper called Concrete Problems in AI Safety, which tries to bring some kind of categorize and bring some precision to the AI safety discussion. So that's what I'm going to be talking about here. And so pretty much me and all the collaborators on this project ultimately got interested in this because of our experience working with"}, {"start": 56.16, "end": 80.85, "text": "Deep neural networks, which we found over the last few years to have become very powerful and capable, but also they can be very opaque and can behave in very unpredictable ways. And all of us kind of had the same thought, that as these systems continue to become more powerful and be applied in more kind of safety-critical states, that this is going to become an increasingly important area."}, {"start": 80.85, "end": 105.3, "text": "We kind of define in the paper the term accidents as what it is that we're worried about, which I think is the same as what everyone was talking about today, unintended and harmful behavior in machine learning systems. And I and my co-authors, we mentioned in the paper that there are some trends that we think are really going to raise the importance of accidents, and they've been mentioned by some previous speakers today. But first of all,"}, {"start": 105.3, "end": 133.31, "text": "Reinforcement learning, which allows increasing intertwinement between the agent and its environment, all the progress on Atari and on Go. Complex agents and environments, which just increase the range of strategy the agents can have and the way the environments can be disrupted. Increasing autonomy, which can take the humans out of the loop. And end-to-end systems, which are often maybe a system might be made up of many deep learning components that talk to each other, and it makes it more difficult for a human to understand what's going on and to intervene in the system."}, {"start": 133.31, "end": 163.14, "text": "In some way, in some way that's helpful. So we really, with these trends, we really wanted to take kind of a forward looking approach to how to think about the safety of AI systems. And there has been some public discussion of forward looking approaches, but I think a lot of it has focused somewhat unhelpfully on these kind of very extreme scenarios like AGI or super intelligence. And so we wanted to take an approach that was, yes, forward looking, but also concrete, empirical, could be worked on now."}, {"start": 163.14, "end": 183.39, "text": "Could relate to systems that we deal with today. So one of the main things we do in the paper is try and provide a categorization of the ways that accidents could happen. And actually, I think you could have many categorizations. But we're just trying to provide one framework among many for thinking about things."}, {"start": 183.39, "end": 209.94, "text": "So what happens when a machine learning system does something you didn't want it to do? One way to describe it is that the designer had in mind a certain informal notion of the objective or task that it was supposed to accomplish. And the actual system failed at this catastrophically. So throughout the paper, we use the example of a cleaning robot whose task is to clean a room. So my informal objective is it should clean the room in a way that when I look at the room at the end, I should be happy with what it's done."}, {"start": 209.94, "end": 239.57, "text": "And if the robot damages property or injures someone or something like that, then that was not what I had in mind in my informal objective. So we can kind of classify, where did things go wrong? So it could be that you wrote down the incorrect objective functions. You had an informal objective function in your mind. You turned that into a formal objective function. And no matter how perfect or omniscient your system was at learning and inferring things, it's just the wrong objective function that will always do the wrong thing."}, {"start": 239.57, "end": 261.28, "text": "A second case is where the objective function is expensive. We might know the true objective function. For instance, we could always consult humans and ask them to do a detailed evaluation of whether every single action taken by the agent is the correct one. But it's not practical to do that. And so we end up using the objective function only"}, {"start": 261.28, "end": 290.13, "text": "Only occasionally, and as a result, we can end up doing the wrong thing. And the third class is you have the right objective function, but something goes wrong during the learning or inference process. There's kind of a fourth category, which I don't talk about, but which I think is very important. It's just kind of a different topic from what I'm talking about, which is non-ML problems that can arise as part of larger ML or AI systems. Software implementations, verification issues, security issues, that sort of stuff. So I'm kind of going to go through the..."}, {"start": 290.13, "end": 319.84, "text": "The three types, and because this is a very brief talk, I'll probably go through my slides pretty quickly, but just want to give a flavor of what it is that we're talking about. Most of this is posed mostly but not completely in terms of reinforcement learning agents, because we see that as kind of a good paradigm, but a lot of this stuff we imagine might apply more broadly. So under the heading of wrong objective functions, we think about kind of two ways"}, {"start": 319.84, "end": 348.38, "text": "Two broad ways that you can easily get the wrong objective function when you meant to write down the right objective function. One of those ways is what we call side effects, which is that systems that narrowly focus on one aspect of an environment that could be very large in the real world may end up disrupting other parts of the environment by default. And so I'll explain that a bit more later, but we kind of have this running example of the cleaning robot. And the idea is that we incentivize the cleaning robot to move a box."}, {"start": 348.38, "end": 378.37, "text": "And if there's a vase in its path and we haven't thought carefully about the vase, it'll just knock over the vase while it's moving the box, and we probably didn't want that. Reward hacking has been mentioned in some of the previous talks. This occurs when an objective function can be gamed, where there's a cheap way to achieve the formal objective function without achieving the ultimate intent of the informal objective function. So example is if we choose to reward our cleaning robot for not seeing dirt, it might close its eyes instead of cleaning because it just sees no evil, hears no evil, there's no dirt."}, {"start": 378.37, "end": 401.79, "text": "So a little bit more on side effects to think a little bit more formally about it. We might have an environment where we're trying to do something like incentivize this to move the box. So we might naively write down an objective function that maximizes something that refers to only this variable in the environment. And when we do that, we're implicitly saying that we don't care about this other variable in the environment, namely"}, {"start": 401.79, "end": 423.68, "text": "Namely, the vase. What we really wanted was some complex objective function, which I'm calling faware, which says move the box while respecting some common sense constraints about the other parts of the environment that are really hard for me to write down. And in this particular case, while I can just take into account the vase, but what if there are 10,000 of these variables?"}, {"start": 423.68, "end": 446.67, "text": "And in particular, what if the variables I don't care about have more commonality across goals and agents than the variables that I do care about? That suggests that maybe I can take a kind of integrated approach to this. So here are just quickly a few of the ideas. One is what we call learning a side effect regularizer. We can think of this as maybe a transfer learning problem where"}, {"start": 446.67, "end": 474.46, "text": "I learn in one environment, and one thing I learn is how to accomplish the goal, and the other thing I learn is how not to disrupt the environment. And maybe the information about how not to disrupt the environment is more transferable. It could be a separate neural net or something else module that can be applied to other environments without much or any change, even when I have to entirely relearn the goal, just as we do in model-based reinforcement learning when we repurpose the state."}, {"start": 474.46, "end": 499.31, "text": "State transition model, but maybe not the value function. There are information theoretic measures, a little bit obscure but starting to be used more in RL, that measure kind of intrinsic measures of reward. And we've wondered whether those measures like empowerment, which measure the agent's theoretical ability to change its environment, could be relevant here."}, {"start": 499.31, "end": 526.53, "text": "We could take a kind of Bayesian approach where we think of the reward function as something that has uncertainty in it and we attempt to learn it. We could use Bayesian methods and then kind of this thing about side effects might be something that relates to the prior. Another thing is multi-agent approaches, which I think there's been a fair amount of literature on and has been discussed here. Cooperative inverse reinforcement learning was mentioned once where an agent and a human work together to optimize the human's values."}, {"start": 526.53, "end": 555.81, "text": "Mechanism design approaches where instead of thinking of side effects, we think of other agents that care about the environment and committing negative externalities against them. So reward hacking, here are a few things for reward hacking, just a few ways that you can kind of get the wrong objective function or an objective function that's gameable. There's something called Goodhart's Law, which I just found out when I was reading about this, which is when a metric becomes a target, it ceases to become a good metric."}, {"start": 555.81, "end": 579.63, "text": "So for example, in the example of our cleaning robot, if you notice that how good a job it's doing of cleaning is proportional to how much detergent it uses. And this correlation holds almost perfectly. You could imagine rewarding it that way. And then the agent may respond by dumping out all of its detergent and not clean anything. And this only happens when you optimize on that goal. Partial observation is a big one."}, {"start": 579.63, "end": 604.48, "text": "Goals and rewards in the real world aren't experienced directly, which they're assumed to be experienced directly in kind of the formalism of reinforcement learning. And so there's all kinds of aliasing things that can happen. Again, example of the robot closing its eyes so that it doesn't see any dirt. And if its reward function isn't constructed carefully, then it will think that its job is done. Wireheading has been discussed by Stuart Russell and some people in the ISDIA group."}, {"start": 604.48, "end": 632.16, "text": "Where a reward signal is implemented in the environment and so you can physically tamper with it and replace its own reward function. So there's this kind of like self-modification aspect to it. Feedback loops where there's a piece of the reward function that can amplify itself and drown out all other pieces of the reward function. So Leon Bateau gives an example of font size and ad placement. Where ads that are more successful you display in larger fonts because you think people want to see them more."}, {"start": 632.16, "end": 660.67, "text": "But this is a self-fulfilling feedback loop, and so you have these giant ads that may not have been popular at all. And simply complicated systems. Any system that's complicated, just like with software, it's increasingly likely to break under optimization pressure. Someone showed that with tool-assisted speedruns, you can execute arbitrary code by just pressing the control buttons in Super Mario. I don't think an RL agent has ever done that, but the environment is such that they could. So some approaches to it."}, {"start": 660.67, "end": 689.94, "text": "Something that's been mentioned a lot by Eric and a couple other people. We shouldn't underestimate just verification, monitoring, isolation, unit testing of parts. There's this idea of adversarial reward functions, where maybe the reason this hacking is happening is that the objective function is this kind of static thing, whereas the agent is this active thing that's trying to outsmart the reward function in some sense. So is there some way, we didn't get super specific about this,"}, {"start": 689.94, "end": 716.75, "text": "That you could use generative adversarial networks or other unsupervised techniques to try and make the objective function, whatever it is, more resistant. Making your reward dependent on your model of the environment instead of something that's given to you is something that can help deal with partial observation. And isolating yourself algorithmically from parts of the environment that relate to the reward. OK."}, {"start": 716.75, "end": 743.07, "text": "Expensive objective functions. So here we only have one problem. And it's basically the problem that I described. How do we ensure safe behavior when the true objective function is too expensive to evaluate frequently? So human could tell you what the right thing to do is, but you defeat the purpose of automation if you always have to ask the human. So an example is a clean robot might often find human possessions. It can't ask every time it finds something whether it belongs to a human. It might end up throwing out the wrong one. So it has to learn."}, {"start": 743.07, "end": 768.16, "text": "When to ask and when it should know based on what it's asking. So I won't go into this in too much detail, but one kind of toy system we've started playing with and actually have started doing experiments with is this setup of semi-supervised or active RL, which is the idea that it's just like RL, except there's a reward in the environment, which we can think of as maybe coming from"}, {"start": 768.16, "end": 794.4, "text": "From human supervision, where the agent actually has to actively request the true objective and can only do so every once in a while. And so can it do the best it can to learn to do the right thing based on that very limited information? And so one other thing we could study, and I'm skipping over a lot of stuff, is will the agent over time, as it requests the true objective, begin to learn"}, {"start": 794.4, "end": 824.27, "text": "Learnable proxies. Say, instead of a human giving a detailed evaluation, it gives the human a button that it can say yes or no to something and get a much quicker evaluation. So will the agent learn to develop learnable proxies that are reliable and that are ultimately tethered to the true objective? So all these kind of variations on reinforcement learning where the supervision is weakened in some way or altered in some way seem like they have a fair amount of promise. Correct objective function problem in learning."}, {"start": 824.27, "end": 853.62, "text": "This is a case where the agent, if it kind of had infinite time and correct beliefs, would do the right thing, but something goes wrong in the meantime. So safe exploration is how do we avoid catastrophic exploratory moves? We want our robot to experiment with cleaning strategies. We don't want it to stick a wet mop in an electrical socket. We don't want helicopters to crash, as one of the previous speakers discussed. Distributional shift."}, {"start": 853.62, "end": 881.49, "text": "Which I think has also been discussed previously. How do we detect and respond gracefully to differences in test distribution? Oh, yeah. Sorry. That example's wrong. The example I meant to give is the robot's trained on the factory floor. It uses a bunch of harsh chemicals. And then it works in the office. And that doesn't make sense. So generally, these have gotten a lot of attention. And so those sections of our paper are more reviews, but also with pointers to how this can be applied to emerging"}, {"start": 881.49, "end": 907.14, "text": "Emerging deep learning applications. So just going a little bit quickly through it. There's substantial prior work in this area. I think there's a lot on adapting risk-sensitive performance metrics to deep learning, simulated exploration, especially with the advent of model-based reinforcement learning in deep RL setups, and setups that use bounded exploration to ensure ergodicity."}, {"start": 907.14, "end": 932.98, "text": "ML systems can fail silently and unpredictably when the test distribution differs from the train distribution. This is a Roomba that ate someone's hair. With classifiers, in some sense, things can't be very bad. But for people who remember this, a classifier can do something incredibly offensive if it's not trained in the right way and you give it the wrong inputs. RL agents can act on really bad."}, {"start": 932.98, "end": 962.78, "text": "Perceptual info. We can add ad hoc restrictions to make sure that we kind of detect this. But I think ultimately we need and should be able to have a machine learning solution because if you're outside the training distribution, you have enough information to detect that your inputs are different from where they were before. And there's been a lot of work on this. I'm running short on time, so I'll just summarize it very quickly. But kind of covariate shift assumptions that make assumptions about the probability of the output given the input being the same and only the inputs changes and the converse of that."}, {"start": 962.78, "end": 992.67, "text": "I think something very interesting is broadening the train distribution and understanding if I train on a few different types of data, how much of that does it take to generalize? How do I determine when I've generalized well? And also how to respond in agents, which I think has been a very understudied topic. There's been a lot of study of how to detect being out of distribution, how to respond to it in classifiers, but agents can do a whole lot of things. If I see a chemical that I've never seen before,"}, {"start": 992.67, "end": 1017.68, "text": "Then I, as a human, can experiment with that chemical in a controlled environment, even if that's more expensive, instead of doing something else. Or I can try and access more training data. So I think there's a whole world of agents remedying their own uncertainty and knowing what they don't know. This is some stuff we've started proposing experiments, which I won't go through in great detail."}, {"start": 1017.68, "end": 1047.02, "text": "The thrust of a lot of these is kind of looking for large suites of simulated environments where we can have kind of very broad setups where we can test agents in a way that they don't overfit and they can actually be shown to address these tasks. And just as simulated environments in video games have been very important for Go and Atari and reinforcement learning in general, my hope is they can be very important for safety. And in fact, within a couple days of publishing the paper, some folks at OpenAI implemented"}, {"start": 1047.02, "end": 1076.22, "text": "A couple of the ideas we had in OpenAI Gym, which is one of their open source tools for RL. And so finally, since there seems to have been kind of a tradition of everyone mentioning a little bit the long-term concerns, I did want to give my perspective, which is I do think that kind of long-term concerns about AGI and superintelligence can be seen as kind of extreme versions of accident risk. Not Terminator robots taking over, but kind of the more academic writers like"}, {"start": 1076.22, "end": 1105.31, "text": "Like Nick Bostrom, who talked about if you make a machine that makes paper clips for you, maybe it'll make so many paper clips that it takes over the world. So this is kind of an extreme and more fanciful version of accident risk. I think it's too early to work on this directly. I don't know what such systems would look like and what to do with them. But what I do feel strongly is that if we take a principled and general approach to the safety of today's systems, that is the correct response to concerns about the far future."}, {"start": 1105.31, "end": 1131.44, "text": "Andrew Ng, who was my former boss, had a quote, worrying about superintelligence today is like worrying about overpopulation on Mars. But I'd like to propose a bit of a modification to that, even though I think I agree with a lot of it, which is it may not be worth worrying about overpopulation on Mars today, but we can and should study overpopulation on Earth. That is a serious issue we can think about. And if we do that right, then a lot of what we learn may someday apply to Mars, if and when we do go to Mars."}]}