{"url": "https://media.transistor.fm/992cbd00/153e53bd.mp3", "page_url": "https://share.transistor.fm/s/992cbd00", "title": "Concrete Problems In AI Safety With Dario Amodei And Seth Baum", "show_or_channel": "Future of Life Institute Podcast", "date": "2016-08-30", "duration_s": 2602, "type": "podcast", "speakers": ["Dario Amodei"], "confidence": "high", "notes": "Recorded around move from Google Brain to OpenAI; with Seth Baum.", "people": ["dario-amodei"], "id": "2016-08-30_future-of-life-institute-concrete-problems-in-ai-safety-with-dari", "audio_duration_s": 2601.2373125, "worker": "jc-w15", "transcribe_s": 32.0, "model": "large-v3", "segments": [{"start": 0.37, "end": 35.02, "text": "From the FLI Audio Files, I'm Arielle Kahn with the Future of Life Institute. This summer, I watched all four of the White House Symposia on AI Research, which were designed to give AI researchers and others in the field a chance to discuss what the status of AI research is now, how it can help move society forward, and what risks we need to figure out how to address. There were many, many hours of talks and discussions, and I couldn't help but notice how often researchers got up,"}, {"start": 35.02, "end": 63.15, "text": "And were very adamant that talks should only be about short-term AI issues. They insisted that long-term AI concerns weren't something that needed to be addressed right now. They didn't want to worry about advanced AI, and they definitely didn't want to think about superintelligence. But then, as soon as they started talking about their research, many of the issues that came up were related to things like control, and transparency, and bias. Now these are obviously issues that need to be addressed for short-term narrow AI."}, {"start": 63.15, "end": 91.92, "text": "But there are also concerns that need to be addressed for long-term, more advanced AI. And so I started to wonder why we were so worried about focusing on short-term versus long-term artificial intelligence. Now, somewhat famously, in the AI world at least, Andrew Ng, who is with Baidu, has compared worrying about long-term artificial intelligence issues to worrying about overpopulation on Mars. And I guess my reaction is that overpopulation is an issue. And it's an issue that we need to address."}, {"start": 91.92, "end": 118.9, "text": "And if we can solve overpopulation issues now so that we don't have to worry about them later, why would we not do that? I realize there are probably some cost issues or other reasons that planning ahead is more difficult, but it seems like a really strange stance to me that we shouldn't try to solve a current problem now so that it doesn't crop up again in the future. I figured the best way to try to understand what's happening would be to turn to two people in the field."}, {"start": 118.9, "end": 147.47, "text": "So I have with me Dario Amadei, who had been working at Google Brain, where he published a well-received paper titled Concrete Problems in AI Safety. And he's recently moved to OpenAI, which is focused on AI research and safety issues, as well as on understanding the social implications of AI development. I also have Seth Baum, Executive Director of the Global Catastrophic Risk Institute. Much of Seth's own research is on ensuring AI safety. Seth, I'll turn this over to you now."}, {"start": 147.47, "end": 176.4, "text": "Thanks, Ariel. And these are really good questions. And we saw it at the White House Symposia. I attended one of those. We also see it in a lot of different conversations across the AI world. And I think it's a really important one because while we might have some reasons for being especially interested in or concerned about the long-term AI issues, at the same time, a lot of people are just more focused on short-term questions. And so it's fair to ask."}, {"start": 176.4, "end": 203.41, "text": "What we can do to get action that works for both short-term and long-term AI issues. And so that's why I think it's good to have this conversation, especially good to be having this conversation with Dario Amadei, who has recently published a paper on this topic, or rather on the topic of AI safety in general and the issues that come up in AI safety, whether it's short-term."}, {"start": 203.41, "end": 227.84, "text": "or long-term safety. Now, Dario, I want to get to the paper in a second, but first, as a little background for our listeners, maybe you could say a little bit about how you became interested in the AI safety topic in the first place? Sure. I've actually been working in the AI field itself for only a couple years, and before coming to OpenAI, I worked at both Baidu and Google, and was drawn to the field by..."}, {"start": 227.84, "end": 256.59, "text": "What I see is kind of the incredible advances, particularly in deep neural networks, to solve problems like vision, speech, and language. And I was involved in research in all of these areas and found it very exciting. But one thing I definitely noticed, I think particularly with deep neural networks and with powerful ML systems in general, is that there are many ways in which they can be very accurate. A speech recognizer can tell you something and can be almost as good as a human, but they can also be somewhat brittle. If I train speech data on..."}, {"start": 256.59, "end": 284.62, "text": "On humans speaking in a clean background with unaccented American speakers, they'll perform great. But then if I test the same data on accented speech or noisy data, it performs terribly. And as these systems get deployed more into the world, having systems that fail unpredictably is not a good thing. And I think that impression was reinforced as I continued my work at Google, where you have issues with Google's photo systems, which was based on a neural net classifier, ended up..."}, {"start": 284.62, "end": 311.26, "text": "Accidentally classifying people of color as gorillas, which of course is an incredibly offensive thing to do. Now, the neural net didn't know that it was offensive. You know, it was a combination of a problem with the classifier and a problem with the training data, but that machines can lack context for something, but the classifier that's produced by the machines can be something that has some very bad real-world impacts. And if it does something that is not what we intended to do, it can be very harmful."}, {"start": 311.26, "end": 339.82, "text": "I think in the last year, I've become particularly interested in reinforcement learning, which is a branch of machine learning that's concerned with interacting with the environment in a more intertwined way and is often used in things like robotics, self-driving cars, autonomous systems. We've seen there was a recent announcement from Google that it's used to control the power in their data centers. So once you're actually interfacing with the world directly and controlling direct physical things, I think the potential for things to go wrong, which often are quite mundane."}, {"start": 339.82, "end": 369.3, "text": "So I became more and more interested in if there were principled ways to reduce the risk or find some theoretical basis for guaranteeing that something bad won't happen as we start to deploy these systems more and more into the world. That was kind of where sort of my interest in AI safety started. And certainly there is the thought that these systems are advancing very quickly and the more powerful they get, the higher the stakes are for something that might potentially go wrong. So as someone who really..."}, {"start": 369.3, "end": 394.05, "text": "Really wants to think about the social impacts of what I'm working on. This seemed like just a very important area. Yeah, that makes a lot of sense. And it does seem that if you look at the systems themselves and what they're doing, that it just naturally follows that they can fail in these types of ways. And we saw it with the Google gorilla incident, which is kind of a classic example of..."}, {"start": 394.05, "end": 418.74, "text": "As you put it, the AI system failing unpredictably. And a lot of my research is on the risk and especially the policy end of AI. And the same issue comes up in that context because who do you hold liable? Do you really hold liable the company or the computer programmers who built this code because they didn't want to do that? And so for the legal..."}, {"start": 418.74, "end": 447.6, "text": "Management of these types of software, it's a challenge because you don't want to hold these people liable. They weren't intentionally causing these harms yet at the same time because these are systems that are almost by design bound to behave in unpredictable ways. Then we're just going to see this happen more and more. My impression, at least, is that that is a little bit of your motivation for, you wrote a paper recently called Concrete Problems in AI Safety that seems to take on some of these topics."}, {"start": 447.6, "end": 476.05, "text": "Maybe you could say a little bit more about the paper itself and why you wanted to write it and how it contributes to these issues. Yeah, yeah. So as I got more and more interested in these safety issues, you know, I did what any researcher does and looked into the literature that had been written in the machine learning literature so far about these problems. And there actually was a fair amount of literature on various subsets of the set of things I was worried about."}, {"start": 476.05, "end": 505.22, "text": "And, you know, in many cases there was a substantial literature, but there was, I would say, kind of a scattered nature to it where, you know, and I'll get into some of the specific problems that I talk about in the paper in a bit, but the four or five different problems that I was kind of thinking in my head are ways to classify things that could go wrong with an ML system were often written in kind of parts of the literature that were not very related to one another and often were kind of very specific to particular applications."}, {"start": 505.22, "end": 528.32, "text": "I felt that what would help a lot would be something that was kind of a combination of reviewing all the existing work that has been done in one place and also kind of having an agenda that talks about what needs to be done in the future. In particular, a lot of this work had been done quite a while ago, and so really writing this review and agenda with a view towards..."}, {"start": 528.32, "end": 557.06, "text": "The cutting edge ways in which machine learning has advanced in the last three or four years with neural nets doing vision, speech, language, game playing, autonomous driving, a bunch of other applications that I felt like a lot of the thinking about making systems safe and controllable and robust and predictable could really use an update in light of a lot of these advances. And then there was kind of another stream where for a while I'd been aware of the work of people like Nick Bostrom."}, {"start": 557.06, "end": 586.21, "text": "And Eliezer Yudkowsky, who come from outside the ML field, have been warning about these very long-term considerations involving AIs that are smarter than humans. I've read their work, and my attitude towards it has always been that I found it thought-provoking. But if as a researcher I wanted to think about ways in which machine learning systems can go wrong, it's very important, at least for now, to stick to the kinds of systems we can build today. And then if future scenarios do come up..."}, {"start": 586.21, "end": 614.64, "text": "Then we're better equipped to think about those scenarios."}, {"start": 614.64, "end": 642.37, "text": "With which we sat down and wrote the paper. And then it turned out that not just me, but there were several other authors in the paper. So my main co-author was Christopher Ola, also on the Google Brain team, who's done a lot of work on visualization and blogs on teaching machine learning to a wide audience. Then we had some collaborators from Berkeley, Paul Cristiano from Stanford, Jacob Steinhardt from OpenAI before I joined John Cholman and another Google Brain researcher, Dan Manet."}, {"start": 642.37, "end": 671.65, "text": "We've all found that we have the same general perspective and vision on it relative to what happened in the space before. So we all got together and spent a bunch of time and eventually we produced this paper. Good. So perhaps you could take us into the paper a little bit. There are five concrete problems. You have avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift, which..."}, {"start": 671.65, "end": 698.62, "text": "Makes sense when you get into the paper, but to just hear him spoken like this is kind of vague, kind of ambiguous. Maybe we could go one at a time through each of these five problems, and he could explain just in basic terms what they mean. So starting with avoiding negative side effects. So actually, before going into the five problems in detail, one frame that I thought of as a little useful for thinking about the problem that we describe a little bit in the paper is that"}, {"start": 698.62, "end": 725.34, "text": "We actually split the five problems into three general categories. And I at least have found it useful to start by thinking about those categories and then going into the problems. If you're trying to build a machine learning system, one of the most important pieces of a machine learning system is the objective function, which really defines the goal of the system or the way of judging whether it's doing what you want it to do. For example, if you're building a system that's supposed to recognize speech."}, {"start": 725.34, "end": 751.66, "text": "Then you might measure what fraction of words it gets correct versus incorrect, the word error rate. If you're building an image classification system, the objective function is the fraction of time that it identifies the images being in the correct class. If you're building a system to play Go, like the AlphaGo system that DeepMind worked on, then it's the fraction of games that you win or the probability that you win a game. So in thinking about when you build a machine learning system and..."}, {"start": 751.66, "end": 781.04, "text": "For whatever reason, the machine learning system behaves in some way that you didn't intend and you don't like. One of the ways I found it useful is to think about exactly where in the process things go wrong. So I think the first place that things would go wrong is if your objective function wasn't actually right. And I'll go into that in a little bit of detail later. But the idea is that you were putting pressure on the machine learning system to do a particular thing and rewarding it for doing a particular thing. And unbeknownst to you, that was actually the wrong thing to do."}, {"start": 781.04, "end": 807.81, "text": "And the system ends up behaving in a harmful way because you had in mind that it would behave a certain way. You tried to formalize that with an objective function. The objective function was the wrong objective function. The second class is if you do know the objective function, but it's very expensive to evaluate the objective functions. This might be human judgment, where we have a limited budget of a human supervising and checking in with the AI system."}, {"start": 807.81, "end": 837.12, "text": "And the AI system has to extrapolate. And so it might do the wrong thing because it hasn't fully seen what the correct objective function is. And the third problem is that a system has the right objective function, but all machine learning systems, as indicated by the phrase machine learning, have to learn. And there is a concern that while a system is in the process of learning, while it doesn't understand the world around it or the tasks that it's supposed to do, it could do something harmful while it's not fully trained."}, {"start": 837.12, "end": 866.98, "text": "That, at least to me, from the perspective of a researcher, has been kind of a natural way to decompose the problem. So now maybe with that, we can kind of go into the five problems. Before we dive into the five, let me just try speaking these back at you to make sure that I understood them correctly. The first one was just the wrong objective function. You gave it goals that, in retrospect, turn out to be goals that you're not happy with. And it's fulfilling those goals despite the fact that you wish it was fulfilling some other goals."}, {"start": 866.98, "end": 889.52, "text": "Correct. Correct. Now, the second one you said, the objective function is expensive to evaluate. That means you gave it good goals, and it is working towards those goals, but it's struggling. And instead of fulfilling those goals, it's doing something else that's causing harm. Right, because it has a limited ability to assess what the right goal is."}, {"start": 889.52, "end": 918.72, "text": "The right goal might be, is a human happy with what I'm doing? But I can't, in every little action that I'm taking, ask if a human is happy with what I'm doing. And so I might need some kind of cheaper proxy that I think predicts whether a human is happy. And if that goes wrong, then the system could do something unpredictable. Okay, good. And the third one you said, harms that occur during the training process. Yes. Maybe say, what is this training process and how can you get harms during it?"}, {"start": 918.72, "end": 946.83, "text": "Sure. So maybe we can go into that more, the two problems in that subcategory. But the general idea is that you can kind of think of a child when they don't really understand the world around them. If they don't know that, you know, if they press this button on a stove, a fire will turn on, it might burn them, it might burn someone else. The child might be using the right process to learn about the world. But if they've never touched the stove before, they don't know what's going to happen and they don't know enough to know that they might hurt someone. Okay. Makes sense to me."}, {"start": 946.86, "end": 976.37, "text": "Shall we go ahead and dive into the five? Yeah. So the first one that you asked about was avoiding negative side effects. So this is one problem under the subcategory of having the wrong objective function. One way to introduce this is to say that in some sense, machine learning systems are very literal minded, right? If you tell them to do X, they'll do kind of exactly and literally X. So the example we gave in the paper was, let's say I have a cleaning robot that is trying to, you know, move a box from one side of a room to another."}, {"start": 976.37, "end": 1001.6, "text": "And just in this kind of very simple example, if all it was trying to do was move the box, I might give it an objective function that basically said you get points, you get reward for moving the box from one side of the room to the other, and that's all that matters to you. But if we literally just give it that one objective, then actually we're implicitly telling it that anything else that's in its environment, it doesn't care about. So if there's a vase in its path,"}, {"start": 1001.6, "end": 1030.08, "text": "I have to make very sure that..."}, {"start": 1030.08, "end": 1056.43, "text": "I'm driving my child to school that I don't run someone over or break the car or do something else, right? I'm never explicitly just doing one task. I'm always doing one task and making sure that the rest of the world is okay and I don't do anything really damaging while I'm doing it. And as a human, I have common sense and I know this, but our machine learning systems, at least at present, are not at the place where they have common sense. And so I see a potential for various things to go wrong in this case."}, {"start": 1056.59, "end": 1084.96, "text": "That to me sounds like the classic genie story, right? So the genie in a bottle, you rub it, you get your wish, and it gives you your wish, exactly what you asked for, whether you like it or not. And what you're saying is that a machine learning AI system has that same sort of taking things very literally. Or could, at least the systems that we built today often have that property. I mean, I'm hopeful that someday we'll be able to build..."}, {"start": 1084.96, "end": 1113.57, "text": "Systems that have more of a sense of common sense. We talk about possible ways to address this problem. But yeah, I would say it is like this genie problem for the specific case of one of the specific things that can go wrong is that the world is big and that it's very easy when you're training the machine learning system to focus on only a small aspect of it. And that gives you a whole class of things that can go wrong. Okay, sounds good. So let's move on to the second one. Avoiding reward hacking. What's that all about?"}, {"start": 1113.57, "end": 1137.58, "text": "So reward hacking is this kind of situation where you write down an objective function, and it turns out that your objective function was kind of trying to capture the ability to do something hard, where you felt like in order to achieve this objective function, it would need to do some hard task that is the task you're trying to get the machine learning system to do."}, {"start": 1137.58, "end": 1163.84, "text": "But often there's some way of cheating the objective function that's been written down. So the example we give in the paper is if you have a cleaning robot that's kind of trying to clean up all the messes it can see, and you decide to give it an objective function that says, well, how much dirt can you see around you, right? You might think that would be a good measure of how clean the environment is. Then the cleaning robot, if it were designed in a certain way, could just decide to close its eyes."}, {"start": 1163.84, "end": 1191.46, "text": "And, you know, again, there's this thing that machine learning systems can be literal-minded, at least in the way we design them today. And..."}, {"start": 1191.46, "end": 1214.5, "text": "There's all types of things that can go wrong. And in the paper, we discuss some general factors that lead to this. And one factor is when the robot or the machine learning system is not able to see everything in its environment. It's very easy for the wrong kind of objective function to give it incentives to hide aspects of the environment from itself or others, like the shoveling things into the closet case."}, {"start": 1214.5, "end": 1244.59, "text": "So we discuss a few of the general ways this can happen and a few thoughts and recommendations for designing objective functions where this is less likely to happen. I'm reminded of those cute pictures you can see on the internet of kids taking quizzes. And it'll be something like there'll be this long, difficult math problem. And then below it, it says, write the answer here. And the student just writes the words, the answer below it and tries to pass it in because they don't know how to actually do the math problem."}, {"start": 1244.59, "end": 1271.44, "text": "We need to get our AI systems to not behave like mischievous little kids. Yeah, there was the blogger who wrote about our work. I think it was Cory Doctorow. And he said, a lot of the problems that I've read in this paper remind me of issues in child development and child psychology. And there is a sense in which a lot of these systems are kind of a bit like savants, right? They're like small children who don't have enough context or common sense."}, {"start": 1271.44, "end": 1298.26, "text": "to know how to do quite the right thing, but are at the same time very, very voracious learners who can process a lot of information. So I do see a lot of commonalities there. Makes sense. Okay, so let's move on. The next one is scalable oversight. Yeah, so the kind of the example we give there is, let's say that we have a cleaning robot again, and it's trying to clean up a room, but there are some objects in the room that might belong to a human."}, {"start": 1298.26, "end": 1324.83, "text": "And there's a surefire way to get the right objective function, which is if every time you find an object, you ask a human, you ask the human, does this object belong to you? You ask every human who could possibly own it, does this object belong to you? Then you'll always do the right thing. But a robot that does that is impractical. No one would sell any robots to do that because if that's going to happen, I might as well just clean it myself, right? I don't want my robot asking me questions every two minutes while it's trying to clean my house. So can we find ways where..."}, {"start": 1324.83, "end": 1351.09, "text": "The robot maybe only asks me like the first few times and gets a good sense of what kind of stuff I would actually own versus what kind of stuff that's okay to throw away. Maybe looks at cues of where I leave the stuff. And so the way to state the problem is if the robot tries to do this, there is the risk that it will throw away things that I really would have wanted. And the solution is, are there ways we can get the robot to think about from repeated experience being able to predict?"}, {"start": 1351.09, "end": 1380.34, "text": "The true objective function, which is what really belongs to me and what I really want thrown away without actually having to ask me every time, which might destroy the economic value of the system. Yeah, I mean, that seems like something that humans face all the time, right? You're cleaning, do you really know whether they want that thrown away or not? I mean, sometimes if it's a candy wrapper, it should be pretty obvious unless it's a, maybe it's a candy wrapper with a winning lottery ticket printed on the inside and some prize competition that the candy company had."}, {"start": 1380.34, "end": 1407.01, "text": "But it seems really easy for robots or AI systems to make the same sorts of mistakes. Yeah. And do you think it's particularly difficult to train an AI to get those sorts of questions right? I mean, that seems like maybe I don't know the systems well enough, but that seems like something that we should be able to train an AI to figure out without too much difficulty. In the particular case of the cleaning robot, I'm pretty optimistic."}, {"start": 1407.01, "end": 1433.34, "text": "But I think, you know, it's designed more to be kind of a parable that illustrates that often there are aspects of human preferences that are quite subtle and that getting all of them right without an unacceptable amount of communications with humans or halting the workflow of the machine learning system can often be quite subtle. A human might be able to look at something like a concert ticket."}, {"start": 1433.34, "end": 1458.75, "text": "Okay, sure. Now, let's move on. The next one is safe exploration. What's that? Yeah, so this is actually, this is a problem that's been worked on a lot in the machine learning community. And so..."}, {"start": 1458.75, "end": 1485.17, "text": "Our work here was more to summarize prior work, but also point towards how work in this area could be integrated with a lot of the advances that we're seeing in robotics and whether it's possible to step up the reach of work in this area. So the basic idea here is that particularly in reinforcement learning, which I mentioned earlier is a branch of machine learning that deals with systems that interact with the environment in a very intertwined way. There's a trade-off between exploring and exploiting."}, {"start": 1485.17, "end": 1510.38, "text": "Between doing the thing that I think is best right now and understanding my environment better, which might lead to me understanding it better and realizing that there are even better things that I can do. But the problem is that when I'm exploring an unknown environment, often there are aspects of it I've never dealt with before, so I can do something dangerous without knowing what I'm doing. The example we gave with the cleaning robot is maybe it's never seen an electrical outlet before."}, {"start": 1510.38, "end": 1539.42, "text": "It wants to experiment with cleaning strategies and it tries to stick a wet mop in the electrical outlet. So obviously this is just going to be really bad for the robot. Another example that has actually come up in actual robots people have built is robot helicopters. So the idea is, you know, if I want to like train my robot helicopters to fly properly, I want to use reinforcement learning to train how to fly. One problem it can have is that, you know, if it's experimenting with spinning its propellers, it doesn't really understand the dynamics of flying very well."}, {"start": 1539.42, "end": 1568.53, "text": "If it does something bad and ends up crashing, it could break its propeller or break its control system or something. And then you can't use the robot helicopter anymore, right? The system is broken and you need to get a new one. And the designers of the system won't be very happy. And yet the system needs to learn somehow. And so again, this is a problem children encounter, right? Children need to try things on their own to understand what works and what doesn't. But it's also very important that they don't do things that are truly dangerous that they couldn't recover from if it went wrong."}, {"start": 1568.53, "end": 1597.41, "text": "To some extent, children have an instinct for this. Part of the role of parents is to keep children from going into truly dangerous situations, but it's something that our machine learning systems currently grapple with and I think are going to need to grapple with more and more. That seems like something that we all grapple with on a pretty regular basis. For myself, as an academic, I'm constantly worrying about whether I'm spending too much time thinking about stuff."}, {"start": 1597.41, "end": 1622.67, "text": "And researching it and learning more and so on versus just going ahead and writing what I have and forming an opinion and getting out there and saying what I have to say on the topic. And it's a dilemma, right? How hard do we try to figure things out before we do things? And the tech community, they have the old Facebook saying, move fast and break things, which..."}, {"start": 1622.67, "end": 1652.19, "text": "For some contexts, it may work well. And for other contexts, it does not work so well. And actually, Facebook has changed it. It's now move fast with stable infrastructure, something that sounds more responsible and not nearly as catchy. And so, yeah, I guess an AI would have to face the same sorts of issues, right? Yeah. I mean, I think the problem of machine learning and AI is to get machines to do some of the same tasks that humans do."}, {"start": 1652.19, "end": 1682.53, "text": "So in some sense, I think it's not surprising that in doing some of the same tasks humans do, they run into a lot of the same problems that humans do. Okay. So we got one more of the concrete problems, and it's called robustness to distributional shift. Yeah. This is the idea that often if you have a machine learning system, you know, we have the notion of kind of training data and test data. So a machine learning system often gets trained on one particular type of data, but then when it's deployed in the real world,"}, {"start": 1682.53, "end": 1712.37, "text": "It often finds itself in situations or faced with data that might be different from the data that it was trained on. So our example with the robot is, let's say we have a robot that's been trained to clean factory work floors, and it's kind of learned that it should use harsh chemicals to do this, that it needs to avoid lots of these metal obstacles. If you then deploy it in an office, it might engage in some behavior that's inappropriate, right? It might use chemicals that are too harsh. It might not understand how the office is set up."}, {"start": 1712.37, "end": 1740.58, "text": "Etc, etc. I think another example of this actually is the gorilla example that occurred with Google a year ago, where one of the problems with that photo captioning app was that a lot of its training data had been trained on Caucasian individuals, and it had seen monkeys, but it had never seen an individual with different skin color. And so it made very inappropriate inference based on insufficient training data."}, {"start": 1740.58, "end": 1766.69, "text": "Interest in robustness to distributional shift is in trying to both detect and remedy situations where you're seeing something that is different than what you've seen before, right? The photo caption system should have said, this is something that doesn't actually look like anything that I've seen before in any of the classes that I've seen before. It's actually something different. And I should be very careful of what class I assign this to."}, {"start": 1766.69, "end": 1795.39, "text": "Because I don't have high confidence about the situation and I'm aware that I'm facing data that's different from the data that I was trained on. It's not always possible to respond appropriately to a totally new situation or totally new percepts that I might receive. But it seems like it is possible to recognize that what I'm seeing is different from what I've seen before. So that section of the paper discusses how to recognize that and how to be appropriately cautious once you've recognized it."}, {"start": 1795.39, "end": 1823.49, "text": "Really interesting to me, just listening to you talk through these different challenges of designing an AI to behave the ways we would want it to behave, how similar, to me at least, it sounds like child development and human behavior and challenges that we all face. It makes me feel like these artificial intelligence systems are already not so different from us, at least in these types of ways."}, {"start": 1823.49, "end": 1849.71, "text": "Well, I would definitely say that certainly the systems that we're building today are very limited systems. I do want to emphasize that we are nowhere near building systems that can replicate the incredible range of behaviors that humans are capable of. However, I would say that within the particular tasks that we assign to machine learning systems, yeah, I think many of the problems that they face in..."}, {"start": 1849.71, "end": 1872.96, "text": "Learning those specific tasks often, not always, have analogies to the challenges that humans face in learning those tasks. Okay, so they're still not at all as capable as we are across the board, but they still face some of the same challenges we do. Yeah. Okay, very good. So I want to bring it back to the conversation we started out with, which is on..."}, {"start": 1872.96, "end": 1899.73, "text": "What we saw at the White House Symposium, what we see in other contexts about this short-term AI versus long-term AI issue, and people care more about short-term and long-term and so on. With that in mind, I'm curious what the reaction has been to your paper, right? Do people say, oh, this is crazy, or do these ideas seem reasonable to them? Yeah, actually, the reaction has been extremely positive, more so than I even anticipated."}, {"start": 1899.73, "end": 1928.3, "text": "There's a few different communities that read and looked at our work. The first was because it was published on the Google Research blog and ended up getting covered by the media. And their reaction was actually quite positive. I mean, most of the stories had titles like Google gets practical about AI concerns or Google addressing AI concerns responsibly. The idea was that it was kind of a set of serious engineers who were really sitting down to think about just very specifically what can go wrong in machine learning systems and how can we..."}, {"start": 1928.3, "end": 1954.69, "text": "So I feel the media understood actually pretty well, which surprised me a little bit. What about from the AI community? Yeah, so the AI community was also extremely positive. So even people like Oren Etzioni, who have been vocal spokespeople against more long-term concerns about..."}, {"start": 1954.69, "end": 1978.27, "text": "AI and risks of AI, they were very positive about this paper. Oren Etzioni was quoted in one of the news articles as saying, these are the right people asking the right questions. And I think a lot of the doubts that people like Etzioni have had about long-term AI risk has just been a kind of vagueness. Well, how do you work on this? And I think the reaction to our paper was very positive because it gave..."}, {"start": 1978.27, "end": 2006.56, "text": "Problems posed in a way where you could actually sit down and write a paper on it. And actually, by the way, we intend to follow up the paper. This was more of kind of an agenda paper, but we intend to follow it up by writing papers trying to address these problems in actual real systems. And that's one of the things I'm working on at OpenAI, and we're going to continue to collaborate with Google on it. So I think that the concreteness part of it and the practicality and the promise of real empirical work that I hope we can deliver on."}, {"start": 2006.56, "end": 2034.86, "text": "made a lot of the AI community seem pretty excited about it. And then finally, the community of people that have been worried about long-term AI risks. I think the reaction there was pretty positive as well, even though the focus of the paper was on shorter-term issues. As Ariel pointed out at the beginning of this, conceptually, a lot of the longer-term risks that they're worried about can be seen of as"}, {"start": 2034.86, "end": 2060.03, "text": "instances of some of the problems we've talked about, particularly the negative side effects and reward hacking, but actually all of them are things that when I try and think about what someone like Nick Bostrom is talking about, I think what they're talking about is the kinds of problems we're talking about in the concrete problems paper, if you had those problems with an extremely powerful AI system that was even more powerful than humans."}, {"start": 2060.03, "end": 2084.51, "text": "Then I think that's how you get to some of the scenarios that Bostrom is describing. And I definitely have a disagreement with the AI safety community, which is it's not that I don't think we may face these extreme scenarios eventually. And I'm glad that there's someone thinking about them. But I at least am most interested in thinking about problems that we can attack empirically today."}, {"start": 2084.51, "end": 2113.6, "text": "I hope that those problems that we attack today will start to shed light on the longer term issues that we have. Again, if we really work on things like reward hacking and avoiding negative side effects, I think if we work on them in the right way, there will be a lot of relevance to the scenarios that people worried about AI risk are worried about, and that eventually many things they're talking about or the things they write about, maybe they'll become very relevant someday. But my difference is more tactical, that I just see a great deal of importance to having an empirical feedback loop."}, {"start": 2113.6, "end": 2141.01, "text": "This is a problem I think a system might have. Let me test it. Oh, it has this part of the problem, but not this part of the problem. Let me do another iteration on it. Just in research and science in general, I feel we've gotten a lot of mileage out of the empirical feedback loop. And so that's something that I emphasize a lot. I'm really glad to hear that the response has been so positive. This seems to me like the sort of clever solution that we need for problems like AI safety that can resonate across."}, {"start": 2141.01, "end": 2167.6, "text": "Seemingly disparate audiences, and we've had a lot of disagreement between the people who are worried about the future superintelligence risk versus the AI researchers who are out there building the new systems today. And my impression is that the difference of opinion between these two groups only goes so far in that both of them, as far as I can tell,"}, {"start": 2167.6, "end": 2183.73, "text": "Genuinely do care about the social impacts of their work. It might not be the core focus of everyone's attention. And I think this is an issue that needs to be addressed within the AI community. Stuart Russell has a great line about how..."}, {"start": 2183.73, "end": 2209.06, "text": "Thank you for having me."}, {"start": 2209.06, "end": 2235.22, "text": "Listened into AI researchers. A lot of them do actually care about the social impacts of their work. They're just not sure about this super intelligence thing. It's remote into the future. It's speculative. It maybe even sounds a little odd. And it's just so far removed from the systems that they're working on. And so to see opportunities to address the general AI safety concerns that may also..."}, {"start": 2235.22, "end": 2264.3, "text": "It makes sense to me that they would respond positively to that sort of message. And I wonder if there's anybody, especially within the AI research communities, that is pushing back against it and saying, no, we should just focus on building AI systems that are more capable and we shouldn't worry about these safety problems. Have you gotten that at all? I don't think I've ever had someone say,"}, {"start": 2264.3, "end": 2292.66, "text": "I think there is probably a healthy debate to some extent in the machine learning community about which social impacts we should care the most about. Some of my colleagues are very interested, and I am actually too, in kind of things like the economic impact of machine learning systems or in fairness. And I think definitely people differ in how much they choose to focus on each of these issues. But I haven't really encountered anyone who says,"}, {"start": 2292.66, "end": 2322.24, "text": "But we shouldn't think about any of these issues or who says this is the only issue that we should think about. I think when properly explained, the risk of AI systems doing things that we didn't intend, everyone says, yes, that's something we should prevent. The risk of AI systems treating people unfairly, that's something, yes, we should prevent. The risk of bad economic impacts of AI, everyone says, yes, that is something we should prevent. Internet security issues that could arise with ML, everyone says, yes, that's something we should definitely prevent. Different people are interested in."}, {"start": 2322.24, "end": 2350.83, "text": "And for some people, this isn't a personal research interest of them, but I actually haven't encountered anyone who says, no, I don't think anyone should work on these things. Maybe such people do exist, but I haven't met any. And maybe that's the bigger challenge with this. It's not people who actively push back against work on these problems, but people who just essentially ignore it. I remember from my own engineering days, and I really like to bring up..."}, {"start": 2350.83, "end": 2379.95, "text": "Social impacts and social issues related to our research. My fellow engineers, they would listen to me and then they would basically be like, okay, that's nice. Now get back to work. Because in their minds, thinking about the social aspect of it was someone else's job. And so it's easy to imagine AI researchers not really disagreeing with the sorts of things that you're saying, but just thinking maybe this is somebody else's responsibility to worry about it. Do you see that at all?"}, {"start": 2379.95, "end": 2406.19, "text": "Thank you for having me."}, {"start": 2406.19, "end": 2430.35, "text": "A decent fraction of the field takes that attitude. I'm fine with it. My concern is more that we as a field collectively that we're on the issue, that we have enough people within the field who do want to think about these issues. If there's no one or too few people in the field who wants to think about these issues, then I think that's a problem because I think it's our responsibility as researchers to think about the impact of the research that we're doing."}, {"start": 2430.35, "end": 2457.73, "text": "But if a particular person says, that's not my cup of tea, that's not my focus area, I'm fine with that. I think that's the way research works. But I would say that right now, the fraction of people doing this, at least a year ago, I would say was too low. Now, I think, thankfully, we're starting to get more and more people into this and maybe getting to a healthier place. Okay, that was what I was going to ask you, because I presume based on the fact that you're speaking up on this topic, that..."}, {"start": 2457.73, "end": 2482.72, "text": "Do you sense that there should be more work going on on this? And it seems like the paper you wrote was, as you put it, an agenda, a call for action, a call for research on these topics. But it's very encouraging to hear that you think that the fraction of people working on these safety problems is going up. Would you still say that it should be going up more? Or do you think that we're actually reaching a pretty comfortable place right now?"}, {"start": 2482.72, "end": 2512.06, "text": "I mean, it's all kind of coming into place, right? Since I've joined OpenAI, I've had a number of people say, you know, I'm interested in these topics. I want to work on these topics. Well, new people coming into the ML field and people who have been in there in a while. So I actually don't know where things will end up. And my main goal is just to get some good technical research done on this topic. Then we'll see if there needs to be more people in the field. And my hope is the usual dynamics of if there's a lot of interesting results to be found in one place, then more people come into the field."}, {"start": 2512.06, "end": 2540.59, "text": "If there's too many people working on something and some people go somewhere else, I'm hopeful that those normal dynamics will get us to a place where we're thinking responsibly. That may be too optimistic, but that's my hope. That makes sense to me, and let's hope that things do balance out there. In my experience, researchers don't necessarily always gravitate towards just where the right topics are, what research most needs to be done. You can end up with too many people crowding in one seemingly popular area."}, {"start": 2540.59, "end": 2569.6, "text": "But we'll see. I'm really glad to hear this and be great if these safety problems could then just solve themselves as more AI researchers work on them. Yeah, that would be my hope for what would happen. Okay, thank you. Any final thoughts you'd like to add to this conversation before we sign off? You know, I think my perspective is that empirical and testable work on unintended consequences of machine learning system is the best way to illuminate."}, {"start": 2569.6, "end": 2599.28, "text": "Okay, thank you. Ariel? Yeah, I want to thank you both for sitting down and having this discussion. I think this helps shed a lot of light, at least on the issues I saw at the White House Symposia. And it's been a really great overview of where we're at with AI safety research today. So Dario and Seth, thank you very much. Thank you. Thank you for having me. To learn more, visit futureoflife.org."}]}