[0:00:00] So I'll be talking and giving some background from my perspective and then Dario is going to come in right after that and then we'll go to questions for both of us at the very end. So just to get started, a lot of this talk is going to be about risks from AI because that's the area where I think we can make the biggest difference right now.
[0:00:30] But I'm really motivated by the potential of AI. I see a really large chance that AI will have a hugely positive transformative effect on society. I think you can already see some of that over the last 30 years or something. But I think maybe the best framing for the talk is going into AI with our eyes open, like trying to really assess the range of outcomes and possibilities that exist in this space, and just trying to consider
[0:00:59] What the different ways we could make a difference are without assuming that it's going to be all about risks or all about benefits or that there's some conflict between those. So all that said, I do think that the biggest opportunity we have for doing good today in this area is recognizing and dealing with risks. I just don't want to lose sight of the goal, which is putting society in a position where we can realize the benefits of AI and avoid the risks.
[0:01:26] Let me set up a timer so that I don't go over here. Stepping back, I think a productive way of looking at artificial intelligence is to think about things that have happened in history and try to figure out kind of what buckets artificial intelligence could fall into. And my general view is that we might see AI systems at some point in the future that are
[0:01:57] Really broadly transformative of society in general. And we might compare them to the agricultural revolution, the industrial revolution, or the advent of language and collective learning in humans. I think this is like the biggest scope that it makes sense to think about the impacts of AI on. And I think it's really worth considering because some things in history do turn out to have this kind of impact. Like there's going to be something that has this kind of impact.
[0:02:26] And if AI is something that could have this kind of impact, I think it's a possibility that's really worth taking seriously. So what kinds of reasons can I give for thinking that this might be the case? So I'll give a few sort of like paradigm cases that I use when I'm thinking about this that make me feel good about thinking that there's some probability that we might want to put AI on this kind of diagram, looking back on it in the future and say like, well, there was language.
[0:02:56] Agriculture, industry, and artificial intelligence, which is a pretty intense claim to make. So the first mechanism that I think about is we could see AI systems that are increasingly capable of doing all of the kinds of tasks that go into scientific and technological research and development. So right now, this mostly looks like amplifying the capabilities of human scientists and human engineers.
[0:03:23] But looking forward, it seems really possible that AI systems could autonomously perform most of the tasks that humans do today in that process. So you could imagine an AI system downloading papers from archive, could design experiments identifying key open questions about how the world works, and designing new technologies that take advantage of the things that it learns. And if that's the case, I think it is
[0:03:52] It's possible that we could see a really rapid increase in the rate of technological and scientific progress in the world because it would be possible to turn capital into new research and development without being bottlenecked on sort of like the really small number of human experts that are available to do this work. So this I think draws a really direct analogy to something like the Industrial Revolution where society sort of reorganized and was able to apply much more pressure
[0:04:19] To making progress on science and technology and where we saw really broad changes in the world as a result of this. Sort of broadening this, it seems like there's a reasonable chance that AI systems will eventually be cost competitive with humans at basically every economically valuable task. And that would be a pretty big deal. The picture that I tend to turn to is this one of fuel use during the Industrial Revolution.
[0:04:49] So you can think of this bottom line with biofuels as being sort of like labor that's mostly done by humans. And then as you see these new energy sources moving in, not only do those energy sources expand energy use a lot on an absolute scale, but as a proportion of energy use, non-biofuel energy sources came to take up more and more of humans' influence on the world.
[0:05:15] I think we could see a really similar picture with artificial intelligence where most of the things that we do right now that are economically valuable or really that influence the world right now go through humans, groups of humans working together, doing things. And we could see something like this where as AI systems can take on more and more of these tasks, more and more of our influence flows through AI systems that are doing things on our behalf.
[0:05:41] Our influence would then be more indirect than it is now and could ramp up much more quickly than it would if it's bottlenecked again on human expertise and the availability of human labor. So this, I think, by itself seems like it could be a pretty big deal. It seems worth paying attention to. But sort of looming over all of this is that in a lot of domains, AI systems don't like...
[0:06:08] Go up to some level of human capability, which is sort of a weird way to think about AI because AI systems have a really different sort of distribution of capabilities than humans do. It's not like a one-to-one mapping between human skills and AI capabilities. But in a lot of domains, AI systems reach human capabilities and then are able to go past them.
[0:06:32] There's not a lot of reason to think that humans or groups of humans are at or near peak performance for most of the kinds of tasks that we care about. So in this case we might see a much stronger version of the picture that we were looking at before and a strong fit with the Industrial Revolution where we saw
[0:06:52] Not just substitution for human labor, not just machines doing all of the things that humans did before the Industrial Revolution, but also machines that could do many things that humans just couldn't do at all in niches where that made sense. So these are sort of like three paradigm mechanisms that I like to think about in this area. And I think it's important to think about many different pathways that AI could take because it is so hard to predict.
[0:07:19] What future progress in AI will look like and where the biggest applications and impacts will occur. As a side note, I don't think that artificial intelligence, if you're looking at the world and at history and trying to figure out what the next thing on this timeline of transformative events is, I don't think it makes sense to just think that there's going to be one thing that is the only thing that can possibly be the next step.
[0:07:46] I think some other areas that make sense to keep an eye on are biology and synthetic biology. All humans have bodies. Dramatically changing our ability to understand and modify the human body and brain seems likely to have a pretty big impact on the world. Extremely cheap and scalable energy sources just historically have, there's a causal story that those cheap and scalable energy sources are behind the industrial and agricultural revolutions. So I think that's worth paying attention to.
[0:08:16] And then there are sort of like transformations that don't look as optimistic as historical ones. So large-scale nuclear war or pandemics, particularly synthetic pandemics, could be significant milestones in the story of human history, but not in a positive direction. So I think that artificial intelligence is not sort of like the only piece on the board if you want to think about history this way. But it is the one that I focus on. As far as predicting when this will happen,
[0:08:46] I think that this question is really hard to make any kind of headway on and feel very confident about. It's not, in my opinion, really reasonable to have a high degree of certainty about what kinds of AI progress will happen if you start looking out a decade or two. For the work we're doing, we think that there is at least a 10% chance that technology that would enable this kind of transformative AI capability
[0:09:15] Could be possible in the next 20 years or so. I'm not going to get into all the reasons. I'd be happy to talk about them later. But it's not really out of scope with the kinds of predictions you get when you ask, for example, like the published authors at major machine learning conferences what they think about these questions. It's not clear that people have really good calibrated methods for making these predictions. So I don't put a lot of weight.
[0:09:41] On these predictions or our predictions, but you have to start somewhere. I also want to reemphasize that I think AI is doing a significant amount of good today. I'm about to get into the part where I talk about potential risks. Most of this good is probably through...
[0:10:01] Through technologies that used to be thought of as artificial intelligence so like a lot of topics in computer science like search used to be thought of as AI now we just think of them as as software engineering and I think it's reasonable to expect future developments even these transformative developments to bring really large benefits in the future as long as we can avoid the risks. Also, I'm hopeful that
[0:10:25] Many people in the future will be aware of potential risks and benefits and will work hard on these problems. So I don't think it's the case that the world is just hurtling towards something and no one is paying any attention. But I think that every bit of incremental work in this area counts. So let's talk about risks. The first category out of the two categories I'm going to talk about are strategic risks. So these have to do with the way that influential actors like governments or developers of AI
[0:10:55] Respond to the prospect of really transformative AI occurring. And what they do as AI continues to progress, what outcomes they hope to achieve, and what policies and methods they use to try to get those outcomes. So going back to this picture from the Industrial Revolution, you can imagine that if a lot of influential actors thought that the world was going to look like this graph.
[0:11:19] With artificial intelligence taking up that increasing slice of the pie. They might also think that groups that have an advantage in AI development would have significantly more influence over the world in the future. There could be this redistribution of influence over the world. And that could lead to a couple, I think, pretty significant risks. The first one is misuse. So authoritarian governments, criminals, or terrorist groups could use AI for high impact cyber attacks.
[0:11:47] Development of biological or other new kinds of weapons, social manipulation or control, or extreme surveillance. And just as a general principle, if artificial intelligence makes a small group of people a lot more powerful than everyone else in the world and a lot more powerful than they've been historically, I think that we just run a pretty significant risk of this type of misuse.
[0:12:12] And also that the effects of this misuse could be really long-lasting or even permanent. So it seems possible to imagine some kind of stable authoritarian state, or that the damage from a conflict incited by misuse of AI could do really significant harm to humanity's trajectory overall, take a long time to recover, and we don't know what shape we would be in when we do recover. The other area is preemptive conflict.
[0:12:41] If governments see AI going this way, they may try to take actions or take steps to make sure that they have the ability to defend their interests in the future. If a strong advantage in AI is like a critical resource that groups need in order to defend their interests, then rapid progress in AI could be sort of a destabilizing force. Could like increase tension between
[0:13:10] Possibly competitive states or other influential groups. And it might even be viewed as a threat on the level of something like nuclear weapons, which really changed the game when they came onto the scene. I'll talk in a little bit about interaction between strategic risks and some other risks that we see of accidents. Because if we see something where multiple influential groups are racing to gain an advantage in artificial intelligence, it seems unlikely that they'll want to invest as much as maybe they would if they weren't racing.
[0:13:41] In making sure that AI systems are safe and reliably controllable. So the prospect of transformative AI could be really globally destabilizing even before we account for the things that people could use artificial intelligence for. And then also misuse of AI once it becomes really transformative could sort of like bend the trajectory of the world towards something that's a lot less good than it would be otherwise.
[0:14:09] As a foundation, our basic strategy in this area has been field building. We think that there's one scenario where these kinds of strategic risks become apparent relatively late in the game. There are basically no professional researchers in strategy who have been thinking about these issues for five or 10 years, devoting 50 or 100% of their career to trying to figure out how we would deal with these kinds of things and what likely scenarios and outcomes are. And there's another world we can see where
[0:14:38] Instead that field has been supported and growing and that there are these researchers who can be called on when they need to do important work to inform influential actors about what kinds of strategies are most likely to lead to good outcomes. So we want to do what we can to sort of like grow those fields and facilitate researchers who want to figure out the answers to these strategic questions or even figure out what the right questions are at this stage really.
[0:15:06] It also incidentally seems like computer security, operational security, and machine learning security are really promising areas in strategic risk. In a world where no particular developer can defend against having their development stolen by another group, it seems much more difficult to keep things stable and to avoid these misuse kinds of scenarios. So the second category of risks that I'm going to talk about are accident or misalignment risks.
[0:15:36] And these have to do with catastrophic unintended consequences. So not anything that any particular group wants to achieve, but something that seems like it could happen nonetheless. This is the focus of Nick Bostrom's book, Superintelligence. It's the thing that most often seems to get talked about in this area. So the way we think about this is that today, most cutting edge AI systems are trained to maximize a well-defined reward or loss function on a particular training distribution.
[0:16:04] So an AI system might be playing a video game over and over again and learning how to change its behavior until it can achieve a really high score. Having that score is key to guiding the learning. We don't really know how to train AI systems without having that kind of concrete score or in a way that will capture the behavior that we want that isn't reflected in that score. So you get what you train for. And in many cases, it seems like we don't.
[0:16:33] We don't know how to train for the kinds of things that we want. And that's because for many tasks we care about, we don't know how to design these cheap, well-defined rewards. So we can imagine a situation where our ability to train AI systems to achieve these well-defined tasks really outpaces our ability to achieve less well-defined tasks like help humanity thrive or act in the way that we'd want you to.
[0:16:59] And in worse cases, so for example, we can imagine a hedge fund deploying an AI system to maximize its profits. Similar to the way we see video game AIs today finding exploits or finding unintended strategies, so it racks up a high score but isn't really playing the video game the way we wanted it to. We could imagine this hedge fund AI system devising some new plan to commit maybe very subtle cyber crime or manipulation of markets.
[0:17:28] It would maximize the score that it was trained to maximize, but not achieve the outcomes that we would really like from it. And in the worst case, you could imagine a system like that figuring out how to develop sort of radically new technologies, including weapons or persuasive technologies or ways of manipulating humans. And in a sense, try to optimize the whole world around the goal of making profits for this hedge fund.
[0:17:56] I don't think that the outcomes would be very good for anyone, including the hedge fund managers in that case. And in that case, our ability to build AI systems that can stop other AIs from doing damage or that can make sure that our aims are achieved in the future, those kinds of like vague and hard to define goals, building those AI systems is not something that we currently understand how to do. So we see a lot of value in finding some way to train AIs
[0:18:26] AI systems to pursue goals that are actually pretty poorly defined because that's the kind of thing that most humans ultimately care about. A second category within misalignment risks, I've just used the illustration. I think you already had a talk on adversarial counter examples. Is that right? A talk on information security. Oh, okay. I think there's a talk coming about adversarial counter examples that will be much better than my explanation here. But basically, we just see on the left an image from the kind of data set that
[0:18:55] That image classifiers are trained on. In the middle are distortions and on the right images after the distortions are applied. So the distortions are calculated to cause the AI system to classify all of these images as ostrich instead of whatever they're supposed to be classified as. I don't think it makes sense to project exactly this kind of failure into the future. It's hard to predict what kinds of progress we'll see. But this is a good illustration of the general
[0:19:24] A situation where an AI system encounters a kind of input that it wasn't really designed to handle well or wasn't trained to handle well. And we can imagine that with powerful enough AI systems that this kind of mistake could lead to really bizarre and harmful outcomes. So returning to the picture above, the real thing that we're thinking about on misalignment risks is a world where most human influence in the world goes through AI systems that are acting on our behalf.
[0:19:54] But we're not able to train them to act on our behalf in a way that respects these hard-to-define values and that's robust against the kinds of inputs it might see when it's deployed that it didn't see when it was trained. These technical problems might be obstacles to capability just as much as they are to safety. So in a sense, it makes sense to just lump all of this into AI research and say, well, the goal is to make machines that do tasks well.
[0:20:21] All of this is normal AI research. It doesn't make sense to separate it out into alignment, misalignment, safety. I think that there's a grain of truth to that. We might hope that normal progress in AI will solve these problems at the same rate that it's solving the problems required to make systems that are really capable within distribution on well-defined tasks. But I think it makes sense to separate these out and make sure that these
[0:20:51] Separate progress bars are keeping up with each other. So our approach in this area is field building as well. And here things are a little bit more concrete because there are, I think, a lot of problems in machine learning and AI more generally right now that are just super relevant to robustness against adversarial examples or distributional shift. Those are topics people are already studying.
[0:21:16] Or that are relevant to training systems to achieve goals that aren't particularly well defined and that we don't know how to define in the typical paradigm. So groups in academia and industry are working on these problems and a lot of these groups have sprung up in the last few years and are growing at a pretty rapid rate. As a foundation, most of what we do is work with academic researchers. They're the ones who we can support most.
[0:21:45] But I think there's also really exciting work going on at industrial labs, particularly DeepMind and OpenAI, and collaborations between all of these groups, which I'm very hopeful for this field to sort of grow into a unified whole over the next few years. So I'm going to pass things to Dario to fill in some of these blanks and look forward to questions afterwards. Cool. So I'm Dario Amodei. I lead a team at OpenAI that
[0:22:24] Does work on making AI systems safe and putting humans in the loop. So I've been in the AI field for about three years. And I think about two years ago, when I first entered deep learning, I was particularly impressed with something about deep learning systems as an outsider to the field, which is that you found they could be very powerful. So the first thing I worked on was a speech recognition system.
[0:22:50] You could see just how many subtle correlations the system could figure out. And yet, if you just change something very simple, you could completely break the system. So if you trained a speech recognition system almost entirely on, you know,
[0:23:04] You know, American accented speech, and then you gave it some British accented speech, it would, you know, it would completely break. And, you know, of course, if you train on a number of accents, then you get somewhat better at generalizing to further accents. But, you know, this mix of incredible power, kind of opacity, and unreliability really kind of, you know, gave me the sense that, you know, an important direction in this field is to...
[0:23:29] You know, to try and figure out that, make sure that these systems do what we kind of want them to do. And that was around the same time there was kind of the, you know, the like Andrewing overpopulation on Mars versus Elon Musk summoning the demon, you know, kind of a public debate on AI. And, you know, something I kind of felt about it was, you know,
[0:23:51] That while many people in the field kind of liked Andrew Ng's response, said, oh, there's nothing to worry about. We're not doing anything evil. I didn't really found it satisfactory, because I feel like whenever we have a powerful technology, it has a lot of upsides, but it also has some downsides as well. And instead of dismissing the downsides, we should be trying to minimize the downsides and maximize the upsides. And that's what truly responsible behavior
[0:24:16] So I eventually found my way to OpenAI and I'm kind of working on that. So often when I have to give
[0:24:24] Give a recruiting talk for OpenAI or something. I start by talking about all the great things OpenAI is doing. But I'm not going to do that here. This is what I think about at OpenAI. OpenAI works on all kinds of wonderful and positive applications of AI. And I think that overall in the world, the wonderful and positive applications will probably dominate.
[0:24:47] You know, my job is kind of considerably more depressing than that I'm the person at OpenAI who thinks about all the possible things that could go wrong. So, you know, I generally kind of divide these into a few categories and I'll mostly be talking about
[0:25:01] The last one here, but we'll try and touch on the other ones. Daniel kind of already made some of these divisions, but one category is misuse. Whenever you have a powerful technology, and particularly one that advances fast so that you have disparities in technology, then there are many ways that technology could be misused by people with bad values or bad intentions. So an example of these are things like drone weapons.
[0:25:27] Mass surveillance that might be enabled by advanced speech recognition at scale, advanced hacking, and ultimately kind of military arms races, where if you have a technology that is the key to scientific and technological progress as well as military strategy, then a lot of governments are going to be interested in it and they're going to use it adversarially against each other.
[0:25:48] The second category is what you might call social side effects. So things where no particular person is misusing the technology, but nevertheless something bad happens that's kind of more related to policy than something going wrong technically with the technology. So worries about unemployment and automation, ideas that...
[0:26:10] Bringing ML systems into their use in the criminal justice system could kind of subtly exacerbate existing biases and create unfair outcomes. Weaponized fake news, I think that in a few years it'll probably be possible to make.
[0:26:25] A purported video of anyone basically saying or doing anything. And it'll be very hard to tell that from a real video. The third category is accidents. So this is kind of undesirable things an AI system does that its designer did not intend. So my general picture of this is we're more and more moving towards things like reinforcement learning, where an agent interacts
[0:26:51] With its environment in a very entangled way, more complex environments where they do the interaction, and systems that are given more autonomy because they're able to do more things end to end. And the more we have that, the more the link from kind of...
[0:27:05] What we designed the system to do, what our intention was when we created the system, to all the little sub-decisions that we've delegated it to, the more there is the potential for some of those sub-decisions to be things we didn't want, because the link gets weaker and weaker. We can trust our AI systems to do tasks and subtasks more, and so that's a good thing. But the longer that chain grows and the weaker the link grows, the more there is the potential for something to go wrong. So these days, things like car crashes and self-driving
[0:27:34] Driving cars, financial things like the flash crash, accidents in factory robots. If you had systems controlling power grids, things that could go wrong with them. And the long run, catastrophes that might go wrong with what OpenPhil calls transformative AI.
[0:27:51] You know, kind of the cartoonish version of this is the kind of Nick Bostrom's idea that the world could be filled with paper clips. That's kind of not really my picture of what I think will happen. But I think the general point that if you have kind of
[0:28:05] A very powerful system doing many levels of things that there's the potential for something to go wrong. And since it's a very powerful system, that thing going wrong could be potentially catastrophic. So actually, personally, I worry about all of these. And if anything, I actually maybe even worry about misuse more than I worry about accidents. But as a technical person, I feel like accidents are a place I can start.
[0:28:33] Where I don't need some politician to implement some plan that I want, which is really hard to make politicians implement reasonable plans.
[0:28:44] And so it's something kind of that we can start on now and maybe even will be helpful for the other stuff. We do do some stuff at OpenAI on a couple of these. I represent OpenAI on the board of an organization called the Partnership on AI, which is kind of a group of AI organizations like Google, Facebook, AAAI, that kind of tries to set policy on some of these and kind of give the industry
[0:29:13] That's kind of a small fraction of my time. I probably spend most of it on this accident stuff. So about a year ago, I and some colleagues back when I was at Google wrote a paper called Concrete Problems in AI Safety that kind of tried to lay out a version of problems that you have with accidents, tried to categorize them and was kind of a call to my colleagues to work more on these problems than we had.
[0:29:42] So there's a lot of stuff we say in the paper, but one way you can kind of break down the way that things can go wrong is, so I'm a designer of an AI system and I wanted it to do something. And something goes wrong and it fails catastrophically. So what are the ways in which that can happen?
[0:30:01] We kind of divided it up into two causes, and a third that we can remind people is also important. One is the wrong objective function. So as I'll discuss, if you optimize for the wrong thing, even in very subtle ways, you can get systems that do something radically different from what you intended them to do. Cases where you have the right objective function, but something has gone wrong in the learning process. So this is like training the system on
[0:30:29] American Accented Speech and then trying to use it on British Accented Speech and having something go wrong with the process. And we're kind of already seeing this kind of thing in self-driving cars and similar systems. And then a reminder that all ML systems are built on top of a software stack. So software implementation bugs or security vulnerabilities with the software.
[0:30:53] All of those things apply to ML systems in addition to the ML specific problems that arise with ML systems. So one way to think about AI safety and AI risk is that we're building another layer on top of a stack of things that are already kind of vulnerable and error prone in a number of ways. So I'll start by talking a little bit about the second thing, which is you had the right objective function, but something
[0:31:21] Something went wrong, and then I'll talk mostly about wrong objective function, which is kind of the area in which my research and those of my colleagues has recently focused. So just a simple example of this is these are two video games where I trained a reinforcement learning system to play them, two kind of simple flash games. So RL is quite good at learning tasks like these.
[0:31:44] Video games, it's amazing what it can do. It's just trying to maximize its score by going along this track as fast as possible. So you can get really good at this particular game, and then I give you another video game that's basically similar to this game. And if I just take the policy that I learned on this game and try and transfer it to another seemingly different game, it's basically the same racing game, but different colors and scenery that's a little bit different.
[0:32:10] All right. Well, this isn't going to be nearly as surprising. But basically, the thing I'm trying to illustrate here is basically you take this policy that's really great at one racing game, and you just deploy it without modifying it to another racing game. And it doesn't do so great. And so the applications to self-driving cars should be pretty obvious here. And in fact, people who work on self-driving cars worry about this problem a lot. There's all kinds of problems that are out of distribution, like
[0:32:38] Google had one story about how one of their self-driving cars ran into a woman chasing a duck across the road. It's like they just never seen anything like that before. But they were able to deal with it because they thought about it. So bad consequences of distributional shift can already be real. So many of you may remember the
[0:32:58] The Google gorilla incident. So this is when Google's photo captioning algorithm basically tagged some African-American people as gorillas. So it turned out, actually, this wasn't them failing to be careful or being idiots or anything like that. They'd actually thought about
[0:33:17] This problem and had rules like if there's an animal anywhere in a picture, don't tag anything in that picture as a human. They had all these kind of hard-coded rules to kind of prevent this kind of offensive stuff. But if something's far enough out of distribution, your neural net won't even recognize that there's a problem at all. I think in this particular case, there were certain photos that were backlit in a certain way that confused the neural net.
[0:33:46] This is kind of merely a classifier, but you imagine if you kind of deploy this in more powerful systems that a lot of things can go wrong. So mostly going to be talking about bad objective functions. So our current AI systems have this really weird limitation. So you're probably familiar with very powerful AI systems that can do things like play chess and go very well. But one thing that's hidden from those systems is
[0:34:14] Curious property of chess and Go is that the optimization and the behavior is incredibly complex. The strategies that you use to play chess and Go are incredibly complex. But the goals are actually very simple. In chess, you're just trying to checkmate the king. In Go, you're just trying to have more territory. And actually, this is a requirement of our reinforcement learning systems. Because if you want to run something by trial and error 100 million times, you have to be able to evaluate your performance in an automated fashion.
[0:34:42] If a human needed to evaluate this, you couldn't do it 100 million times. By contrast, real world tasks often have very high level, fuzzy, or complicated goals. So what if I want to arrange the furniture in a room to look nice, engage in realistic and informative dialogue with humans, or something that I would want a very powerful AI system to do, plan and execute a mission to Mars.
[0:35:04] We currently kind of fudge around this by finding very simple proxies for our complex goals and trying to do the best we can with them. But this often leads to unexpected and, in fact, often very unsafe behavior.
[0:35:18] A general principle here is Goodhart's Law. When a metric becomes a target, it ceases to be a good metric. What that means is if I have some metric that's simple, it's 99% correlated with what I want or appear so, and I optimize it really hard, that correlation is dynamic and it will go down, and I don't get the thing I intended to get. So hopefully this video will play. It will not.
[0:35:41] This is an example of a boat race. So it's like a little flash game where you're racing a boat and you're trying to complete a course. And you get points along the way for doing something. So because the flash game is kind of enclosed, the only really available objective function is the points that you get for being along the boat race.
[0:36:07] You might think that because the points are laid out in front of the entire race that it would cause the agent to actually complete the course in the right way, because it looks pretty correlated. But actually, RL is incredibly clever. And so instead of completing the course, it goes backwards. These are the things you need to click to get points. And it finds this lagoon that's just kind of pretty far off the course. And it can get points by doing that.
[0:36:35] And so if you train this for a few hours, it figures out that despite not completing the course at all, it can get most of its points just by kind of going around in circle, getting these turbos, accelerating, setting itself on fire, and just doing it all over again. So you could say, well, you know,
[0:36:56] You get what you ask for, right? I mean, you know, you didn't actually incentivize this thing to finish the course, so of course it's not going to optimize for finishing the course. What's so interesting about that? It's just that the video game is broken. But I think my response to that is that in some sense reality is broken, right? If you have a robot, you only have sensors on things that are very close to you. You don't see the whole state of the world. And it's always easier to see simple things than more complicated things. It's easier to comprehend simple things than it is to...
[0:37:24] to comprehend more complicated things. So I think this is a very generalizable and general problem that can happen. So as something
[0:37:36] Very closely related to this, this thing which in the press was called a back-flipping noodle. So this is something that looks like a pretty simple task. This thing only has a few joints. The action space would look simple. But something that's totally crazy is despite the fact that we can play chess and go better than any human player,
[0:37:54] Until our paper, it was literally not possible to make a backflip as elegant as that, nowhere near. So why is this hard? It's because it's hard to specify the objective function. If I wanted to go joint by joint and say, what exactly do these joints need to do to make something that constitutes a backflip, it's actually very difficult. We had engineers spend hours trying to hand design a reward function, and you end up with this kind of
[0:38:20] Thank you.
[0:38:43] And at first, it's just doing totally random things. And every once in a while, it shows an example of what it's doing to the human. Two examples. And the human clicks whether the left or the right is more like the behavior it wants. Nothing about the behavior is pre-programmed into the system. The agent is literally just trying to figure out from what the human clicks what it wants. And then in the background, it's using reinforcement learning to try and display the behavior that the human wants. So it starts out random.
[0:39:13] And then humans give some feedback on it. And then it goes off and says, let me try and manipulate the environment to do what the human wants the best I can. And then it brings back two more examples of behavior to the human. Do you mean this or do you mean that? The human says, I meant more like this. The agent updates its model. And in the background, it tries to manipulate the environment to do something more like what it thinks the human wants.
[0:39:41] Behind the hood, the normal way an RL algorithm works is the environment gives observations to the agent, the agent takes actions, and it observes, and normally there's a reward coming in, and the reward is something fixed, like did you win or not in Go? Or in the case of the boat agent, how many points did you get? What we did was we replaced the reward with a reward predictor, which is kind of an avatar for what it thinks the human wants.
[0:40:11] So we had this reward predictor, which is another neural net in addition to the policy and value function that run the RL algorithm. And this thing is fed examples of human feedback. And then as the agent acts in the environment, it gives more examples of behavior, asks the humans to label it.
[0:40:31] That trains the reward predictor. So the human is kind of involved in every process of training the AI system. It doesn't have one objective function. It has an objective function that adapts and learns over time as it figures out what it is that the human wants. So one of the tricks is you could never do this if you wanted to basically have the human look at everything that the reinforcement learning
[0:40:56] Learning Agent produces. So things like AlphaGo and even very simple Atari games, I have to go through something like 100 million time steps in order to learn. So that ends up being many weeks or months of real human time. So our method is able to basically request human preference feedback on only about 1% of the experience.
[0:41:21] So instead of 10 million frames or so, you have 10,000 frames. And if a video is 25 frames, then it really only comes down to about a few hundred pieces of feedback that you have to give on those short video clips. So we're able to train some of these simple robotics tasks in about half an hour and Atari tasks in about an hour or two. So just to make the point a little better,
[0:41:50] So this is the game of Enduro and Atari. So the usual goal of Enduro when you have a reward function is to basically pass the cars in front of you as much as you can. And so what this was intended to show is that you can train this to two different goals. So here I basically train the system not to pass other cars but to keep even with them.
[0:42:13] There's nothing in the emulator that gives me a good handle to figure out how to reward that. I just watched the thing play Enduro a bunch of times and rewarded it when it stayed exactly even with other cars. And this was designed to be a video of an agent that is exactly trying to race past the other cars, where I've reinforced it and rewarded it for racing past the other cars. So the idea is I can have basically the same set of algorithms, the same environment, and with...
[0:42:40] By merely changing what I decide to give feedback on, I can cause the agent to do two very different things in the environment. I'm not limited to a reward function that I can code. I can let the agent do anything that I want. Here are some more tasks we're able to make it happen. So this bar on the right is our reward predictor. So basically, this thing goes really high when the agent is happy and thinks that it's done something really great. So like in Pong, you know,
[0:43:07] It goes up a little bit when the agent returns, and then it goes way up when the agent scores a point. In Sequest, when the agent shoots one of these evil fish, the reward predictor goes up. So again, it doesn't know anything about score. We completely hid the score from it. We basically taught it on its own to understand that winning a point is good and losing a point is bad. Knocking out these blocks is good. Shooting the fish is good. Dying is bad.
[0:43:36] So yeah, here are some graphs from the paper where the purple is basically how well you do.
[0:43:47] When you use this human preference learning, even in a case where the game has this exact reward that you can just learn from the exact reward. And so the takeaway is that most of the time you can do almost as well, even learning a pre-specified reward, if you hide that reward from yourself and just have a human learn it, you can learn in almost as few RL steps. And in fact, sometimes you can actually do better.
[0:44:17] One reason you can do better is that often humans are better at providing rewards than the actual game. So in Enduro, a problem with Enduro is that you don't get any rewards at all until you pass some cars. When you start, the cars are zooming past you. And so the agent often doesn't know how to explore enough to figure out how to pass cars at all.
[0:44:38] But a human will give it a reward for being passed by other cars a little bit more slowly, right? If a human looks at it and it's like it's going forward a little bit versus it's not going forward very much, a human will reward that. So the human kind of corrects the flaws in how the agent rewards tasks.
[0:44:55] And so rather than relying on just one reward function, often the human can give a more naturalized reward function that makes it easier for the agent to learn. So for example, in tasks like Enduro, you can actually learn better using this method. Even though this is a method that's kind of designed for safety purposes, you can actually learn an ML task, one where you already have a defined reward and shouldn't even need a method like this, better than
[0:45:21] You know, than you otherwise would. And this is also true of, so we tried this on a bunch of simulated robotics tasks, not just the back flipping noodle. And, you know, it turns out mostly it does kind of a little worse or about as good as the method where you cheat and know the reward. But sometimes it even...
[0:45:40] It even manages to do better. So where are we kind of going with this in the long run, right? So right now, we're just kind of clicking left or right on these agents and saying, is left better than right? That often feels like a very thin communication channel, where I really wanted to say, wow, that thing you did was really good.
[0:45:59] You move the paddle here. What you needed to do was move the paddle a little lower. So you can imagine cases where the human draws little football play like sketches on the two videos and says, well, you needed to do more of this and less of this. And if you did that, you could probably learn more complicated things with less feedback. And I think the ultimate direction this is going is that we should be able to give our agent instructions in natural language.
[0:46:27] What would it mean to tell a machine learning system to plan and execute a mission to Mars? It's a totally ill-defined question, right? Like, you know, I'm taking the humans to Mars. Do they need to be alive? Should I accept a 1% chance of them dying in exchange for them going faster? Or do I need it to be a 0.1%? How many humans are going to Mars?
[0:46:47] What should, if the humans are radioing back, does that really mean they're alive? What's going on? So it's this terribly ill-defined task where the only way I can imagine actually having a system that's useful to us, that doesn't constantly need to be told what's going on, is where a system that asks questions to us.
[0:47:09] That asks us to clarify what the particular goals are. So, you know, if we say plan and execute a mission to Mars, it'll say, what's most important to prioritize? Do you want to be extra safe? Do you want to go fast? How many people? Do you intend that they'll come back?
[0:47:22] Or that they'll stay. And that somehow, in some way we don't know how to do yet, that these words given by the human will be compiled through some language interpreter to some reward function in some very large vector space. And that the agent will develop a policy based on that.
[0:47:40] That reward function. A lot of the details of this are vague, but I think this is the kind of thing we should be aiming for in the long run. Because we need to interact with environments. And for that, we need something like reinforcement learning. But reinforcement learning itself, it just won't cut it. I think it's not going to be able to do tasks of this complexity. And even worse, if it does, I think it'll be very unsafe.
[0:48:05] So that's kind of the general direction we're going with our research. Next, we're thinking about things like robotics and dialogue and those kind of tasks, and hopefully we'll have some work on that soon. Just to finally return a little bit to the division I gave in the beginning between accidents and misuse and other things, there's been all this stuff in the last few weeks about China making pushes in AI.
[0:48:31] Even if we manage to solve all these technical risks, we still have this policy risk where we have two very powerful countries that are both
[0:48:41] Racing very hard to develop AI. There's a concept in international relations called the security dilemma, which says that you can often have a situation where neither side really wants to fight or to be in an adversarial relationship, but they're afraid that if they don't prepare themselves that the other side will attack them or the other side will have an advantage. So you can get into a situation where basically no one wants to do something destructive, basically no one wants to be adversarial, but it happens anyway because
[0:49:11] Because you have two very powerful entities that see the potential for a very powerful technology. And if they don't get ahead in it, the other side will. So I think this is one of the looming issues that I think deserves much more attention than it's getting, maybe even more attention in the safety stuff. I just don't really quite know what to do about it other than to make people more aware of it.
[0:49:37] And try and have something happen with it. So yeah, lots of acknowledgments on the Concrete Problems paper. Basically, Chris Ola, Jacob Steinhardt, Paul Cristiano, John Shulman, these two are now at OpenAI, and Dan Manet at Google. The Learning from Human Preferences paper, the first author on that was Paul Cristiano. It was a collaboration between
[0:50:00] OpenAI and DeepMind, who think similarly that safety is an important topic that we need to study more than we did. Actually, one of the co-founders of DeepMind, Shane Laig, participated in this research. And the various institutions that I've been at or collaborated at, OpenAI, DeepMind, and Google Brain, have all been extremely supportive of this work. It seemed weird to everyone when we first started thinking about it. But I think there's been kind of increasing acceptance and institutional support for DeepMind.
[0:50:29] Are you doing it and scaling it up? Yeah. Yeah. Because you have a limited amount of worrying time. And you need to make sure that you're using it the right way. Yeah. Yeah. Oh, definitely. No, I totally see the question. So actually, we think about this a lot. I'll be interested if you have thoughts on this. But we think a lot about, first of all, I think you can actually do some estimation of these things and come to numbers that are maybe qualitatively meaningful. A lot of that has to do with trying to think more concretely
[0:52:03] How situations could play out. I don't feel like I have great examples off the top of my head about how to do those kinds of comparisons. But the other factors here are tractability. So there are some problems you look at. So for example, we might worry that there could be a high energy physics experiment out in space somewhere, naturally occurring, that results in vacuum collapse to a new level of vacuum. And that wave is coming towards us at light speed. And we can't do anything about that.
[0:52:32] So you could worry about it if you like worrying about things, but you're not going to do anything about it. And I think there are less stark tractability differences in different areas. So maybe there's a particularly promising looking research path in artificial intelligence that gives you an extra reason to spend more effort working on or thinking about those things. Then the third point here is I think there is a sort of like a low hanging fruit effect that can incentivize portfolio approaches.
[0:52:59] So in each of these areas maybe you're worried about both risk of pandemic and risk of artificial intelligence and you also think that there's outstanding potential for like some cheap scalable energy source and you want to figure out how to balance among those things. Often it's the case that the first bit of support you put into something will result in like a really great amount of learning about that area or will result in sort of like
[0:53:25] Step change in whether there's any progress happening, whether there's any growth in the field. And I think that sort of pushes you in the direction of, even if there were really significant differences in the probability of these risks, it could still make sense to spread your resources among several of them. That's not a super concrete answer. There's also your comparative advantage. So if you work in artificial intelligence, then your comparative advantage is probably not in managing synthetic biology or something.
[0:53:55] So these are all tools that I find I don't know that many people who, when they take all of these things into account, still feel sort of paralyzed about what they can productively engage with. Although a lot of people I know spend a lot of time worrying about a lot of things. So feel free to worry as much as you please. So a point I often make when thinking about this is
[0:54:21] I think humanity faces a lot of risks. And in fact, if I think about human civilization, it seems basically unstable to me. The last year, I think one of the lessons you could take from the last year is the idea that I'm actually not entirely confident that in the long run,
[0:54:40] Digital communications is compatible with democracy and complex human society. We invent new technologies every day, and they often have subtle effects, and sometimes we can't handle those subtle effects. So it seems to me that if we were to just keep going on for 200 or 300 years, just kind of the way we were.
[0:55:07] Gradually advancing technology. That's actually a very dangerous situation. And there's a lot of ways that we could destroy ourselves. We've talked about risks from AI. But I think one of the benefits from AI is it's one of the few things that could
[0:55:24] Reduce and maybe even end many or all of those risks if we really got it right. That it could help us to make ourselves smarter, could change a lot of things that have been fixed about the human condition since we evolved from chimpanzees. And so whatever the risks are in AI, one of the reasons I'm most interested in AI is that I think it has the potential to eliminate most of the other risks that we face.
[0:55:53] But of course, if we're going to do that, then we have to worry about the risk that that technology itself has. And so that's what takes me to caring about this more than I care about other risks, although there are a number of other risks that I find quite disturbing. I think that you shouldn't take the things I say on this point as coming from much expertise.
[0:56:51] One of the things I want to do is find people who know about these things and give them money so they can figure them out. But I think that there's a lot that we can learn from history and from history of politics and history of political philosophy. That a powerful tool here is depessimizing. So for example, when we're trying to set up rule of law, there is a lot more concern for finding some minimal set of things that we want to try to make sure won't be happening.
[0:57:21] Like making sure that people aren't in fear of physical violence all the time. And then try to make those things happen. And then as society gets better as a result of those things, we're more able to get together as a community and figure out what things we want to do next. So there's this sort of incremental progress that seems pretty promising to me. And what I think that would imply is like starting with the things that are most widely agreed upon that are most clearly
[0:57:50] Things that would improve the human condition and then sort of move up from there and hope that the effects from people getting richer and incurring less risk in their day-to-day lives and having more hope for their children's lives being better than theirs are, that that makes people more able to step back and think in a serious way about how to cooperate with people who are more and more different from them. But I'll repeat again at the end.
[0:58:18] I really don't think you should ask me this question. Yeah. So I mean, one of the ways I think about misuse is there are many technologies today that you can misuse. So the real question is, what's new about AI? So I think the way we've traditionally handled misuse is a kind of game theoretic or balance of power solution.
[0:58:40] Where, you know, I can take a gun and shoot someone. No one's stopping me from doing that. But if I do that, you know, the police will arrest me or shoot me. And so, you know, I have very little incentive to actually do that, right? There's a lot, you know, like my power is very small compared to the power of the rest of the people in the world. And, you know, even if I was, you know, much more rich or powerful, it would still be the case that my power was very small compared to the power of the rest of the people in the world.
[0:59:07] My concern about AI is discontinuities of power. So I think when thinking about misuse, the things that we should put most of our attention on and that we might need to address in novel ways are cases where a discontinuity or a very, very rapid, very sharp concentration of power could occur. Of course, whenever there's a novel way to attack people, even within the balance of power, we have to address those.
[0:59:32] But the things that worry me the most are the cases where you would have this sharp discontinuity. So talk a little bit about the research being done. Is there a chance that, like, you create an evil AI, but it's going to, like, a child that's going to take you two years to learn anything, or it's getting faster where you can create it in little two hours and knows everything as tough as happening there? Yeah, yeah. I think that's a good question. So some sets of distinctions here. You know, I think that, so right now where we are,
[1:00:22] is exactly as you say, that to train a system to do one thing takes an enormous amount of training. So if you wanted to train a robot from scratch to do something in the real world, it takes a lot of interaction with the real world, more time than you could possibly take. If we want to train the system to play Go, then of course we can speed that up, but it's still playing 10 million or 100 million games. People, I think, at both OpenAI and DeepMind are very excited about this concept called meta-learning, which is the idea that
[1:00:51] Instead of training an agent in a single environment, you train it on a wide distribution of environments. So like the Flash games I showed, you could train an agent from scratch over and over again to play a thousand Flash games, and then it learns how to learn a Flash game. So then you go through this long training process initially once for a wide range of tasks, and then when you want it to learn a new task, it learns that new task quickly because you've taught it to do the learning activity of learning new tasks quickly.
[1:01:20] So I expect that at some point, perhaps some point soon, we will make a lot of progress on this kind of learning fast thing. So the model I would think about, and you might call it a threat model if you're talking about what you're worried about, you might call it a product model for all the positive effects, is a system that takes a long time to train, but then once it's trained in kind of a very broad way,
[1:01:44] Can then learn to do new things in new environments very quickly. Those are the kinds of systems that I think we will have in the coming decades. Yeah, so I mean, I'm employed by Elon Musk. I shouldn't like, you know, and I work with OpenAI and our policy stuff. I think I have to say here that we're thinking about what we'd like to do and that I don't have.
[1:02:39] That much to say at this time. I think if you think about regulation, I think what I can say is there may be some areas like military use of AI where some type of regulation or laws make sense. And I also think that in the long run, if we're training powerful AI systems, at some point society is going to have to grapple with who is allowed to use them and for what purpose.
[1:03:08] We're not used to thinking of AI systems as being like drugs or something that can have a dangerous impact. But at some point, compute, hardware, and trained models are going to be some of the most powerful things in the world. So in some form, we're going to have to grapple with that. I don't know what the right answer is, but it might prove that.
[1:03:32] Having such things distributed everywhere is that there are going to need to be at least some rules about how it's used. I don't know what those rules look like, and I certainly wouldn't know how to implement one now. But this may very well happen eventually. So I think up until about a year ago, the general approach to multitask learning was that you would basically train a system on kind of different tasks and then try and fine tune it.
[1:04:56] And transfer it to a new task. So it was kind of like, I would train the system to play Space Invaders. And I would train the system to play Pong. And I'd take a network that was kind of an average of those. And then I'd try and transfer that network to some new task. What we do now sounds the same, but is actually subtly different, which is that I take a single network and I put it through
[1:05:22] The process a thousand times of learning from scratch, not just playing an episode, but learning from scratch a thousand games a thousand times. So it's actually learning the process of how to learn a new game. Yes. So no, no. Multitask learning just means that you have like a single model with many heads that kind of learns to do a number of tasks at the same time.
[1:05:55] What we have here is like I might have some metaparameters in a meta policy. And basically the metaparameters describe, the metaparameters are basically a formula for when I see a new environment, the metaparameters unroll into some policy that absorbs information in the new environment that it's never seen before, and then learns over time a policy in that new environment. So it's a little bit like,
[1:06:23] The process of development or something. You can think of the metaparameters as being a little bit like a genome. They basically, when you're put in a new environment and you start as a blank slate, the metaparameters tell you, they're basically a recipe for constructing parameters that allow you to learn and interact with a totally new environment. So like a human brain is a tool like this, right? If I take, or the human genome is a tool like this. I take the human genome, I like...
[1:06:51] Instantiated in a person who's in a totally new environment. They have a different set of experiences than someone else with the same genome might have. And they go off and become a different person. I think the classical multitask learning is more like I'm one person, and I have like 100 different kind of views on the world. And my policy is mostly the same, but you fine tune me a little to do each of these tasks.
[1:07:20] Yeah. Sure. Yeah, so about a year ago, OpenAI published a paper called RL squared. So the idea is that you basically have an RNN, and the parameters of your recurrent neural network, or an LSTM, and the parameters of the RNN are...
[1:07:52] So basically, you start with this RNN, and every time you're put in a new environment, every observation you make goes into the hidden state of your RNN, and then comes out with an action. And the set of things you've learned as you proceed through the environments, the RNN doesn't just proceed for one episode, it proceeds for the whole process of learning over that environment.
[1:08:15] And the activations of the hidden state of the RNN is used to store everything you've learned about that very one specific environment. And so the metaparameters are the weights of the RNN, and the object level parameters that are specific to an environment, rather than persisting across environments, are the hidden state of the RNN. So that's an example of how you could divide the two. So you meant to make sure that they don't change? Yep.
[1:09:04] So I think the general philosophy with the human preference learning stuff is to try and develop, instead of things that work in one situation, try and develop general widgets that can be kind of adapted to whatever models end up being built or that only need to be changed a little bit. So if you forget about safety and just think about machine learning, examples of these widgets are things like convolutional neural networks, LSTMs, attention models.
[1:09:32] Neural Turing machines, optimization algorithms like Atom, where they're kind of things that they're very general in their nature. And so even if they were developed for one particular problem, they may work for lots of other problems. And even if I develop lots of new ways of doing things, like let's say I shift from model-free RL to model-based RL, or I shift from object-level learning to meta-learning, it's still the case that I can use things like LSTMs.
[1:09:59] In those models. So in particular in the human preferences, this is kind of a widget where it's something that transforms a certain amount of availability of human time plus an RL algorithm into an aligned RL algorithm. And so we will, I think, and the directions we're thinking about are, how do we make this compatible with meta-learning? How could you use this to guide the meta-learning process?
[1:10:21] To learn to explore. Would it be compatible with, say, a model-based RL algorithm instead of a model-free RL algorithm? So we're thinking about all these modifications. And I think the model is, you know, whatever is the latest powerful thing that OpenAI comes out with, you know, like the bot that just beat Dota. A few months after that, I want a way to...
[1:10:41] General modules that you were considering?
[1:11:07] I think robustness and generalization. If we found a way to get better at robustness and generalization or at least detect when a system wasn't likely to generalize. So if instead you had an RL policy that basically told you when you trained it to play this car racing game and then you give it another car racing game, it basically says, sorry, I'm not designed to do this. I'm not going to take any actions in this environment. Or at least tells you that it needs a certain kind of data to be trained on. That seems like something pretty...
[1:11:36] Pretty generalizable, where it's not going to be fixed to a particular architecture. Okay, so I think this is a really important research area. The culture novels by Ian M. Banks, they're not perfect as a world, and I think he doesn't do as good a job of presenting a utopia as you could, but I think they might be the closest thing that exists, where machines are incomparably powerful, and they exist alongside humans and seem to benefit.
[1:12:53] The humans and the world that's created is a pretty good world that persists for a long time. It's a little better. Yeah, I mean, I think maybe ultimately AI could help humans themselves get smarter. There isn't that much of that in the culture series. I think that's one of the flaws. Yeah, there's a... Oh, what is this author? Oh, it's going to bug me. I'm going to have to get back to you on this. Werner Wenge? No, no, different one. I think one of the problems here, though, is that like...
[1:13:24] I feel like that makes your job.
[1:14:52] Quite a while on the funding side to feel like we got enough familiarity with some of these issues to start doing some basic steps where the basic steps are like find researchers who care about these things and who have reasonable plans for having an impact and then funding them. And I think often it's the case that with, if there's a mechanism by which AI has impacts that go through societal forces by reshaping norms or by having a sort of like
[1:15:22] Complex social effect. It just feels much harder to form a concrete picture of what you could do that is reasonably likely to have a good impact on the world. It's not an area that has an awesome track record for reliably figuring out which interventions, like if you're worried that your culture is going to spin out of control in one direction or another, maybe as a result of some technology, maybe as a result of economic forces. It's just like...
[1:15:48] In a lot of ways, it feels like a much harder problem than actually trying to figure out how to get AI systems to learn complex human goals, just because there's this meteorological system that you're trying to interact with. So I don't feel on the funding side like we've gotten a clear enough picture to be confident and start moving forward on these things. It's certainly something that we keep talking to people about and keep trying to figure out more, but it's just less mature in my view.
[1:16:17] I'm pretty worried about these short-term things. I hear the concern, could the world fall apart before we even get to transformative AI? I think in general, all of these things will still exist within the usual balance of power framework of humans, which of course hasn't always worked that well. So I feel like it does seem like a lot of bad things could happen when we can generate weaponized
[1:16:46] Weaponize fake news and make drones that do anything. I guess I'm hopeful that these things won't be so severe that they'll literally derail or destroy civilization. Civilization has been around for a long time. It doesn't have to be around for all that much longer before we create truly general AI. So I hope we'll muddle through. But it's definitely something that I'm
[1:17:13] It's definitely something that I'm concerned about. As a technical researcher, I really have no idea what to do about it. Sounds like Daniel has only slightly more idea what to do about it than I do. When we're looking at, like when I was talking about strategic risks and we're looking at field building, you'll notice I didn't have a slide describing like all the cool things we're doing to build that field right now. I think it's just because it's like a much more confusing and difficult problem than this case where we're like, we have these technical problems.
[1:17:38] Oh, great. There's like a field of AI and machine learning researchers who are already pretty interested in a lot of these problems or problems that are close nearby. Whereas, yeah, it's actually, I should clarify it. There are a bunch of people who are working on related issues from a variety of perspectives. It just feels less like there's a unified narrative that we can use in order to really be confident about making progress. I know that there's some attention.
[1:18:42] I don't know that there is a really, there's not like a really large amount of attention as far as I know on any kind of misalignment besides the kind that just manifests as like your product breaks completely. This issue of like alignment, for example, like maybe it would be nice if YouTube would check after you watched a video, like wait a week and then say, hey, remember you watched that last Saturday? Are you like glad?
[1:19:08] Is your life better because you did that? And then maybe it would be more aligned with your on reflection values or something like that. Is that the kind of thing you're thinking of? Yeah, well, it's interesting. I never thought of that as an exception. Yeah, yeah. I think this is totally a cool area to think about. So yeah, I think we should probably divide it into two cases, right? So I think one case is easier to solve technically and harder to solve socially, and the other is the reverse.
[1:19:45] So one reason it could happen is because Facebook has some ads and what they actually want to optimize for is what causes people to click the most. And so then it's not like they had the wrong objective function by accident. It's that the objective function they're trying to optimize you for is not the objective function you really want to be optimized for. Yeah, it's at least in part a policy problem, right? Where it's like, you know,
[1:20:12] We're not really great at controlling our own reactions and our own attention span. I mean, you can see this all the time. You can imagine how it radicalizes people. Let's imagine you're a liberal or a conservative who's mildly offended by some things the other side does. And Facebook shows it to you, and you get really mad, and you click on it, and then it shows it more to you, and you get more mad. And obviously, that isn't a great thing. But in some sense, that gets optimized for.
[1:20:41] The other case is where it's actually really hard to tell what does the human want. What do I want from Facebook or Google or whatever? What do I actually want when I use the internet? I would hope that it makes me a more enriched, a smarter person in some way. It helps me fulfill my goals. But those things are a lot harder to define than clicks. And so I think that has something in common with these misalignment problems.
[1:21:08] I worry that it may be almost as hard, that to really solve those problems means describing what it means for a human to live a good life, which is what we're trying to solve in the general case. Although maybe if we solve the general misalignment problem early on, long before we have AGI, it'll also help us solve that problem. But I'm not super optimistic about that possibility.