[0:00:00] Hi, I'm Robert Wiblin, Director of Research at 80,000 Hours, and welcome to the podcast. If you want to make sure you never miss an episode from us, you can subscribe by searching for 80,000 Hours in whatever app you use to get podcasts. That way, you can also speed up the episodes, which is how I much prefer to listen to interviews. Next week, I'm scheduled to speak with Alex Gordon-Brown about working in quantitative trading in order to earn to give, which I expect to be very engaging. Today's conversation really goes into the weeds, and I learned a great deal from it.
[0:00:30] If you're looking for personal advice on how to pursue a career in technical AI research, stick around because we get to that in the second half. I apologize for the audio quality on my end. I think we'll have that fixed up by next time. If you'd like to offer any feedback on the podcast, please do email me at rob at 80000hours.org. We're still figuring out how we can best use podcasts to help our readers, and I'll try to respond to everyone. Without further ado, here's my conversation with Dario Amadei.
[0:00:59] Today, I'm speaking with Dario Amadei, a research scientist at OpenAI in San Francisco. Prior to working at OpenAI, Dario worked at Google and Baidu and helped to lead the project that developed Deep Speech 2, which was named one of 10 Breakthrough Technologies of 2016 by MIT Technology Review. Dario holds a PhD in physics from Princeton University, where he was awarded the Hertz Foundation Doctoral Thesis Prize.
[0:01:22] Dario is also the co-lead author of the paper Concrete Problems in AI Safety, which lays out in simple terms the problems we face in making AI systems safe today. Thanks for coming on the show, Dario. Hi. So we plan to talk about the motivations behind technical AI safety research, the Concrete Problems paper, and how someone can pursue a career in this field themselves. But first, we're at the OpenAI offices here in SF. So tell us a bit about OpenAI and how you ended up actually working here.
[0:01:49] So OpenAI is a nonprofit AI research lab. It was originally founded by Elon Musk, Sam Maltman, and a few other folks. And generally, we're working on kind of following the gradient to more general artificial intelligence and making it safe. So I joined around...
[0:02:13] What was it, July of last year? So about a year ago, which was a few months after it started. And I came here because, you know, there were a number of, you know, I thought there were a number of really talented researchers here, and it was kind of a good environment in which to think about safety in the context of AI research that's already being done. So OpenAI was only founded about 18 months ago, is that right? It was about 18 months ago, yeah. And how many staff does it have now?
[0:02:39] So I think there's, last I counted, about 55 people here. Right. So has it been difficult hiring that many people that quickly? I mean, I've actually never worked at a startup before. Our CTO, Greg Brockman, was previously CTO of a startup called Stripe, which now has around 1,000 people or so. So it's definitely hard. He's really good at it. It's not something that I've been super involved in except on the safety side. Yeah.
[0:03:07] I guess it's the Bay Area way to have explosive growth in organizations. And yeah, who's backing it? What's the budget like? So I don't know if I can give exact numbers in the budget. The main donors at this point are Elon Musk, Sam Altman, and Dustin Moskovich through OpenPhil.
[0:03:26] Cool. And it's what you do pretty similar to what's going on at DeepMind? Or are there other important differences? You know, I would say that the general kind of research agenda at OpenAI in its kind of focus on reinforcement learning and in learning across many environments and trying to push forward the boundaries of what's done instead of just focusing on supervised machine learning, I would say that's very similar to DeepMind and probably is one.
[0:03:55] OpenAI is kind of trying to be a smaller organization that focuses on hiring just the people that we want the most.
[0:04:19] Okay. You studied physics, right? You did a PhD in physics?
[0:04:44] And then you switched into AI, or was your physics... So, yeah, my physics work was, in particular, I specialized in biophysics. So I was thinking about models coming from kind of statistical physics and applying them to kind of models of the brain, and then also using techniques from physics and electronics to make measurements to try and validate those models. So, you know, I come from a physics background, but I've been thinking about intelligence for quite a while and how intelligence worked, and I think...
[0:05:13] You know, when I did my PhD, I wanted to understand that by understanding the brain. But, you know, by the time I was done with it and by the time I did a short postdoc, you know.
[0:05:24] AI was starting to get to the point where it was really working in a way that it hadn't worked when I started my PhD. And so I felt like maybe it was starting to be the case that the best way to understand intelligence would be to actually directly work on building parts of it rather than kind of studying the messiness of the brain. So that was what kind of led to that switch. Do you want to give a quick pitch for why working on artificial intelligence is so important?
[0:05:48] Yeah, I mean, you know, I think, you know, you can give, you know, kind of the standard arguments that a lot of people are familiar with, which is that, you know, if you think about any technology.
[0:05:58] that humans have created, what's allowed us to create that technology, sanitation, flight, medicine, improvements in human health, improvements in ability to feed the world. All of this has been generated by our intelligence, and our intelligence is relatively fixed.
[0:06:21] And so if we were able to build something that was able to match or exceed our intelligence, then that would really be increasing the engine that produces a lot of the great things that we do. And ultimately, maybe immediately, maybe it would take a long time, would give us a much more complete control over our own biology and neuroscience, could make us whoever and whatever we want to be, could end conflict, war, disease, that sort of stuff. That sounds a little utopian, but I think if we push this...
[0:06:50] Technology far enough and all goes well, then that'll be the result either immediately when we build it or over a somewhat longer period of time. But I don't see any reason why those things can't happen. So I think that's the basic reason to work on AI. I mean, as I've kind of written, there are these safety issues where we can imagine situations in which...
[0:07:13] It doesn't actually go well. And so to the extent that that's a risk, that's also a risk that we can reduce. And we can also have leverage by focusing particularly on reducing that risk. So on both the positive side and the negative side, it seems like there's just a huge amount of leverage to be had. So the previous stuff I was doing was in biology. And it's great. You can help people. You can try and cure some disease. But this feels like it's more.
[0:07:43] What does the name OpenAI mean? Does that relate to the approach the organization is taking?
[0:08:11] And there's kind of another set of people who I don't think many people think this anymore, but who for a while kind of thought that it was about kind of like making an AGI without any safety precautions and just kind of giving a copy to everyone and that this would somehow solve safety problems. So, you know, yeah, these were kind of two early misconceptions that were around long before I joined OpenAI.
[0:08:41] It's meant to indicate the idea that OpenAI wants the benefits of AI technology to be widely distributed. I see. Rather than only going to the owners. Assuming that safety and control problems are solved and we build AGI, there's then a question about, well, who owns it? What happens with it? What world do we live in after it's created?
[0:09:05] Again, I wasn't the one who named this or set the specific mission statement, but I think Elon's intention with it was trying to think ahead to, given that we've built an AGI and it's not wildly unsafe, how are its benefits?
[0:09:24] OpenAI is a non-profit. It is a non-profit, yeah. So if you developed a really profitable AI, how does that work? OpenAI becomes incredibly rich and then gives out the money to everyone? Yeah, I mean, personally, I have no interest in getting rich from AGI. I mean, I think it would do so many interesting and wonderful things.
[0:09:53] To humanity that I think the meaning of money would change quite a lot and even maybe the psychological motivations that would want me to get a larger share are things I could change and might want to change.
[0:10:11] Right. In many ways, shares in terms of money are maybe not the right way to think about it. But I think there's all kinds of stuff that could happen when AGI happens. And so some of the things I think about are where that could go or what that could mean. And the summary is we don't know very much because it's something we haven't done yet. So a lot of it's speculation. What do you research here at OpenAI?
[0:10:38] So I mainly work on safety. So we have a safety team that so far, myself and Paul Cristiano. So Paul was a co-author of mine on the Concrete Problems paper and has also written a lot online on his blog about AI alignment. He's probably one of the people who's, I think, done the most to kind of promote.
[0:11:07] Clear thinking about the problem and tie it to current AI. We have a third person joining in a few weeks who I'm super excited about. So we're kind of trying to build up a team that focuses on technical safety. We also do kind of a little bit of strategy stuff, which is how do we get different organizations that are working on AI to cooperate with each other? How do we cooperate with policymakers on questions like these?
[0:11:36] We're also thinking a little bit about those issues, but mainly technical safety. I also do some stuff that's not.
[0:11:44] Not strictly technical safety, but is generally done to kind of stay up to date on where AI is currently going. So I did some work on transfer learning a while ago that was, you know, kind of really a little bit safety motivated, but, you know, trying to make environments that are broad enough that it's possible to see kind of a distributional shift or out of distribution problems. So that's kind of the range of stuff I work on.
[0:12:12] What's the organizational culture like here? What kind of people does OpenAI attract? Yeah, I mean, I think we've generally been very selective in who we pick. So I think generally it's people who are very talented machine learning researchers.
[0:12:29] But also people who, I would say not everyone, but a large, large faction of people here really do think in terms of eventually getting to AGI. And at least some people, a significant fraction are quite interested in or at least supportive of safety work related to that or related to what we do. Now, there's kind of a wide, wide range of beliefs on how to work on safety.
[0:12:58] How possible it is to work on safety from our current vantage point. So there's kind of a wide distribution of views, but broadly people are pretty supportive. Right. And OpenAI recently moved away from software development. Is that right? To more focus on machine learning? So that's not quite right. So I think what you're referring to is we had a project called Universe.
[0:13:22] Which actually I was somewhat involved in on the machine learning side. And the idea of that project was to make a lot of environments that agents could learn using. And the way we did this was using something called the VNC protocol to connect directly to a browser through pixels. And so that would allow you to play kind of thousands of flash games and navigate web tasks.
[0:13:46] It turned out to be the case. So I was actually really excited about this because I saw it as a test bed to study safety, right? If you have 100 flash racing games, you can train the agent on one flash racing game and then see how it behaves badly when you transfer it to another flash racing game. You can kind of study some of these open world problems where an agent has a very wide space to explore and a wide range of actions it could take. So this is one thing that...
[0:14:10] ML researchers have been working on is teaching computers to play computer games really well. Yeah, yeah. Like at a superhuman level. Yeah, this is – so DeepMind has worked on this with Atari games. And so we were kind of taking it to another level with kind of any game you can find on the internet. So this ended up being a project that I think could probably be described as a little bit ahead of its time. So it turned out that in order to connect this way, we needed all the different –
[0:14:38] Workers who were applying the R algorithm to be asynchronous with one another and for reasons that were kind of complicated and we only figured out later. Actually, such asynchronous communication was really hard to make it play well with ML. And it led to a lot of complexity. So we're, you know, we're to some extent de-emphasizing that project now. But we've, we're kind of, you know.
[0:15:04] We're actually trying to move to doing the same thing with more synchronous environments. So basically kind of the same idea, but more in a way that's more amenable to ML benchmarking and to measuring how well we're doing and doesn't have this kind of hard to interact with property. So it's more kind of like we made a tool.
[0:15:28] Okay.
[0:15:47] So turning now to the broader issue of superhuman AI development, what do you see as the potential dangers here? Why should anyone be worried about this? Yeah, yeah. So I mean I – my attitude to start off with has always been – although I do think about –
[0:16:08] AGI, which is a term I prefer to use than superintelligence because I think no one knows whether a machine will rocket past human level or not. So that's something that could happen or not. But AGI is something that I think definitely will eventually happen. So I kind of prefer to talk in terms of that. And then even within safety, you know, in concrete problems, I've kind of...
[0:16:30] I explicitly tried to think in terms of not how powerful the systems are, but conceptually what can go wrong with them. So the same kind of thing could go wrong with an AGI as could go wrong with a very simple agent playing a video game or a robot cleaning your house. If it has the wrong objective function, if you don't specify its goal correctly, it can do something unpredictable and therefore dangerous. And so in general, when I talk about safety, I talk about safety kind of...
[0:16:59] Generically, whether it's in powerful systems or very weak systems, all that said, with respect to powerful systems in particular, I mean, I think there is a possibility that if we either do a bad job specifying the goals of complex systems.
[0:17:16] Or just they're unreliable in the way that self-driving cars are unreliable. A self-driving car has to have a very high standard of safety in order to trust it to drive on the road. For almost a decade now, we've had self-driving cars that are 99.9% safe, but that's not enough. We need them 99.999% safe. And so with AGI, which is something that it's going to take a lot.
[0:17:42] More novel strategies than self-driving cars. The space in which it operates is a lot broader than self-driving cars. If you just transpose that kind of safety testing from, you know, self-driving cars to general intelligence, then kind of even with all the controls you put on and even with all the safety standards, it's clear that kind of at the very least we're going to have a big challenge in making sure that something doesn't go wrong. And, you know, if something does go wrong, you know...
[0:18:08] It would be easy for, you know, or it might be easy for a large amount of harm to be done relatively quickly, right? So, you know, you have your AGI controlling the stock market or the economy or something and it just doesn't know how to do it very well yet and something goes wrong and it takes a long time to unwind that. So there's kind of a long tail of, you know.
[0:18:26] Things of varying degrees of badness that could happen. I think at the extreme end is the kind of Nick Bostrom-style fear that an AGI could destroy humanity, and I can't see any reason in principle why that couldn't happen. If it was sufficiently powerful or sufficiently good at accomplishing its goals. If it was sufficiently powerful and safety had been handled sufficiently badly, that's definitely something that can happen.
[0:18:54] So I think there are folks at places like MIRI who say that this is the default outcome or this is really likely to happen or there's almost no way to avoid it or you have to solve some incredibly hard math problem to avoid it. I don't generally agree with any of those things, but I think this is a possible outcome. And at the very least, as a tail risk, we should take it seriously.
[0:19:20] I think another thing I'm worried about is that even if we manage to make a superhuman AI safe or an AGI safe, then it might be used for the wrong ends. Deliberately. Yeah, deliberately used for the wrong ends by a disturbed individual or an organization whose views are not aligned with humanity or nation state whose views are not aligned with humanity. So that's, in my mind, the range of risks. Right.
[0:19:48] So do you think there's much of a chance that the risks are being overblown here and, in fact, we're just going to end up delaying something that could be incredibly useful and make life a lot better? So, I mean, you know, it may very well turn out. Maybe it's more than 50% chance that as we get closer and closer to AGI, then, you know, kind of it becomes kind of clear how to make something safe. Maybe it's just like, you know, the goals are specified in a way that's kind of very cordoned off.
[0:20:17] From the tasks that are done, you know, that there are certain, you know, problems of nature like, you know, scanning brains or something that we need AIs to do for us in order to get, you know, in order to gain control over our biology or control over resources. And then there are human values and maybe there can be an efficient division of labor where there isn't much.
[0:20:38] There isn't much confusion or maybe safety problems are just a bunch of research and they're just a corner of machine learning research where we haven't done much yet and so we haven't tried. And so I can think of lots of ways and maybe it's even the most probable way where things turn out totally fine.
[0:20:56] But, you know, I wouldn't in any of those worlds say the risk was overblown, right? So, you know, it's like, you know, suppose you have a fire alarm and someone's cooking a barbecue and, you know, it's smoke. And, you know, you wouldn't call like installing the firearm overblown, right? It's just sometimes you'll have a fire and sometimes you won't. But, you know, like installing the firearm is the right course of action. So, yeah, I think of this as a precaution. And, you know, I don't...
[0:21:25] Really think of it, I don't think of anything I do as slowing down the rate of AI progress, or at least I'm not trying to do that. I think of it as...
[0:21:35] Broadening the scope of AI progress and thinking about AI in a more kind of interaction and human-centered way. If anything, maybe it accelerates progress a little bit, although that's probably a minor effect. But if people are worried about progress being slowed down, I don't believe anything I do is doing that.
[0:21:56] So, yeah, how much of OpenAI's work is focused on these kinds of problems? It sounded like 5% of the staff, but I guess other people are worried about it too? Yeah, I mean, I think broadly most people at OpenAI are worried about or at least think these issues are worth thinking about. But, you know...
[0:22:17] That's different from, you know, who is actively, you know, doing their technical work on it. So, you know, I would say it's, you know, three or four people now. And, you know, I'm hoping that grows somewhat. And, you know, we're actively looking for really talented people. But, you know, I think OpenAI as an institution has the general idea that, you know, in order to work on AI safety, you have to be at the forefront of AI.
[0:22:46] That, you know, also if you're at the forefront of AI, you have a better ability to implement AI safety in the final system that's built. So, you know, many people are kind of interested in safety in the long run. But I think until recently and even so now, I think many people here don't, you know.
[0:23:03] We're trying to show that there's...
[0:23:25] Concrete work that can be done. And that's kind of had a variety of reactions. Some people are like, yes, this is exactly what you meant with safety work. Now I see how it can be done. Some people are like, well, that's good machine learning work. I don't actually see how it connects to AGI. And so then, you know, we'll try and write another paper and say, okay, this is how, you know, this is the line we're drawing and this is how we think it gets us there. And, you know, it actually could turn out that this is mostly just ML work and the final systems we build are different enough that...
[0:23:52] For whatever reason, this ends up not being relevant to safety. But again, I'm pretty happy in that world. If there was nothing concrete it was possible to work on in safety and I instead ended up doing a different direction in machine learning, then that ends up being fine. Then it will turn out we couldn't have worked on safety until later and then we'll work on safety later. Whereas in the world where it does matter, it's really great and really impactful to get a head start on it. Yeah.
[0:24:19] So I'm curious to get your view on this debate that I've seen online. So you have this contrast, some people like perhaps Bostrom could be accused of this in the book Superintelligence, of talking as though once we have a superhuman AI, then it will get very much smarter very quickly, and it could potentially just solve all of these problems. It could solve war, solve aging, solve all of our health issues.
[0:24:42] But I've seen some people criticizing this online saying, you just think this because you're a bunch of nerds and you think that thinking is the way to change the world, the way that everything gets done. But it's not going to be so simple. Even if you had a very intelligent machine, it wouldn't necessarily be able to solve those problems. Do you have a view on that debate? Yeah, I mean, I think I'd actually, so I think, yeah, not quite. I'd rephrase the debate a little bit.
[0:25:07] I think there's an interesting technical question of like, let's say I built an artificial general intelligence tomorrow. And because it's software, let's say I made 100,000 of them. How much does that fundamentally change our…
[0:25:21] Our society and our technological capability. So, you know, a lot of it is just, you know, you can look at individuals throughout history that, you know, that managed to discover a lot more than other individuals, right? You know, you look at, you know, you look at, you know, Von Neumann or, you know, or Einstein or one of these figures who just, you know, managed to be leaps and bounds ahead of others. And the question is like, what's the ceiling on that? You know, if we...
[0:25:50] You know, invented AGI tomorrow. Would it take like a couple days to scan all of our brains into software, upgrade us, give us indefinite life extension? Or, you know, would it just be kind of like, oh, it's more humans to talk to? And I think it's actually complicated. I think some people act like it's obvious one way or another, but it's not really something I have.
[0:26:14] A lot of certainty on in part because I think, you know, modern science has experienced a lot of kind of diminishing labor, like diminishing return, like the depletion of low hanging fruit. And so, you know, it could turn out that like solving biology is just this kind of.
[0:26:32] When you put it all together, do we get...
[0:26:50] Zoom, you know, to do something much, much, much faster than we ever could? Or do we get just some kind of mild acceleration of what humans can be doing? I feel like many people act as if the answer is obvious. But as someone with a background in biology and even thinking about all the directions in which machines can optimize it, my guess is machines could probably make things happen pretty fast. But I think there's huge uncertainty here. And I don't really think anyone knows what they're talking about on this question.
[0:27:19] My background is in economics, and I imagine if you had an incredibly smart AI and it was trying to figure out macroeconomics, like understand recessions and booms and bust cycle.
[0:27:30] I suppose it could have lots of conceptual breakthroughs, but you can only take the measurements so quickly, and you can't really run experiments. So it could end up being that the processing of the data that we get is extremely good and very fast, but then the data only comes in so quickly, and there's only so much you can actually learn. There's some subtle stuff, which is like, I wouldn't be surprised if, for example...
[0:27:50] I'm pretty agnostic on it, and I don't really...
[0:28:17] I don't really have a good answer on the kind of like nerds think AI can solve everything question. I mean I think there are some deep set problems in human nature. But – and so just solving resource constraints isn't going to solve war. We've probably in some ways already solved resource constraints.
[0:28:40] But maybe having true AGI will allow us to redefine what it means to be human, and we'll ultimately elevate ourselves above our petty human bickering, or maybe the petty human bickering will prevent us from being able to elevate ourselves, so we'll be stuck. So I don't know about that either. We'll reach superhuman levels of petty bickering, perhaps. Yeah, I don't actually know. It's actually very hard to know. Yeah.
[0:29:09] So we've talked, we've mentioned a few times this paper, Concrete Problems in AI Safety. So let's dive into that. But before we discuss those problems, what was your impetus for writing it? So, I mean, you know, I kind of had been aware of the work of the AI safety community for a while, but had, you know, had in general felt that I wasn't particularly happy with the way they were phrasing things. It didn't seem like what they were describing was actionable. And there wasn't a lot of tie to like,
[0:29:39] AGI was generally discussed in these very abstract terms as having a utility function and kind of having incentives to do this or that. And discussing things at these very abstract level, I couldn't help but feel that there were a lot of implicit assumptions that were not really being discussed.
[0:29:57] At the same time, the mainstream machine learning community, which I've been a part of for about a year and a half, having a lot of experience with speech recognition systems, one thing that I found about neural nets is that they're very powerful, but they're very unevenly powerful. So the key example I gave early on was you can train a speech recognition system on...
[0:30:21] 10,000 hours of American-accented data. And then for someone with an American accent, it gets it perfectly. Then you give it someone with a British accent or an Indian accent or something, and it just does terribly on it. And, you know, of course, if you train it on enough...
[0:30:35] Diversity of accents, then it starts generalizing better. But generally, when we build engineering systems, that kind of silent, random failure, it's not something that we see as desirable property in systems we build, particularly safety-critical systems. So the idea that fixing those kind of problems was not just kind of...
[0:31:02] A one by one thing where we're like, oh, we're using a neural net again in a self-driving car on a statistical test for everything we can get. Or, you know, we're using a neural net now in a drone. Let's make sure it doesn't shoot someone that, you know, that we could have principles behind what gives us guarantees on the behavior of a system or at least what gives us statistical guarantees. That seems super interesting to me. And it really didn't seem like seemed like very.
[0:31:32] It seemed like very few people were actually working on it. So, yeah, so, you know, me and some of my colleagues, Chris Crissel at Google, Paul Cristiano, who's now at OpenAI, Jacob Steinhardt at Stanford, John Shulman here, and Dan Manet, who is another Googler, kind of, you know, had...
[0:31:58] All thought a little bit about this problem. And so we decided to kind of get together and write down all of our ideas in a paper that would kind of lay out an agenda for why we think this is a thing. And in particular, I think I felt that the machine learning community as a whole was like a little bit confused. I think that...
[0:32:16] They largely thought AI safety was about, you know, fears that AIs would kind of like malevolently rise up and attack their creators. And even when they didn't think it was about that, they worried that the people who talk about AI safety will feed into fears that it's about that. And so, you know, I felt like this was kind of a silly state of affairs and that, you know, like of course we can do research on making systems safer and more reliable that doesn't kind of trade on these fears.
[0:32:45] And in particular, we can even do research that ultimately points towards AGI. I think the important thing is that we shouldn't go around with every other word we say being AGI. And in particular, the research itself shouldn't be specific to AGI. You can't really research AGI now because we can't build an AGI. So I think a very standard technique when doing research on a topic is if you want to think about a topic that's kind of...
[0:33:11] Yeah, so what are some of the concrete problems? Do you want to tackle one or two here?
[0:33:39] We kind of made a distinction between problems that relate to what happens if you don't have the right objective function and what happens if you do have the right objective function, but something goes wrong in the process of learning or training the system. So, you know, not having the wrong objective function, the extreme version of that is kind of what, you know, what's talked about in, you know, kind of the classical AGI safety stuff, which is, you know, you want to specify a goal and you, you know.
[0:34:08] You kind of, for whatever reason, you know, you had some simple instantiation of the goal and it kind of ends up not quite being the right thing. So, you know, we call that. The genie problem? Yeah, yeah. We call that reward hacking. And so, you know, a few months ago using an environment in the now de-emphasized universe.
[0:34:28] I had kind of an example of like a boat race where the boat's supposed to go around and do a few laps. And what it's trying to do is finish the race as fast as possible. But the only way it's able to get points, and you kind of can't change this because it's the way the game is programmed, is you kind of...
[0:34:48] Thank you. Thank you.
[0:35:11] Well, this is the reward function to this is the behavior that it leads to. It's a very twisted mapping. Yeah. And so the point is that it's not what we would have intended. It's not what you would have intended to be.
[0:35:22] The lesson of that is that it's very easy to make small changes in the reward space and have that lead to big differences in the behavior space and also for that mapping to be very opaque. For you to look at a reward space, think you know what it means, and in actuality, it leads to something very different.
[0:35:41] Yeah. Than what you kind of would have expected. So, you know, we called that generalized reward hacking. And then kind of there was another problem called negative side effects, which is a little related to that, which is just that if your reward function relates to a few things in your environment and your environment is very big, then there's kind of a lot of ways for you to do destructive things. So it's one particular way in which it's easy to specify the wrong reward function.
[0:36:11] Because you haven't put in side constraints? Yeah, you haven't explicitly put in the 10,000 other things you care about. Then there was this thing called scalable supervision, which is if you're a human trying to specify a goal to a machine learning system, even if you have a clear idea of what it is that needs to be done, then you don't have enough time to...
[0:36:38] Control every, you know, to control or give feedback on every action that an AI system does. And, you know, and therefore, you know, limits to your ability to control and supervise can lead to a system behaving in a way you hadn't intended because it's interpolated in the wrong way. So those are kind of the problems with, you know, like the, you know, the classical AI safety type problems. Like you had the wrong, you know, you somehow, you gave yourself the wrong goal in a way that was hard to understand. And then...
[0:37:07] You know, the kind of more technical-ish problems that relate to, you know, your system was trying to do the right thing, but something went wrong. It's things more like this thing we call distributional shift, which is when your training set is different from your testing set. And so, you know, the classical example of this is like, you know, when I was at Google, there was an incident where, you know, Google's photo captioning system had been...
[0:37:34] You know, they had this photo capturing system that was trained on a lot of photos. And it turned out that most of the photos were...
[0:37:41] You know, statistically biased to be photos of Caucasian people. And, you know, and there were also a lot of like animals and monkeys in it. So unfortunately, the system, you know, the system reacted, you know, when a black person took a picture of themselves in the photo, it tagged them as a gorilla because it had only seen humans with white skin. So this was, of course, incredibly offensive. And Google had to apologize for it. And, you know, they even had, you know, they'd even kind of thought of this.
[0:38:10] But the neural net ended up being so screwed up that it didn't even warn them that it was kind of in a region of the state space where it was kind of dangerous. Because the algorithm has no concept of what's offensive. Yeah, the algorithm, yeah, it's just a statistical learning system. It doesn't know about racism. It doesn't know about racial slurs. It doesn't know about what's offensive.
[0:38:38] It's just a learning algorithm, and it just learns from the data it was given. And there turned out to be some problems with the data that it was given, and there turned out to be some problems with the algorithm. And so it just kind of innocently produced this extremely offensive result.
[0:38:57] And the world of neural nets is full of this. I think something related to this distributional shift is adversarial examples, which my colleague Ian Goodfellow works on a lot, which is kind of when you...
[0:39:11] Intentionally and adversarially try to disrupt an input to a machine learning system and make a very small change to it that kind of causes something bad to happen. They're a little complementary. Adversarial examples is like a small but carefully chosen, like, you know, perturbation to it, whereas out of distribution is this kind of like holistic perturbation to it. So resistance against those two is kind of, it's separate, right? You're talking about two orthogonal directions in the perturbation space.
[0:39:41] But, you know, these are all issues with, you know, making sure that when you train something that it, you know, it behaves in a new environment the way you intended to behave or if it goes wrong but it fails gracefully. And we haven't put a lot of, I mean, you know, we put some work into this area. We cite a lot of papers in the concrete problems paper. But I think, you know, kind of relative to the stampede of work in mainline AI, I'd like to see more of this stuff.
[0:40:10] So I think you did an interview with the Future of Life Institute where I think you talk about this paper for about half an hour. So people who are interested can go and listen to that and you'll get more detail on each of those different five problems. But yeah, how do these problems tie together the long-term concerns with the short-term ones that we have today? Yeah, so I think the attempt was kind of to come up with some conceptual problems that relate to both.
[0:40:34] That have long-term and short-term versions, right? So, you know, with something like a distributional shift, the short-term version of it is something like the gorilla. The long-term version would be something like, well, I've, you know, I've trained an AGI in a simulation and then I put it in the real world and a lot of things are different. So does it break a lot of stuff?
[0:40:52] Without meaning to. And the super intelligent version of it is it's like whatever. It's the same but more extreme. It's building a Dyson sphere. It's never built a Dyson sphere before. Something go wrong. Just whatever outlandish thing you can think of. So I think the point and the explicit strategy was that people often kind of contrast.
[0:41:20] Long-term versus short-term approaches as if working on short-term safety and long-term safety are like different topics and like they trade off against each other. What I'd rather do is have a thread running from long-term to short-term things where you kind of –
[0:41:37] Identify what the fundamental problems are, right? And then you work on them on short-term problems. And then as the systems get more powerful, you update your techniques. And it creates this kind of more symbiotic where you're following along. I think safety shouldn't be anything different from reinforcement learning, right? Reinforcement learning is a general paradigm for learning systems. You can use it to do something as simple as walk across a grid all the way up to playing Go, all the way up to, you know.
[0:42:03] Perhaps building a system that's as intelligent as humans. I probably wouldn't literally use reinforcement learning, but the reinforcement learning is a general paradigm that runs from things that are very simple to things that are very complicated. So I guess the idea was to do the same thing for safety. Come up with some general principles that will carry across towards very powerful systems. And I wouldn't say these problems, you know,
[0:42:30] Tell you everything that could go wrong with powerful systems. I think there are most certainly things that are very specific to powerful systems. But my general view is I'm much less confident in our ability to identify those problems. Maybe we can. Some people are trying. But my view is just there seemed like there was a lot sitting on the table. Let's identify the problems we can identify. Let's work on them. And then whatever's left, we either have to work on them very late in the process.
[0:43:00] Yeah, so...
[0:43:07] So the hope is that in order to solve the long-term problems, you want to find cases that are similar today where you can get feedback on whether it's actually helping. Yeah, yeah, exactly. I think that there's kind of a magic of empiricism because it's very easy to kind of engage in long chains of reasoning about a topic that kind of don't get tied back to reality. Of course, the risk of working on short-term stuff is that it doesn't matter or it doesn't generalize. And so the compromise I've come up with is,
[0:43:36] Try and think of things that are conceptually general and then try to tie them into empirics. Yeah, so to that end, has OpenAI made any noticeable progress on these problems or other problems? Yeah, so I think about three weeks ago, Paul Cristiano and I and Tom Brown here and three people at DeepMind, including Jan Lieke, Martin Milczyk, and Shane Laig.
[0:44:04] came out with a paper called Deep Reinforcement Learning from Human Preferences. So this kind of works on the reward hacking, scalable supervision side of things. So the way this paper works is normally you have a reinforcement learning algorithm. It has a goal or a reward function, and the agent acts to maximize that reward function. So this works.
[0:44:27] Pretty well for something like chess or Go, where the behaviors are incredibly complicated, but evaluating the goal is pretty easy, right? With Go, it's like, are you in a winning position? Do you have more territory? And with chess, it's have you checkmated the king, or have you been checkmated? And so it's really easy to evaluate these simple goals with a script, and so you can run the algorithm through millions or even hundreds of millions of games, and the goal evaluation is easy.
[0:44:56] Most of the stuff that we do in real life, the goal is complicated. It's like carry on the conversation or be an effective personal assistant to a human, which means scheduling things for them, making their life easier, but not emailing all their private information to their boss or whatever. There's a lot of context-sensitive stuff, which is part of what leads to safety problems. If I take a complicated...
[0:45:25] Set of goals like that. And I try and force it into the framework of a hard-coded reward function. It's going to lead to something that makes everyone unhappy. Because just the two things don't fit together. Is it that it's maximizing on one dimension and then fails on all the others? Yeah, or just that the intrinsic number of bits of complexity in something like hold a good dialogue is very high.
[0:45:51] If I try and program that in, I'm either going to be programming for a very long time, in which case I'll probably make an error, or if I try and make what I program simple, then, you know...
[0:46:02] Then there's just not going to be enough bits of information to fit the actual complex nature of the goal. So I'm either going to be very error prone or I'm just not going to be capable of learning what I need to learn. And so that's why people talk about strategies for absorbing values and things like this. So what our paper basically does to address this is it replaces the fixed reward function with...
[0:46:30] A neural net based model of the human's reward. So the idea is you have a reinforcement learning agent that's learning. And at the beginning, you know, it starts acting randomly. And every once in a while, it gives it.
[0:46:46] gives some examples of its behavior to a human. So it'll come out with two video clips. And the human looks at the video clips and says, is the left better or is the right better? The human says left is better or right is better. So if it's playing pong or something, then if the left is point got scored on you and the right is you scored a point, then the human will say the right is better. And then the agent builds a model of what...
[0:47:15] Reward function would lie behind the human's expressed preferences. So the reward function becomes something kind of implicit and learned, observed from the human's behavior. And then the RL agent gets to work saying, yep, this is what I think the human's goal is. I'm going to go and try and maximize this. But then it comes back to you and it gives you more examples of behavior. And then, you know, the human decides on those. And so over time, it...
[0:47:40] The human is given kind of more and more subtly different examples of behavior, and the reward predictor in response learns to discriminate them and gets a more refined understanding of what the human prefers, and then the RL algorithm then tries to maximize that.
[0:47:55] And then the consequences of its behavior are then given back to the human. So it's this kind of like... So there's kind of three steps, like the human and then the AI that's trying to figure out what the human's optimizing for. And then the thing that does that. But then like most of the time it's asking the intermediate AI. Is that right? Yeah. Most of the time it's feeding it back into this intermediate model of what the human wants. You have three parts. You have a model of what the human wants. You have the R algorithm that's maximizing that model. And you have the human.
[0:48:24] That feeds, that trains the model, but also the RL algorithm feeds back to the human so that it basically, whatever the RL algorithm has learned to do goes back to the humans. So, you know, it basically says, okay, is this what you wanted? Or of the things I'm now doing, which do you want more? So it's this kind of...
[0:48:45] You know, by no means does this, you know, solve all safety problems. You know, it's just, it's just, you know, it's just.
[0:49:09] One little bit of progress on one safety problem. But this is an example of the sort of thing I'm talking about. And so we use this both to solve ML tasks that you couldn't solve before because the reward functions were too hard to specify. And then kind of the impact on safety is obvious because it allows us to specify goals more easily.
[0:49:32] Other problems you can have with it. It has to scale. There are other safety problems. You don't want AI systems tricking you. There's so much. But this kind of thing is an example of what we did, and I think we're going to try and do a lot more of it.
[0:49:48] So how many times do you have to get feedback from the human to solve these problems? Is it a reasonable number? Yeah, so it depends on the task. But on some of these Atari games, which take about 10 million time steps to learn, usually a human has to give feedback a few thousand times. So less than 1% or a tenth of a percent, the human actually has to pay attention to. We managed to train this kind of simulated...
[0:50:16] We're trying to get that number down.
[0:50:30] So this is the Learning from Human Preferences paper? Yes. Okay, we'll put up a link to that, and you can take a look at this little worm thing here that learns to jump progressively from just flailing around. Yeah, it got a little bit of media coverage, and my favorite headline was what this back-flipping noodle can teach you about AI safety.
[0:50:48] It seems quite a bit. I think it was here's what this back-flipping noodle can teach you about AI safety. That's some good clickbait. Yeah, so apart from those five issues that you talk about in the paper, what do you think are some other kind of important problems or open problems in the field? Yeah, so one thing we didn't discuss in the paper is the issue of transparency of neural nets. So this is kind of trying to figure out why a neural net does what it does, which you could eventually extend to.
[0:51:16] Why is a reinforcement learning system taking the actions it takes, right? It just kind of has a policy. It's in a situation that runs a bunch of things through its neural nets and it says, I'm going to move left or I'm going to bend my joint and doesn't really have much explanation for what it does. So if we could explain why, break down the decisions made by neural nets, then that could help with feedback, could help with making sure that systems do what we want them to do and that they're not doing the right thing for the wrong reasons, which might mean they would do the wrong thing.
[0:51:44] In another circumstance. So I think that's a pretty important problem. My co-author on the paper, Chris Ola, did a lot of work in that area with Deep Dream, which is just kind of like all the back-propagated images generated by neural nets. That was originally designed to be a way to...
[0:52:05] Visualize what maximally activates a given neuron within a neural net. So it was initially a transparency technique. So that's an area that Chris is very excited about. So that's another area I think we should work on. I mentioned before adversarial examples.
[0:52:22] I think that's an area that's already getting a decent amount of attention, but probably should get more, like everything in safety should get more. So I think that's an area we should work on, and also that has short-term safety implications. Someone could sabotage a self-driving car with adversarial examples, and we certainly wouldn't want that to happen. So we can't have that. Yeah. Interesting. So is that a problem for the rollout of self-driving cars now, that someone might put up a...
[0:52:50] A road sign that confuses them? I'm not the expert on it, and I definitely don't want to give anyone any ideas about how to do that. I have no idea. I guess it would certainly end up being criminal, I would think, to do that in the same way as hacking a computer system. Right. It would be extremely illegal. So I don't actually know the details of whether that's feasible or not and wouldn't discuss them if I did. Of course, yeah.
[0:53:17] As we progressively work towards being able to control AIs that we're developing, do you think it's going to be possible for people to understand the solutions that we've developed? So here you've discussed this like...
[0:53:27] Three-step process by which you train a reinforcement learning algorithm to understand the humans, and then that trains the machine learning algorithm on the other side. So I can kind of understand that. And there's other big breakthroughs in history that you can kind of get. Like quantum physics, it's a particle and half a wave, and broadly you can grasp it. Do you think it's going to look like that, or will it just be impossible technical details? The AI itself or the safety?
[0:53:49] I guess the way that we're going to get machine learning or other AI technologies to do what we want rather than flip out in some way we don't expect. I guess there's two possible questions. One is, are we going to understand at a very granular level every decision that's made? And then there's, are we going to understand the principles by which the system operates?
[0:54:10] I think we better understand the principles by which the system operates. If we don't understand those, I don't know how we can build them. And if we did build them, I would definitely worry about their safety. So I think it's realistic to understand the basic principles on which something is built. But then there's the question of on what level of abstraction do we understand it, right? The principles on which a visual neural net are built are very simple. It's back propagation and alternating linear and nonlinear.
[0:54:35] That's pretty much all there is to understand. So then the question is, how much do we know about what goes on inside the neural net? And that's kind of the question of transparency. So I'm optimistic that we'll gain a better understanding of transparency inside neural nets. And the question is, how does that actually help us on safety? How do we actually use it? There's a lot going on inside neural nets. Even if we could individually understand every piece of them, how does that actually help us? There's more units than I can...
[0:55:05] that I can read and understand. So I have to have some way of transducing that into something actionable, like correcting bad behaviors or something. So somehow that component of it has to fall into place as well. And I don't know yet how that's going to happen. I don't know if it's possible, but I think it's an urgently important research area.
[0:55:26] All right, so let's turn now to how someone might be actually able to pursue a career in AI safety. What are the natural paths to getting a job at OpenAI or other similar organizations? Yeah, so I think my advice is going to be focused on kind of...
[0:55:44] The kind of AI safety work that I'm excited about. So for example, you know, Miri does some safety work that's kind of more based on, you know, mathematics and formal logic. And so, you know, if you wanted to do that, you'd need kind of different background. But the safety work that I'm most excited about, I think, you know, just, you know, it sounds obvious, but the two things you most need are, you know, an extremely strong background in machine learning.
[0:56:13] And a real deep, deep interest in AI safety. I think to break those down, I think the first one is certainly at OpenAI, we really try hard to have a really high bar for hiring people.
[0:56:32] Just because someone wants to work on safety doesn't mean that we lower the machine learning bar at all. We have a lot of people here who are very good. So going to get a PhD in machine learning, going to get a PhD with...
[0:56:49] Trying to work with the best people you can work with, do the most groundbreaking work you can do. There's kind of no ceiling to how much of this helps. My sense has been that people who have a deeper understanding of machine learning, if they're interested in AI safety, also tend to really grasp the AI safety issues.
[0:57:13] And that's the second component, which is, you know, I want people who really have a deep interest in safety, not just, you know, oh, you know, oh, you know, it would be good if systems, you know, didn't, you know, it would be good if self-driving cars didn't crash, but have kind of a broad view of, you know, where we're going with AI, which could be totally different from my vision, might not involve AGI, but this general idea that we want to build.
[0:57:42] I think that idea is a broad one, and I want people working on safety to have a broad view of that issue. In the EA community, I don't think the second one is lacking. There are many people who are passionate about the second one, so I think the limiting factor is just very strong machine learning talent.
[0:58:09] Yeah, so we just wrote a career review of doing a machine learning PhD, which we'll put up a link to and you can have a read. Is it machine learning or bust? Are there other options, like other PhDs that people could do that could be relevant, like computer science or philosophy or data science? My PhD wasn't in machine learning. And we have a number of people here who have backgrounds in neuroscience or another area of computer science or mathematics or physics.
[0:58:38] So it's entirely possible if you happen to be educated in another area to go into this field. But I think going forward, if you're a young student, I don't particularly see a case for doing a PhD in another field if what you want to do is machine learning. I mean, I guess I'm saying it's pretty easy to convert skills in related areas. And, you know, sometimes it gives you perspectives that you don't have. But if you want to do machine learning, you should get a PhD in machine learning.
[0:59:08] I think another thing I'd add is we do have some people working here who don't have PhDs. My co-author, Chris Ola, actually never even went to college. He just straight went to Google. And he had to do a lot to prove himself, right? The level of technical...
[0:59:33] I would even go so far as to say it's not my expertise, but even the people doing safety work that doesn't involve machine learning, I get pretty nervous when they don't have a strong background in machine learning because even if they think that,
[1:00:02] If it relates to AI safety, I would say that even those people, I would encourage them to learn as much machine learning as possible, if only because they should understand approaches that they're critiquing.
[1:00:29] Is it fair to say that you think that the approach you're taking where you study machine learning and try to actually improve AGI is the best way to make AGI safe? That you'd rather see someone do that than go into these kind of other adjacent areas?
[1:00:42] It's a little complicated because I think that as systems get more complicated, there may be ways in which we kind of combine neural nets with formal reasoning. So there's been some work by my friend Jeffrey Irving and some of his colleagues on doing theorem proving. So basically using neural nets to select dilemmas to be used for a next theorem. So if you take that far enough, you can imagine versions of...
[1:01:09] reasoning systems that basically, you know, they traverse some well-defined reasoning graph and, you know, they make kind of logical conclusions that are tractable, but it's all driven at the bottom by neural nets driven kind of intuition where the neural nets decide what conclusions you draw and where your thinking goes. And so I think this is how humans...
[1:01:30] Do symbolic tasks like physics or math or anything like that, right? We're neural nets at the bottom, and then we have a layer on top of that that is kind of we use those neural nets to represent symbolic reasoning. And a computer could probably do that even better because it can make sure that it never makes a mistake in the symbolic reasoning, that the symbolic reasoning engine is there. So you can imagine having formal guarantees on that kind of formal reasoning, but I think...
[1:01:55] I actually don't know. Maybe the stuff that's being done now is productive.
[1:02:21] I'm pretty suspicious of anything where you can't get that tight empirical feedback loop because I think it's really easy for people to fool themselves. Okay, yeah. So it sounds like the key piece of advice is to do a PhD in machine learning. So what universities can people go to? What supervisors can they have? Do you have any suggestions there? So I do want to repeat again that it's...
[1:02:41] There are ways not to do a PhD. And in particular, a number of people go to PhD for a year or two and then do an internship here or at DeepMind or somewhere and then get hired. So even partial PhD can work. But that said, I think the usual suspects are places like Stanford, Berkeley, Cambridge, or Oxford in the UK.
[1:03:04] You know, Montreal, Yoshua Bengio's group is, you know, pretty well known for, you know, for doing a lot of good stuff. And then, you know, there are kind of a number of other places, right? And definitely, you know, the PhD pools are where we hire, you know, a lot of our folks because, you know, we know a lot of the relevant professors.
[1:03:30] But again, we have some people here who didn't do the PhD work. I think the most important thing is being able to keep up with the literature and make creative, original discoveries that...
[1:03:40] are novel and that stay on pace with what everyone else is doing. If you can do that, then that's the best thing. When I talk to people who I want to switch into machine learning from another field, the advice I always give them is just get every possible model you can. If you're trying to learn supervised learning, get all the ImageNet models. Implement them yourself. Read a paper, implement it. Read a paper, implement it. Same for supervised learning. Same for generative models. You just kind of get this knack for it after you've...
[1:04:07] Yeah. So I might be showing my nevity to ask this, but machine learning is kind of only one way that you can approach AI, right? There's like other paradigms of how you produce artificial intelligence. Yeah, yeah. This has historically been the case. It's gotten a little bit...
[1:04:36] I think complicated. You know, in the old days, we had kind of things like expert systems that were based on kind of purely unlogical reasoning.
[1:04:45] And, you know, generally we found that those systems were very, very, very brittle because, you know, they couldn't represent the kind of high dimensional space that we see. So, for example, a vision system that's based purely on rules, it's difficult because, you know, like if I'm trying to identify a face or an object, I mean, you know, I'm trying to identify these kind of blobs in distribution space. And so, you know, in some sense, these problems are inherently statistical. And so, you know, the rule based systems.
[1:05:11] Actually don't end up working all that well. But historically, there were statistical systems and there were rule-based systems. I think we can say now that the statistical systems have pretty decisively won. I sometimes hear people say things like, I don't think...
[1:05:34] You know, AGI could be built using machine learning, or I don't think AGI could be built safely using machine learning or something. I think when people say that, they aren't really thinking kind of carefully about the alternatives. So I'm quite sure that kind of like a pure rule-based system is just not going to work because of the thing I said, that you have to ground what you're doing in sensory information, and the sensory information is just inherently statistical and fuzzy.
[1:06:03] So pure rule-based systems, I think, are not going to work. What could happen is the thing I described before, where you have machine learning systems, deep neural nets being used to drive logical reasoning systems. So that would be a hybrid of the two, but people are working that. That would be considered within the field of machine learning. So I think we'll often...
[1:06:27] In the future, machine learning may include logical reasoning processes, but they'll be kind of at a higher layer. And so what ends up happening, it will kind of involve reasoning in the same sense that a Go playing system involves Go, right? But it won't really be like those rule-based systems that we had before. I mean, you can't be 100% sure of this, but I think just the basic argument that like...
[1:06:56] What percepts are these statistical blobs? And so you have to use a statistical system, at least at the beginning, to measure them. And then whatever concepts you draw from them end up being fuzzy statistical concepts. And that if you want to bring those back to logical reasoning, the reasoning has to exist on kind of a...
[1:07:14] So there's not some other AI paradigm that people should be doing deep study in?
[1:07:43] Okay, so assuming someone has been studying machine learning or one of these other related areas that that's a potential path in, is there a natural path from studying to actually getting a job at OpenAI or another organization or other intermediate steps that people have to take?
[1:08:10] You know, again, PhD students who, you know, who do impressive first author work on papers are people who, you know, were generally very excited to interview. So, you know, if you're in a good PhD program and you do some work, then I definitely, you know, do some good work. I definitely encourage you to apply here. For people who are relatively early in their careers or come from another background, there is a program at Google called the...
[1:08:39] What's it called? The brain residency program that allows you to kind of study machine learning with experts for a year. And then that kind of allows you to, you know, to like, you know, to train your skills. And we've had a number of kind of residents who have applied here or elsewhere. Yeah. And so that ends up being a good thing. Yeah.
[1:09:01] Yeah, speaking of which, so there's a bunch of different organizations, right? So there's OpenAI, there's Google DeepMind, Google Brain, Vicarious was another company.
[1:09:10] Maybe more in the past. And there's the human-compatible AI group at Berkeley. Do you want to go through a couple of these that you might recommend working on? So as I said, OpenAI and DeepMind are probably the most focused on reinforcement learning and probably spend the most time thinking about AGI. Not everyone here does, but it's a focus here more than it is elsewhere. Google Brain is where I was before coming here. That was kind of the original research group at Google. And they're, I would say, a more kind of...
[1:09:39] So, you know, we collaborate with them some. We have some interns from there come here. So, you know, I think, you know, Stuart's been someone who's been thinking about safety for a while.
[1:10:04] That's another good place to work on it. Vicarious, to my knowledge, doesn't think about safety. I could be wrong. Are there other groups that we've missed here?
[1:10:16] Are there any government research projects or anything in China? Not that I'm aware of. I mean, of course, there's MIRI and FHI. Right, yeah. So the Future of Humanity Institute and the Machine Intelligence Research Institute. But they're less doing machine learning. Yeah, their focus is less on machine learning, although I think...
[1:10:39] Stuart Armstrong at FHI collaborated a bit with DeepMind on something that I think was like broadly machine learning related. This was like studying interruptibility, incorrigibility or something like that. Yeah. So, yeah. Okay, yeah.
[1:10:54] At 80,000 Hours, we talk to people reasonably often who would be interested in doing a job like what you're doing. Is there any way that they can get indicators early on about whether that's possible or whether they're just kind of wasting their time and should look at other options because it's not going to be a good fit? Either they don't maybe have the machine learning chops or culturally they're not going to be a good fit or some other reason? So I think the thing I mentioned about implementing lots of models very quickly. If you want to know whether you're good,
[1:11:24] A way that is a good proxy for how well you'll do in grad school, as well as for the tests we give when people apply here, is find a machine learning model that's described in a recent paper.
[1:11:37] Implement it and try and get it to work quickly. If this is a painful process for you and you really don't like doing it, then you aren't going to like the research that we do either on AI safety or other AI stuff. If you find you can do this quickly and or you really, really like doing it, you find it addictive, then that's an indicator that this might be something you really want to do.
[1:12:04] I wouldn't worry about the cultural stuff. Like if you're...
[1:12:09] I don't think you'll have any problems. We try and be as open and welcoming as we can. I mean, we don't have the luxury of selecting people on anything other than their... You don't like our favorite TV shows. Yeah, I mean, why? That's just wasteful and pointless. Yeah, absolutely.
[1:12:37] So how early can people do that test? Can they do that as an undergraduate? Is it more like a PhD level thing? I mean, people can do it in high school. Like, you know, if you meet a 17-year-old, like, how do you get into machine learning? I'm like, just go home and implement these models. You know, you actually don't need any kind of formal education. You probably need $1,000 to buy a GPU. I have kind of considered at various times, like, you know, should we have like a, like,
[1:13:04] Yeah. Yeah, why not? Yeah.
[1:13:32] So, okay, so if there's a 17-year-old listening here who wants to go and build their machine learning model, what should they Google for? What should they start reading? I mean, you know, because I'm from the kind of, you know, OpenAI slash DeepMind direction research, you know, thinking about kind of reinforcement learning, you know, trying to... So what's the difference between reinforcement learning and machine learning? So...
[1:13:57] So machine learning is kind of the broader topic, and within it there are several different areas. There's supervised learning, which is where you kind of try and predict some data that's been labeled. So an example of a supervised learning problem would be like you're given images, and they correspond to objects. So this is an image of a dog. This is an image of a cat. This is an image of a computer. And you train the network on lots of pairs of here's the image, here's what it is, and then it learns over time to map the two to each other.
[1:14:26] Supervised learning has this kind of static quality where it's kind of like a one-off. You're trying to predict one thing from another. Reinforcement learning is more a setup where you're...
[1:14:37] So interacting in a more intertwined way with an environment. So the game of Go is like this. You make a move, and then the opponent or your environment makes a move, and then you make a move again, and overall you're trying to win the game. And the reward or figuring out whether you've won the game or how well you're doing can be delayed by a long time. And so the reason I focus a lot on reinforcement learning and why OpenAI focuses a lot on it is that
[1:15:06] You know, reinforcement learning and things like it, extended versions of reinforcement learning.
[1:15:12] Okay, so sorry.
[1:15:41] Cut you off. We were figuring out what the 17-year-old should read to get their foot in the door. So lots of papers in reinforcement learning. I read about what's called DQN, just Google DQN. It's not a common acronym. Deep Q Learning, this was a paper done by DeepMind in 2013. Policy Gradients, in particular A3C, and just kind of follow the trail of recent reinforcement learning.
[1:16:07] Things that have showed up on Archive. Just go to rmachinelearning on Reddit and look at some kind of...
[1:16:13] And what kind of program are you running? Presumably you're not doing this in Excel.
[1:16:42] You know, you'll use the typical is, you know, Python with TensorFlow. So TensorFlow is this tool that the Google Brain team made for, you know, for doing.
[1:16:54] General computations, but in particular deep neural net computations. And so you'll find a large fraction of the stuff is implemented in TensorFlow or some similar framework. So Python is pretty easy to learn. TensorFlow is pretty easy to learn. So read some TensorFlow code of some stuff that's been implemented. Learn TensorFlow and implement some stuff yourself. OK. Great. Yes. So what's the range of roles available?
[1:17:18] Yeah, how do they vary? Okay, so you're talking about within machine learning or within safety-related stuff? I guess mostly within safety, I think. Okay, so I mean, you know, I can talk about what's being done here at OpenAI. I would say there's two main directions. There's the technical safety stuff, and there's, you know,
[1:17:39] The kind of policy side of things. So on the technical safety, there's not a lot of people working on it yet, but the human preferences paper that I showed you is a good example of it. A lot of the papers we cite in Concrete Problems are good examples of this work. DeepMind has had some recent good examples of...
[1:18:03] So, you know, that has some downsides in that.
[1:18:27] You have to set more of your own direction, but it also has some upsides in that you can be one of the first people in a totally new field. And, you know, that's what excited me about writing concrete problems, right? I'm like, you know, I could work on something that 200 other people have worked on, or I could try and set a new direction. And maybe it won't be exciting at all. And, you know, oh, well, at least I did something interesting.
[1:18:50] Or maybe it turns out to be really exciting, and that's a bet that I'm perfectly happy to take. Okay, so what would you recommend to someone who is considering entering the AI safety industry doing machine learning work, but they're worried that they're not going to have such great long-term career options elsewhere, especially compared perhaps to doing machine learning work in a more commercial way with less of a safety focus or just going into whatever kind of pays the most or has the best career?
[1:19:18] So I think ML work is so hot right now that anyone who goes into it, particularly on the fundamental research side, it's easy to transition to applications. And I think the kind of safety work that we're doing has many of the same skills as any other area in machine learning, even though the subject matter is very different. And so I think someone who does that is going to be in a very strong position to do very well in the future. And I think it's probably...
[1:19:48] Even if you're going into it for altruistic reasons, it probably just also happens to be one of the most secure, financially secure career areas you could go into. I mean, we recently had someone leave OpenAI who became head of AI at Tesla, like head of all of AI, reporting directly to Elon Musk.
[1:20:10] You know, I mean, I myself want to stay at OpenAI and work on safety. You know, I want to keep working on the research end all the way until we get AGI whenever that is. But if I didn't want to do that, if I wanted to leave, you know, there's like plenty of wonderful things that I can do. And, you know, the same will be true of other people who come here.
[1:20:30] Right. Is it more of a concern for people who would, say, be working at Miri doing non-machine learning safety research? Yeah, I mean, I think most of the people there are smart people who either had or could have really great careers as software engineers. So they probably have great, great, great options as well. I generally get the sense people who go to Miri are really passionate about Miri's mission in particular and tend to worry about this less.
[1:20:59] But, yeah, I mean, you know, the amount of kind of buzz and hype is definitely not as high as it is for, you know, the machine learning approach. Right. So, yeah, pretty often we talk to people who are, say, in their mid-20s and they did a fairly quantitative degree, maybe like economics or, you know, logic.
[1:21:19] But they don't know machine learning in particular. Is it possible for them to retrain to get into this or is it all just over for them at 25? Yeah. So, I mean, you know, my own example is, you know, until I was 28 or 29 or something, like I hadn't done any machine learning. So it's definitely possible to do this. My main advice is the same advice that I'd give to, you know, like the 17-year-old that we talked about earlier, which is just.
[1:21:45] Just implement as many models as you can as quickly as you can just to see if you have the knack for it and you really enjoy doing it because this is going to be greater than 50% of what your job consists of. Just knowing how to have the real intuition to implement neural net models and have them work, how to put together new architectures that do new things.
[1:22:08] You know, first implementing these papers and then tweaking them. So that's a really cheap way to find out whether this is a career that you're good at and that you'll enjoy.
[1:22:20] Going back and doing another PhD in machine learning, I think once you have a PhD, there are some positions where Google wants you, for instance, wants you to have a PhD when they hire you, but they don't really care what area it's in. They do want to know that you're committed to some new area that you want to go into, but it's more important for the places that care about whether you have a PhD, which we don't care that much, but even for the places that care, I think it's more important that you have a PhD than that you have it in some particular field.
[1:22:49] Right, right. So if you already have a PhD in philosophy, then you should just go and learn ML directly or do some kind of internship somewhere? Yeah, I would say learn ML, implement a bunch of models, then go do an internship or the brain residency program at Google or come do an internship with us or at DeepMind. All these are viable options, and each step gives you a better idea of whether this career path is really for you.
[1:23:15] And so really it's just the case that someone who did an undergraduate degree in economics can kind of jump in and try to run machine learning, try to train machine learning on their own computer. Yeah, I mean I think there's – if you know how to program Python and you can learn TensorFlow quickly, then it's a very empirical field. So of course there's lots of kind of hidden knowledge that researchers know that they kind of –
[1:23:38] Tell each other, but it's hard to express in the papers. And so you won't pick up on everything, but you can certainly get started this way. And then talking to people about models you've implemented, talking to professional researchers to get a sense of what's an exciting thing to work on next. That's kind of enough to get you started.
[1:24:01] And so how does working at OpenAI or Google compare to a machine learning role in academia? So I generally tend, I'm a bit biased, but I generally tend to give people the advice to come to industrial labs. I think one reason is the industrial labs have gotten, you know, and by industrial labs, I kind of mean OpenAI, even though it's not for profit, but just kind of the large non-academic research centers.
[1:24:29] They tend to have more resources, more compute, and in part because of this, I think they've been winning the talent war recently. So I think it still makes sense to go do a PhD. I think staying in academia your whole life, I mean, I guess if you become a professor, then it becomes a lot easier to collaborate with.
[1:24:55] The industrial labs, so both we and DeepMind have people who were or are professors and spend part of their time there and part of their time here. So it's kind of all very feasible, and there's a lot of mobility between the two. But in general, I felt, many people will disagree with me, but I felt that the most groundbreaking work has tended to happen at the industrial labs over the last couple of years at least.
[1:25:20] Interesting. Is there any effort to change that? Are universities trying to catch up or is it just too expensive? Yeah, I mean, they are. You know, there's Yoshua Bengio's group in Montreal is, you know, is quite large. He's one of the few, you know, major figures in deep learning who's kind of resisted the pressure to go into the industrial world. And his lab does, you know, a lot of great work. Peter Beale at Berkeley, Percy Liang at Stanford, and just kind of a number of others, including folks who, you know, don't, who do work.
[1:25:49] There's a lot of interesting work everywhere. But at least the kind of safety that I work on tends to play best with kind of the cutting edge of ML work and explicitly tries to keep up with the cutting edge of ML work. Yeah.
[1:26:09] That reminds me, given the cost, how is OpenAI funded? Is it just donations, or are you also selling products at this point? No, no, we're non-profit. So I think I mentioned earlier the major donors are Elon Musk, Sam Altman, and Dustin Moskovitz at this point. Yeah, so it's just donations. At this point, it's just donations, yes. Yeah, interesting. Is it possible for you to sell things to get extra computational power if you need, or start selling services?
[1:26:38] Or is that a legal issue that's outside your area? Yeah, I'm not an expert in this. I'm not sure you can sell stuff if you're a nonprofit. Right, right. Yeah.
[1:26:48] So is the work frustrating because you're not sure whether your solutions actually exist and you can beat your head against a wall for quite a while before you figure out, well, maybe that there isn't even a way of solving it the way that you thought? Yeah, so I think actually that's the case in any area of machine learning where you're trying to do original research. If you're trying to do something worthwhile, then you don't already know if it can be done and you have to try stuff that seems crazy and it might not work. So it's true of any area. I think it's especially true of an area that's very new, like AI.
[1:27:17] So yeah, I definitely agree that one of the trade-offs for working in AI safety is like, on one hand, you have this exciting ability to work on a new field that's just starting and it could be very impactful. But at the same time, no one's kind of defined what successful work looks like. We're still kind of laying out what the problems are and what the work is that needs to be done.
[1:27:48] It definitely requires an attitude of being willing to do more to define problems yourself, and you need to be more creative instead of doing something that's an incremental improvement on the thing that was done last. To me, that's a good property.
[1:28:11] Yeah, yeah. So is it a good kind of role for someone who has a lot of grit and, you know, willingness to persist with things despite adversity? Yeah, I mean, I think that quality is useful in any, you know, important or original work. And it is here as well. Yeah.
[1:28:29] All right, so turning now to non-machine learning approaches to tackling the problems in AI safety, what kind of non-technical approaches do you see as promising? I interviewed Myles Brundage at the Future of Humanity Institute recently. Do you have a view on any of the AI policy topics that we spoke about? Yeah, so I don't know precisely what you guys spoke about.
[1:28:53] I spend a little bit of my time speaking about the relevant policy issues. So I think if humanity at some point builds AGI, then we're going to have to think about both how to handle safety issues as we're building it and some of the coordination issues that are going to come up with respect to safety and also the question of who uses it, what it's used for.
[1:29:21] One example of this is you can imagine that maybe if it's possible, a really good way to build AGI would be to build an AGI instead of doing anything with it in the world. Try and, if it's possible, first develop a capability to have it advise you on the situation you've put humanity in by building it. And say, look, we've just opened this can of worms by creating you. Can you analyze our strategic situation and say what we should do? Because we're kind of aware that if we don't...
[1:29:49] Right. Get the AI to make the world safe for AI.
[1:30:18] Something to say about AI. Someday they'll have something to say about AGI when we're more in a world where AGI is going to happen. And so what strategies should we take towards all of these actors? How do we make sure that when everything's put together, it leads to a good outcome? And is there anything we can do today to deal with these kind of distant problems?
[1:30:46] And so those are kind of the set of policy issues that we tend to think about. There's also some more thought on kind of short-term policy issues. So how can we get people to think about more mundane issues of safety? Should the government regulate things? What should policies be on self-driving cars and stuff? What should policies be on automation and job creation?
[1:31:13] We do some of that stuff, but a lot of people think about that. So we tend to focus more on the long-range stuff. It's less actionable, but there aren't that many people thinking about it. So we might as well do whatever thinking we can on it, which might be there's nothing that can actionably be done, but we want to at least consider.
[1:31:35] Do you have any thoughts on how we can ensure that all of the players can cooperate and avoid having an arms race where they just try to improve their machine learning techniques really quickly without regard to safety? It seems like you're collaborating a great deal with DeepMind. Things are all very collaborative. Yeah, this was in part motivated by the idea of the organizations working together.
[1:31:57] You know, myself and some of the founders of DeepMind have known each other for a while. We all kind of think about AGI and think that safety issues are important.
[1:32:14] When people at the major organizations are friends with each other and work to actively collaborate, then that reduces the probability of any kind of conflict. Because people know each other, there isn't fear or uncertainty, and if there's a disagreement, we can work it out. So then the question is, how does that scale to there being a lot of...
[1:32:35] How does that scale to there being a lot of organizations? And how does that scale to others who get involved once they see how powerful AI is? Can we make them cooperate as well? And my hope is that we can. But no, it's not an easy thing. Yeah.
[1:32:52] So do you think it would be a good or bad thing if AI were developed sooner? So there's been kind of this explosion of investment in machine learning and improvement. Is this something that we should be pleased about or concerned about or just kind of neutral, not sure whether it's good? Yeah, it's kind of hard to say. I mean, the obvious bad thing is if you're like really afraid that there will be safety problems with AGI, then you might think it was a bad thing. And a lot of people think it's a bad thing for that reason. I mean, my view is, you know,
[1:33:22] We're kind of relatively early in the game, and I think there's a substantial probability that the gloomy analyses are really misunderstanding the safety problem and how the safety problem works. Some counter risks to worry about are that something bad happens to the world in the meantime while we're trying to develop AGI, or that AGI is used in a bad way.
[1:33:51] I guess a couple years ago, I often made the argument that we were in a relatively peaceful geopolitical time, so it would be good if AGI would be built. But I'm starting to wonder, in the last year or so, that maybe we're not in such a peaceful geopolitical state. As we're recording this, everyone is flipping out about North Korea developing intercontinental ballistic missiles. Yes, yes. No, I mean, these things are pretty deeply concerning.
[1:34:17] There's been a lot of political instability in the Western world in the last year. And aside from the usual reasons why this might make me unhappy, it's made me unhappy because it creates a less stable political environment in which AGI would happen. So I don't know. I mean, I will say I think we're better off if AGI is developed in a stable...
[1:34:41] Political environment with leaders who are intelligent and have reasonable views. And so I'd like that to happen. And I no longer know whether that means that AGI should happen soon or that it should happen in a long time. I guess it depends whether the current trends.
[1:34:55] Negative trend. Yeah. I guess it's a difficult thing to tell.
[1:35:25] The hardest safety problems to solve may be problems we can't solve until the last couple of years until we build AGI. So in that case, delay doesn't really help us. It just kind of delays the crunch period that we'll have to face. Because it's like someone trying to finish an essay by a particular deadline. If they know they're only going to do it the night before, it doesn't much matter when the deadline is. It doesn't much matter when the deadline is. So I think it's a complicated question.
[1:35:51] It's not a variable that I have a lot of control over. It's happening at the field level. So I prefer to...
[1:35:58] Try to control variables that I have some control over. One thing I have control over is that it seems like there's at least some safety work that can be done now. And so I'd like to do it. And it seems like there are some ways that different AI organizations are not collaborating now that we can encourage them to collaborate. And so I've also been working on that. And I think those efforts have been successful. And so I feel like it's been good to cause things to happen that wouldn't have caused otherwise. But then there are all these other things that I feel like I have to do.
[1:36:27] No control over whatsoever. Yeah. Well, I've taken up an awful lot of your time here, and I'm sure you have to get back to your research. Yeah, hit these deadlines. So is there anything you'd like to say to people who are considering following your example and doing this kind of research before we finish? Yeah, I mean, we're, of course, hiring for very talented machine learning people who care a lot about AI safety.
[1:36:53] Welcome applications at OpenAI. We collaborated a lot with the deep mind safety people. So I'm always, as part of this collaborative spirit, I think that's a really great team as well. And people should apply there as well. It's kind of convenient to have a place that's in Europe and a place that's in the US. So I think there's a lot of good work going on at several different places. Yeah.
[1:37:21] Yeah, I'll just add 80,000 Hours has been doing a whole lot of research into this question of how can we positively shape the development of artificial intelligence.
[1:37:29] And we're coaching some people to try to help them get jobs at places like OpenAI. So if you feel like you're in a really good position to do that, then fill out the application on our website. We think it's one of the most high-impact roles that someone could take if they're able to do it, which is one of the reasons why we've looked into it so much. Hopefully, over the next few years, we'll see quite a lot more people go into this field and it won't be so neglected and we can be a bit more relaxed about it.
[1:37:58] But it's been fantastic to have you on the show, Dario. Yes, thanks for having me. Hopefully we can check back in in a couple of years and find out what OpenAI has been up to, and hopefully you've found lots of new talented people to work in the area. Hopefully. Fantastic. All right, thanks so much.