[0:00:00] Chris is one of the co-founders of Anthropic, an AI lab focused on the safety of large models. Previously, he led interpretability research at OpenAI and worked at Google Brain. We're very pleased to have you, Chris. Chris, can you hear us? Yes, yes. That's great. Thank you. I will hand over to you now. Fantastic. Well, thank you so much for having me. It's really wonderful to be here.
[0:00:29] And there's my slides, excellent. So I wanted to talk today about something a bit different from what I normally talk about, because usually when I'm presenting, I'm speaking about technical research. But today I wanted to talk about something that I think is very important, which is the safety of AI models. And I'm going to be sharing some thoughts that a number of my colleagues and I have been thinking about. So I'm sure that I'm not the first person speaking today.
[0:01:01] And probably won't be the last to sort of express that AI seems to be going remarkably quickly and really progressing very remarkably. And of course, none of us can know if that's going to continue. But it seems increasingly possible that AI will profoundly impact society and that we're going to build very powerful AI systems. So this might sound really grandiose, right?
[0:01:32] Like historically, if you think about it, most people who believe that their work is going to go and have some kind of highly consequential effect on society, probably most of the time they're mistaken. And so it sounds kind of arrogant to worry about this kind of thing. But I think that the trend, both the capability of the systems that we've already produced and the trend of us producing more and more powerful systems,
[0:02:04] So these brought me to the point where I don't feel like I can dismiss the possibility that we're going to build very, very powerful AI systems. And I think if you're willing to take the thought that we're going to build powerful AI systems seriously, then a natural concern is that we're going to go and build, is to worry about risk.
[0:02:34] Sorry, I'm noticing that there was an image that I was presenting that didn't come through on the last slide. But hopefully that won't continue to be a problem. And already, we don't know how to build safe, reliable, and steerable AI systems. We don't know how to do this for present systems. And it seems like it may be very difficult to go and do that. In fact, it may get more and more difficult as AI systems become more powerful.
[0:03:03] So this makes one quite worried. And I think the truth is we actually have a very limited understanding of the large models we're building. We're often surprised by them. We're often caught off guard by their abilities. We know that neural networks often suddenly develop new capabilities. New capabilities emerge as they get larger, sometimes quite abruptly. I think that sometimes an analogy here can be helpful. So I often like an analogy.
[0:03:39] To biology, where in evolution, you have very simple rules, survival of the fittest, that produce incredible complexity. And it seems to me that in some ways, the situation of machine learning is similar. Of course, we understand neural network training, and we understand neural network architectures. But those very simple structures give rise to really remarkable complexity.
[0:04:10] Sometimes, at least in the West, I see people say things like, oh, you know, deep learning and large models, they're not interesting. You just make them bigger and they get better. And it seems to be actually that's kind of missing the point, that in fact, it's the fact that these simple rules create such remarkable systems and such structure and such capabilities that's so beautiful. And I think there's an aesthetic way in which that's very beautiful. But I think the fact that we're having
[0:04:39] These sort of systems emerge means that just as we shouldn't think that because we understand evolution that we understand all of the organisms that are going to be created, so too we shouldn't expect that we're going to understand all the systems that we build that are created by machine learning. And so we end up in a situation where I think we know relatively little about the risks of these models that we're building and that we're going to build.
[0:05:12] And we know relatively little about how to make them safe right now. And it seems to me that actually a very wide range of possibilities are plausible. So I sometimes like to think about this with a little cartoon. Because you see lots of people with very different views on AI safety, and they often have various arguments for why they see things one way or another. And there's some people who I think are very
[0:05:47] I really believe that safety isn't going to be a problem. That if we can build powerful AI systems, it'll be easy to make them safe. And I think that there are plausible ways in which you could imagine that to be true. I think you could imagine that all you have to do is to go and prompt models. I don't think that's likely, but I think you could imagine that.
[0:06:11] And on the other extreme end, there's many people who are very pessimistic about the safety of AI systems. They really believe that no matter what we do, it's going to be almost impossible to make AI safe. And that also seems to me kind of possible. It seems plausible. But I don't know how I could know that one of these situations is true. I don't know how I could know that it was easy or that it was hard. It seems to me that we just don't have the evidence at this point to know that. And so I think to me and many of my colleagues, it seems more
[0:06:42] We have to be very uncertain. There's a very wide distribution of possibilities. And there's a way in which this creates an interesting picture where I often think of a lot of research on safety as progressively eating probability and moving us towards being able to go and have AI safety work out in progressively harder and harder scenarios. And we don't know how hard things will ultimately be.
[0:07:15] But every time we come up with better technologies for making AI systems safe, we go and we move ourselves a little bit to the right and move a little bit further towards more and more difficult situations. So in the most extreme, easiest situations, it might be that all we have to do is ask the systems to be safe, that we prompt them and we say, ah, you know, you are a brilliant scientist who's wise and kind and peaceful and loves humans and would never hurt humans.
[0:07:45] And then the AI system just does that. And that's all you had to do. And that would be a very lucky world. I don't think that's very likely, but that would be a very lucky world. And maybe we're in a slightly more difficult situation. And then we can go and do reinforcement learning on human feedback. And we can use that to go and make AI systems safe. But I think there's a variety of ways in which that type of work also might fail. And then maybe we can go and use, there's a method we call constitutional AI, where AI give
[0:08:15] AIs give feedback on how the AI should behave. And you could imagine that working in slightly harder situations as well. And with each step, we push the margin of AI safety research forward. And we can go and deal with slightly harder situations. But there's still a very wide range of situations, of different difficulties. And so another way we can think about this is we could try to break it up into different situations. We could go.
[0:08:48] And sort of break up that distribution. We could think about the easy safety scenarios and the intermediate safety scenarios and the pessimistic safety scenarios. We could talk about what we want to do for each of those scenarios. And so we can start with the easy safety scenarios. And in those scenarios, we know more or less actually already how to make AI systems safe. And that leaves many other issues. We have to worry about toxicity, about people deliberately abusing these systems, about the economic impact they're going to have.
[0:09:21] About their geopolitical implications, maybe. And a lot of these are questions for people other than me who think more deeply about policy and issues like this. But just because even if safety, at least technical safety, was solved, that doesn't necessarily mean the problem is easy at that point than with all these other issues. But then we can go and ask about the intermediate safety scenarios. So these are the ones where
[0:09:53] We don't yet know how to make systems safe, but there's a lot of progress that we can make on the margin. We're close to the margin, and maybe if we work really hard, we can figure out how to go and make AI systems safe. And there's actually a lot of natural things that you could do here. So we could go and work on scalable supervision. So this is research where one of the worries that we have about training AI systems.
[0:10:23] Is that as AI systems become smarter, it'll be harder and harder for us to give them feedback. It'll be harder for us to say, you know, you did a good job here because we might not be able to tell if they did a good job. And so we need to somehow address that. And there's ideas like constitutional AI where you have an AI system give feedback. And there's lots of other ideas in this space. So that's one thing that we could work on. Another thing that we could do is we could do process-based learning. So we could say, rather than going and training models based on the outcome, we train them by how they come to the outcome.
[0:10:52] And if we could get really good at that, maybe that's another way in which we could go and make systems safer. And so these are ideas that maybe in more intermediate difficulty scenarios could help. But there's a final kind of scenario that we need to think about, and it's the scariest one. We might be in a pessimistic safety scenario, a scenario where solving safety is very far away, and we're not going to be able to do it on a short timeline, and where perhaps we'll build very powerful systems.
[0:11:25] Before we know how to make them safe. And that's a very worrying thing. And unfortunately, I think that the most pessimistic scenarios one might worry about, often, I think they might look a lot like the optimistic scenarios on the surface. They might fool us. So for example, if a model was very good at manipulating or deceiving us, it might appear safe, even though it wasn't.
[0:12:00] And we've actually already seen small hints in this direction. It's not total speculation. For instance, there's this paper by Ethan Perez et al from Anthropic showing that large language models can exhibit psychophancy, where they go and they infer what you believe and then say things that you'll agree with and try to go in. And even though they obviously don't necessarily do that, because if you believe the opposite thing, they would say the opposite thing to you. So that's, in some ways,
[0:12:29] Sort of moving in the direction of deception, and that's something you might worry about. But it really seems like if you believe this, if you believe that situations might appear, you might have systems that appear safe even though they aren't, then it seems like a really important goal needs to be figuring out whether we're in one of these optimistic scenarios or whether we're in one of these pessimistic scenarios and building tools that can help us tell which of these worlds we're in. Because we want to do very different things.
[0:13:02] If you were in one of these worlds, you know, if we were in an easy world, then we'd want to go and think about really hard about economic impacts. Of course, we should all do that as well. But you could focus on some of these issues. Whereas I think if we were in a world where we knew that things, these systems were really dangerous and that we weren't going to be able to go and solve safety, then we'd need to figure out how we could go and avoid some kind of catastrophe. So how could we tell these apart? How could we know if we were in an easy world?
[0:13:34] In an easy scenario, in an optimistic scenario, or if we were in a pessimistic scenario where it was going to be really hard, how could we know if we have a system that is actually safe or if we just have a system that appears safe? Well, there are a few ideas. I'll go through a few of these in more depth in a minute, but very broadly, one thing you might do is you might try to just test for dangerous failure modes. So as you go and you build more and more capable systems, you might try to test them for things like...
[0:14:05] Deception for their ability to go and do dangerous things for the extent to which they want to do dangerous things. And you could try to test them in various ways that you might think are less vulnerable to them trying to hide things from you. You could also try to understand what's going on inside of them. You could try to understand what algorithms are actually running that are causing this behavior. And there's many types of interoperability. There's a particular type of interoperability that I work on called mechanistic interoperability, which is kind of targeted at this.
[0:14:35] Another thing you might try to understand is how neural networks generalize and how we should expect them to behave in new situations. And maybe that could go and give you some tools. So these are all some things that you might do to try to tell these things apart. So of course, one could try to test models for dangerous capabilities and also for traits like manipulation or dishonesty. With regards to reverse engineering neural networks and trying to learn why they behave in particular ways, I think that it's really worth
[0:15:09] Trying to understand what really are the algorithms that are running. So neural networks are in a lot of ways like we get something kind of like a compiled computer program. We get the weights, the parameters of the neural network are sort of like a binary computer program that runs on the neural network architecture. And a question you could ask is, can we reverse engineer those weights into algorithms? And what I've shown you here is
[0:15:38] There's a vision model, Inception V1, and there's a car detector neuron. There's a neuron that really quite reliably is detecting cars. And we can look at the three neurons in the previous layer it's most connected to. And there's a window detector, a car wheel detector, and a car body detector. And what you see is it wants to see the window at the top. The weights just say that it's going to excite the car detector if there's a window at the top. The wheel is going to excite the car if they're at the bottom, and they're going to inhibit it.
[0:16:08] If they're at the top. And you can see this as a kind of algorithm that's just written in the weights of the neural network. And we can just read it off. And of course, this is only a tiny little fraction of a neural network. But if we can do this for larger and larger portions and go and understand more and more of the network, then we can start to be confident that we understand what it's going to do. And we could tell maybe if it was going to go and do something dangerous or if it wasn't. OK, so in conclusion.
[0:16:44] It seems to me that AI may have a very profound impact on society. We can't know that for sure, but it seems harder and harder to be confident. Well, I don't know how I can be confident that it wouldn't. And it seems harder and harder to me to not be very worried about that. And if AI, if we're going to build very powerful AI systems, I think we should be aware that we don't yet know how to make the AI systems that we build.
[0:17:13] To make systems that we're confident would be safe. And finally, I think if we're willing to entertain these kinds of ideas, we understand safety very poorly. And so rather than fixating on a particular theory of safety or a particular picture of it, I think we should take a wide range of views on many axes seriously. That includes how difficult safety will be, but also just the nature of safety is a problem because we don't yet know.
[0:17:43] There's a lot more in the core views post. So I just summarized a few things, but there's much more in that post, which I was discussing. And yeah, thank you very much for your time. Thanks so much for sharing your insights, Krit. So we have collected some excellent questions from the audience, and we'll have around 10 to 15 minutes for the Q&A. So let's start with the more optimistic scenario. Can you explain what constitutional AI is?
[0:18:17] And what are the pros and cons of this method as compared to RLHF, Reinformed Learning from Human Feedback? Sure. So in RLHF, you have human evaluators go and say, well, you have the model generate two responses. Then you have a human evaluator say, which of these two responses is better, A or B? And then you train the model to go and produce responses more. If the human evaluator says, oh, A is better.
[0:18:45] And then you train the model to go and produce more things like A. And if the human evaluator says B is better, you train the model to go and produce things that are more like B. And this has a few challenges. So one is that the evaluator needs to be able to tell if the model did a good job. And so if the model is doing something subtle where it's hard for humans to go and tell whether it did a good job, that might be a problem. And also,
[0:19:14] I think another disadvantage is it's just not very legible what's actually being optimized for. So it's what the person who's evaluating it likes. But it may come down to the idiosyncratic preferences of the particular group of people who are giving these evaluations. So constitutionally, AI can help with both of these. So the basic idea is that rather than having a human give the feedback and choose which of A or B
[0:19:44] We're going to go and have an AI system, which was trained to be helpful already. And we'll go and we'll say, you know, which of these two responses was more consistent with, and then we have some sentence that describes some goal. So this could be, you can actually read the constitution that we use online, but it contains all kinds of things like, you know, avoiding bias.
[0:20:12] Going and being consistent with various kinds of values. And you can go and say for each of those, you know, is it was response A or B more consistent? And then you go and you select that one. You can do more sophisticated versions as well, where you have it rewrite it to be sort of more consistent with these. But that's the general idea. And so the hope then is that you can avoid the...
[0:20:37] Well, first you can by having an AI system evaluate things. As the AI systems become more powerful, the evaluation can also become better and better. And I think the other thing that's really cool is you end up with this document that describes what the model is doing. And so it's no longer the idiosyncratic preferences of the people who are going and giving the feedback. Instead, there's a document, there's a constitution that says what rules the model is following. And that's really fundamentally what it was trained to do. That's a very helpful response. Thank you, Chris.
[0:21:09] You have been doing interpretability research for many years, and Sam Altman recently mentioned that OpenAI is trying to use GPT-4 to explain some of the neurons in GPT-2. What do you think about the promise of this direction? Yeah, well, maybe just a step back. I think there's a very wide space of approaches to interpretability.
[0:21:34] The approach that I'm most excited about is this mechanistic interoperability where we try to really carefully reverse engineer the neural network, sort of working in terms of small pieces and building it outwards. The disadvantage of this kind of approach is that because we're dealing with such small pieces, there's a question of whether you'll ever be able to go and understand an entire neural network this way. So if I'm reverse engineering the neural network piece by piece,
[0:22:03] One worries about whether they're going to be able to understand the entire model. And so this is the problem of scalability. Can we go and scale mechanistic interoperability to be able to go and work with and fully understand large models? And one proposal for how you might do that is you might do automated interoperability. You might have AI go and automate, help you with the interoperability.
[0:22:34] And go and automate it so that you can go and apply it to very large models. And I think it's very cool. OpenAI sort of had a very, very neat demonstration of this, which sort of demonstrated that this could work to some extent in language models. So that was very neat. I think the challenge is, well, maybe first, I'll give a bit of an analogy for why I'm not. I think it's very exciting, and I also have a little bit of hesitancy. So maybe just in terms of,
[0:23:06] I feel like my hesitancy maybe comes a little bit from a similar place as to why mathematicians maybe are a little nervous about theorems that humans can't understand, where a mathematical theorem is proven by a computer, but we don't, as humans, understand it. And if I'm trying to say that a model is safe, I'd really like to understand myself why it's safe. And I don't want to give that up to a neural network. And I think there's pragmatic reasons for that. I think that
[0:23:35] If I'm going and using a neural network to automate interoperability, probably the model that I'm using is also very powerful. And if the thing that I'm trying to do is test, should I trust this model? Then I think I need to worry about whether the model that I'm testing it with might also be untrustworthy and trying to deceive me in some ways.
[0:24:00] You end up with a sort of reflections on trusting trust. There's this very famous essay in computer science about how if you don't trust your compiler, then you can't trust any of the software that you build with it. And you end up maybe with something kind of like that. So that's a reason I'm a bit hesitant. Now, I think there are other approaches to scalability as well. And a lot of these sort of rely on there being some kind of large scale structure in the model that you can use to go and organize your understanding of the model.
[0:24:29] And there's a lot of research risk there as well. But yeah, in any case, I think scalability is a really important problem, and it's one that we've far from solved. I have a little bit of hesitancy about these sort of automated approaches, but they're certainly better than nothing. And I hope that we'll be able to come up with really reliable solutions to mechanistic interoperability at some point.
[0:24:56] And yeah, I think it's very exciting that we have this work. Great. On the more pessimistic scenario for AI safety, about two weeks ago, you signed a statement on AI risk, which suggests that mitigating the risk of extinction from AI should be a global priority, among other societal scale risks, including pandemics and nuclear war. Other signatories include Geoff Hinton, Sam Altman, Sir Russell, Professor Zhang Yaqin, all at this conference.
[0:25:25] Why did you send this statement and why now? Yeah, I'm very, very deeply worried about the systems that we're building. I think that we understand safety is a problem very poorly right now. I think we understand how to go and build safe systems very poorly right now. And of course, we're working very hard to improve that. But I think that as we build more and more powerful systems,
[0:25:55] I'm quite worried about these things. And I think it's really incumbent on all of us to take this very seriously. You know, it might turn out that we are in fact in an easy scenario, and that safety won't be that hard. It could also be that AI progress grinds to a halt. But I don't think that we could be sure of any of those, either of those things. And I think there's a very significant chance that
[0:26:20] But we're not in the very optimistic scenario. And I think there's a very significant chance that AI progress is going to continue. And so we need to take that very seriously. Yeah. And what will you take for humanity to be safe from this type of extinction risk scenario? And how do we know whether groups like Anthropic is succeeding at this mission? Yeah. Well, I mean, I think that a big part of it is we should continue going and doing good technical work.
[0:26:51] I think it's a very tricky situation. I think it's hard to tell as well because we just understand the situation very poorly. So it's tricky. We have methods that I think probably improve the situation, but I don't think we yet sort of understand it clearly enough to really
[0:27:20] Even be able to tell if we don't know whether we're in a really hard situation or not, we're not going to know whether we've solved the problem. So I think we can make the situation better. And then we can also go and do these more ambitious projects, like mechanistic interoperability, which is something that I've dedicated my career to, to go and try to get to a point where we could really reliably know whether systems are safe. But I think on that front, we're still
[0:27:49] Quite a long ways from that. So yeah, it's a tricky situation. I guess that's why you were suggesting that as a research community, we need to be gathering more information about a type of scenario that we are in. And one of the suggestions you made was we need to be testing for dangerous failure modes. Can you give some examples of the type of dangerous failure modes that you are concerned about? Yeah, well, I think there's a, I guess there's a lot of difference between what are the
[0:28:21] Thank you for raising these important questions. And thank you again for being here with us today, Chris.
[0:28:52] We will wrap up this session. Of course. Thank you. It's been a pleasure to be here.