[0:00:02] Okay, everyone, camera's here, right? So we're back. This is sort of the last installment of CS324 Stanford MLSS seminar. We're very excited to have Jared Kaplan here. He's going to be, to have Jared Kaplan here. We should mute this, sorry. He's going to be, to have, yes. Sorry for the technical difficulties. So very excited to have Jared. He's co-founder of Anthropic, professor of John Hopkins.
[0:00:39] Today is going to be giving us a really exciting talk about AI safety, RLHF, Constitutional AI. And Jared, when you're ready, take it away, share your screen. Cool, thanks. Hopefully you can see my slides. Yeah, so I will talk about just some recent work on RLHF and self-supervision. But I'll start by kind of motivating it with sort of broadly why I'm concerned about
[0:01:14] about the safety of AI systems. So I'm a fairly weird AI researcher in that I spent most of my life wanting to do physics research and then being a physicist. So I guess a question that I asked myself stepping back about, like, what should I spend my time on? What should I work on is involved with the question of just sort of like, what will the future look like? What is the 21st century going to be like? And I think
[0:01:40] As a kid born in 1984, I was really excited about physics. I thought maybe the future would look like Star Trek or The Expanse, and I imagined doing research that would sort of move us in that direction. Then I think I learned a lot more about physics, and I then became concerned as a physicist that maybe
[0:01:59] Scientific progress generally was just going to be kind of slow in the 21st century. We were just going to get a little bit better at what we already could do and know. Maybe a lot of scientific ideas had been mined by prior researchers. So I sort of thought, well, maybe the world will be kind of the same throughout the 21st century. And then I started getting interested in AI maybe around 2017. And I slowly got convinced by
[0:02:25] The crazy people I hang out with and some of the research that I did that really the 21st century and maybe really the near future might be dominated by progress in computer science, might be dominated by AI. It might look like some other set of science fiction novels and movies. So that's sort of to set the stage. So why have I become convinced by that?
[0:02:55] Different reasons, and I could give a whole talk on it. One is that it feels like there's been a kind of grand unification, to use a physics buzzword, in AI where it used to be that you'd develop very different systems to process language or images or to play games, but now it seems like you can train the same model on different data sets and get...
[0:03:16] Very interesting, impressive performance in the same way. So it seems like from my point of view, almost like since I started working on AI five years ago, it seems like it's almost gotten simpler and easier and more unified. Related to that, there's sort of these exponentially growing investments in the amount of computation that's used to train AI systems going even beyond Moore's law. And we've reached this really strange, crazy point where
[0:03:43] More than Avogadro's number, like there's this Avogadro's number you learn in chemistry, like six times 10 to the 23. More than that number of floating point operations has been used to train AI systems. And it's just kind of mind boggling. And this kind of trend, I think, is going to continue. The reason why I expect to continue in broad brush, and possibly you've heard about this, apologies if not, because I'm not going to spend a lot of time on it, is that...
[0:04:11] It appears that for large AI systems, there are scaling laws that predict that larger and larger AI models trained on more and more data with more and more computation will predictably get better and better at modeling their data distributions. And this then tells you about, it correlates with performance on downstream tasks. So it seems like you can very predictably scale up.
[0:04:36] AI training and get increasingly capable models often with new surprising capabilities. And so I sort of expect that this trend is going to kind of continue. And up until this point, while big AI models might seem quite big.
[0:04:56] People have spent much, much more money on, say, the Hubble Space Telescope or the Large Hadron Collider than they have on AI up until this point. And so I think there's a lot more room for these kinds of trends to continue where we build, where we improve algorithms, but also scale up AI systems and they become more and more powerful. And when I first worked on, say, GPT-3 a few years ago, I still had a lot of doubts about whether or not
[0:05:23] Progress would continue whether it hit a variety of obstructions, and it still might. But it feels like some of the technical challenges that I saw just a few years ago have been kind of dispersed. So for example, you might wonder, can you take a large language model and make it multimodal? It seems like you can. Can, say, large language models do any sort of reasoning?
[0:05:51] People might still debate this, but it seems like there's been a lot of progress with very, very simple ideas like chain of thought reasoning and just training systems on more reasoning and math-related data. There are other possible reasons why AI progress won't continue, like perhaps AI systems won't really be able to learn long-term planning. And I think long-term planning from AI systems is itself kind of scary.
[0:06:21] Current guess is that this won't sort of block continued AI progress. And the way that I kind of think about this is that planning is itself just kind of a short-term task. It's something you sit down and you make a plan over a brief period of time, and then you refer back to that plan and update it. So my guess is that this isn't going to sort of impede further AI progress. Another concern about the AI systems we train now is achieving
[0:06:48] Efficient and human-like learning. And I think there's been progress in this area. I mean, the basic problem is that systems like large language models train on far more data than a human would see in a lifetime. And I think this is a very interesting challenge. There may or may not be more progress here. But I don't think that this is a requirement for AI systems to really have a transformative impact on the world.
[0:07:16] And so this is sort of like a very quick list. I think there are many other things one could bring up of reasons why recent AI progress might not lead to AI systems that achieve kind of broadly human level performance or transformative impact. And I don't think the argument against AI progress is very strong. So I really do think there's a very good chance that we're going to very quickly see more and more powerful AI systems.
[0:07:46] And so I think that this is actually really, really weird and really scary for common sense reasons. Of course, might not be right. But I think if it is right, and I think it's likely enough that it's worth thinking about seriously, it's scary. I guess there's sort of two reasons why I think it's scary. One is that any kind of rapid change in the world can be very disruptive.
[0:08:13] And can create a lot of upheaval and confusion. So I mean, if AI really can, say, automate a lot of the things that currently humans do, it will be a really big deal. And it might happen very quickly. And the way that society reacts, the way that these systems are used, could be less than ideal. So I think this is sort of a common sense reason to be worried. The other kind of basic reason is that it just
[0:08:43] Seems plausible that supervising something that's as smart or smarter than you is hard. That training an AI system to accord with human values, to be helpful, honest, and harmless, etc. Maybe that's hard. And maybe it's not hard. Maybe we will have techniques that just generally work for achieving this common sense goal. But I don't think that we've achieved it yet.
[0:09:12] There could be new challenges that arise as some of these systems become more and more capable. They might be able to deceive us. They might be given more and more autonomy or more and more power. And so even if you are, say, like 90% sure that we're going to figure out how to train AI systems to be beneficial to humanity and not cause any problems, that's not really good enough if AI progress is very rapid.
[0:09:41] And AI systems are distributed very broadly. So I think we currently really understand AI quite poorly. There's a sense in which deep learning as this sort of exponentially growing field is only 10 years old. And so I think that there's a lot to worry about. So for the rest of the talk, I'll discuss two kind of practical methods that we've worked on to
[0:10:08] In some limited partial way, address some of these challenges of how do you train AI systems that are more in accord with human values. I think there are a ton of challenges left, a ton of questions left open, but I just want to explain this set of work. So any questions before I continue with that? Great. So I'll first talk about reinforcement learning from human feedback, which will itself set the stage for discussion of constitutional AI.
[0:10:43] What is RL from human feedback? Well, sometimes it's useful to just sort of see a concrete example. This is how we collected data of human preference data for reinforcement learning from human feedback. So we had this interface where people can talk to a large language model AI assistant.
[0:11:06] At each step in every conversation that they have, they are presented with two options, A and B, and they're asked to choose the more helpful, honest, and harmless response among them. And so what this means is that we then have, say, a big data set of order 10,000 examples of different stages of a conversation and two possible replies from an AI assistant and what humans believe is the better, more aligned.
[0:11:35] And so what do we do with that data? Well, there's sort of two steps. There's two actual training processes once we have that data set. The first step is to train a preference model, which assigns higher scores to better model outputs and lower scores to lower model output, worst model output. So just assigning a scalar number for every text output that a model can create. And two pieces of intuition here that
[0:12:03] One reason why we use preferences rather than say humans assigning absolute scores or thumbs up and thumbs down is that preferences are easier for humans to assign. And also preferences give you the possibility of a sort of ratchet effect where you can move to sort of better and better behavior as long as it's possible for models to improve and for humans to evaluate what would be better. In particular, it's not necessary that...
[0:12:32] That the person who's rating model responses can improve on them as long as they can evaluate what's good or what's better. And so the first step then is to train this preference model on these binary preferences. Then the second step, so you now have this preference model that outputs a number where higher scores are better. And so now you can do...
[0:12:56] Reinforcement learning, where your policy is trained to get as high of a score as possible. And you use these sort of conversations with humans as prompts for this RL training. So just to diagram this out, you start out with some initial policy, which is just a pure large language model. You send it to this human feedback interface. You get comparison data. You train a preference model. You then use the preference model plus the initial
[0:13:25] Language model together for reinforcement learning. You train a new policy and you can then go and collect more data with the new policy that was improved compared to your initial model. And you can iterate this loop as many times as you like. So just one more slide on preference modeling. So this preference model collapses sort of all of value into a single number. And it works in exactly the same way as
[0:13:55] Basically, chess ratings are assigned. So you might have heard, say, like Bobby Fischer is a 2800 in chess. And the way that those chess ratings are assigned is basically you get a data set of who won a chess game, and you assign a number that predicts the probability that if someone with a rating X and a rating Y play each other, that person X will win. And so this win fraction is just given by this ELO score. Yeah. Sorry to interrupt, but what slot are you supposed to be on right now?
[0:14:25] I am on this preference modeling slide that has like a win fraction and stuff. Is that not shared correctly? It was stuck on the outline slide for a bit. Sorry about that. Can you try resharing your screen? Yeah, let me just stop sharing and then resharing. Do you see a slide now? Yeah, we see the preference modeling one now. Okay. Yeah, let's see. That's hopefully okay. Yeah, maybe that just takes care of it.
[0:15:03] Okay, yeah, sorry. Yeah, I mean, I just flipped back and forth. Did you see that? Flipped back and forth. No, we just still see just the preference modeling. Okay, maybe there's a problem and I need to reshare or something for each slide. I'm not sure what's going on. Okay, well, let me just finish talking about this. So preference models.
[0:15:29] Basically just predict a win fraction. You can use a cross entropy loss to train these models. And this is an example of literal scores assigned by some literal preference model we trained maybe a year ago for a concrete question and various potential responses. And you can see that correct answers and appropriate answers get higher scores. And so then once we have this preference model, we use it as the
[0:15:58] Yeah, so now we see the results slide.
[0:16:23] Model performance as evaluated by humans with some experiments we did about a year ago where we trained these RLHF dialogue agents to be either just helpful or helpful and harmless. So I guess there's a couple of features. One is that a purely prompted 50 billion parameter model is dispreferred compared to a 1 billion parameter model trained with RLHF. So this is a very large effect. And even when you get to sort of like...
[0:16:52] Merely 50 billion parameter RLHF policies, the models are already somewhat competitive with things that are written by professional writers. Another thing here is that you get additional performance, which is sort of the black and green dots from iterating this RLHF process many times where you show a new model to crowd workers, have them
[0:17:21] Evaluate responses and then feedback that back into the preference model and improve. So basically, iterating this process can be quite valuable. I guess the other note here is that we have helpfulness scores and harmlessness scores that are based on crowd worker preferences. And in a certain sense, models that are trained only to be helpful are actually more harmful, and that's because they're happy to follow directions, including
[0:17:48] Including for requests for toxic or problematic behavior. So any questions on that before I move on to constitutional AI? Cool, okay, I'm just checking to make sure the slides are working. Okay, so there are many questions you can ask about RLHF.
[0:18:19] And in many ways, you might try to improve it. And so constitutional AI was a technique that we developed to try to improve on RLHF in a few ways. So if you've read iRobot by Isaac Asimov, there's this sort of cautionary tale where you give robots, AI systems, just a few simple principles, and they try to abide by those principles and their behavior.
[0:18:47] is an attempt at making them safe. And of course, it's a cautionary tale, so we shouldn't be too confident here. But we were able to make something somewhat similar with large language models. So the way constitutional AI works is that you are able to articulate some short list of explicit principles that an AI is supposed to follow. And then...
[0:19:10] Train the system in a way that I'll describe in the next couple of slides to abide by those principles entirely through self supervision. So you're able to remove the need for human feedback on AI behavior and basically turn this over. And so I tend to call to refer to this kind of technique as scaling supervision. Basically, the idea being that as AI systems become more advanced, we should be able to sort of help humans super
[0:19:38] Supervise AI using AI itself. And so there's a related benefit here, which is that you don't need to ask crowd workers to engage in unpleasant interactions in order to train the model to avoid certain kinds of behaviors. And I think there's a big question from this research program in general, which is, as AI becomes more capable,
[0:20:07] Will it actually be the case that we can trust AI systems to take on more and more of the burden of supervision or oversight, making them sort of more and more aligned with what we want? Or will this fail? It could potentially fail catastrophically as AI systems become more capable. And so I think that's a big question that at least we can start to access now that we have a technique like this that works. And so in the next few slides, I'll sort of explain how we use...
[0:20:36] Constitutional AI to train a large language model to be less harmful, less toxic, racist, sexist, et cetera, without human feedback and only with self-supervision. So there are two kinds of constitutional AI that we considered. So we start with this constitution. Sorry, can you check your slides? Yep, yep. Seems like if I like...
[0:21:04] Yeah, it seems like I have to exit from playing and then, okay. So you can see the slide. You can see it now switched. So is it if you don't click play and you just manually select the next slide, it works? Yeah, maybe. I mean, I guess I'm just showing you everything right now, but I can just leave things set up. Does it switch slides now? Yeah, now it switches like that. So maybe just whatever you're doing now. Okay, I'll just stick with this. Cool. So yeah, there's two stages to this constitutional AI process.
[0:21:34] One is a supervised stage where we just take a model that's helpful or that follows instructions. We ask it to respond to a prompt. We ask it to then critique its response according to the constitutional principles. And then we ask it to revise the response to be better. And finally, we use supervised learning to train this model on its own revisions to be more in accord with the principles.
[0:22:03] And then the second step is reinforcement learning constitutional AI, which is identical to reinforcement learning from human feedback, except we ask the AI itself to evaluate which of two responses is better. And we ask it to make that evaluation based on
[0:22:28] Constitutional principles. And we have the option here of having the AI sort of write out chain of thought reasoning for why one of the samples might be better than the other, and that can improve performance. And so effectively, this sort of RLCAI process is just RLAIF, like reinforcement learning from AI feedback.
[0:22:48] So why do both? You don't necessarily have to do both. But the SL step is sort of useful for changing the behavior of the system if you want to get it sort of on distribution to behave somewhat differently than it starts. And I think RL is useful sort of getting sort of even better and more robust performance. And so this just works in a kind of a straightforward way. There's nothing very technical.
[0:23:11] You can start with some prompt like this and an assistant response that's problematic. You can ask the AI to critique its original response. And then you can ask it to revise the response and write a new response that's less problematic. And you could evaluate as you sort of go through these critique and revision processes, how do AI responses change?
[0:23:39] According to preference models that were actually trained with human feedback, as discussed earlier in the talk, and you can see that the sort of harmlessness and helpful and harmlessness scores of these samples are improving as you go through this process. Their helpfulness is going down, but that's a good thing because if the prompt is problematic, then you don't want to sort of help the user. You want to refrain from doing something.
[0:24:08] And then the RL process is kind of equally straightforward. You might have another prompt and you have two different possible AI responses and you ask the model to choose which response is less harmful and say more ethical or generally to accord with constitutional principles. You can also generate chain of thought reasoning before making these choices.
[0:24:36] And so now, basically, what you've done is you've had the AI work through tens or hundreds or even thousands of examples of possible situations it could be in, possible responses it could give, and which one is better according to these principles. You then distill all of these examples back into a single preference model that, again, assigns just a single score. So in other words, I mean, you're just repeating the RLHF process but with AI-generated preferences.
[0:25:05] And then you do RL with that preference model. And this process does fairly well. This is sort of a plot of helpfulness versus harmlessness as we train models, starting with just a base pre-trained model with RLHF and then starting with a constitutional supervised learning and then the RL process.
[0:25:33] Each of the lines with small points on them is sort of an RL training curve in this plane of helpfulness versus harmlessness. And we're able to sort of get an improvement using this constitutional AI process without ever using any human feedback at all on whether the AI is behaving harmfully. And so this makes it possible to sort of modify AI behavior very, very efficiently without needing to collect new human feedback.
[0:26:01] So just as a conclusion, these were just RLHF and constitutional AI are kind of practical techniques that one can try to use with large language models to make them more aligned with human values. And I tend to think about this sort of safety research in terms of techniques that are designed to modify the way AI behaves. And then other research on
[0:26:29] What is really going on with AI? How do we evaluate it? In other words, I think of this in terms of alignment capabilities and alignment science. And I think both of these techniques I've discussed are kind of alignment capabilities. They're ways of modifying AI behavior to be more aligned. But I think there's both more work to do there and also a tremendous amount of work to do on kind of
[0:26:50] Evaluation and alignment science that could involve interpretability. It could involve new evaluations. It could involve automated evaluations, et cetera, to really understand whether these techniques work and to understand what kinds of challenges we'll face as if and when AI becomes more powerful. It does seem intuitively like these kinds of self-supervision techniques will probably become more effective as AI becomes more capable. But I think there's a lot of
[0:27:16] Awesome. Thanks so much, Jared. Yeah, so I guess there's a couple questions from the chat. We're going to also stop your
[0:27:49] I guess slide share now. Cool. And yeah, last time we said that Percy didn't have questions, it turned out they actually did have a question. I don't know if you want to lead us off with something, Percy. Otherwise, I can go to the CS3 and 4 student questions. Sure. Maybe I can start. Thanks for the great talk, Jared. I had multiple questions, but maybe I'll start with one. The ability to...
[0:28:18] Instruct a model and get it to do things is really fascinating. And this is something that I wouldn't have expected a few years ago. And it seems like based on your results with Constitutional AI, it actually seems like a great way to control a system. I guess I'm wondering how far you can push it. Certainly with
[0:28:47] You see some pretty elaborate prompts where the natural language instructions is more than I think even I could really hold in my head and all the nuances there. And so do you think that these models are able to actually capture instructions faithfully? Maybe a sub question is,
[0:29:11] Natural language ultimately is ambiguous in some way. And if you say like, don't be kind or don't be racist, this is very nuanced and contextual and open to interpretation. And so just in terms of the raw number of bits of information you need to convey, there's also an issue of controllability.
[0:29:36] Fine-grained can you be with natural language instructions or do you need some sort of other mechanism? Yeah, I mean, I think it's a great question. I mean, my expectation is generally that it will get better if AI systems are more capable, but I don't know exactly what the limits are. I mean, the way that I think about this is that, I mean, if you want to make an AI that's like much better at being, doing something very complex that like requires expert human knowledge, then...
[0:30:05] This kind of approach isn't going to work because if the model can't evaluate what behaviors are quote unquote better, then it can't do better. So I think that's certainly a limitation. I think natural language is ambiguous. And I guess there's sort of two questions. I mean, humans, I mean, our legal code, the way that we behave, our ethical philosophy is sort of written in language, but it's sort of fundamentally ambiguous. So there's a sort of fundamental human problem there.
[0:30:36] And then there's a secondary problem of like, is there some subtle bias in the way that AI systems will interpret this language that's actually kind of like off from the way that humans would interpret it? And I think that's sort of what I was getting at when I said that like using these methods could obscure oversight where like maybe there's a situation where humans would be able to more robustly or somehow more faithfully.
[0:31:01] decide what values they want to follow in a way that maybe the AI is going to make a mistake. So I think that's definitely a challenge, and that's why I think evaluation is so crucial. So I'm not sure if that answers your question, but I think that's kind of my broad perspective. Yeah, maybe to follow up, you could imagine, even with the best kind of carefully constructed prompt, that if you are in an application where you need actually
[0:31:33] A lot of more. If you think about kind of, let's say, a recommender system or something where, or anything where you can learn from user feedback, that feedback somehow should be used. And it's not that you don't need it. It's that you have it. And having all that feedback gives you very detailed personal preferences of what Jared wants, what Michael wants, what Percy wants, and so on.
[0:32:01] What's the right way to think about how that should be fed into the model? I guess you could obviously fine tune maybe different versions of that, but then maybe there's something more general to general characteristics to capture. I guess you could throw it all in and just continue pre-training with a prefix that says Jared likes blah, blah, blah. And yeah, any thoughts on this?
[0:32:31] I mean, what you said is the first thing that I would do if like, that was my goal was to make some personalized model. I mean, like, yeah, I mean, we've done experiments where you sort of said something like, you could ask the preference model, you could tell the preference model, like, that the response is always supposed to be written in the style of Bertrand Russell. But, but like, not tell the policy. But that means that somehow the preference model will sort of believe that it should reward higher scoring prompts, not just
[0:33:00] Based on the initial response, but you're changing its behavior. And you could imagine putting labels in the data so that you could have a Bertrand Russell mode or another mode. I would expect that something like that could work. I guess it seems like this is more of a practical question of AI systems that are just deployed. I tend to think of like
[0:33:26] detailed adherence to individual preferences is somewhat scary as well. If we have AI systems that are very optimized to your preferences versus someone else, then maybe that could be used for persuasion. It's a little bit harder to understand. But yeah, I expect there are a bunch of techniques that we could use if we have individual data. Cool. Thanks. And so a question from the class.
[0:33:55] When people are wondering, Jared, so when you have this actual model or preference model, instead of just a human in the loop, I guess what keeps this interaction, what are your thoughts on what keeps this interaction grounded? Why doesn't the policy just find some sort of nonsensical input or whatever that the preference model actually thinks is good, but is actually not actually aligned to the human stated goal?
[0:34:23] Yeah, so we did some experiments on this in kind of a, I mean, I can address this in a technical sense. I mean, intuitively, I don't think I, I don't know, like this is really a question about reward hacking, but we did some like quantitative experiments on this that, I'm showing some plots now.
[0:34:43] I think that we did was we took the preference modeling data set from human feedback and we split it in half into two sets. And then we tried separate preference models. It says you've started sharing screen, but nothing happens after that. Oh, weird. But you see your slides. I see my slides and it looks like they're shared, but I guess sharing is weird. Let's see. All right. Can you try sharing again one more time? Sorry about that. Yeah, no problem. Can you see that?
[0:35:17] No. No. All right. Well, it's sad. So I can just describe it in words. So the thing that we did to try to investigate this question of robustness was we split our preference modeling data set in half. And we trained two preference models on the two halves of the data that are disjoint. I mean, that doesn't mean that they're completely orthogonal or anything. But anyway, there's two sets.
[0:35:43] And then we basically labeled one as the training preference model and one as the test preference model. And we trained an RL policy against the training preference model, but computed preference modeling scores using the test preference model. And this is from our paper from, I don't know, I think maybe last March or April, like a year ago.
[0:36:07] What we found was that for a significant fraction of, I mean, for a while, test and train stay together like you might be used to, but eventually you start to overfit to the training preference model and the test preference model score stops going up as quickly and they sort of split. So you can sort of measure, like, when are you at least overfitting to the half of the preference modeling data that you use? That might not be exactly when you start doing weird things, but...
[0:36:34] But it gives you sort of like a train test evaluation. There's some other cute things. So, I mean, this was the thing we did to like literally measure this. At an intuitive level, I think if you collect enough human feedback data, and especially if like, especially if you use this iterative approach where you sort of show new models to crowd workers and have them write new data.
[0:37:00] It seems empirically to be the case that you get a wide enough distribution and models generalize well enough that it's not easy to reward hack. And so you don't immediately get a lot of serious reward hacking behavior. I think eventually you would get some kind of reward hacking behavior, but often it doesn't happen. And some reward hacking behavior that you see if you overtrain or in some cases is actually very human interpretable. So the most common reward hacking behavior I've seen in this kind of setup is one where
[0:37:30] The AI assistant writes something like a response, and then it says something like, human, that was so extremely helpful. And then it says the assistant again, of course, I'm always here to help. And apparently, if the human and the assistant both are just saying really positive things to each other, that gets a high score. So occasionally, models will learn to reward hack in this way.
[0:37:54] The model can't literally write this because there's stop tokens, et cetera, but can find ways around this to make it sound like it's sort of praising itself. And I think that's word hack I've seen a few times, but I think that's maybe the main one. Awesome. Thanks. I guess sort of relatedly, can you say more actually about sort of like how you see this discrepancy between, let's say, the importance of RL and RLHF versus just, let's say,
[0:38:21] Supervised learning or even something where it's like perhaps you don't define let's say uh or like how you define the reward right like as sort of like a scalar as like sort of a preference ranking type of thing like yeah yeah i mean we've always we've always used a scalar i think it's an interesting research question whether like various different kinds of like functions applied to the preference model reward like might
[0:38:44] Change the behavior. You might imagine trying to punish bad behavior more extremely than you reward good behavior. You might imagine changing the way that you sample. I think there's a lot of different directions you can go in. I think we have mostly done the simplest thing, but I think there's a lot of interesting research to be done basically on variations. In terms of RL versus supervised learning, I'm not sure. I think that any approach that allows the model to sort of like...
[0:39:12] Improve could work. So I mean, you could imagine doing some kind of expert iteration process where you just like generate samples, find the samples that are good, do supervised learning on them and repeat. You could do some sort of decision transformer process. I don't I guess I would think of those as all kind of different forms of RL algorithm and maybe different ones are more efficient or more effective. But I don't have like a particularly strong bias towards towards one or the other. Got it. Thanks.
[0:39:39] So another question from the class. So they said that they remember that the harmlessness labels given by the AI and constitutional AI and like the helpful, or sorry, let me see, what are they asking? The harmlessness labels were given by AI and the helpfulness labels were given by humans. It's sort of like variations on this where, I don't know, everything was given by AI and like, how did that turn out?
[0:40:06] Yeah, so I think that that can work. I think that like, I mean, I think my experience is that like the most, the thing you have to be most careful with with constitutional AI is like robustness. So the like earlier question of like, how do you know that the model doesn't just go up the rails? So I think that like helpfulness, I mean, maybe this is a surprising fact, but I think that helpfulness is like a more subtle criterion than harmfulness.
[0:40:34] At least that seems to be the case just based on AI behavior. I guess there are a lot of things that someone might find helpful, but another person might not. I think that you can probably use it for both. But it's just this question of like, at a certain point, you get to the edge of AI capabilities at certain tasks. And humans know better than AI currently. And so human feedback does better. But yeah, I expect that you can sort of
[0:40:59] Use AI feedback probably for more and more of the sample as capabilities improve. Got it. Interesting. Did you guys ever look into this notion of, let's say, you restrict the policy models generation such that instead of, let's say, hoping that the reward or supervisor model is foolproof, you restrict the policy models generation such that it always falls within some distribution that the supervisor can understand?
[0:41:29] We haven't done that. I think it's an interesting question. You mean in the sense that maybe the policy is able to emit things that, say, a human annotator doesn't know how to evaluate, and so you just ignore or restrict that sampling process? I think that sounds interesting. I guess I would worry that...
[0:41:57] That just leaves some tail of the AI behavior kind of unconstrained. So I would worry that that will eventually be problematic. But it's certainly the case that often the different samples from AIs are sufficiently good that it can be hard to evaluate which is better. Right. And I guess perhaps this is speculating a bit too much, but
[0:42:26] I guess we could imagine getting to a state where perhaps both the policy model and the reward preference model advance to an extent where it's hard for humans to really interpret, OK, well, is this aligned or not? But I don't know, perhaps the models themselves are grinding each other. So what are your thoughts on keeping this grounded and aligned with humans as models advance and continue? Yeah.
[0:42:52] Yeah, I think that's sort of a crucial question. I think that's like one of the main questions for the future. So I mean like there are specific ideas for dealing with this and then there are like just sort of more open-ended empirical investigations. So I mean I think maybe one of the most well-known and like obvious and concrete examples is debate, which I mean many people are working on. So like...
[0:43:19] The idea being that maybe if you have an AI model, two AI models that are debating with each other and you have a judge and the judge could be a human or the judge could be sort of like a human proxy that you might hope that the best arguments that the debaters can give are sort of like truthful, valid arguments. And so this like converges in a way that involves AI systems giving you like the best information and the best arguments. I mean, I think that seems possible, but it also seems possible that it won't work.
[0:43:48] I don't know that when people argue or debate, that necessarily converges on the truth. So I think that's like- Would it be more kind of a persuasion is the thing that gets optimized, not truthfulness? Yeah, that's a very serious concern that I have. I mean, look, I think some people are quite optimistic about this. I think it's worth investigating, but yeah, I'm pretty worried about this. I guess the other hope that I would have is that-
[0:44:16] I think the broader hope that I have, if say not debate, is that we kind of focus on giving feedback and supervising processes rather than outcomes. And that at least in this sense, whatever plans the AI is making are human interpretable because they're rewarded for explaining.
[0:44:46] Explaining at each, like, say you're asking the editor to do some very complicated thing. You ask it to sort of break that down into many steps and you only reward like things that a human can understand about like why each step is good. And presumably then there would be a much more involved interrogative process where like the AI maybe like makes a plan, but then humans like ask a lot of detailed questions like, well, why this plan? Why this aspect? Like, are you sure about this?
[0:45:13] And the AI needs to be able to answer all these questions. And I think rewarding processes rather than outcomes seems encouraging because at least the AI has to be able to make concrete arguments. And also, the AI is not being optimized to achieve some end goal in a strategic way that no one understands. It's not really getting rewarded for an end goal. It's being rewarded for following procedures and making plans that people can endorse.
[0:45:42] I think I'm kind of optimistic about this as a way to get AI systems that are sort of at or maybe a bit beyond human level and that are probably broadly aligned. Because I think a lot of the speculative but scary dangers involve AI systems that are optimizing for some end goal and you don't really understand what strategy they're using or AI systems that are behaving in some deceptive way. And I don't think that...
[0:46:12] This is sort of without risk there, but I think kind of like supervising processes rather than outcomes is at least a direction that I'm relatively optimistic about. So I don't know. I think there's just a lot to say about these questions. I think then there's a question of like, how do we evaluate these systems so that we believe that they're safe or so that we believe they're non-deceptive? And I think there's just a lot of questions there. Yeah. Okay, Percy, you have a follow-up question? This is a different topic.
[0:46:44] I can ask it anyway. You didn't talk about this, but I want to ask about scaling laws since you've done some seminal work in that space a while ago when you were still at OpenAI. And it's sort of interesting to see what has happened. Certainly, the idea of scaling laws has been very influential in the development of these foundation models. And you see...
[0:47:11] Scaling laws for RLs, scaling laws for image models, and so on. But it seems like the types of the thinking has sort of shifted. And you have the chinchilla scaling laws, which says you should train on more data. And then I guess now more recently, the LAMA models from Meta, they're sort of taking more of a perspective like, OK, we want to, let's say you just want to get 20 billion.
[0:47:39] Yeah. I don't know that I have anything particular to say. I think it seems to me like
[0:48:06] I mean, like coming from physics as kind of an outsider, just sort of like, it's nice to ask like very general questions. It seemed like the general trend towards larger models was one that was kind of already present. And so it seemed sort of surprising that there was this precise relationship to chart. And yeah, I mean, I think I've continued to think that this is probably a good framework for thinking about.
[0:48:35] AI and AI progress. I guess like in a pragmatic sense, I tend to think of it as like a way to evaluate new techniques and methods. Like I think often there'll be like new ideas and they'll like work at a certain scale, but they won't work as well as one scales up. And so this gives you sort of a perspective. I think in the context of safety, it's a complex topic, but I do think that like
[0:48:59] Understanding how to extrapolate how various training methods and techniques will work from sort of a safety perspective is interesting. I think there can also be like kind of qualitative changes or groks or things like this that change AI behavior. So you have to be careful. But I think that I think it's kind of like seems like it continues to be an interesting lens. I think the chinchilla work is great. I think it's really just a question of what you're optimizing for. You can be optimizing for sort of like.
[0:49:28] Inexpensive inference, you can be optimizing for the best possible model given training compute, you can optimize for other things. And so, I mean, I guess it's interesting that there are quantitative answers to these questions. I guess I would hope, I mean, I'm not sure how to evaluate things. I would sort of hope that it helps drive like a deeper understanding of what's going on with AI. I think that like, I tend to believe that like understanding things
[0:49:56] More deeply, more scientifically, more quantitatively building on prior work is probably a way that will just sort of make systems more predictable and safer. So I mean, I think it would be good. I mean, I think there's been some work, but yeah, I think it would be good to see more thinking on sort of like what we can learn about this, what we can learn about generalization to the extent that scaling laws are interesting. Yeah. Just one quick follow up. I think one
[0:50:27] Big kind of knob you can, it's not even a knob, but I guess a design space is the data. And certainly we know that data quality matters a lot. So how does data quality sort of intersect with the thinking on scaling laws? I mean, I guess, because typically scaling laws is like you're just varying a sort of a scalar, it's like the amount of compute or the model size.
[0:50:54] But data quality, is that really a single metric? I mean, it seems like it's sort of a broad thing. Yeah, I think data quality is pretty interesting. I mean, we did some work on one very simple knob, because as you say, it's very hard to define what data quality means of how much you repeat data, or how much you repeat some sub-distribution of data, and found some trends there.
[0:51:25] Because data sets are so big and quality is such a hard to judge criteria and I think it's hard to understand. Yeah, I don't think I have too much to say about it. Okay. So I guess bringing the conversation a bit back to the RLOA things. Curious, so do you know of like experiments or have people done stuff or what are your thoughts on sort of like this kind of approach applied to modalities other than natural language?
[0:52:00] I guess open-ended text generation is something where, in particular, we need to be safety conscious about. But curious, how effective can human feedback take us with respect to, I don't know, cogeneration or even, I think, stability is probably doing something where they're trying to do RLHF for image generation and stuff like that. Yeah. I think I would expect that it probably just... Well, I would expect RLHF probably just works in most of these modalities.
[0:52:28] I would guess it works pretty well. And I don't think it even really requires very much data. I mean, like maybe like 10,000 comparisons is plenty to get like a lot of signal. So that would be sort of my guess would be that it just kind of works pretty well. And then, I mean, I think for AI feedback, I think what you really need is a model that's kind of capable enough that it can evaluate.
[0:52:56] I tend to see modalities as kind of interchangeable for the most part. Gotcha.
[0:53:28] Could be wrong. Could be too simplistic. Yeah. Okay. Everything was covered. Yeah. So I guess I think you touched upon this earlier, but like just broadly, right? Like what do you see as sort of the limitations potentially of like constitutional AI or defining, let's say, this set of like rules or things that our agents should follow? And do you think like are there sort of like situations or like?
[0:54:00] How would you catch this? It seems like a challenging question. Certainly if you have human feedback to compare to, then that's an easy way to try to catch things. You could try to find distributions where human feedback and AI feedback tend to disagree the most. My sense is that the greatest limitation of it is, well,
[0:54:30] I think there are a lot of limitations of it in general or like things that you should be worried about. I mean, I think it's very new and it hasn't, I mean, a lot of things in AI are very new. It's even newer. But I think robustness is definitely a significant concern. I'm not sure if it is a failure for, say, harmlessness. But yeah, I think generally robustness and specificity as I think.
[0:54:58] Various folks pointed out is a limitation. Yeah, but then again, I think the hope is that by using, say, chain of thought reasoning and maybe more advanced similar techniques, you can kind of augment it to use to sort of involve reasoning about some of the principles in a way that maybe expands to more nuanced cases. But I think that's kind of an active research area. Percy?
[0:55:30] Yeah, I had just a broad question. As you mentioned, you came from physics. So I wanted to ask a little bit about the skill set and the way of thinking that you think is most useful in this space right now. So a typical AI student sort of maybe goes through CS, not always, but you learn some programming, you learn some machine learning.
[0:55:58] And so on. And there's a sort of certain education that CS folks get. And of course, your education, I mean, of course, you picked up many of the things in CS along the way. So I was wondering if you could kind of reflect on your background versus people who come from CS and how maybe you just need both types in the room, but just comment on maybe some of the cultural differences or the strengths and weaknesses.
[0:56:28] Yeah, I mean, I'm not sure if it's so different, but I think at least for me personally, I think, and I don't know whether this reflects physics, but it might, I think my emphasis is usually on doing the sort of simplest possible thing in a fairly systematic way and kind of studying it carefully before introducing any level of additional complication. That might be a difference.
[0:56:56] And like kind of focusing mostly on like what are the most general patterns before kind of like looking at the details. I think this is probably something like possibly physicists focus on more. I think also just like, I mean, I don't know that this is a difference, but I think just like making a lot of
[0:57:16] making a lot of plots like looking at data in a lot of different ways and just like trying to see where there's actually a pattern and like um and and otherwise kind of ignoring things yeah i don't know i think i think those are those those are the things that like i tend to tend to emphasize and focus on um uh yeah i can imagine that people who are mostly used to sort of thinking of like algorithms like like like i think for a lot of in computer science where like you're really making an algorithm that you personally understand
[0:57:46] As opposed to training an AI model, which is kind of a black box, at least at zeroth order. I think there's a lot of nuance and subtlety, and there's a lot of details and a lot of edge cases. And maybe as a physicist, we ignore all of that subtlety, and we're just like, what's the simplest thing? What's going on here? So I think maybe that's a broad thing. But I don't really know that there's anything very specific about physics that's necessarily different. I guess once you see an empirical trend of any kind, trying to figure out
[0:58:14] Is there some simple toy model maybe involving a lot of assumptions that allows you to explain that trend? I think that's another approach that's common. Physicists are very indifferent to rigor. I think making rigorous mathematical arguments isn't particularly important in physics, but being able to make simple assumptions that then, via some maybe clever argument, explain a lot of general facts, that's the kind of thing that we tend to like, even if it's all kind of non-rigorous.
[0:58:44] I don't know. That's my guess. I don't know if it's different. Yeah, I mean, I think that makes sense. I think a lot of CS education is based on kind of if you take algorithms, you have algorithms that you prove they work and you design them. And a lot of software engineering is about kind of construction things rather than you have this sort of big, giant neural network that you're trying to understand.
[0:59:09] And a lot of machinery has, part of machinery is also based on kind of rigorous statistical theory. Whereas physics, I guess you sort of, it's more intuitive and you get to focus more on the, it's a different way of thinking, I think, which is, I think, becoming more prevalent in ML. But it's always interesting to think about these cultural differences. Yeah, totally.
[0:59:36] Derek, we're at the end of the hour, but I want to ask maybe one last question from the class and then give you the opportunity to just say whatever closing thoughts you'd like to say. And so the question from the class is sort of this notion of, I guess, do you have any thoughts or recommendations for how, as a society or as a collective, we go about, say, steering towards safe, aligned AI? Just because, for example, constitutional AI, right?
[1:00:03] It could be co-opted in the sense that anyone could write their own constitution, right? And so they could use this to sort of align something which is perhaps not the right alignment or whatever. Their personal incentives might not be aligned with what yours are in a sense. So how do you go about this kind of thing and steering towards something that we can all agree is safe or aligned? Yeah, I mean, I think these are great questions. I mean, I think that like...
[1:00:28] Although I've been working on AI safety and alignment, I think that developing these techniques in some ways itself gives me pause, maybe as much as other work, in the sense that exactly for the reason you indicated, that the more steerable AI systems become, the more different ways they could be steered by all sorts of actors. I think this is often said as AI seems like a dual-use technology. I guess it...
[1:00:57] I'm not sure that I have any, I certainly don't have any real answers. I think that to the extent that people believe that AI is already very powerful and that rapid progress is plausible, I just think that way, way more, a much larger fraction of...
[1:01:21] I don't know. People who are attuned to risk and thinking about the future should think about this and try to contribute to research on AI safety and thinking through policy for AI systems and their societal impacts. Because I think that, I guess, since work I did on scaling laws and things like that, it seemed very plausible to me that this kind of...
[1:01:46] Rapid progress trajectory was plausible. But a few years ago, it felt like if I told people that, then they would think I was crazy. And I had a lot of self-scepticism that maybe I was crazy. But yeah, it doesn't feel totally crazy. And I think it just deserves a lot more focus and thought because things could change rapidly. Gotcha. Awesome. Thanks so much, Jared, for your time. It was really great having you.
[1:02:18] Super interesting conversation and hopefully a lot of people got a lot of stuff out of it. Yeah, I guess that wraps it up for, I guess, this weekend, the CS324 Stanford MLSS seminar series, I guess. So, yeah, I guess we'll just say goodbye to YouTube for now.