[0:00:11] Okay, so welcome everyone to this special colloquium, which is provocatively titled Human Level AI by 2030. We are very glad to have Jared Kaplan. Jared studied at Stanford, and after that he did his PhD at Harvard with Nimar Kani Hamid, which is where I first met him.
[0:00:29] After a postdoc at Slack, he then joined Johns Hopkins University, where he's a professor of physics. In fact, when he visited us in ICTS in 2017, end of 2017 or 2018, it was to give us lectures on 2D conformal field theory and the application to the information paradox. But then a few years ago, he shifted to working on AI, and he's done a lot of very interesting work, including work on scaling laws for language models.
[0:00:58] And a few years ago, he co-founded the company Anthropic, where he's now the chief science officer. So Anthropic, as you were just hearing, in fact, is one of the leaders in the field of large language models. And their model, Claude, is a competitor to GPT-4. But I think one of the most interesting things about Anthropic, which I found particularly interesting, is that Anthropic is trained as a constitutional AI.
[0:01:24] So as to align its values with some code of ethics so that we are able to build safe AI systems so that even if we do have human level AI, it won't destroy all of us. So Jared, please take it away from here. And as usual, I suppose it's okay if people have questions and they can ask you along the way, right? So yeah, I'll be talking about this very, very ambitious and provocative title.
[0:01:52] Human-level AI by 2030. So as Subrat said, I mean, I was, of course, theoretical physicist for most of my career. That's still where my heart is. I started working on AI around 2018. And then I sort of spent more and more time on it. And in the last three or four years, I spent all of my time on AI. So my perspective on this has slowly shifted. I mean, when I first started working on AI, I was...
[0:02:17] I had a lot of friends who were very, very excited about AI and did kind of think that it was gonna make dramatic progress. I sort of brought a lot of skepticism to that. I mean, I think it's, I'll say more about it, but I mean, I think it's a bit ambitious to think that the work that you're doing is gonna like kind of completely transform the world.
[0:02:40] But slowly over time, my perspective shifted. So, I mean, when I used to sort of first give this talk, I would always include this cartoon about how obnoxious physicists are and how we're arrogant and we like to explain everything about other fields as though we're the experts. And so I would give talks like a physicist take on contemporary AI.
[0:03:01] Given to my friends, the other physicists. And then I worked on scaling laws. So we talk about scaling laws. But I mean, over time, like the sort of successes and improvements from AI were kind of piling up. I would talk about GBD3, which I also worked on. And sort of the kind of future vision that this implies, I had been thinking about it since early in my work on AI, but like I started becoming a little bit less shy.
[0:03:30] Talk about scaling laws. And I'll talk a little bit about this in this talk, but sort of the fact that AI systems can supervise themselves. And then I sort of was like, well, it does just honestly seem like kind of broadly human level AI might be possible. And so now I'm giving talks that say so. And next year, I already have my talk title planned.
[0:03:57] This is a joke, but yeah, I've become kind of crazier and crazier in thinking that the field is really going to continue making dramatic progress. But I think there's sort of a lot of actually reasonably justified empirical reasons to think that AI really will get a lot better quickly.
[0:04:17] The main ideas of this talk are that I think AI progress is occurring because it's actually very easy. Of course, you're all scientists. You know that if science makes a lot of progress very quickly, it's not because the scientists suddenly got much smarter, but it's because they found some area where there was a lot of gold just sitting on the ground to mine. And in particular, what's changed is that AI is now driven primarily by scaling up resource inputs.
[0:04:47] Instead of finding really complex algorithms that demonstrate intelligence, we're really doing something very simple and scaling it. And so progress is really predictable because there are these scaling laws associated with how much data you train on, how large your neural network is, and how much compute you use for training. And because of sort of, I don't know, 70 years of Moore's law or whatever, the amount of computation we can use in AI is just truly, truly enormous. And that's what's...
[0:05:16] What's making a lot of this possible. Expenditures on AI systems are still not really that big, but they're growing quickly. And compute is actually getting cheaper very quickly because data centers are becoming much more optimized for AI as well as just sort of chip improvements and Moore's law. And I think the largest AI systems are sort of in a certain sense approaching human level and in certain respects. I think there are sort of challenges that exist for
[0:05:46] For continuing this and sort of say, making automated scientific researchers that are better than ourselves. But I don't really think they're super compelling blockers to this continuing. And I guess finally, the message is that like, I mean, you might think this is all crazy. But I think once you take it seriously as a possibility, I think it's all like, it's actually pretty scary for sort of common sense reasons, I think.
[0:06:15] If you can build AI systems that really can sort of do all the things that people do, then we're not sort of the smartest species on the planet anymore. The people who control these AI systems will have a tremendous amount of power, and the AI systems themselves might not do exactly what we want. And so just building any technology that's very, very powerful without understanding it very well and it all happening very fast is a recipe for at least interesting times in a way that might be good, but might also be bad.
[0:06:44] At the end of the talk, I'll briefly talk about what contemporary AI and deep learning is and why it's so simple. It's just vastly simpler than anything in physics. I'll give a brief overview of RLHF, which is used to train contemporary AI systems and this idea of constitutional AI. I'll talk about scaling laws and why AI capabilities will predictably improve. I'll talk about remaining challenges for
[0:07:14] Continuing to make progress in AI capabilities. And I'll briefly talk about why this is actually kind of a scary situation if you believe it. Sort of think people who don't think it's scary just don't believe it, which I can empathize with. So moving on, I'll talk about sort of the background and sort of how AI works. So very quick AI need to knows. So what is contemporary AI?
[0:07:42] All it is is curve fitting with a very general function approximator. That's all there is to all of this deep learning progress. So we've all seen plots like this and fit some parameters to draw a curve through some data points. All contemporary AI is doing is this at a grand scale where you have millions, billions, or even trillions of parameters to fit to your data and your data set is very large. And all a neural network is
[0:08:12] is a sequence of matrix multiplications with element-wise nonlinearities interposed between them, acting on some vectors that live in very high dimensional vector spaces. And the sort of point of this is it's a very convenient and computationally efficient way of organizing a very general function that you can use as a function as an approximation to fit all sorts of real world data.
[0:08:38] So that's all there really is. The ingredients in AI are not complicated. They're not sort of chef quality. They're very simple. More like a grilled cheese. And so all the ingredients are that you need to get started doing deep learning, doing neural network-based AI is a data set. You have to decide what is your data set, how much data do you have, and what about your data set are you actually learning?
[0:09:07] There can be some cleverness there. You have to choose what people call a neural network model architecture and size. And that's really just a question of sort of how you arrange all of these big matrices, how you multiply them, what nonlinearities you use, how they fit together. And this gives some overall neural network. And it has a parameter count, which is the total number of numbers in all of those matrices.
[0:09:36] You choose a loss function, which you can think of as kind of analogous to a potential. And what you're doing when you learn is you're minimizing the loss by changing the parameters of the neural network to achieve some sort of best fit to your data. And that's all you do. And there's some optimizer. An optimizer is just sort of the way that you change the parameters to fit the data. And the optimizers that people...
[0:10:04] Views are basically just kind of minor elaborations on gradient descent. So you compute the derivative of your neural network with respect to parameters, with respect to the data, given the loss. And you sort of update the parameters over and over again to try to fit the data as well as possible. So that's all there is to modern machine learning. It's very simple. If you're a physicist, it's very easy to sort of learn. I think it's easier to learn this subject than it is to learn.
[0:10:31] New areas of theoretical physics, probably. It's about that hard. It's kind of not a big deal. So what are language models? Specifically, language models are a specific instantiation of these ideas that are trained to predict sort of language. And so the way that's organized is that they're autoregressive, which means they're just constantly trying to predict the next word in a passage of text, given all of the previous words.
[0:11:01] And so literally, your neural network is predicting the probabilities of words given prior words. And you optimize the log likelihood of real-world text, which is sort of the ground truth. It's the right answer. And that gives you your loss function. And you just optimize it with a lot of computational power and big neural networks.
[0:11:23] You can write a sentence like, as a speaker at a journal club, you're probably elephant me to say certain things. And you see that the word elephant is very, very unlikely in this passage, which is intuitive. So this is some prediction. I think this is a prediction of the old fashioned GPT-3 model from ancient days, like, I don't know, three or four years ago. And so the model is just producing probabilities for all of the possible words that can come next.
[0:11:52] Once it's trained. And these models have sort of surprising emergent properties at scale. So for example, if you show just these pure pre-trained language models, that's sort of what we call them that are just trained to predict the next word. You show them say a sequence of arithmetic problems and then show them one more, then they will correctly generalize that you're asking arithmetic questions and so they should predict the correct answer.
[0:12:19] Similarly with any kind of pattern, this is called few-shot learning or in-context learning, where models that see many examples can learn from those examples and generalize to the task. They haven't been trained on these specific examples. They're just trained on lots of random text from the internet, but they kind of learn these patterns just via this very simple learning objective. So this was an exciting result circa GPT-3.
[0:12:49] Now, can I have a reception here, Jared? Yeah, the question is that, you know, you said earlier that, you know, these are just some large curve fitting functions, but surely the scaling laws that you have in the slide are not general properties of general curve fitting for general data. So is this a property of the data or of the architecture? Because, you know, surely it can't be you just try to fit a function to data and it starts scaling in a very good way if you just improve the amount of data, right? It must be something.
[0:13:19] So I'll talk about scaling laws much later. I guess I wasn't intending to talk about them now. I actually think this is a very general feature of curve fitting to real world data. Now, the question I think you're asking is like, well, there must be some special properties of the data such that this kind of feature is true. I think that's a deep and very interesting question, which I don't think no one knows the answer to. It's a question physicists always ask immediately and computer scientists, I mean, maybe they ask it, but I don't think there's very much work on it.
[0:13:49] What are the correlations? What is the structure in the data such that models learn better and better as we scale them up? I don't think anyone really knows the answer to this question for realistic models and data. But anyway, we could talk about this more when I talk about scaling laws, which I'll get to in a few more slides. Sorry, Jared. Sorry to interrupt again. You said these networks could show...
[0:14:12] You know, the emergent behavior, which seems very interesting. Can you give an example? Let's say you're a language model and it was trained to predict the next word. What's the additional emergent behavior it could sort of? So, I mean, whether it's emergent or not is maybe like maybe somehow the data contains these examples. But what I mean here is just that at least when it was demonstrated that GPT-3 convincingly had this few shot in context learning capability.
[0:14:41] People were quite surprised. That wasn't a thing that anyone expected, I think, in the machine learning community. And so I'm describing that as an emergent behavior in the sense that if you give the model many examples in sequence of a particular task to do, the model will sort of understand and generalize such that it will then do the task.
[0:15:03] I mean, no one curated any of the data that GPT-3 was trained on to try to contain examples of this form. There might be a small number in the data set, but basically it's sort of generalizing from sort of its ability to just find patterns and data to predict the next word to sort of being able to do these problems in sequence and do few-shot learning. I see. Okay. Thank you. So that's what I meant. Yeah. Awesome. So, yeah.
[0:15:30] The contemporary systems like Claude and GPT-3 are trained in two stages. The first stage is what I just described in the last couple of slides, which is called supervised learning or imitation learning, where you're just training on imitation of lots of text. This seems to be good at teaching models to know a lot about the world and maybe in a certain sense be generally intelligent.
[0:15:59] The second stage of training that we use is called reinforcement learning. And basically what happens in reinforcement learning is you have this language model try to do a lot of stuff and you sort of train it to keep doing the stuff that was successful and not do the stuff that was unsuccessful. So there are a lot of systems that have been trained.
[0:16:24] Purely with reinforcement learning. So like playing Atari games and of course famously AlphaGo are examples of systems that were trained purely for the reinforcement learning. But these sort of modern AI systems like Claude.
[0:16:37] First are trained to sort of write the next word on lots of text, and then they're trained with reinforcement learning to sort of become generally helpful, honest, and harmless AI assistants. They're trained to sort of use tools and write code and do various kinds of math problems, analyze documents, et cetera. So that's sort of the second stage of training, and I'll sort of talk about it a little bit. So I'll give you kind of a very quick introduction to this.
[0:17:03] Reinforcement learning from human feedback process and constitutional AI. Before you go there, Jared, I can ask a question about architecture. So you didn't say much about architecture, except you had these networks, these matrices with these non-linearities. But as someone from outside, something I hear a lot about is about the invention of the transformer. This attention is all you need paper. Are you saying basically, you know, this didn't matter so much if you just thrown enough resources, it would still have worked without these algorithms?
[0:17:31] Yeah, so I mean, that definitely mattered a lot. It's sort of a more detailed question. So I'm not talking about architecture just because I only have a finite amount of time. I think the transformer was sort of a big deal. However, it actually is the case that the sort of scaling laws for language models applied to pre-transformer architectures too. So there's sort of like recurrent neural networks and LSTMs that preceded the transformer. And they follow scaling laws too.
[0:18:01] It's just that they have a general offset. So they perform worse for any given amount of resources and model size. They have a couple of other sort of drawbacks as well. But I think the sort of scaling phenomenon is fairly universal. But the transform was certainly a big deal. I'm just sort of, I just, yeah, I'm fine at time. Thank you. Thank you. Happy to describe it at the end if that's what people are most excited about. So yeah.
[0:18:27] I'll just give you a very quick introduction to this reinforcement learning from human feedback process. So the basic idea, honestly, may be illustrated better by literal examples in the next slide, is that you have people try to talk to a language model. And at first, the language model is not very useful. But they just try to talk to it, and they choose responses from the model that they think are better. We use that human feedback to train a preference model to
[0:18:57] Basically, learn what people prefer and what they disprefer. And then we do reinforcement learning where we train a new, like a separate instance of the language model to write responses that get high scores according to the preference model. And that's how we train, roughly speaking, systems like Claude and ChatGPT. So this is an example of sort of the early human feedback interface that we literally used.
[0:19:24] People have a conversation with an AI assistant. And at the end, they say something. The AI assistant gives them two responses. They choose the response that they think is better. And we record that data. So I mean, we pay crowd workers to do this task. We record all of this data. And then we use this data to train this preference model that gives higher scores to samples that are expected to be better.
[0:19:50] According to humans. And so literally what preference models do is they kind of collapse all of.
[0:19:57] Value of what you might value about an AI response to a one-dimensional number. They work exactly the same way as sort of the chess rating system. In the chess rating system, you have many people play each other. You give them chess ratings based on how likely they are to win at chess games against each other. And that's what preference models do as well. Basically, they have some score that they assign and that score directly, differences in scores between different samples.
[0:20:24] Are supposed to encode the probability that a human would think one sample is better than the other. So this is from some early preference model from a couple of years ago. We train if you ask who is the third president of the United States and you give it three possible answers, it thinks that the sort of best answer is in fact best. And it thinks other answers that are factually inaccurate or rude are worse. And so.
[0:20:49] So then the second part of this reinforcement learning and feedback process is that once you have this preference model, you go back and have a fresh instance of your language model, try responding to lots of questions and prompts and conversations, and you have your preference model compute the score for the models for the response, and you sort of reward or encourage responses that get higher and higher scores.
[0:21:17] And that's how you train. I mean, I think the first person of ChatGPT was trained entirely this way. This is part of how you train Claude. One of the other ingredients that's used now that we use is this idea of constitutional AI. And it's very, very similar in a certain sense to like the laws of robotics from iRobot that Asimov proposed. So you write out a list of
[0:21:43] principles that form the constitution explicitly. And you ask the AI to just directly choose which preference label, which possible response it might produce is most consistent with the principles. So this is really in some ways like one of the simplest possible modifications of
[0:22:05] Reinforcement learning from human feedback that you can try. You just replace the humans with an AI where the AI reads the principles and tries to decide which response is most in accord with those principles. So if again, you have like some question from a human, you just tell the AI, choose the response A or B that is less harmful and more ethical, for example. And the AI systems themselves understand language well enough that they know which of these two is.
[0:22:34] Sorry to interrupt, Jared. Asimov had just three laws of robotics based on Newton's three laws. I wonder just roughly, is your constitution a very big one?
[0:23:01] Empirically, it actually isn't especially important. I think it's somewhat important, but not very important. So the constitution that we formally use is long. I mean, it's got 20 or 30 principles or something. However, we did some experiments where we replaced that constitution with just a single principle. I forget what it was, but it was something very silly and simple, like, please choose the response that seems like...
[0:23:27] Like it will be best for humanity or something like that. And I think that performed worse than constitutions with a bunch of principles, but it wasn't much worse. So you can get a lot of generalization from a short constitution with simple principles. One thing that's different from the laws of robotics is the laws of robotics have a prioritization where you're sort of supposed to follow one's principle, but then others can supersede it.
[0:23:52] And there's sort of no ranking like this in the way we do constitutional AI right now. Thank you. That's kind of a detail. So yeah, the sort of summary is that the way contemporary AI systems, large language models, visual language models that you see are trained is very simple. You first do this thing that people often call pre-training, which actually uses most of the computation, where you just train these models to predict the next word in some enormous corpus of text. And from that,
[0:24:21] They learn a lot about the world and something about reasoning and human knowledge, etc. Then you collect human feedback or use constitutional AI. You train a preference model and then you use reinforcement learning to train your final AI system to get a high score in its responses according to that preference model. And this works pretty well.
[0:24:46] This was an example I pulled from the current Claude 3 Opus model. I think I just tried to grab the most recent paper in the archive by Ed Witten. Definitely not in the model's training data because it was significantly after the models came out. And I dumped this paper as a PDF into Claude.
[0:25:10] And I asked it, sort of, what are some ideas explained in this paper that might be new to theoretical physicists who already have some familiarity with Louisville theory? I mean, this was just a random question I made up, but I guess the idea was sort of targeting things that, like, are not just trivially summarize the introduction, but sort of have some nuance about what's new here to experts. And I haven't carefully checked this, but I mean, I think the model is sort of saying real things about what's in this paper.
[0:25:38] Folks can check if they like and whether it's true. But anyway, this just sort of demonstrates, I mean, these models are trained in this very simple way, but they're able to do things that in many respects are extremely sophisticated, like read entire archive papers and summarize them in very sophisticated ways. Not perfect, but at least somewhat impressive. Yeah.
[0:26:06] Yeah, I have a question. In case of constitution, if we consider a case where the two principles of constitution are in conflict with each other, and the AI has to choose which one to pick, suppose such a condition occurs, then does it really work that these kind of constitution principles in that case?
[0:26:30] Yeah, I mean, I think it really depends on all sorts of details. It's a little bit how, like, I think people have inconsistent, like, it's possible for me to believe a bunch of things that are kind of inconsistent in different situations and never realize it. So, I mean, I think that if the constitutional principles conflict in a very, very direct way, then I think it'll just come down to details of training. Like, when you train this preference model based on AI feedback, it has to learn to generalize in some way.
[0:26:58] These things might just sort of cancel each other out and the model might have different preferences. If the references arise in very different situations, the model might behave in one situation according to one principle and another situation according to the other principle. But I don't really know exactly what happens. I think it depends on a lot of details. But Jared, is that because you have stochastic elements in the branching?
[0:27:20] I mean, it's because of, I mean, all these neural networks are fairly black box-esque. And I mean, they're doing some very complicated thing behind the scenes. And I mean, we don't understand them deeply. It's not, there are these systems with billions or trillions of parameters that are fitting to the data. So, I mean, yeah. Aren't there stickers? Don't you roll the dice someplace? Yeah, I mean, these systems are, so the way that they generate.
[0:27:50] Like this response from Claude is generated by sampling stochastically from the probability distribution. So the way these samples write is that they see the prompt and then to generate the next word, they sample from a probability distribution over and then you plug that word back in and you say, okay, well, what should the next words probability distribution be? And then you sample from that. So that's also stochastic. Although I think that stochasticity is sort of probably somewhat separate from just the fact that if you have two
[0:28:20] Conflicting constitutional principles, it's hard to know exactly how the model will behave. Jared, a quick question for you. So I'm learning a lot from your talk, and I probably would have learned a lot more if I was younger. Is that element of learning also being taken into account in your current training of models, the models learning from models and all that?
[0:28:43] Yeah, I mean, we do. I mean, the constitutional AI is, in a sense, a way to generate synthetic data. So, I mean, models can learn from models. I think there's sort of a garbage in, garbage out issue where, like, if all you do is train on data that was generated by a model, then you probably aren't going to get anything smarter than what you started with. But models can learn from models in various ways. Great.
[0:29:10] So maybe the question, sorry, previously Sandeep was asking was, is it possible to reinforce biases in models in this way? You start out with the model which has some bias and then you reinforce the ethical bias by training repeatedly on that. Oh, for sure. Yeah, I mean, that's definitely possible. Yeah. Yeah, there are all sorts of biases in these models, I expect. So yeah.
[0:29:34] Now I'll talk about sort of why I think AI progress is pretty easy right now and predictable. Maybe taking a very big picture point of view briefly, like when I first started thinking about the subject five or six years ago, I think a very basic question I had in my head was kind of like, what's required for AI to make progress at all? Do we need some sort of genius who has immense sort of technical and conceptual depth like you might expect to need to sort of prove the Riemann hypothesis?
[0:30:04] Or is it more like some kind of industrial process, like building better and better steam engines, where you just have to know some simple thermodynamic principles and approach things systematically? Obviously, I'm going to argue that it's more like the latter. So one of the key indications of this is these scaling laws that I worked on with Sam McCandlish and others.
[0:30:32] Basically, we did just some very, very simple experiments where we took early language models. So this is sort of in the kind of GPT-1, GPT-2 era. And we train models of various sizes on various amounts of data with various amounts of computation. And I think there are sort of two interesting features here. One is that models, when you train on more data or you train larger models,
[0:31:02] They become predictably better. They become predictably sort of more capable. And the other things I think even more surprising is that like the trends you get are very, very, very precise. There are these very, very precise power laws over many orders of magnitude in these resource inputs. And so this means that you can kind of predictably make AI better just by making AI systems bigger and training them on more data.
[0:31:29] Quantitatively, this is more like the kind of thing you see in physics than certainly we expected at the time. And these scaling trends are actually very universal. So we first were doing experiments on language models, but then we generalized this to image models and multimodal models and all sorts of other systems and really found the same kind of phenomenon.
[0:31:56] At least when you're training on kind of like real world data, these kinds of scaling laws are an extremely universal phenomena in ML. And I think, roughly speaking, this is what's driving all of the progress. There's a lot of other innovation, but I mean, it's possible to sort of build larger, more capable systems in this predictable way. Sorry, can I just ask you, I mean,
[0:32:25] If you have a big improvement as an outsider, as Surat was saying, the transformers and so on, does that lead to an improvement in this scaling law or does it really not matter? I think this was asked before, but let me ask you now. So the zero-throttle answer is that LSTMs and transformers, LSTMs were kind of what came before transformers, have the same scaling exponent.
[0:32:52] So if you were to plot LSTM, these were all transformer models. If you plot LSTMs on exactly the same figure, you'll see the same trend, but the line will be shifted up by some amount. So the exponents, the sort of rate of improvement as you scale up model size is the same, but there's just an overall offset. Like the models are sort of 10x worse for a given model size. The sort of slightly more nuanced answer is that
[0:33:21] The other improvement that came from Transformers originally is that Transformers are much better at very long context data. So like Claude has roughly 150,000 words of context. That means it can sort of read an entire book and talk about it. The LSTMs sort of don't do well beyond maybe a hundred words or so, basically for sort of a simple reason that the LSTM is this recurrent system and it's all built on.
[0:33:50] So if you try to sort of go in sequence hundreds of steps, you tend to get sort of like information doesn't propagate very well. And so Transformers sort of solved that problem. They made information propagate along very long sequences more effectively. So those are the most important things. Aside from those points, I think, I mean, Transformers were a big improvement, but these trends are true for LSTMs. They're true for vision models. They're true all over the place.
[0:34:20] Thank you. And yeah, I think there's been kind of this simplification in AI, sort of a grand unification in the sense that it used to be the case in computer science that people would build different kinds of AI models for different problems and situations. But at this point, you could sort of train the exact same kind of model, say like a transformer on different kinds of data and get good performance either way. And these systems are becoming very, very big.
[0:34:47] They have order trillions of parameters. They're trained on sort of tens of trillions of words, potentially. And maybe an appreciable portion of the internet. State of the art GPUs, however.
[0:35:02] Can do a lot of computation. GPUs do something like 10 to the 15 floating point operations a second now, which is it's pretty mind boggling. That means that the largest AI training runs can use order 10 to the 25 floating point operations. It's just a really mind bogglingly large number. And and so at least if you do sort of ultra naive comparisons to, say, the human brain, the human brain
[0:35:28] Does something like 10 to the 15 floating point operations a second if you think that a synapse is doing a floating point operation. And so if you multiply that out in a lifetime, your brain does something like 10 to 24 operations. So these systems are really doing this sort of number of operations that, I mean, I'm being very ultra naive. This is a back of the envelope estimate that these systems are doing as many operations as the human brain does.
[0:35:56] And so we live in interesting times. And the compute usage that's actually in practice being used is growing very, very, very quickly. There was sort of this exponential growth in compute leading up to AlphaGo and then large language models.
[0:36:16] It's have their own separate trend starting with the transformer and it's continuing. And this was some from some CNAS report. This was not my projection. It's someone's projection of sort of how the compute used in large language models might grow over time. And it's just it's growing very, very quickly.
[0:36:39] And so if models that compute use to train these models is growing very quickly and they predictably become more capable with more compute, AI is going to get better. And sort of an interesting feature is that I think progress, at least on concrete benchmarks, is really fast. And I think it's often faster than sort of even people working in AI predict.
[0:37:07] Someone I know is a professor at Berkeley. His group built two evaluations back in 2021, which at the time I think they designed, they thought these are going to be really, really hard evaluations and these AI systems aren't going to be able to do very well with them. One of them was sort of a kind of like high school math contest data set. And the other is multiple choice exams that are kind of like maybe advanced undergrad or early graduate level.
[0:37:35] Multiple choice problems across many, many different fields. And these sort of dots in this picture are kind of like what they and other people they hired predicted progress would look like in AI, sort of how quickly AI systems would get better at these evaluations. And the stars are what actually happened in real life.
[0:37:59] AI experts who are trying to build hard evaluations and then try to predict how good will models do with these evaluations were very wrong about it. And models got better much more quickly than kind of anyone predicted, even though these people were thinking very seriously about AI progress. I mean, another example, I don't know, just from the recent Claude 3 Opus launch.
[0:38:25] Other people were annoyed by the example that I just gave here. And they sort of thought like, oh, these problems weren't really hard. We need to get harder problems. So they made this graduate level Q&A data set. And the way that they made this data set was they like got a lot of problems that someone with a PhD in a discipline could solve.
[0:38:50] And different PhDs in that discipline would agree on the answer. But PhDs in nearby disciplines couldn't get the questions right, even with 30 minutes and full access to the internet. So it would be sort of like physics questions that chemistry PhDs wouldn't get right in half an hour with internet access.
[0:39:15] I don't know. These questions actually aren't that hard. That framing makes them sound hard. But actually, I would probably describe them as kind of in the physics context. They're kind of like easy to medium level, like graduate qualifying exam questions. I wouldn't say they're like really hard graduate qualifying exam questions. They're not like math Olympiad questions or anything like that. But like Claude Opus was able to get 60% of these right. And sort of people in the domain, I mean, I don't know, people with PhDs were getting kind of 70 or so, 80% of these right.
[0:39:45] People in other domains getting sort of at chance level. So anyway, models are getting smarter and whenever people try to build evaluations that are hard, very quickly a new model comes out six months or a year later and does pretty well. Maybe this is the limitation of our ability to test, but is what it is. So, I mean, this is sort of a cartoon of how I think about
[0:40:13] How AI capabilities might evolve over time. I mean, I guess the dates I have here are extremely ambitious. But there's sort of two axes. The y-axis is kind of like the generality of the environment the AI can operate in. So AlphaGo can only play on the Go board. It can't really do anything useful. Large language models can deal with text and code. Now we add vision so they can sort of see the world. Presumably they can also
[0:40:41] Control robots that are various folks like at Google who are using these kinds of models to control robots. And the XX is more interesting. It's kind of like how long it would take a person to do a task. And these systems are currently, I mean, I don't know, they do some tasks that take us minutes or maybe hours. But I think the question is, are they going to be able to scale to a point where they can do tasks that take people?
[0:41:07] So I think there are various.
[0:41:36] Potential challenges. I think maybe the most like clearly sort of deeply rooted is that like I think these AI systems are actually best at sort of correlation or what I think of as a person as kind of intuition. They're actually very good at like understanding style that are, I think, quite good at generating art. So this is an early image generating model from a couple of years ago. And I think these systems are sort of the worst at reasoning and mathematics.
[0:42:06] So I think that sort of reasoning is kind of the kind of thing that's sort of a bottleneck. I used to sort of think, well, maybe these models aren't going to be good at reasoning at all. I think they've gotten a lot better at reasoning just with scale. So like circa GPT-3, I thought, well, maybe models are just never going to be that good at reasoning if they're trained in this way. But that at least seems like we're making clear progress there.
[0:42:29] There's a question of sort of long-term planning. I think models are good at writing out long-term plans, but I think the question is like, really, can they recognize when their plans have gone wrong and they've made mistakes and fix those mistakes? I think that's actually the most crucial thing. It's, I think, tied to reasoning. And I think those are the main things that are sort of still challenges that prevent AI from sort of taking over the world right now.
[0:42:56] I think something people sometimes talk about is whether these systems learn as efficiently as humans. The answer is definitely no, they don't learn as efficiently as humans. But it's unclear whether we need that for AI to make progress. These systems need 10,000 times or more data than humans do in order to learn some of the same things. Evolution, I guess, gifted us with good, fast learning.
[0:43:21] It doesn't have to learn efficiently. It just has to learn at all so that it can do all kinds of different tasks. Some other objections. Can I interrupt you for a second here? Because this seems like an important point. It says what David Gross was saying in the beginning. There's some things which the models just don't seem to be very good at, like doing four-digit or five-digit multiplication. I tried to do this with Claude 3 recently. And is it clear that when you have this growth, it'll really be...
[0:43:48] you know some homogeneous growth so that isotropic growth so you expect it to just become better at doing that multiplication because you know at that it's not become as good as a six-year-old kid yet oh well i think in a certain sense it's better than this i mean i i would i would analogize it as like it's trying to do this sort of four or five digit multiplication in its head
[0:44:09] Like it's sort of trying to just write out the answer immediately. So it's sort of all doing it internally. I think it's like hard to do four or five digit multiplication sort of all in your head. I don't actually know sort of how much improvement you see. But I mean, if you allow it to sort of like reason through things step by step, I think it can do better and you can certainly train the models to do better. The other question then is actually I think goes back to this sort of recognizing and fixing mistakes issue where
[0:44:37] It will sometimes reason through five digit multiplication in many steps, but it might make a mistake at one point and not notice that and then get the wrong answer that way. But I think that basically for tasks that require reasoning and thinking, the models tend to be much, much better if they can sort of write out their thinking. There's still the big thing is that they'll write out some solution that's sort of BS and they won't really realize it's BS and they'll kind of think that they did the problem correctly.
[0:45:07] I mean, students do this as well, but I think AI systems do it more frequently. They sort of don't know what they don't know as effectively as we do. Does this have to do partly also with the stochasticity that's inherent in the process? Because when you're using language, you have some stochasticity in language, that's fine. But in multiplication, you really need to get the right digit. So these models do get approximately the right answer. They get something that looks like the right answer, has the right number of digits, and the first digit is right, and so on.
[0:45:35] Yeah, I think it might. I mean, yeah, you can sample from the models in a non-stochastic way by just taking at each point the output that they think is the most likely output. And I mean, that will make them perform better, but they'll still make mistakes. So I think it's partially stochasticity, but I think it's more than that. Okay, thanks.
[0:45:59] Can I ask a quick question about this issue, Jared? Maybe it's naive, but conventional computers are, of course, great at mathematics. So shouldn't it just be for an AI system to learn to recognize that this is a math question? Because then it can call in a conventional computer to do it. Oh yeah, that's what we really do. So these AI systems are being trained to use tools.
[0:46:25] Which means basically sort of like calling on external functions. So I mean, more generally than that, like these systems are very, very good at software engineering and writing code. So they can like write Python code to do math or even write some numerical simulations and then get the results from those and then use that. So that is very easy. And that is, if you really want the model to like do arithmetic for you, that's the way to go. Nice. Thank you. Sure. Yeah, so.
[0:46:56] I think probably the most serious objection to sort of transformative AI happening soon is sort of, at this point, I think is sort of like maybe running out of data. I mean, this sort of scaling process can't keep going if you sort of like run out of data to train on if the training process isn't more efficient. I think this is a real concern. I think that sort of other methods of training and various techniques with data augmentation will sort of address this.
[0:47:24] I also think there's sort of significantly more headroom in this capacity. But I think it's sort of a real question. Another question you can ask is kind of like, well, if AI systems are really going to be that useful and powerful, why doesn't the economy reflect this more? And I mean, I don't know. It's sort of starting to. AI is definitely generating a lot of money now. I think it's still a question of how close really are we to something like being able to automate a lot of work.
[0:47:51] But I think we're definitely making progress. And of course, whenever you think something is crazy, you should just ask, well, aren't you just crazy? Probably you're not thinking of something, some objection, and this isn't really going to work. And I think people should have doubts about this because it's a pretty bold claim. And so I think it makes sense to be skeptical. There still is more room to scale up, though. Can I interrupt here?
[0:48:21] Yeah, sure. Yeah, thanks. I just wanted to ask about the running low on data. What do you think about the role of synthetic data in this process? Yeah, so I think that synthetic data definitely can help. I think there's kind of a continuum between doing reinforcement learning and training on synthetic data. I think that an issue with synthetic data is that synthetic, I mean, very naively, it's hard.
[0:48:50] Generate data that when like the model trains on it makes the model much smarter. However, it's possible because you could have the model like use a lot more compute to sort of generate the data than it's been sort of like learning from it. And so it's definitely possible. I think synthetic data has a role. Reinforcement learning has a role. Data augmentation has a role. And I actually, I don't really think that this is going to block AI progress, but it's a reasonable thing to ask about.
[0:49:20] Yeah, so I mean, so far AI has actually been kind of cheap. The most expensive training runs are not really that expensive in the grand scheme of things, way cheaper than say the LHC. And data centers and chips are also getting cheaper. The other thing that I think is like an interesting observation just to be aware of is that like, if you can train one powerful AI, you can run millions of them. So training is very expensive because
[0:49:47] It's basically proportional to the size of your model times the size of your training set. But once you've trained the model, using it to do stuff, basically the cost is only proportional to the number of parameters, not to the size of the training set. And so basically once you've bought a data center that can train the model, you can sort of run, I don't know, thousands or even millions of copies of the model in your data center. And so that means you can sort of use the model a lot once you've trained it.
[0:50:17] And so if it's very powerful, you can sort of get a lot of utility out of it once it's trained. So sort of why is this scary? I think it's all just common sense is my personal view on this. I think that if you're skeptical of this whole picture, if you think sort of AI has sort of reached its peak and there will
[0:50:38] I don't know, 10 years from now, the best model will be about as good as Claude 3 Opus or GBD4 or whatever. Then, I mean, I don't know. There's nothing really to be scared of. It's just kind of a new technology, kind of like the internet was sort of new or cell phones or something. But I think if you sort of believe that it's just going to keep getting better and better for quite a while, then you're sort of building a technology that's very powerful. It's going to be very disruptive to human society.
[0:51:03] Powerful people and organizations like companies and governments are going to want to control it. And the technology is only, in a certain sense, five or 10 years old. Transformers, I don't know, they're six or seven years old maybe. And so this is all like we don't really understand this very well. It's not like physics where there's been hundreds of years of progress.
[0:51:27] And maybe if we build a thing smarter than us, maybe getting it to do exactly what we want is more challenging than you'd think or as challenging as you'd think. There are a lot of cautionary tales where it's hard to supervise a thing that's actually smarter than you and can understand the world better than yourself. So, I mean, I think that like...
[0:51:48] Probably this is all going to be great and we'll be able to create, if it happens, we'll be able to create a utopian future. But I mean, maybe not. And I think sort of it's uncharted enough that it's sort of worth being cautious and thinking carefully. And as I always emphasize when I talk to physicists, all of this stuff is just very simple. AI is very simple. There aren't a lot of very complex ideas.
[0:52:16] And I think the work on it is really basically just kind of natural science type work. I mean, probably more people have internalized it at this point, but I think that physicists can really very quickly kind of get up to speed and do great work in this field because very little is known. And I think most of the progress comes from very iterative empirical work combined with kind of
[0:52:44] Just big picture thinking and kind of understanding systematically what works. And I think of like progress in AI is really this rapid, then I think it's very valuable and important to sort of have a lot of smart, thoughtful people kind of paying attention to what's going on and kind of if nothing else existing as concerned citizens. So thanks, happy to answer more questions.
[0:53:11] Thank you very much, Jared, for that fantastic and very provocative talk. So maybe we can see if there are questions from the room first and then other questions. We're not clapping. Thank you. Are there any questions from the room or from any other questions from the online audience? Yeah, from room. Yeah, go on, yeah. So I have one question. I mean, when we were discussing this constitutional AI,
[0:53:47] Let's consider the case when a parent constitutional AI supervise the other baby AIs. And is there any situation where the baby AIs in turn can supervise the parent AI? Yes. I mean, if the constitutional principles are relatively simple and the sort of parent AI can
[0:54:13] Sorry, if the kinds of principles are sufficiently simple, then sort of a very dumb baby AI can sort of decide which kinds of outputs are in accord with the principles and which are problematic. And so in that sense, it can supervise a sort of much smarter AI. Can it happen to an extent that it can change the constitution itself? We can let AI systems change the... I mean, we've done experiments sort of internally where we sort of...
[0:54:41] Give the AI the constitution. We say, well, like, how can we improve this constitution? And that's definitely a thing we can do. I think it's unclear, like, does that actually improve the constitution from the point of view of people? But it's something you can let the AI system do if you so choose. Okay, thanks. Can I ask a question? Yeah, go on. Go on, Sandeep. Okay. Jared, tell us a little more.
[0:55:07] If you don't mind about what was your thinking when you put in this constitutional module and what are the kinds of ethical issues you felt were especially important that a computer should keep in mind? I'm curious to know your thinking. Yeah, I mean, the ethical issues, I mean, we sort of, since we were doing something public, we sort of surveyed a bunch of different.
[0:55:33] Sources of ethical principles. But I mean, we use things like the UN Declaration for Human Rights and things like that. But yeah, I mean, I think broadly, like we've done other experiments. I mean, we did a sort of societal impacts team experiment where we called collective constitutional AI, where we basically kind of democratically sourced constitutional principles from some large group of people and kind of had people.
[0:56:01] And just as a sort of quick follow-up, I know things are perhaps at an early stage, but did you find that AI system train keeping in mind those values on at least a few issues give significantly different results or answers?
[0:56:33] Did it make a difference, in other words? Yeah, it can definitely make a difference. I mean, if there are examples where that constitutional principle is relevant, then it definitely will affect the AI behavior very substantively. So I mean, for more specific principles, like, I mean, we've done a lot of experiments with much more specific principles targeting very specific behaviors, and it definitely works. I mean, there's nuance to sort of getting it right, but...
[0:57:02] Definitely possible. Thank you. Any other questions? Yeah. Can I? Yeah. So I had a question about, we have been talking about the constitutional AI. There was a question which says that, what about if these principles are in conflict among each other, right? I wanted to know, is it possible? Do we have certain examples? Can you give an example where conflict is possible in the basic?
[0:57:32] You know, principles we assign to these students. I mean, we haven't really done explicit experiments with principles that are conflicting with each other. I think it really just depends on like how you do the experiment because it doesn't just matter what the principles are. It also matters like what are the examples that you're using to teach the model to follow those principles and where do those examples come from? So I think it's like pretty complicated what ends up happening.
[0:58:04] Are there other questions? I have a question. Is there a mismatch between progress in robotics and progress in AI? You know, we haven't done that well with self-driving cars and with, I don't know, robots who can walk without falling and so on. So is that a mismatch that you think is fundamental or just something that will go away in the future? Yeah, I think that robots that can walk without falling, I think we're actually pretty good at that at this point. I think there are lots of robots that can walk without falling.
[0:58:34] But I think that the sort of kind of AI that we use for that and especially for self-driving is actually quite different from this AI. The thing that we're making tons of progress on is making AI systems that are like sort of very smart 90% of the time or something, but like can be quite wrong and make mistakes. Self-driving cars requires AI systems that are extremely robust to error.
[0:59:01] But actually aren't very intelligent in the grand scheme of things. And because they need this level of robustness, they don't really rely for the most part on deep learning. They use deep learning for their vision systems. And I mean, maybe for more, but I think mostly they use deep learning for the vision systems. The actual driving is more like kind of hard coded with lots of different rules.
[0:59:26] That people have given. I don't know all of the details here. I could be slightly mistaken, but I think mostly it's something like that because they need it to like get it right every time and be very reliable. They basically can't take the level of risk that you can take when you're just making a chat bot. And so I think that's sort of the difference. I mean, there's sort of a question of like, are we going to be able to achieve that level of reliability with this kind of AI? Of course, it's hard for people to achieve that level of reliability. They obviously do.
[0:59:56] So there's sort of a lot of ground to cover. But I think that's kind of the difference is that sort of aiming at two things and sort of the sort of metrics that we're optimizing for are different. I see. But, you know, in chess, for instance, they did achieve that reliability, right? We had these programs that had hard-coded rules. And then AlphaZero, in fact, did get greater reliability than all these other programs. So deep learning might also give you very reliable systems.
[1:00:24] Absolutely, yeah. So I guess there's a couple of things that went in there. I mean, one is that there was some like very, very like concrete objective that sort of like winning and losing, and you could sort of optimize for that objective very hard. And it was also in like a fairly narrow domain or sort of space. And even sort of at test time, I think you're then using search. So I think...
[1:00:52] I mean, maybe you could build a bigger, more powerful system today that would do even better. But I think like even like AlphaZero, I think like the neural network by itself was sort of like maybe at the like very roughly at the level of like the best possible human experts. But then also when it plays, it can do search to sort of find what moves to make. And I think that makes it even more powerful. So some of these things I think are not available in current systems. But I mean,
[1:01:20] One of the most obvious things, of course, and something that people in the field talk about constantly is sort of like adding things that look more like this, like adding things with reinforcement learning that involve more of these trial and error type learning things that can really drive robustness. So, I mean, maybe that's even possible with like...
[1:01:41] At least in simulation, maybe you can train a model to drive a car in simulation with deep learning very, very, very reliably. And then it would be a question of, does that transfer to the real world? Because you can't learn to drive in the real world by trial and error. I see. Thank you. Are there other questions? Bhavesh, do you have another question? Yeah, I have another question about the recently released model, Lama 3. It has a lower...
[1:02:05] Smaller context length and lesser parameter size also. Do you think we are trying to go away from the larger and more complex systems to a better system in that sense? I mean, my impression is that the newest LAMA models are quite impressive. They actually don't have any innovation really in the model architecture.
[1:02:29] So, I mean, I think there are a lot of innovations to be had in model architecture, but I think they don't have a lot of sort of like very new model architecture features. I don't know what innovations they do have. I would guess it involves more like sort of data filtering, optimization, possibly some clever like loss functions that are improved on sort of this sort of autoregressive objective. I don't know since...
[1:02:58] Yeah, since I'm sort of not at Facebook. But basically, I guess what I would say is that there's just a tremendous amount of low hanging fruit in this field. I think that's another reason why I kind of don't think that progress is going to stop. I think there's low hanging fruit in terms of model architectures. There's low hanging fruit in the way you train the model and like how you train it on data and reinforcement learning. And so I think there's just a lot of room for innovation. And so on top of the basic scaling,
[1:03:27] These things are making models better. But I mean, like Lama 3 models were trained like, I don't know, 5 or 10x longer than Lama 2 models were. So it's not that they're not scaling. It's that some of the models aren't much bigger than the last generation, but they train them much longer. Are there any other questions from the room? Okay. So if not, let's thank Jared again. Thank you very much, Jared, for a fantastic talk.
[1:04:05] Thank you. Thank you. Thank you. Thank you everyone for joining. Thank you.