[0:00:00] So yes, on Friday we will have a discussion about the future of string theory, and today I guess we'll have a talk by Jared Coupling about the future of string theorists. So we are very happy to have you, Jared. So please, the floor is yours. Thanks so much. Yeah, I appreciate I was following a very serious topic, and now I think...
[0:00:26] I'm kind of more like the court jester, so hopefully this isn't entertaining. Sorry, Jared. Sorry to interrupt you, but we see a little show slide, show window in the middle of the screen. Okay, now it's better. Maybe you can do it like this. Okay, is this better? Yeah, it is better. Thanks. Okay, great. Yeah, so I will be talking about...
[0:00:51] Some of my work on AI and whether perhaps we will have human level AI by 2030. So hopefully many of you remember me as a physicist. I started working on AI around 2018 and sort of over time my perspective has shifted. I sort of used to apologize for being a physicist working in another field and how...
[0:01:16] How sort of arrogant we all are. And I was very new to it. But over time, I've sort of become more confident. I've worked on some things I'll talk about, like scaling laws and GPT-3 and this idea of constitutional AI and AIs supervising themselves. And so versions of this talk have evolved.
[0:01:39] Now we come to sort of today and I have this talk title is now just Human Level AI by 2030. And next year, if you invite me again, I can give a talk on this subject. So basically, I'm sort of very aware that as I've worked on AI longer, it sort of seemed more clear to me that AI progress was going to continue and potentially be very transformative.
[0:02:08] I've always been very self-skeptical that maybe this is all totally crazy, but I'll kind of explain some of my thinking in brief anyway. So the main ideas from this talk are that AI is progressing, AI progress is occurring because it's very, very easy. And it's sort of driven predominantly by increasing the sort of resources we use to train AI systems.
[0:02:37] And a lot of actual sort of algorithmic ideas are sort of being replaced by things that are much simpler and involve kind of scaling up simple systems. So sort of as everyone probably knows in science, when a sort of scientific field is making a lot of progress, it's not because the scientists became smarter. It's just because we found some direction where progress is very easy. And that's really what's happening.
[0:03:03] And in particular, progress in AI is predictable as there are scaling laws with respect to the amount of data you have, the size of your neural network, or the amount of compute used for training. And the amount of compute used is really, really very vast. I think that in certain respects, AI systems are kind of approaching kind of human level performance.
[0:03:28] I think there are sort of challenges for continuing this progress. But I think there aren't any really kind of compelling blockers for AI doing things like helping to sort of automate science. And I think this is all fairly scary for sort of obvious common sense reasons. As humans, we're sort of used to being the smartest species on the planet. And so if we aren't anymore, then that really kind of changes the balance of power. And I think it's...
[0:03:57] If any of this is true, then it's something we're kind of just kind of sleepwalking through because it's happening so quickly and it's worth thinking about and working on. So as an outline, I'll just talk briefly about what contemporary AI is and why it's kind of trivial. I'll talk very, very briefly about language models and how we train systems like Claude and ChatGPT.
[0:04:26] Then talk about sort of why we think AI capabilities will continue to predictably improve and sort of where we're headed from there. So some very quick sort of need to know pieces of information about the way that AI works. So contemporary AI is very simple. And in particular, it's literally just kind of curve fitting with a very general function approximation. Sort of at a high level, all AI is
[0:04:55] As it's practiced today is you have a lot of data points and you're just trying to fit some kind of function approximator to those data points. And the sort of potentially sort of impressive progress of contemporary systems is just using very, very general, very large function approximators on lots of data in slightly clever ways. And if you didn't already know, neural networks, which are sort of what are used
[0:05:24] For these systems, they basically just are a way of parameterizing very, very general functions via a sequence of matrix multiplications and element-wise nonlinearities. So you can make a very general function approximator by having very big matrices and kind of multiplying them together. That's really all that's going on in contemporary AI and neural networks research.
[0:05:49] And so the only ingredients you need to do sort of AI research or train an AI model is you have to decide sort of what is your data that you're trying to fit. You choose what people refer to as sort of a model architecture, but that just means kind of in what order and what configuration do you kind of multiply matrices together and sort of how big are they. Then you have to choose some kind of loss function, which determines sort of what
[0:06:18] What parameters, basically some function where if the parameters when applied to the data achieve a low loss, then you're doing well. And then you just use some slight variant on gradient descent to adjust your parameters until your loss is low and your function fits the data well. And that's how you train everything from sort of large language models to image generating models to...
[0:06:45] So that's basically kind of really all there is to modern AI. It's very easy. It's easy for physicists to learn. And in particular, if you look at language models and large language models, they work like this. So if I say a sentence like, as a speaker at a journal club, you're probably elephant me to say certain things, then the word elephant was very unlikely. It sort of stood out in that sentence. All these large language models do is,
[0:07:15] Predict the probabilities of words given the prior words in a sentence or paragraph or book or whatnot. And so all they're really doing is what we call autoregressive sort of next word prediction, where given all of the prior words in a sentence, they predict the probability of the next word. That, of course, means that you can sample from that probability distribution for the next word.
[0:07:40] Write it down, put it back in the list of words you already have and sample yet another word. And that's how large language models write and generate text. So that's how you get a pure large language model. You fit this probability distribution to the data where the data is kind of a huge, huge corpus of text, books, potentially text from the internet.
[0:08:07] If you want to train a system that's actually useful like ChatGPT or Claude or Gemini or whatnot, then you can do another slightly different thing, which is once you have this large language model, you can put it in front of people and ask them which behavior they think is better. So this is sort of an example from research we did a few years ago with a very, very early version of Claude.
[0:08:37] Where we created this interface and we sort of paid people on say like MTurk and other crowd workers to talk to the language model and they would be given two possible responses to various questions and they would sort of choose the one that they like better. And from this, we would basically fit a version of a large language model to these predictions in order to assign a score to
[0:09:05] Samples from the language model that people tend to prefer that are more helpful and honest and harmless. And we use this to sort of train a final version of Claude. So there's sort of this kind of flowchart where we put a version of a large language model in front of people. They choose what kind of response from that model is better. We use that to train a model that assigns scores.
[0:09:32] to better performing responses. It's actually sort of exactly the same function and fitting function that you use to rank chess players based on who wins in chess games and give them sort of chess ratings or ELO scores. And then we would do reinforcement learning against that preference model. So we basically sort of train another yet another copy of the LLM, the large language model to
[0:10:00] Write out responses that get higher scores by encouraging responses that get higher scores, discouraging responses that get lower scores. And that's how you train contemporary AI systems you probably have played with. So in summary, you first pre-train these models to predict the next word in some giant pile of text. You can collect human feedback on what kind of responses are better.
[0:10:26] Or in fact, you can use the AI itself to give itself feedback in this way. And then finally use reinforcement learning to sort of fine tune an LLM to write responses that give high scores. And so then you get sort of, if you sort of do this at large scale, you get kind of state-of-the-art large language models such as Claude. So this is sort of an example from Claude 3 Opus, which is this model that we at Anthropic released a few months ago.
[0:10:55] And I sort of chose this recent paper of Ed's because it came out after we trained Claude 3 Opus. Sort of I asked, you can of course go do this at claude.ai, but I sort of asked it, I just uploaded the PDF of this paper and asked it this slightly non-trivial question, what are some ideas explained in this paper that might be new to theoretical physicists who already have some familiarity with Louisville theory? And you can evaluate for yourself.
[0:11:23] If this response is decent, I didn't check all of the points, but I think it was fairly reasonable. And so sort of this extremely simple process that I outlined allows you to generate a system that seems like it can at least sort of grapple with and explain, if not create new kind of ideas from science. And of course, this isn't where I think things are stopping.
[0:11:50] But this is sort of where we're at, where we're at now. And I'll say a bit more about that. So now I want to talk about sort of why I think AI progress was predictable. I mean, I founded this company, Anthropic, with a bunch of friends, many of them also physicists like Sam McCandlish.
[0:12:07] Basically, so that we would sort of have the resources to continue kind of this kind of scaling process and kind of understand these systems, because we believed that this was all kind of a predictable trend. And so the main reason why AI systems are getting better very quickly, or at least predictably better, is that there are what we call scaling laws for AI performance.
[0:12:37] So this was from some work that Sam McCandlish and I did in 2019. These plots are from language models, but this is really true for kind of almost any kind of AI system. And what we found is that as you scale up the number of parameters in your neural network or the amount of data you train on, or simply the amount of computation you're doing during the full training run of your AI system, you get extraordinarily predictable behavior in terms of
[0:13:07] How big the sort of resource inputs and the performance of the system is measured by the test loss, which you can think of sort of as the ability of the system to kind of compress information. And I think the thing that was most striking about this when we first worked on this was that these scaling trends are not rough. They're extremely precise. They're kind of like the kind of thing that you'd expect from a physics experiment.
[0:13:33] And furthermore, they span many, many orders of magnitude in the size of these neural networks and the computational budget. And so this sort of suggests that this is going to continue and this is going to allow you to predictably make AI systems better. And the way that we do AI has also actually simplified. I think there's been a kind of grand unification in the way that AI works, where it used to be the case that computer scientists would use very different specific systems.
[0:14:02] For language modeling, for image modeling, for image identification, for playing games, for all sorts of different tasks. Whereas now we use sort of the same kind of architecture and the same kind of methods for all of these data modalities. And so this is sort of an out-of-date example. This is sort of GPD-3 writing a poem versus a system that's almost identical to GPD-3 trained. Instead of training it on...
[0:14:31] We trained it on predicting the next word. We trained it on predicting the next pixel in images. And just as GPD3 can kind of complete text, similar models can complete images. It's possible to build better image models now that are more efficient, but basically the same ideas applied everywhere work well and kind of scale well. And so we're sort of in this regime.
[0:14:57] Where basically the amount of compute used to train AI models is growing exponentially. And so there's sort of this trend of exponential growth in AI in red from early AI systems kind of culminating in AlphaGo. And then more recently, there's been this sort of exponential trend of increasing compute used to train language models. Sort of the estimated GBD4 compute is here. And various people are kind of forecasting.
[0:15:26] That this compute trend will continue and will continue to grow for a while longer. And the y-axis here is sort of powers of 10. So we sort of have already crossed like Avogadro's number. We've already crossed numbers like, I don't know, the number of stars in the observable universe. This is just measuring literally the number of additions and multiplications that are going into training these systems. And this is sort of what's powering in part.
[0:15:53] The sort of growth and capability of frontier AI systems. And even sort of researchers who work in AI are generally very, very surprised by how fast AI is making progress. Some people I know at Berkeley specifically made back in 2021 some evaluations that they thought would be really hard and where they thought progress would be fairly slow.
[0:16:19] And so if you sort of go back, I'll discuss this a bit, but if you go back to sort of 2021, AI systems were a lot worse at math and not nearly as smart. And so they made these sort of kind of relatively hard math competition data set and this other data set of kind of undergraduate level knowledge that includes science, math, and many other subjects. And they tried to predict sort of how
[0:16:44] Quickly, AI progress would go in terms of how well state-of-the-art systems would do on these tasks. And in this sort of figure in the left, their predictions as a function of time were sort of these blue dots, whereas the sort of stars are what actually happened. And so kind of like very quickly, within roughly a year, AI systems made as much progress as they thought would be made in four or five years.
[0:17:10] And if you sort of look at where we are now, we're sort of like far beyond what they expected to be possible by now. Another example, I mean, kind of in response to this trend, some other people made a data set of kind of graduate level questions. These were sort of intended to be questions where people with a PhD in one domain could answer the questions correctly.
[0:17:36] But people with PhDs in neighboring domains, like, I don't know, physics versus chemistry, couldn't get the questions correct, even with sort of 30 minutes and full access to sort of web search. So that's what this data set was, this GBQA data set. And it was created, I don't know, six months, maybe nine months ago. And at the time, they thought sort of the best models were a little bit better than chance, but they weren't doing very well. And sort of this...
[0:18:06] Model Claude 3 Opus we released recently can be used to get sort of 60% of these questions and sort of PhDs were getting, I don't know, 70% of the questions or something like that. So progress has been very rapid and people keep being kind of surprised by it. But it's sort of driven by predictable trends. And so the way I kind of think about progress is sort of...
[0:18:30] There's kind of the generality of the environment that AI systems can operate in. And it's been very, very easy, as probably you've seen, to build multimodal AI systems that can understand images, they can understand your computer screen, and probably various researchers are using them to sort of control robots. And so they'll probably be able to move to real life. And then the other axis is kind of like what tasks these systems can do correctly.
[0:18:57] And how long comparably it would take a person to perform the task. I think that's in some ways the more interesting axis. And systems are not that great at doing tasks that take people days or weeks or months. But I think it's very plausible that systems are going to get better and better at this. And that it's going to happen very rapidly. So I don't know. I may be crazy. But my guess is that...
[0:19:24] My kind of median guess is that we're going to sort of be moving up and to the right on this plot quite rapidly. So what are the remaining challenges for AI? I mean, is it all kind of done or what could sort of make this picture of rapid AI advance false? I think that like one of the things that at least maybe was initially surprising about contemporary AI systems is that I think these AI systems are sort of,
[0:19:53] Best at correlations and things that you might associate with intuition. They're very good at generating things that are artistic and mimicking styles. This is, again, an image-generating model from Google from a couple of years ago. But they're sort of worst at reasoning and math and at self-correction in particular. They're poor at noticing when they've made mistakes and fixing those mistakes.
[0:20:21] So this slide sort of exists. I mean, I made this first version of the slide a while ago. I've crossed out multimodality as a remaining technical challenge. But I think that sort of reasoning and long-term planning are kind of the main challenges that still exist for AI systems. Back when we created GPT-3, it sort of seemed like GPT-3 was just terrible logical reasoning and math. And so it seemed like maybe that was going to be a big blocker.
[0:20:48] I think as we sort of scale AI systems, it seems like they're getting better at this quite smoothly. And then there's just kind of long-term planning. And by that, I sort of don't really mean the planning itself, but really the ability to sort of self-correct. The reason why we as people are able to kind of robustly do complicated tasks isn't that we get it right from the beginning, but it's that we encounter challenges and we make mistakes and we recognize that and we fix them.
[0:21:15] I think AI systems are currently kind of bad at this, but I think they'll rapidly improve. So I think there are other reasons why you might think that AI won't continue to make rapid progress as it has in the past. One that people sometimes talk about is that maybe we'll kind of run out of data for these systems to trade on. I think this is a real concern, although I sort of don't expect it to be a blocker for progress because...
[0:21:44] We'll be able to use things like reinforcement learning to accomplish useful real-world tasks, and we sort of therefore won't really run out of that kind of data, the ability of the system to practice doing useful things. I think people have said that it seems implausible that we'll have kind of human-level AI soon, because if we were, then you would think that you'd see AI contributing more to the economy. I think we are seeing that more now than we have in the past.
[0:22:13] But I think that's kind of another objection for why AI might not make as much progress. And I think generally, like, if you think that sort of you're about to usher in the greatest transformation in human history, then you should question whether you're sane. And I kind of question this about this whole line of thinking frequently, but nevertheless, it seems sort of plausible. And in particular, like...
[0:22:40] Although people talk about AI becoming big, in some sense, like, we haven't spent nearly as much money on sort of an AI training run as we have on the LHC thus far. And another just sort of interesting fact is that, like, once you've trained a very powerful AI model, you definitely can deploy sort of millions of copies of it. So although training these large AI systems is expensive, it's not that expensive to run them and sort of use them at scale.
[0:23:08] And so sort of before I end, I could just say sort of like, why is this kind of scary if it's true? I mean, it's not scary if it's not true. So I mean, this might all be crazy. But I think if it's actually the case that AI is going to sort of achieve kind of broadly human level performance and be able to do things like automate scientific research, I think it's sort of scary for basic reasons. Like we don't really understand how these systems work very well.
[0:23:37] I think progress in this field is happening very, very rapidly. And yet this would have a very big impact on society and the world if AI systems are as smart or smarter than us. And again, we sort of aren't that good at sort of supervising and training these systems. The sort of best AI models out there make mistakes and sort of can be...
[0:24:00] Jailbroken or corrupted. So, I mean, maybe it's difficult if you have a thing that's sort of smarter than you to train it to obey you and kind of do exactly what you want. It's possible that this isn't that hard, but because everything is moving so quickly, we don't have a lot of time to think about it and reflect on it and kind of evaluate it. And so, going back to Sasha's comment about the future of string theorists,
[0:24:29] I sort of can't overemphasize how simple AI is. I mean, many of the people that I work with in AI at this point are physicists who've kind of transitioned to work on AI and it takes them like three months. And I think also AI research is really natural science. It's not a situation where we sort of like traditional computer science where you sort of
[0:24:53] Write down an algorithm and you work out its complexity and you understand exactly sort of what that algorithm will do. It's much more like condensed matter physics or biology or something where you're rapidly iterating two experiments at a very complex system and trying to understand its kind of general behaviors. And so it's really sort of well suited to the kind of thinking that physicists employ.
[0:25:16] And I think if progress is as rapid or as kind of I expect, then I think we really just do need more of our best people kind of thinking about this and trying to make sure that it goes well. So that's it. Thanks. Thanks, Jared. We have some time for questions. Please, Julian. Hi, Jared. Thanks a lot. Just a quick question. So these scaling laws, can you comment whether there's some kind of analytical understanding, as we might call it, in this room?
[0:25:52] Yeah, it's a great question. Yeah, it's something that I thought of and other people that you might know, like Ethan Dyer has worked on this. I think that there's sort of a toy model that seems to work in simple contexts for this. The toy model, like, again, sort of makes neural network learning seem kind of simplistic. Basically, the toy model is that if you imagine that all of your data lives on some manifold of some dimensionality, some high-dimensional manifold,
[0:26:22] And then you imagine that all your neural network is doing is kind of slicing that manifold up into pieces and building some piecewise linear approximation of the function on this data manifold. If you postulate that as a toy model, then it has the property that it obeys these kinds of scaling laws. And in particular, the exponent, so these scaling laws are power laws.
[0:26:52] And the exponent of these power laws in the scaling law is basically something like one over the dimension of the manifold that the data lives on. So that kind of toy model seems to sort of fit a lot of the patterns that we're seeing quite well. And there are other more sophisticated ways of talking about this. But I don't know that we sort of have a proof that that's really what's going on in kind of state of the art systems.
[0:27:22] So I have two questions. The first one's only as serious as you want it to be. You have probably as many alumni at your company as we have in a given group here. And if you're going to replace us by 2030, if you want to propose to host strings in one of the intervening years, you're welcome.
[0:27:37] But the other question is kind of related to that. What do you see as possible like kind of partnerships and or like collaboration between industry and theoretical physics? I think it's interesting that we're in a time where the engineering challenges that we face are actually of interest to industry. And so it seems like a good opportunity to collaborate or drive more domain-specific advancements. Yeah, yeah. I mean, I think that like...
[0:28:02] I mean, the main way that we hire physicists is we have sort of a residency program where people can come for six months and do AI research. Of course, like, I mean, most of them don't leave after that and they get sort of full-time positions and stay with us. But I think that's the main mechanism. I think that the main, like, difference between... So I've sort of...
[0:28:27] Very much emphasized how like all of this research is quite easy. It's sort of the opposite of my experience of working on particle physics and string theory where it's really, really hard to make progress in physics. There's just so much low-hanging fruit in AI research. And what that means is that like, I think AI research is very iterative. It's like very fast moving and it's very like, because there's so many obvious things to do, we mostly are just very busy doing kind of obvious things.
[0:28:56] So I think that means that like the directions we're going in are kind of guided by that. And like, and I mean, Anthropic as an organization has kind of a specific goal of figuring out how to make sort of safe, transformative AI systems. So we're very like kind of like laser focused on that goal rather than sort of like, I don't know, in physics research, we kind of do whatever is most interesting to us.
[0:29:19] And so I think that's the main difference. But I think there's a lot of opportunities for potential collaboration. Yeah, I mean, because for example, like semantic search, better symbolics, it's going to be mathematic or a large language model that'll help us make it a lot easier to do our job or to get up to speed as graduate students going into postdocs and stuff like that. So I think there should be more opportunities than just we work for you guys. If we could even get the grant money, you said that CERN has more money for our collider than you guys have for...
[0:29:48] Whatever your large language models. I mean, if you want to accelerate physics, I feel like there's opportunities to collaborate and not just poach. Yeah, yeah, yeah. Yeah, no, absolutely. I mean, I think there's a lot of opportunities. I just think that we're actually a pretty small organization, so we're relatively focused. And obviously, I mean, I miss doing physics. I mean, as I said at the beginning, I hope that...
[0:30:15] Please feel free to not work on AI. I'm talking about AI. Obviously, I understand if people want to continue doing string theory. It's extremely exciting. In view of time, I think we should finish. Let's thank Jared again.