{"url": "https://www.youtube.com/watch?v=I5mC4nDDp2I", "page_url": "https://www.youtube.com/watch?v=I5mC4nDDp2I", "title": "206. Jared Kaplan on Scaling Laws", "show_or_channel": "AI Safety Reading Group", "date": null, "duration_s": 846, "type": "talk", "speakers": ["Jared Kaplan"], "confidence": "medium", "notes": "The transcript snippet sounds like Kaplan speaking, so this may be an excerpt of one of his talks. Unverified, and the date is unknown.", "people": ["jared-kaplan"], "id": "0000_ai-safety-reading-group-206-jared-kaplan-on-scaling-laws", "audio_duration_s": 845.5895, "worker": "jc-w09", "transcribe_s": 37.8, "model": "large-v3", "segments": [{"start": 0.0, "end": 23.33, "text": "There's a lot of freely available writing on the internet to train on and in books. And if an AI knows language, then you can ask it about anything or try to communicate with it about almost anything. And you can therefore try to get a lot of intuition from its responses and what it's doing as compared to some other kinds of examples of tasks. And that might help us make it both general and safe."}, {"start": 23.33, "end": 52.48, "text": "So what are language models? And basically all of the models I'll talk about in this talk are always autoregressive and they always have an autoregressive loss where there's say a bunch of words and the model's trying to predict the probability of the last word given the earlier words in the sentence, paragraph, passage, et cetera. And this applies to other modalities like say pixels. It can apply to the answer to a math question. It can apply to both images and texts that are joint."}, {"start": 52.48, "end": 75.1, "text": "Together, as we'll see. And so here's an example, like as a speaker at a journal club, you're probably elephant me to say certain things. And when I say elephant as a human, we notice that was a weird word to insert in that sentence. And the GPT-3 model also thinks that it was a weird thing to say given the preamble. And so all we're doing is optimizing these log likelihoods."}, {"start": 75.1, "end": 101.66, "text": "Almost always be talking about transformers except when I compare to LSTMs. Transformers are based on this idea of self-attention where you, when you're making a given prediction, you kind of look through all of the prior words or tokens and you up-weight or down-weight them kind of like intuitively a human might highlight certain words in a passage that are most relevant to what's going to happen next. And then you look at that weighted linear combination of"}, {"start": 101.66, "end": 130.88, "text": "What you highlighted, how much you highlighted it, you process it. This occurs over and over again for layer upon layer. And then finally, you make an actual prediction. So that's 30 seconds on Transformers. And you can get very, very impressive seeming performance on a lot of different data modalities. So on the left, we have a sample from GBD3. We provided the title and author, and it wrote this poem. On the right, we have a bunch of completions from iGBT, which is just training."}, {"start": 130.88, "end": 164.51, "text": "Exactly the same kind of model on images, pixel by pixel. And it seems to know a great deal of semantic and other information about images. So this seems to sort of work pretty well. For some reason, my slides are stuck. I don't know what happened there. Okay, so scaling laws. So scaling laws for neural models. I guess there's..."}, {"start": 164.51, "end": 189.63, "text": "Two different levels of motivation. A super high level motivation is sort of why does machine learning seem to work well? What actually matters and what doesn't? And I think this informs what kind of research problems we should work on and also what we should forecast or expect for the future. Question I had in mind when I started working on this subject three or four years ago was something like, is making progress on AI more like proving the Riemann hypothesis where you might expect"}, {"start": 189.63, "end": 219.09, "text": "Progress comes from a few lone geniuses who obsessively work on the problem. It's very hard for outsiders to gain insight about what exactly they're doing. Progress is kind of maybe seems like it comes through flashes of insight rather than being predictable or incremental. Or is it more like building a very powerful steam engine where there is a lot of incremental progress. You don't have to actually spend your whole life obsessing about the problem to understand it. And there are kind of simple basic laws underlying."}, {"start": 219.09, "end": 245.54, "text": "What leads to progress and what doesn't. And so what I'll be showing you is evidence that there are fairly precise scaling laws for the performance of AI systems. I'll be focusing on macroscopic variables like how many parameters you have, how big your data set is, how much compute you use for training. But I think there are actually a lot of other scaling laws with other variables in machine learning. I think they're kind of ubiquitous."}, {"start": 245.54, "end": 269.44, "text": "I also kind of argue weakly that a lot of the other details don't matter very much. And in particular, it seems like the scaling laws basically stay the same even when we make a lot of algorithmic progress. And the main thing that changes is some kind of constant pre-factor a lot of the time. And achieving good performance is mostly about avoiding bottlenecks. So the simplest bottlenecks are like you have a model that's really big but not enough data."}, {"start": 269.44, "end": 294.11, "text": "Or vice versa. Or you just don't have enough compute to train your model. Or you have literal bottlenecks in your network, like information kind of doesn't propagate well through your network. And the classic example of this is that if you have many, many layers, or if you have the same layer repeated many, many, many times, then maybe you face this kind of problem where you raise a matrix to the power of a thousand and you mostly just end up projecting onto the largest eigen..."}, {"start": 294.11, "end": 327.42, "text": "The eigenspace of the largest eigenvalue and therefore information doesn't propagate well. And I think a lot of the most highly cited papers in all of machine learning are really solving these kinds of problems like batch norm, resonance, layer norm, to some extent transformers versus LSTMs. I think they're kind of avoiding these kinds of bottlenecks. And I don't know why I keep having this problem. Some kind of, I keep having some sort of weird, anyway."}, {"start": 327.42, "end": 354.5, "text": "No big deal. So what are these scaling laws? The simplest scaling, so I'm just going to kind of explain some empirical results rather than tell you a lot about any kind of theory. So the simplest scaling law is pictured at right. So we have a lot of data, and we train a lot of different transformer language models on the same pile of tons of data."}, {"start": 354.5, "end": 382.77, "text": "And we train them basically to convergence. And then we plot the loss, the autoregressive log likelihood loss as a function of the model size, which is the number of parameters, not counting the embedding matrices. That doesn't matter a lot for larger models anyway, but that's what we have here. And we find super, like very, very precise fit to a power law relating the test loss."}, {"start": 382.77, "end": 411.97, "text": "So that's one example. In the center, we have a fairly big model. We train it with early stopping on different data set sizes, and we also get a very nice clean scaling law. And the most complicated example is sort of test loss versus compute, where we have different amounts of compute in our compute budget. And so what you see here are blue lines, which are learning curves. The learning curves for bigger models are shifted to the right because"}, {"start": 411.97, "end": 433.66, "text": "Per training step, you need more compute to train a larger model. And what you can therefore look at is what is the best performance you can get from all of these different models of different sizes as a function of the compute budget. And there's an asymptotic"}, {"start": 433.66, "end": 460.27, "text": "Which is what this orange curve is predicting for what the best is that you can do with a given compute budget. And that also seems to be a very clean power law. And an interesting thing that you can do here is check what was the model that was most optimal for a given compute budget. What is the model size that was optimal for a given compute budget? And that's something that we see here on this slide in kind of cartoon form. So you can ask."}, {"start": 460.27, "end": 488.78, "text": "As you scale up your compute budget, what is the optimal allocation to increasing model size versus increasing the amount of data you actually process? And something that I at least found surprising was that most of your compute budget should actually be allocated to making bigger models and only a small amount to training longer or with a larger batch size. And furthermore, it seems like architecture is somewhat less important."}, {"start": 488.78, "end": 512.7, "text": "On the top left, we see Transformers versus LSTMs. Interestingly, it seems like these trends kind of parallel each other until we get to quite large models. And then I think LSTMs stop being able to improve as rapidly as Transformers, basically because they're not as good at handling very long contexts. And that's what's illustrated on the right. And at the bottom, we have a bunch of different other"}, {"start": 512.7, "end": 538.03, "text": "Hyper parameters you can tune that are associated with the transformer. Maybe the most interesting is in the center, which is the width of the model divided by the depth of the model or the aspect ratio. And it looks like, first of all, getting the aspect ratio wrong doesn't hurt performance very much. And there's a wide range where you get fairly similar performance. And then furthermore, all the different model sizes sort of want"}, {"start": 538.03, "end": 564.43, "text": "A similar aspect ratio. So you should basically scale up by keeping the width over depth roughly constant. But you won't even suffer very much if you get that wrong. So in other words, these other hyperparameters don't matter a huge amount compared to just the overall scale. And there are all sorts of other interesting scaling laws that you can find. So there are multivariable scaling laws like where you change the"}, {"start": 564.43, "end": 592.35, "text": "Size of your data set and the model size together. And you can therefore predict the amount of overfitting like the test loss with finite data versus the test loss with infinite data. All these things seem to be relatively predictable and you can make up an onset that fits them very well. And these things aren't really just true for language. So there's some further questions. Is this specific really to language as a data set? Does it eventually break down?"}, {"start": 592.35, "end": 613.38, "text": "Does it actually improve performance and downstream tasks? And these are the compute plots for many other modalities, including video images, solving procedurally generated math problems, multimodal models that model an image from the text or the text from the image."}, {"start": 613.38, "end": 636.35, "text": "And also language. And it just color coded where bigger models are yellow and smaller models are purple. And you see that there is some kind of steady trend here. A thing that I also found surprising in this vein is that you can look at what the optimal model size is versus compute for all of these modalities. Since model size and compute don't really care about what domain you study, you can..."}, {"start": 636.35, "end": 666.02, "text": "Plot all the data modalities together in a sensible way. And it seems like, roughly speaking, the optimal model size as a function of your compute budget is actually pretty much the same for all of these different data modalities as you scale up. And I wasn't particularly expecting that. There is a further question, which is, do you really benefit on downstream tasks? And so an example here is you can train on image classification."}, {"start": 666.02, "end": 689.18, "text": "Using the pre-trained model that were generative models for sort of modeling image pixels. And if you do that, you see that you get another predictable power law scaling for classification error on ImageNet. This is 32 by 32 pixel ImageNet. And furthermore, pre-training helps you a lot to avoid overfitting. So the orange line is when you train without."}, {"start": 689.18, "end": 713.57, "text": "From scratch on just ImageNet, the blue line is when you take a model pre-trained on image generation and you see that that helps you to continue this trend much further. So pre-training is helping and scaling laws are relevant to downstream tasks and scaling laws are everywhere. So this is, I'll just flash this quickly. This is mutual information between image and text for multimodal models. And that also has kind of a nice scaling law."}, {"start": 713.57, "end": 746.77, "text": "Which I think is kind of cool and interesting. And so then you can ask what happens if you really just do scale up language models and that's what GPD3 represents. So this is the compute scaling trend from GPD3. And we see that we get continued smooth scaling. And then the other cool thing about GPD3 is that it can learn in context. So this plot at the bottom shows performance as a function of model size."}, {"start": 746.77, "end": 776.67, "text": "For colors, but it also shows how many examples of a task were provided in the context. And this is the few-shot learning of GPT-3. And the solid lines show when you give instructions in natural language to the model, and the dashed lines show no instructions. So you improve when you see many examples in the context window, and you also improve when you see prompts. And you can see this then for many different domains, arithmetic,"}, {"start": 776.67, "end": 805.23, "text": "SAT analogies, the kind of tests American high school students take to go to college. Trivia and Winograd schemas. And there's steady performance improvements of model size on all of these tasks, although the exact form of that is quite different. So there's, with arithmetic, there's this sudden takeoff where the model suddenly kind of learns arithmetic in a kind of discreet way. For these other data sets, it's smoother and you can have fun trying to project these things."}, {"start": 805.23, "end": 831.71, "text": "Humans have difficulty telling the difference between GPT-3 generated news articles and real news articles. I think these results suggest that we can continue to get a lot more performance by scaling up. They suggest certain abstractions for thinking about machine learning performance. I think they suggest sort of how we should measure improvements in algorithms where you really care not just about the algorithm on"}, {"start": 831.71, "end": 844.56, "text": "This is a slide I added to answer a question, and so why don't we move to Q&A."}]}