[0:00:00] Language models are two-shot learners, popularly known as GPT-3. I was one of many, many authors who contributed to this. And really, this project was a huge undertaking with almost all of the OpenAI team and all of the sub teams within OpenAI. So I think all credit goes to these people and any mistakes during this talk.
[0:00:28] I take full responsibility for, so bear with me. So first off, what is this talk going to be about? Basically, I'm going to talk about what led to GPT-3, what the actual GPT-3 paper is about, and I'm going to talk a little bit more about what might happen next. David and Adam gave me permission to wax philosophical about what might happen next and like further into the future. So if you're not into that.
[0:00:59] Just feel free to mute and drop off of it. I won't be too offended. But if you stick around for that, I think it could be entertaining too. So step one, what kind of led to GPT-3? Well, unsurprisingly, GPT-2, you can tell from the number. But there was a particular thing about GPT-2 that was kind of interesting, which was GPT-2 talked about this like zero shot behavior. And so y'all might recall this.
[0:01:29] This figure from GPT-2, which showed four different tasks that were being done zero shot with GPT-2. And this was like pretty interesting and cool, but something that careful readers might've noticed was that the baselines here were things like embedding the nearest neighbor for translation. And y'all know about translation, like that's not a super strong baseline.
[0:01:59] And if we look over at summarization, you might see random three. And for people who work on summarization, random three is you're trying to make a summary of a big block of text. And random three is the cutting edge technique of taking three random sentences from the whole long article and just saying, these are the summary. So GPT-2 had some pretty bad.
[0:02:28] Zero shot performance. And I think that this was something where, like if I talked with Alec about GPT-2 when he was writing it, he said that this was actually something that he really, he struggled with writing this paper because Alec is like the machine learning engineer par excellence and he is like known for making things really great. And like, I think he really loves like squeezing every last bit of performance out and doing like really great on a task.
[0:02:57] And so it was hard for him where like GPT-2 was starting to do some of these things, but it was just really terrible at them. So GPT-2 wasn't really about like being impressive on these tasks. GPT-2 was about making some predictions and in particular pointed out that like, yes, it's still terrible at these tasks, but it's going up into the right. So it's like starting to get better. And so it was kind of making this rough prediction.
[0:03:27] That if we continued to make these models bigger, then they would get better at some of these tasks. And it didn't have a lot of evidence behind that, but it was like pointing at this and kind of like waggling its eyebrows suggestively. So this was kind of what GPT-2 was about. And so after GPT-2 and yeah, like as we were starting to get involved in GPT, like the beginnings of GPT-3, a different group.
[0:03:56] was putting together these scaling laws for neural language models, which I'll just refer to as scaling laws. And it really was about trying to make like scaling laws. So it's a bunch of physicists who were like, okay, like machine learning, it seems like has a bunch of different people who are like mathematicians proving things very concretely and a bunch of people who are like engineers who are gonna try to
[0:04:25] Throw things at a wall and kind of like see what works. What we're going to try to do is kind of bridge a middle ground where it's like we can't actually put together a super concrete theory about how things work, but we can gather a bunch of evidence and try to fit lots to it and try to make some sort of theories that have some predictive power, even if they aren't, even if we don't know why it is that these theories work. So kind of where GPT-2 made rough predictions.
[0:04:51] The scaling laws made incredibly detailed predictions. And so I was only kind of tangentially related in this, but I was a like end consumer of it where many of these plots kind of informed how we ended up doing GPT-3. So just walking through these a little bit from like, I kind of like to think about the scaling laws paper from like an economics point of view where
[0:05:21] It kind of gives like a production function for a transformer. So this is like, if you have some amount of, yeah, this is like, what are the inputs to a transformer that result in a certain loss? And so they study these kind of by breaking it up into the different components that you need, like parameters, data size, and compute. So here's like three of the main plots from the scaling laws paper.
[0:05:50] And as you can see, this parameters is showing how loss scales with parameters as long as you have enough data set size. So for this, it's trying to represent the infinite data set size limit and when you have infinite compute. So if you just train a model for as long as you want with a given amount of parameters, what is the loss that you'll end up with? And I think pretty astonishingly, they just found these incredibly, incredibly good fits.
[0:06:19] For a power law. And yeah, as you all know, a power law is just like a polynomial where you have exactly one value raised to a power plus some constant. And so the parameters seem like they fit well. The data set size, again, this is like if you have as big a model as you want and you have as much compute as you want, what loss can you get with a given amount of data?
[0:06:46] And then they put those together to also give like if you have a given amount of compute and you can make your parameters any size you want and you can make and you have as much data as you want, what is the efficient way to train a model given some compute? And they found that this also followed this really consistent power law. So that was one kind of prediction that they made.
[0:07:13] And then another prediction that they made was just by plotting the same thing, or sorry, just by plotting the loss of each token, they noticed that there was like, yes, these things were going down for any given model size. So like this is the smallest model. And so this is to some degree, this is basically what ended up in GPT-3 being the in context learning.
[0:07:41] So the model is picking up on a pattern as it goes along. I think this was not really surprising to people, but it was something that hadn't been, or at least to my knowledge, hadn't been studied very carefully. And also they showed that this happens more for the larger models, and the larger models actually sweep under the big models. So the amount of in-context learning that happens for these larger models is also more than for the small models.
[0:08:12] Then I think the other thing that was quite surprising and interesting about this was that scaling laws fit the loss over seven orders of magnitude. And seven orders of magnitude is just a crazy large amount. I think that was something that was quite surprising. And if you put GPT-2 here, it strongly implied that we could continue this out at least another half of the graph, maybe, and expect it to go
[0:08:41] To continue to be mostly straight. Okay, so that's kind of like what led to GPT-3. There were basically those like three predictions that were made. And then what GPT-3 was about was about validating those three predictions. And just to spell this out, I see there being like three main predictions that GPT-3 was about validating. And the first one was that the loss improves with the model size.
[0:09:12] So that was the last thing that was shown in the scaling laws paper. Or yeah, that basically just like we have these very straight lines where the loss will improve with model size or amount of compute that you have. Also from the scaling laws paper, in context learning improves with size. So you get more in context learning or like meta learning as the model scales. And finally, that like a lower loss.
[0:09:40] is correlated with better performance on many downstream tasks. And this was something that the scaling laws paper didn't look at, but that was like part of the like waggling suggestive eyebrows of GPT-2, which is basically like we had looked at a bunch of different tasks and for all the ones that we'd found so far, if you did better on the pre-training loss, you also did better on the downstream task. And so kind of these three predictions having been made and proven out enough or like
[0:10:11] Proven out to some degree was enough for OpenAI to be willing to invest another two orders of magnitude and compute to do this. So the bottom of this little red block is GPT-2 and the top is GPT-3. And so what we found was then, I guess, looking at prediction number one was that, yes, like as we scaled up.
[0:10:40] This is the same plot from the scaling laws paper with GPT-2, but now this is the GPT-3 version. And GPT-3 more or less follows this curve, even though we've added another two orders of magnitude of compute. I guess this actually, sorry, this is three orders of magnitude of compute, two orders of magnitude of model size. And I think going into this, we thought that, I don't know, my personal guess was that there was maybe like a...
[0:11:06] 60% chance that mostly this would continue to follow this curve. I think the physicists had more trust in the law of straight lines, so they were more optimistic about it. But I think it was worth putting that bet into it based off of the evidence that they'd had before.
[0:11:35] The first prediction was, yes, that loss does indeed improve with model size and it continues up to the scale of about GPT-3. The second one was that the scaling loss predicted this in context learning to improve. And the way we looked at that is we did this on just a ton of different tasks. And you can see here that as the models get bigger, they look
[0:12:03] The spread between the few shot, the zero shot, and the one shot starts to increase. So yeah, that's that one. One thing that you might notice is that this plot is kind of a weird plot. So this is an aggregate plot of 42 different accuracy denominated benchmarks. The original version of the plot that we had also had a blue score confusingly averaged into the accuracy. I'm glad that we cut it to just our 42 accuracy.
[0:12:38] Based benchmarks. But this is kind of like a weird way of showing this information. So by page count, the like majority of GPT-3, the GPT-3 paper is just like plot after plot that is just showing each of these things for individual tasks. So there's like a large appendix about this. And I think that was also a large amount of like the actual work that we put into this paper.
[0:13:06] was just running all of these different tasks following the text-to-text format that T5 pioneered. And what we found there was that really, for most but not all of the tasks, that lower loss really did yield better performance on these tasks. So those were the main three predictions that GPT-3 was about validating. So now I'm going to get to the fun part, which is talking about some limitations of GPT-3.
[0:13:39] There's too many to fit on one slide, so this is part one. So first thing is that it's incredibly expensive and inconvenient. The main innovation here was just making the model very, very big. And the two implications of that is you have to spend a lot of money to train it, you have to spend a lot of money to do inference on it, and you need a crazy distributed system to run it.
[0:14:06] And all of that is just very, very raw right now. It also had a quite outdated architecture. We did this. We were working on this as T5 came out. And I think T5 is the best evidence that we have that our model actually is using a totally outdated architecture and that it would do much better if we improved the architecture. And I appreciate all of the help that
[0:14:37] Colin and Adam gave over email to me as I was trying to figure out ways to improve this. I think that that was something that the T5 did a great job of sharing just best practices there with the community. And finally, GPT-3 is just really bad at NLI and NLI-like stuff. You can see here it's just flat near random guessing. I don't know why that is, but it's bad at that.
[0:15:07] So all these things I think will take work, but they definitely seem solvable. So let's get to limitations of GPT-3 part two. And here I'm going to start to get a little philosophical. GPT-3 is totally not interpretable. So we can't figure out what's going on inside this crazy black box that we have. It's not robust. It gives completely miscalibrated predictions. And if you get it out of a domain that...
[0:15:34] Yeah, if you get it going down a weird path, it'll do very weird and unexpected things. And of course, it retains all the biases of the data where it's been trained on the autoregressive language modeling task. And if you want to use it for something else, it's not going to be into that. What it wants to do is just predict the next token correctly. So as you might notice, yes, these are all limitations just of deep learning in general. And unfortunately, they probably won't be solved by GPT-4.
[0:16:06] So this kind of brings me to the future section, which is just like what might happen next. And I'm going to kind of like talk about like what might happen in the next five to 10 years. And I think the short answer of it is that I don't know. And I think that what I'd like is for all of you guys to make kind of your own predictions.
[0:16:32] And this is something that I think I've found useful to do every couple years as an exercise of just maybe setting aside two hours sometimes, get a cup of coffee, something like that, quiet space, and just write down what you think will happen with AI. I think that's actually something that is pretty rare for people to do. And then I think something that's even rarer is to write down what would change your mind.
[0:17:01] Because then I think you actually have a trigger to notice if something actually different or weird has happened with AI. And then maybe it would be time to actually update what you think is going to happen. And then finally, if you do that, just email me what your predictions are at Tom at OpenAI. I'm just very curious about this, and I will guarantee that I'll read them. I don't know if I'll have any useful feedback on it, but I really love just thinking what different people
[0:17:31] I think is going to happen, because I think there's a ton of uncertainty. I think one thing, though, that I am pretty certain about is that the stakes are becoming high. It seems like there's way more resources now being pushed into AI. And it seems like AI is likely to permeate every aspect of our lives within the next five to 10 years, unless something greatly updates what I think.
[0:18:00] This is what maybe to paraphrase Uncle Ben, with great parameters comes great responsibility. And I think that we have a chance as researchers now to kind of do some things to help out. And one of those things would be, yeah, what I described before, which is make your own predictions, come up with your own ideas about what you think might happen and share those things and be willing to update them as things change. I think we're kind of in like a...
[0:18:29] We're living in the future a bit, and we have a vantage point where we, I think, have some perspective on what types of things might happen that other people won't. I guess I talked about three big limitations that I don't think will be solved by GPT-4. I think those are things where I'd like to see more people working on that.
[0:18:58] And I think that that's something where, since those are such big limitations and they're slowing down everybody, there's something where if anybody makes progress on them, everyone would be useful to just snatch them up and immediately use them. They're like the most obvious public good where they just would have a huge impact on the world. And finally, just keep lines of communication open between these different research groups.
[0:19:28] I think that helps both from just spreading what the good ideas are faster. And I think it also helps out just in terms of if there are new things that we're worried about, it's a way of making sure that people know about what's going on. I think I've personally been super great. Yeah, I think we're all in this together.
[0:19:52] And I'm personally incredibly grateful for all the help from Adam, Colin, and the rest of T5 folks as we were putting together this project. And I'm grateful for all of you guys for inviting me here to talk. So thanks. I'm Tom.