[0:00:08] We're joined next by Greg Brockman, President, Chairman, and Founder of OpenAI, and Alexander Wang, CEO and Founder of ScaleAI. OpenAI is a research and deployment company whose mission is to ensure general-purpose artificial intelligence benefits all of humanity. Before OpenAI, Greg was the CTO of Stripe, which he helped build from four to 250 employees.
[0:00:34] Please join me in welcoming to the stage Greg Brockman and Alexander Wang. Hey Greg. Hey. Thanks for making it. Absolutely. Glad to be here. I want to start actually, I don't know if you remember this, but we first met at this summer camp called Spark where you gave a presentation about, at the time you were the CTO of Stripe and you gave this presentation about sort of like everything that you had accomplished and I was a member of that camp and it was extremely memorable. You had a lot of...
[0:01:14] A lot of good soundbites. I'm glad that it landed. Kind of a full circle moment. Well, I think to start out with, I mean, you've been CTO and now you're president of OpenAI, but CTO of two incredibly iconic companies, Stripe and OpenAI. In some ways, probably two of the most iconic startups of the past decade. I wanted to start out just by asking, in what ways I have...
[0:01:41] Are the two organizations the same and being CTO the same and in what ways are they different? Well, thank you for the kind words. Yeah, I think that one thing that's very interesting to me about kind of having been part of both of these organizations is seeing how much groups of people are kind of the same regardless of what the problem in front of you is. So, you know, I think that a lot of how we approached Stripe was thinking from first principles. Like I remember when we were pre-launch and we...
[0:02:09] We had some buzz going because we had some early customers, and one of my friends took me out to lunch. He was a VC, and he was like, all right, I've been hearing about this Stripe thing. What's your secret sauce? And I was like, I mean, we just make payments really good. And he's like, no, no, no, come on. You can tell me. What's the secret sauce? And really, that was the secret sauce, right, is that we had just rethought every single piece of what we were doing from the ground up, from first principles, not sort of locked into the way that people had been doing it.
[0:02:36] And we asked, how should it be? Where's all the pain? And does it need to be there? And I think that in AI, we did much the same. We really thought about, OK, there's this field that we're entering. And we hire a lot of people who had been in the field. But a lot of us also hadn't been in the field. And we came to it with beginner's eyes. And I think that that approach of just not being beholden to all the ways people were doing it, but also becoming expert in the way that things have been done. Because if you just throw everything out, you're also just going to be starting from scratch in a not helpful way.
[0:03:04] So I think that that maybe is the deepest commonality between them. But obviously very different organizations. For Stripe, I think that we ran the traditional startup playbook. You basically come up with the innovation and you just build, build, build. You get in front of customers from day one. The story is that we gave the first API to a customer who charged a credit card and he was like, I would like my money now, please. And we were like, huh, I guess we should build that.
[0:03:33] OpenAI, we had research to do. Where's the customer? And it really took us, I guess, five years. We started in late 2015, and it was really not until 2020 that we had our very first product. And so I think that that sort of figuring out what you're even supposed to work on, did you do a good job? Should you feel good on a day-to-day basis? I think that all of that had to come from within rather than from without. Yeah.
[0:04:01] Well, actually, I want to go back to this point that you mentioned around first principles thinking. It's very interesting because even I remember like maybe 2020 or 2021, you know, you would sort of post GPT-3, you would talk to other researchers in the field and even they would still, you know, there's still like some degree of skepticism over the sort of like core concept of scaling up these models and if there were still gains to be had, et cetera. And I think, you know, I don't know the story, but it seems like the sort of research.
[0:04:31] Sort of intuitions that led to GPT-3, Dolly 2, that have really ushered in kind of a new era of AI were probably somewhat against the grain or somewhat unintuitive at the time. One question I have for you is, I think now looking back, it's obviously very obvious to point out GPT-3, Dolly 2, basically have fundamentally accelerated AI.
[0:04:57] Progress and its relevance to the world and its relevance to every industry and sort of have created the sort of most recent AI wave. How is that matched up against your expectations when you were building these technologies? Yeah, well, I think the thing that's most interesting to me is that those
[0:05:16] Models you mentioned are kind of overnight successes that took many, many years to create. And so from the outside, it looks like, wow, you just produced this model and then that model. And really on the inside, the GPT arc, that's a five-year arc. It really started with the sentiment neuron paper, which is back in 2017. Do you remember that paper? I remember the paper. It was very cool, but it felt very novel. It felt very novel. Yeah, very few people remember it. It was this very early result where we basically had been training.
[0:05:44] A LSTM at the time to predict the next character in text. So we basically showed a bunch of Amazon reviews. We said, what's the next character? And of course, it's going to learn where the commas go. Of course, it's going to learn where the periods go. But of course, it's not going to understand anything. But we found a single neuron in that model that had learned a state-of-the-art sentiment analysis classifier. I could tell you, is this a positive review or negative review? That's understanding. I don't know what understanding means, but it's semantics for sure. And that for us was like, OK, this is going to work.
[0:06:12] The transformer came out late 2017, and my co-founder, Ilya, immediately was like, that's the thing. That's what we've been waiting for. So you take this sort of very early, nascent result, put in a transformer, and then that's GPT-1. GPT-2 is you just keep pushing it. And I think the algorithm we kind of run internally.
[0:06:32] Is that we do these little get signs of life. And you have to be very, very careful to distinguish signs of life from just pushing too hard on a specific data set that isn't really going to keep going. But if you build those right intuitions, then you know, OK, now is the time to put in more compute. Now is the time to put more researchers. Now is the time to really scale it up. And so GPT-2, obviously, was exciting. And we were all like, well, we look at the curves.
[0:06:55] The bigger we made this model, the more compute we put in, the more data we put in, the more we just got all the engineering details right, those curves just got better. And so actually, our goal was just to break the paradigm. It was just push it until the curve stopped looking good. And we still haven't managed to accomplish it. Yeah. Well, I think one of the things, at least for me and probably for many people who initially played with GPT-3, the shocking thing was not, I mean, it wasn't necessarily that even the model
[0:07:24] Got better and better performance on established tasks is that it sort of had all these qualitatively new behaviors that felt very magical. And even now, there's prompts that you'll see on Twitter or whatnot that are sort of really shocking. I mean, did you have these sort of early moments when you had the early model results where you're like, holy crap, this is magic? Yeah, well, I think the earliest one that I remember was around code.
[0:07:52] And just, you know, at the time, totally mind-blowing that you could just write a function name and a doc string, kind of describing what the function should do, and actually write it. Not super complicated functions, right? But just that it was able to, you know, you'd ask for something that would take a couple lines, and it would be able to really do it. You modify things a little bit to make sure it hadn't just memorized it, and sure enough, it would write out the modified code. And I think, you know, the overall thing that's really interesting about the paradigm of a GPT-3 is that
[0:08:20] Where it really comes from is that we kind of had this picture that, look, the problem with these models is that they're great within their data distribution. But as soon as you're outside of that distribution, all bets are off. And so what if you just make the whole world, the whole universe, be the data distribution? You put the whole internet in there. And I think that what we've really seen is that these models, that they really...
[0:08:44] are able to generalize extremely well within the kinds of things that they've seen. Again, different question if it's never seen anything like it. I mean, humans are also not very good at things you've never seen before. But I think that that picture of just like all of the different things that it's seen in all these different configurations is almost unimaginable. There's no human who's been able to consume 40 terabytes worth of text. And so I think that we just keep seeing surprises where you just ask for, one of my favorite ones actually was this.
[0:09:12] Teacher-student interaction where I was the teacher, Model was the student, and I managed to teach it how to sort numbers. And you just kind of have these experiences where that's what it should be like to interact with an AI. Yeah. I mean, it's incredibly shocking. One of the things I'm curious to get your thoughts on is I think in the path of developing GPT-3, I think probably the jump from GPT-2 to GPT-3 required a lot of conviction because you all were spending probably a fair amount on
[0:09:41] Yeah, and to your point that scale was not an obvious thing, not the company, but the scaling things up.
[0:10:11] At the time, the funny thing is actually our very first scale result that just sort of convinced us that this is the right way to approach things, you push until it breaks, not necessarily that more compute is just magically always going to solve your problem, was Dota. That was, you know, playing competitive video games. And there we kind of went through this whole, that was a three-year arc where we started out with something that didn't do anything, finally beat like, you know, the in-house team, then we managed to go beat the pros. And at each step...
[0:10:38] It was just kind of pushing in all dimensions, right? It's make the model bigger. It's to, you know, sort of, again, fix all the bugs. And you just kind of keep iterating on every single dimension, and every single dimension yields returns. And so I think that we did very much the same thing, where for GPT-2, you know, it's not as simple as saying, okay, like, clearly you just need to, like, you know, crank up this one variable, and you just do it in one shot. It's this, like, sort of iterative, like, stepping through the space on each axis at every single time.
[0:11:08] And so I think that on the one hand, it does require conviction because you do need to say, we're gonna like carve out a big compute budget so you're not constantly not kind of fighting other people for the big supercomputers. But on the other hand, I think it's also very iterative and you don't have to make scary irreversible decisions because in each step you get feedback from reality. And I think that that key of like both the big picture thinking of what if this works and make sure that you're really set up for success, but also don't blindly
[0:11:36] Spend a year of your organization on just pursuing a thing that might not pan out. I think that balancing those two is what was really key. Yeah. I mean, one of the cool things as you sort of walk through this and talk through the insights is that the sort of organizational learnings were really critical in this entire sort of path-dependent sort of path to GP3. It makes sense when you say it that sort of insights from
[0:12:04] Dota 2 and insights from the sentiment neuron, these were the key nuggets that led to the crystallized idea of scaling up and building GPT-3. But it's very unintuitive from the outside, and it's almost a statement of innovation in some sense, is that you're going to piece together this disparate collection of insights that you gather from a wide variety of experiments.
[0:12:29] Get the ingredients together and you build something magical. That's the first principle is thinking and action. Yeah. I think that the story of AI, I don't know if you think about this at all, but I think about this a little bit. I think the story of AI to date, and especially the past few years, and the story of OpenAI is probably going to be something that historians are going to study for decades and decades to come. Are there any fun stories from the journey of creating some of these foundation models that
[0:12:58] You think deserve to be in the history books? Well, I'll tell you my actual favorite story from the Dota days. So we'd been working on this system. And actually, the funny thing is, at the very beginning, we wrote down our list of milestones. On this date, we're going to be Jonas, our best OpenAI employee who also had many thousands of hours of Dota 2 gameplay. This date, we're going to beat the semi-pros. This date, we're going to beat the pros.
[0:13:25] And so it was supposed to be June 6th or something. June 6th rolls around. We don't have anything. He just crushes us. And two weeks go by, three weeks go by. We keep pushing back that deadline by a week every week. And then one day, we actually do beat him. And I think my conclusion was that it wasn't actually actionable to set those goals of outputs. You can only control your inputs. You can control the experiments you run. And so we just managed the project very differently after that.
[0:13:51] And the thing that was so crazy to me still is that, you know, so a week before the International, which is the, like, world championships, we're going to show up, we're going to play 1v1 against the best players in the world, we finally started beating our semi-protester. And we're like, okay, maybe this is actually going to happen. But then we learned that he actually was on, like, vacation. He didn't have his, like, real setup. And so we were like, oh, no, like, this is not going to go well. So we show up, you know, we continue to train. We, like, kind of do, like, a Hail Mary of, like, scaling things up, biggest scale we've ever done.
[0:14:21] We show up at the International, and we play against a low-ranked pro, like a previous pro. And we go 3-0, 3-0, 2-1. So we basically win-win, and then we did have one loss. And we take a look at it, and it's like this item that we'd never trained with, we'd never seen before. Like, oh, wow, OK, we need to add that and do it fast. And so the team stays up all night.
[0:14:46] Putting this thing into the training, getting the whole thing launched. And again, we doubled the scale. We're basically maxing out our CPU cores at this point. And start training. And we're supposed to play against the top pros in the world. Fortunately, they can't do the next day, so we get an additional day of training. And the number two person comes in. He plays against us. And we win, win, win, win, win. And he's like, OK, but.
[0:15:14] I beat this, but the top player is never going to lose to this thing. Or sorry, yeah, he beat me, but the top player is going to crush this thing. And fortunately, because he had spent so long playing, that guy couldn't come that day. So we got one more day of training. And that one more day of training was enough. And so I think it's just the story of you can really see the improvement.
[0:15:35] And at each step, we could see new behaviors of the system have learned. And I think that that experience of just sort of watching it grow up in front of you is just something that was really amazing. I'm actually surprised that because you would I sensibly you'd probably train the sort of like agents for.
[0:15:52] A long, long, long time going into the international. I'm surprised that each incremental day. Yeah, so this is something I think has changed over time. So at the time, we basically had two weeks worth of training, was like the whole model run. And so you'd start from scratch each time. And the thing that was really funny in the middle was that we'd put in this new item, we were training it, and when we took out of training, it was the best bot we ever saw, except that our semi-protester was looking at it and was like, this bot is doing something really dumb.
[0:16:17] It's just sitting there in the first wave and taking all this damage. It doesn't have to. I'm going to go beat it. He ran to go fight it, and he lost. I was like, that was weird. And he did it like five more times. He lost each time. But then he figured out a strategy that actually does work, which is you realize what was going on was it was baiting him. It had learned to deceive. It actually learned that what you do is that you pretend, oh, I'm just a weak little bot. I don't know what I'm doing. And then a person comes in, and you're just like, smack. And so the way you defeat that is that you actually don't fall for the bait.
[0:16:46] You let the bot take all this damage and sit there and get weaker, and then you finally go in for the kill. And so there we actually stitched together our good bot for the first wave with the deceived bot thereafter. And so there was a lot of this really examining what was going on in the systems because it's such a limited domain. It's a complicated domain, but it's very, very interpretable. It meant that we could observe behaviors like this and figure out how to engineer around them.
[0:17:11] But once we graduated from the 1v1 version of the game to the full 5v5, much more like competitive basketball or something rather than heads up, suddenly all of our analysis of the behavior stopped working. We used to have someone who just literally would watch the bot play and be like, oh, we have this bug in the training. We've got to go fix that. For 5v5, we just could not do that. And I think that's kind of where we've graduated as a field, is that when you look at GPT-3 and the mistakes it makes, sometimes people ask, well, why did it make that mistake?
[0:17:40] And sometimes you can interpret it, but sometimes it's also a little bit like asking, well, why did you make a mistake on some test? It's like, well, you think you know, but your explanation isn't always very good. And I think that to do complicated behaviors, sometimes there's a very complicated explanation. Yeah. Have you read this short story? I think it's like The Life Cycle of Software Objects by Ted Chiang.
[0:18:01] I think I have, but I don't recall the details. It's about how there are these AI pets, and they sort of keep learning new and new behaviors. It's very reminiscent of what I described as these Dota agents. Yeah, I think that we'll see that kind of thing in our future somewhere. Yeah. I want to kind of go back. One of the things, we've known each other for many years, long before these foundation models, and even before this competition, Dota 2. And one thing I vividly remember is how...
[0:18:30] Optimistic and confident you were in this path of increasing and increasing AI capability. I remember the time was maybe 2016, 2017. It felt very striking because it was like, with these algorithms, they're still pretty weak. You were always very confident, like, oh yeah, they're just going to keep getting better and better and better. You had a lot of confidence in that.
[0:19:00] Sort of the resolve or confidence in the, and the optimism in the technology. Yeah, I mean, at some level, you know, to have that kind of belief and conviction in something that hasn't happened yet, it's a very intuitive thing. I mean, I remember when I was in school and showed up excited about doing NLP research. I went, tracked down an NLP professor, and I was like, please, can I do some research for you? And he's like, okay. He shows me these, like,
[0:19:27] I was like, this is never going to work. And to explain why does it feel like it's not going to work, it just doesn't have the right properties. It just felt like you're going to pour all this human engineering and intuition and effort into the system, and I know I can't even describe how language works. It just feels like there's just something inherently missing. I think neural nets have the opposite property. Neural nets is very clear.
[0:19:52] This is a system that just absorbs data. It absorbs compute. It's like a sponge that just slurps everything up. And so it has the right form factor. But the thing that's always been missing is, well, can you train it? Do you have enough data? Do you have enough compute? Do you have enough ability to have a learning algorithm that can shove all this stuff in efficiently in a way that it comes out in some way that generalizes? That's the thing that's been missing. And I think what became kind of clear, the field really, I think, got its most recent resurgence in 2012 with the
[0:20:21] The AlexNet paper. And I think that there, that was the first time where you had a neural net that really just crushed a task, right? That it was like people had spent decades on computer vision and suddenly it's like, well, I'm so sorry, but this approach has just supplanted you by this massive gap. And I think that you just started to see it spread, right? That it was almost like you had all these disparate departments and there was this wall that was being knocked down day after day. And I think that...
[0:20:51] When you see a trend like that, where things that have been longstanding and very deeply established, and these ways of thinking, these great debates that have gone on for a long time, and suddenly you're seeing a repeated result that is consistent with the history, I think that that, for me, is maybe the most clear sign that something is going to work, and there's a real sort of exponential that is waiting to unfold. And then, you know, were there, what were the moments, if any, of doubt?
[0:21:17] Let's chart the path. I think OpenAI started in 2016? Yeah, yeah, I'd say December 2015, 2016. Okay, great. December 2015 till now, were there any moments of doubt in the technology? Or was it sort of always, hey, this is clearly the way of the future? Yeah. I mean, I think that doubt is a strong word. There's definitely moments, like, I think to build something, you're always doubting, right? That you're always like...
[0:21:43] You've got to be questioning every single bit of your implementation. Anytime you see a graph that's wiggling in a weird way, you've got to go figure it out. You can't just be like, ah, I'm sure that the AIs will sort it out. And so I think there's lots of tactical doubt, lots of worries that we're not quite doing it right, lots of redoing the calculations to figure out, hey, how big of a model do you think you're going to need? Lots of mistakes, for sure. A good example of this is the scaling laws.
[0:22:10] So we did this study to actually start to really scientifically understand how do models improve as you push on various axes? So as you pour more compute in, as you pour more data in. And one conclusion that we had at one point was that basically that there's sort of a limited amount of data that you want to pour into these models and that there's kind of this very clear curve. And that one thing that we realized only years later was actually that we'd read the curves a little bit wrong.
[0:22:37] And you actually want to be training for way more tokens, way more data than anyone had expected. And there's definitely these moments where these things that just didn't quite click, where it's like it just didn't add up that we were training for so little and that some conclusions that you drew downstream. But then you realize there was a foundational assumption that was wrong, and suddenly things make way more sense. So I think it's a little bit like physics in some sense, where like, do you doubt physics? It's like, I kind of do. I think all physics is wrong, right?
[0:23:05] Only so wrong, right? It's like we clearly haven't reconciled like quantum and relativity. So there's like something wrong there, but that wrongness is actually an opportunity. It's actually a sign of you have this thing that's already useful, right? It really like has affected our lives and it's actually like pretty great. I'm very happy with what physics has done, but also there's fruit. And so I think that for me, that's always been the feeling that there's something here and that...
[0:23:26] If we do keep pushing and somehow the scaling laws all peter out and they suddenly drop off a cliff and we can't make any further progress, that would be the most exciting time in this field because we would finally reach the limit of technology, we would finally learn something, and then we would finally have a picture of what the next thing to do is. It actually reminds me of one of the Stripe operating principles, which is, I think, micro-pessimist, macro-optimist.
[0:23:52] Yes. And it's very, I mean, it's very resonant, but obviously like very related to what you're talking about, which is these, you know, you have to be extremely pessimistic or extremely questioning in the moments of the technology, but then obviously on a long enough time horizon, you know, incredible stuff pops out. Yep. And you got to be excited. Like, I think that this is just an exciting field and it's a scary field as well. You've got to have some amount of just like awe at the fact that you have these models that they started as just random numbers.
[0:24:22] And then you build these massive supercomputers, these massive data sets, and you do a ton of the engineering work. You do a ton of these algorithmic developments. You put them all into a package. And we don't really have other technologies that work like this. I think the fact, to me, the most fundamental picture, this sponge that you just kind of pour stuff into, and you get this model that's reusable and works across all these different areas. You can't do that for traditional software. Traditional software is just...
[0:24:49] You know, human effort writing down all the rules, and that's where the return comes from. But you can't, you know, maybe you have like a Spark cluster that does some stuff, but that's not the cake. And in neural networks, it really is. Yeah. You know, I want to kind of switch gears to thinking about the sort of future and looking forward at what's next. What do you think, I mean, I'll ask this sort of as broadly as possible to start with, what do you think the future of AI holds? Yeah, I think that...
[0:25:17] The future of AI is going to, again, be both exciting and a source of a lot of change. And I think that that is something that, you know, part of our mission is to try to help facilitate that in as positive way as possible. I think that kind of, you know, the super high level, I kind of feel like AI was like, you know, something that for the, you know, 2010s was like kind of cool. You know, it's the game of like publish some papers and you play some video games and like, you know, it's just like fun, good science.
[0:25:46] I think it's really interesting that 2020 kicked off with GPT-3, which is really the first model that was commercially useful just as the model. Literally, you put an API on top of it, and people just talk to it. And people build products on top of it. And one of our early customers just raised it at $1.5 billion valuation, which to me is a really wonderful thing to realize that you build this model, and it creates so much value for so many different people.
[0:26:14] We're still in such early days for what these models can do. And so I think that what I'm most excited about from just seeing GPT-3, seeing DALI, is thinking about the sort of economic value that it can create for people. And I think that there's a lot of other pieces to it in terms of that everyone's going to be more creative if you want to. Like, I can't draw, but now I can create images. Now I can take a picture of this in my head, and I can actually see it on a page. And one of my favorite applications of DALI is actually people who are
[0:26:43] 3D physical artists, you know, somebody who's like a sculptor. And now they can actually get a great rendering of the thing that they have in mind by kind of just like iterating with this machine and then they go build it, right? I think that this sort of amplification of what humans can do is what these systems are for. And so I think that for this decade, I think what we're really gonna see is these tools just sort of proliferating. They're going to be everywhere. They're gonna be baked into every company. I think it's kind of like the internet transition that, you know, that it was kind of like.
[0:27:11] If you're a company, what's your internet strategy? In 1990, it's like, what even is this thing? And in 2000, it's like, huh, maybe it's interesting. And there was a little boom and bust. And here we are today to even talk about internet strategies. It's just so integral to every business. It's not even a separate thing. It's just part of your payroll strategy. It's not a separate part of your business that you can pick or choose whether you're going to have it. And I think that AI is going to be much the same. I think there will be a transition point.
[0:27:40] I think that it's interesting. Our mission is really about building artificial general intelligence, really trying to build machines that are able to perform whole tasks, that push this technology to its limit and build machines that are able to, our charter definition is outperform humans at most economically valuable work. And there's a question of the timeline, but I think that that picture of you have these tools that are creative, that help everyone amplify, and what happens when they do become so capable that they're able to perform these tasks even autonomously?
[0:28:10] And I think that actually the implications of that are different from what people expect. I think that it's much more like, you know, I think there's still going to be this amplification, but I think that there the change is going to be just very hard to predict and unexpected. And I think that really thinking about how all of that sort of value gets distributed, how to make sure that it's sort of pointed at solving these like hard challenges that humans, you know, maybe are unable to solve ourselves, you know, the climate change and, you know, universal education and things like that.
[0:28:39] And really transitioning to this AI-powered world, I think is going to be just like a real sort of challenge for all of humanity to work together on. Yeah, I mean, I totally agree. One thing that I think is almost funny with how the timing of all these technologies have worked out is that last year everyone was talking about Web3 as crypto, and now it feels very obvious that AI is the actual Web3. We'll take Web4. Yeah, Web4. We'll skip over Web1. But sort of like Web1 was just reading.
[0:29:09] Web 2.0 was reading and writing, and now Web 3.0 or 4.0, depending on what we want to say, adds computer read and computer write, and it's sort of this incredible new phase. I think you mentioned two directions here that I think are really interesting. One is the sort of advancement and proliferation of GP3 and DALI and the existing tools becoming more and more economically useful, and there's sort of this...
[0:29:36] I mean, I think that humanity to a large extent has been on the AGI roadmap for a very, very long time. I think even looking at just the history of neural networks in particular.
[0:30:00] On the one hand, we say, hey, 2012, that was the moment. Everything changed. We have all these curves of how much compute people put into the landmark results. It was going like 10x year over year. Still continuing, by the way. That's a decade of 10x year over year. That's insane. And the thing is, we actually did a study to then look back at previous results, all the way back to, let's say, the Perceptron in 1959. And you actually find...
[0:30:29] That there's basically a very smooth curve back there as well. The amount of compute going into all the landmark results was exactly Moore's law. And it kind of makes sense, right? It's like that people were not willing to spend more money. They wanted to spend a constant amount of money on these experiments because you're starving grad students. Like, you know, you can only get so much computer time. And that the results got better and better the more compute was available to them.
[0:30:55] And I think that that is so interesting that, yeah, basically what changed in 2012 was that we said, OK, we're just going to like, you know, we are going to spend more money. We're going to build massive supercomputers now because the ROI is there. But that fundamentally the curve, if you control for that cost factor, it looks exactly the same. And so I think that basically this picture of building more capable models by pouring more compute into them, by getting better at harnessing this technology of neural networks back propagation, I think that has been very invariant.
[0:31:22] And the details maybe change a little bit. Do you want to work on GPT-3? Do you want to work on Whisper? Do you pour in your speech data? Do you pour in text data from the internet? And to me, those details, I think they matter in the sense of what are you going to work on today? And what are you going to download? But if you zoom out and you look at the scale of this technology, I think it actually sort of doesn't matter so much.
[0:31:46] And I think kind of what we're building, it's almost like building computers. Like you think about the heyday of Moore's law, right? Where it's just like, there's a new chip that comes out, and then there's a new chip that comes out. And it's kind of like, what's the path to building the best computer? The answer is, well, you just keep building the next best chip, and you keep building the next best chip, and you keep getting better peripherals, and all these, you know, you keep working on every single piece of the technology. And so I think this full stack of better GPUs, great software for utilizing them,
[0:32:10] Neural networks that we learn to harness more and more. The scaling laws, doing all the science. Alignment, extremely important. Making sure these models not just are smart, but actually are aligned with what humans intend. All of that, I think, is the stack. And so I think that what our goal is is just to keep doing something that was previously impossible every single year. So I guess you should check back in a year. But hopefully, 2023, we'll all forget about DALI 2 and GPT-3, and we'll be talking about something new. And I think as long as we continue that,
[0:32:40] You cannot continue that path without ending up somewhere amazing. Yeah. I mean, I think I actually remember this in probably 2017. You were sort of very still quite, you were very excited about sort of the sort of Moore's law continuing and that sort of creating a lot more opportunity for neural networks and AI. And that's sort of played out. Are you worried about the sort of
[0:33:08] Proverbial end of Moore's law, kind of causing a stall out in progress. So I'm not worried about it per se. I think the way to think about this, right, because I think we often get caught in this debate of like, is it all about scale or is it all about algorithms, is it all about data? And the answer is, that's the wrong question, right? It's really like you multiply together these factors and the best thing to do when you're multiplying together multiple terms is that you actually kind of want them all to be equal. And I think that the answer is like, it's been great for the past, you know,
[0:33:38] Seven years that we've been able to just pour more dollars to build bigger computers. That's one way to get ahead of Moore's Law. At some point, there just aren't more dollars. There aren't more grains of sand that have been turned into these wonderful computers that we use. So there is a limit there that we have not yet hit.
[0:33:58] But when you do, that does not stall all progress. You still have algorithmic progress. And there we've again done studies. And we've shown that actually if you take, like an example is, if you look at the amount of compute it takes to hit the same performance, so to train a state-of-the-art 2012 or 2014 vision model, that that compute's also falling exponentially. We're basically making exponential progress in algorithms. Not at the same rate as we are able to build bigger computers, but that is a...
[0:34:27] Amazing force, too. It's like, I've got this exponential. I've got that exponential. Let's not even talk about the data exponential. So I think that the truth is that we will find a way. I think that the history of this field is just so consistent. And I think that humanity is just so innovative that I think that we're not going to hit a wall for the foreseeable future. And do you think that one of the interesting juxtapositions of today, just from a scientific perspective, is
[0:34:56] A relative slowing in nearly every other science. And there's a lot of research that sort of demonstrated that science on the whole is slowing. And then comparatively, the sort of acceleration of artificial intelligence and sort of this, in many ways, this renaissance that we're entering right now. Do you fear that at some point AI will similarly sort of reach these points of diminishing marginal returns and slow relative, like much in the same way that other sciences have? Or do you think that's so far away that, you know? Well, I think.
[0:35:25] Two things. I mean, I think that there's always S-curves. Although I think that something that's also interesting about S-curves is that there tends to be paradigm shifts. Like, have you ever read Singularity is Near? No, I haven't. Yeah, so this is the Ray Kurzweil book from like 2004 or something. And I always thought, just based on the reputation, it's going to be kind of a crazy book. But if you actually read it, it's the most dry, boring reading you'll ever do. And it's basically just curve after curve of different...
[0:35:52] Industries within computing showing how the performance has changed over time. And it's, you know, basically the conclusion he comes to is that there's this repeated pattern that seems to happen across, you know, memory, across number of transistors on the chip, you know, et cetera, et cetera, where you kind of have an S curve of the current paradigm. And then you have a paradigm shift. And that example he talks about is, you know, thinking about.
[0:36:18] Let's talk about CDs. So you talk about great CD adoption. It's great S-curve. It's suddenly everywhere. It's like everyone's got a CD player. It's just the technology of the day. And people get really excited about doing more of the same thing. It's like Blu-ray. That's the thing. And so then everyone starts investing in Blu-rays, and somehow it just doesn't take off. And it's because it's just more of the same, and it's not backwards compatible, and so it's just not really worth it. But the real paradigm shift was streaming. Suddenly you have this new adoption curve, this new S-curve that just is this totally different way.
[0:36:47] And the way we got to fast computers was basically five different paradigm shifts across 100 years. And so I think that that's maybe a story here, too, which is like, there's going to be an S-curve in what we're doing right now, and that there will be a paradigm shift when you hit it. And I think that that, again, speaks to the ingenuity of humans. But I think there's also a second thing, where my other answer is, to some extent, it doesn't matter, because the thing about this field is that it's useful now.
[0:37:12] The goal that I think we've always had for AI was to actually make it so computers are just way more helpful. You think about what computers have done for humanity, how many problems they've helped us solve. They've created new problems as well, but I think that on net that they've helped us solve way more problems than they've created, and I think they've just changed the nature of how we interact with each other, about how it's just hard to get lost anymore. You just plug Google Maps, and I think there's really amazing problems that are now within our reach that just would not have been otherwise.
[0:37:40] And I think that AI, like, we're starting to crack that nut. We're starting to be able to, you know, like, I think it's kind of interesting. You've got GitHub Copilot, you know, which we power. We have the models that power it. And the way that that is useful to people is it provides very low latency suggestions, right? It's basically an autocomplete for code. And that, you know, there's a very strict latency budget. You know, if you're more than, you know, 1,500 milliseconds to get an autocomplete suggestion, it's worthless. Like, no one wants that. You've already moved on.
[0:38:10] But I think that what we really want to build the next phase is machines that help you produce, that are able to produce artifacts that are materially interesting on their own. So not just interesting because it's like a fast suggestion to you, but because it's actually a quality answer. And you're starting to see it. If you talk to our current GPT iteration, you can ask it to write some poems. And it writes way better poetry than I can. It actually wrote a poem for my wife that made us both cry. Like, you know? Yeah.
[0:38:37] I cannot do that myself. But now I can, you know, by partnering with this machine. And I think that that's the real story, right, is really trying to get these tools out and everywhere. And yeah, you know, if what we're doing right now stalls out, I don't think that removes the value from what we're able to create. Yeah. By the way, it's depressing that the attention span of most engineers is only 1,500 seconds, or 1,500 milliseconds, but it is what it is.
[0:39:06] What, if anything, you know, I think one of the things that, if I recall, spurred you to work on OpenAI was sort of also being concerned about the sort of potential negative consequences of the technology. At this point, looking forward, what are your sort of biggest concerns, or what are you afraid of with artificial intelligence that you sort of urge everyone in the field to sort of help avoid? Yeah, so I think that one thing that's very interesting about AI is that
[0:39:36] You know, if you talk to, certainly, you know, 10 years ago, if you looked at every article about it, you talked to someone on the street, Terminator is the main thing that comes up, right? So I think there's always been this feeling around AI that is sort of, you know, that there's an element of fear mixed with the, you know, sometimes people don't see any potential or sometimes, you know, they realize that there is the potential, but like really trying to figure out how to navigate it. And I think that that picture, the specifics,
[0:40:01] I think that we're starting to see a little bit more, but I think the high-level picture of this is technology that's very powerful, and it can be powerful in positive ways and negative ways, I think is extremely correct. And I think it's very important not just to be starry-eyed optimist, everything's just going to work itself out, but also not to be...
[0:40:17] You know, sort of doomsday, like everything is terrible and, you know, humanity is over because I don't think that's at all true. I think this technology can be the best thing that we've ever created and help us be the best versions of ourselves. But I think that it requires very careful navigating in the space. And it's not something that just is for, you know, companies in Silicon Valley to figure out. I think it's really all of humanity kind of challenge. So I think that we're going to go through different phases. I think that right now, you know, we're kind of starting to build systems where the
[0:40:47] You think about misuse is the most clear problem. And the systems themselves are still not very powerful. The kinds of things you worry about for GPT-3 are important problems. You think about bias and representation. You think about the system sort of saying the wrong thing. But its action is really in your mind. It's sort of words on a page that then, words on a page are very powerful, but that they don't themselves have direct action in the world. But you think about something like Codex, our code writing system.
[0:41:16] Which is a little bit more like a robot because it emits code. And if you were to just execute that code directly, it can actually directly have actuators into the world and making sure that that's aligned and doing the right kinds of things, not having buggy code and not writing viruses and that kind of thing. That's really important. And so I think that figuring out what values go into these machines and that they're operating according to those values, that's going to be very, very critical. Figuring out how to avoid misuse and sort of regulate that, both at a sort of societal level, at a...
[0:41:45] You know, technical level, all of that is very important. And I do think that there is also a point where the technology itself, you have to think about that it's going to be extremely powerful. And you think about a system that's, you know, sort of talking to lots of humans and is operating unchecked. That's the kind of thing that you should worry about. You know, we already worry about that. Think about the companies, right, that lots of people are using us, you know.
[0:42:08] Social media platform or any of the technologies that we use and how much influence those can have in the world. And those aren't systems that have deep behaviors that are emergent from what they've learned. And so I think that figuring out the technical controls to make sure that these systems remain in service of humanity and to actually empower and accelerate all of us, that I think is also a very critical thing. So it's kind of this ramping set of stakes.
[0:42:37] And making sure that we're building systems that are aligned with our values and figuring out what that even means. What is the values of humanity that should be in the system? And that, I think, is not going to be an easy problem. One question, and this may be the last question that I have for you, is one of the conclusions of if the technology is such that scale continues to be one of the more important things, whether it's data or better algorithms or scale of compute,
[0:43:07] Then the technology itself will tend towards sort of this game theoretical proliferation mode where it's sort of like people are going to compete. And you see some of this today even with the large tech companies and you guys. Obviously people are going to compete to sort of build the bigger supercomputers that have the better performance. And you have the bigger supercomputer, you have sort of supremacy over the other supercomputers. And sort of there's like this laddering of the stakes and proliferation is really the right word.
[0:43:36] Do you think that that is a version of the future, or do you think that there's sort of some path in which this becomes a much more open and useful and not this sort of tool for nation states or large companies to compete with one another? I think that the future that seems to be unfolding is kind of a replay of how, say, computing technology has played out more broadly. I think that it is.
[0:44:05] Still going to be the case that you're going to have these increasingly massive supercomputers that are in the hands of only a few that are able to just create models that can just do crazy things that no one else can do. But I don't think that that removes the value from the massive set of things that people are going to do with these models. And so I think that balancing the super powerful, very dual use, extremely, think of these almost like these massive sort of.
[0:44:32] Systems that are, you know, think about like a nuclear reactor and it's like, you know, it's like these like giant like sort of, you know, systems that you should approach with great care. And you think about like, you know, by contrast, think about wind turbines and like there's lots of wind turbines everywhere. And actually, if you add up the amount of value from wind turbines versus nuclear reactors, like I think actually that the balance is probably in favor of wind turbines. And so I think that that's kind of the future we're going to is that the AI technology is going to be everywhere and there's going to be.
[0:44:57] Lots of value that's delivered by having open source models that are integrated to every business and that people are building all sorts of crazy applications on top of. And that's something that we really want to support and promote. And you also have to have this dual answer for what you do with the new, extremely capable stuff that's just a mile ahead of everything else. And that's something you have to treat with kid gloves, with more care. And I think that that balance is tricky. It's not easy. That's something that we as an organization have been trying to straddle. And I think that we've had real existential struggles internally trying to figure out
[0:45:27] Our goal is to empower everyone. It's really to bring everyone along to this AI transition. And the best way to do that, I think that our picture of it has changed as the technology has unfolded. And I think that we're starting to get a sense of where this can go. It's really exciting to see all the energy of all these builders coming in because I think that, like you said, people are starting to realize AI is really going to work and it's time to build. Yeah. Well, this was an incredible conversation. Thank you so much, Greg.
[0:45:54] Next time we speak, I'll make you read the poem that you wrote. All right, there we go. Cool. Thank you so much. Thank you so much. Cool.