[0:00:00] Thank you to Chris for organizing this event. I'm super delighted to have Greg Brockman, who's the CTO of and the co-founder of OpenAI. Before I dive into a brief introduction with him, I just want to introduce myself just so you know who is asking the questions. So my name is Nii here, and I'm a third year PhD student at Imperial College, where I study multimodal artificial intelligence. My work primarily involves generative tasks.
[0:00:25] The intersection of computer vision and NLP. And I guess similar to OpenAI, my passion for research comes from this key milestone of creating a brain like or a general intelligence. I'm also a co-founder of an early stage startup called Feather. It's not about me, though, but if you do have any questions about that, feel free to reach out and DM me.
[0:00:45] Anyway, so as I said, I'm super delighted to host Greg Brockman today. So most notably, he's the CTO of Stripe and also the co-founder and CTO of OpenAI. He was a key part in the team that developed OpenAI 5, which is an algorithm that beat professional Dota 2 players and teams. And before his experience in the corporate world, he has done a vast amount of things before, including bug fixes for Git and network and infrastructure systems for Harvard.
[0:01:14] So while he is quite technical in nature, we won't be diving too much into that. This talk will be more focused around his journey from student to entrepreneur and how his learnings across the way. So thank you for joining us, Greg. As I said, we are super delighted to have you. Glad to be here. Cool. So kind of let's start with Stripe, because I think that's one everybody has now heard of Stripe. It's a massive company. And if I did my research correctly, I think you were the third employee.
[0:01:45] Yeah, there were four of us hacking in a room. So two founders, first employee, and I was kind of the first engineer to have joined. Okay, cool. So I think when you joined, Stripe didn't even have any connection with banks or payment or financial institutions at that point in time. I remember watching Patrick Collison's talk, and he was saying how they hired someone for business there as the sixth person who had contacts with these.
[0:02:10] So how did it feel like working in early stage startup where you didn't actually know whether this thing might succeed or not because you were very much working in the dark and so many things which were out of your control weren't in your control? Well, when I joined, the way that we would sign people up for a merchant account was that we had a contact.
[0:02:34] At just some random merchant processor. And we'd ask them, hey, can we send some merchant account applications your way? And they said, okay. And so you would sign up on our nice website. You punch in your details and what your business is. And then frantically on the back end, we would fill out a form and fax it over to these people. And so we were really just proxying and sitting in the middle. But of course, that I think is the right way to do it, right? That we were kind of really trying to figure out what was going on in the rails and to...
[0:03:02] Provide the experience that we wanted to build to users and see what actually people wanted. And I think that that actually kind of exemplifies maybe the feeling, right? Is that there's a lot on the back end that I think that you're like, is this going to work? Is it actually the way to do it? Do people even want this thing? And you have to try it, right? That's the only way to figure it out. And I think that one thing that for me, I joined that company not because I thought that it was particularly likely to succeed.
[0:03:31] But because the people were people I wanted to work with, that I'd kind of spent my time in college just bouncing around from project to project. And from each one, I learned another thing not to do. But when I met the people at Stripe, I realized, like, these are the people that I've been looking for. And I just want to learn from them. Like, I kind of assumed that I would go there and, you know, maybe it wouldn't last that long and I'd go back to school. But I actually ended up dropping out and staying there and it ended up working. Yeah, watching some of Patrick's videos and the way he talks, I definitely can...
[0:04:00] Get a sense for how inspirational it must have been to be around somebody like that. So and obviously what you said seems like a classic example of do things that don't scale and a true MVP approach. So I'm glad that actually worked out. Did you ever feel in the dark or unsure of the future of that startup? So I know you mentioned working with the people was important, but kind of what was it that gave you faith that this would succeed or this might? What was the tipping point where you were like, oh, damn, this could actually work? Yep.
[0:04:29] Well, I mean, I do think that my kind of algorithm for figuring out what to work on is to find something where, you know, I was thinking of the upside of what if this works, right? And so for Stripe, I was thinking about, okay, like if this actually works, you've got all the payments in the world. That seems pretty big. This whole startup thing seems like, you know, there's lots of startups being started and they all are going to need to charge for things. And so you can really tell the dream of, you know, that upside case. And then you think about the risks, right? Think about, okay, like what happens if it goes wrong?
[0:04:56] And honestly, the thing about startups, you know, that I think that, you know, that there's a certain sort of, you know, level of security that you need to have to be able to make the statement. But, you know, if you're kind of in a place where if it fails, you know, that you're okay, right? That you can go to school, you can go to another company, that it's kind of nothing catastrophic to you or your family, then actually the downside is pretty limited. And so I think that that was kind of the calculus for me. You know, for me, I was still a student. I was still in school, studying things. And basically, rather than spending my time...
[0:05:25] Thinking hard about classes was spending my time thinking hard about startups. And so this actually seemed like a much more efficient use of that time than sitting around in school. So I think that for me, there was this question of, you know, I don't think that I had any particular, I wouldn't say I had faith that the whole thing was going to work, but I had the excitement that it might. And the downside seemed very capped. And, you know, the thing is, I'd actually applied this algorithm to maybe like 10 other ideas and none of those panned out. But we're not here talking about them because, again, the cost was just not very high.
[0:05:55] Yeah, there's a stat that like one in 10 startups fail. So I just say, why don't you start 10 startups? And if you're just constantly investing in yourself, then again, the ROI is actually pretty good. Yeah, it's a good way of phrasing this asymmetric bets as well. And especially while you're young, obviously, you have more of an appetite for risk because you know, you can fall back on cottage, as you said, you were still a student at the time. So throughout your career, you taking a look at your website, it seems that you've
[0:06:24] Focused on building and managing like more advanced networks and systems and infrastructure kind of roles. So what were the reasons that you jumped to OpenAI, which is like a completely different vertical of technology than, you know, the classic infrastructure and network stuff? Yeah, so I like to work on problems that I feel like are not going to play out in the same way if I were not there. And, you know, for Stripe at the very beginning, I felt very critical, right?
[0:06:51] There was like many, many, many times where the barrier between us and success was doing some sort of very hard technology work. For example, we needed to sort of do the integration with the payment processors that usually takes like nine to 12 months. And we wanted to launch in like a month. And so I basically wrote all of that in a day. And yeah, it was a very fun night. It was just like doing a problem set in college. And as we grew and sort of...
[0:07:19] Got more certainty on that future, as you say. Then I started to feel like, well, I think this thing is going to succeed with or without me. And empirically, I've been gone for six years and it hasn't crashed and burned. So I think I was right. And so then I really had a chance to step back and think about, well, what is a problem that I really want to work on? And I think that another thing that really flagged for me is if you...
[0:07:43] You know, I spent a lot of time really thinking about career, right? So if you think about your career, you chunk it up to like five-year blocks. You know, you have like maybe eight of those. And, you know, I think five-year blocks is about the right length because shorter than that, it's just really hard to have a real impact. Longer than that, like you probably just have diminishing returns. Like I have some friends who have, you know, done startups and been there for a decade. I remember thinking like, you know, like, okay, like your startup is still kind of the same startup it was, you know, five years ago. And I just kind of feeling like that's not what I wanted for me.
[0:08:13] But I think that what I started really thinking about when I reached that sort of point at Stripe that I felt like I was complete was thinking about, well, what is a problem that I'd actually be happy spending my remaining seven chunks on and just be fine with marginal impact because I just care so much about the end goal? And for me, there were three things on that list that I could think of. Number one was AI. And I just didn't know if it was the right time. I didn't know if I had skills to contribute. Maybe the right thing to do is just try to go make a bunch of money somewhere else and donate it back.
[0:08:43] VR and AR was another thing on the list. And programming education, because I just think it's like totally broken how we learn to program right now. So I kind of investigated each of these. VR and AR, I just kind of concluded like, I just am not excited about the applications of that. No offense to anyone who's working on it, but it just wasn't the kind of thing that I wanted to contribute to directly. Programming education, I'm actually still very passionate about. But the thing that really just sort of blew my mind was realizing that AI
[0:09:10] was really happening. Like in college, I remember I went to a professor and asked him if I could do some NLP research. He showed me some like parse tree stuff. And I was like, this is just not going to do it. And so I focused on system design and programming languages, you know, which kind of capture the same idea of AI for me, you know, computer solving problems that I can't, but also worked. And in 2015, I was poking around deep learning because, you know, you see deep learning on Hacker News every day, but I didn't know what a deep learning was. And so I started getting interested people in the field.
[0:09:38] And I just kept getting introduced to my smartest friends from college. I've realized how small the world is as well in AI too, yeah. Yes. But I just realized like, huh, these people are all working on this. There's got to be something real. And so I kind of dove in and realized that this is, I think, you know, we're kind of at the beginning still of something that's going to really change the world. And I wanted to just help contribute to it. So you said that you break your career up into five-year chunks and you felt like you played your role out there with Stripe.
[0:10:08] Do you feel like you've played your role out with OpenAI, or is this something that you want to dedicate more time to? Yeah, well, I'm six years in now, and it definitely feels like we're just getting started. And I think it's interesting because I think that I really went into this thinking, what is a problem that I just care so much about that I care less about? I'm happy to sweep the floors if that's the thing that it will take to get this thing moving.
[0:10:34] And AI for me, just you think about the impact, you know, back to that algorithm of how do you select the thing you want to work on? You think of the thing that has the biggest upside and just make sure that the downside is not too bad if it doesn't pan out. And AGI is just the highest upside technology you can imagine, right? It's just like, you know, and there's all sorts of debates that people love to have about whether it's going to happen, how long it's going to take, exactly how it's going to play out. But fundamentally, for me, I'm just like, this is still, I just...
[0:11:00] You know, after getting a little taste, I just can't imagine working on anything else. So, you know, I'm here six years in. I hope that, you know, in 60 years, I still work on the same stuff. Yeah, cool. Happy to hear that. I hope the next six years are as fruitful as the, or if not more fruitful than the previous six years. So kind of extrapolating this out to our viewers, would you say that passion is a key motivator, you know,
[0:11:26] How fruitful your work is actually going to be? If somebody wanted to start a startup, what traits would you say are necessary for somebody to grow something to a big scale? Yep. Well, I think that there's the things that people usually say, and I think I have maybe a different slight angle on them. And I think that for me, number one algorithm has always been just find, again,
[0:11:49] Problems that you just think are going to be high upside. Let yourself dream. Don't worry about the like, well, what if people don't like it? Or what if this goes wrong? Or there's a lot of regulation here. This sounds really hard. Just don't let yourself think those thoughts. Just get really excited about something. And then again, just make sure that then you let those thoughts come crashing in and make sure that you're like, you know what? Look, if that happens, it's not the end of the world. So I think that's definitely number one. Number two, I think is just always like, I think probably the people in this audience are pretty young. You've got a long career ahead of you.
[0:12:19] And that you've just got to realize that you're constantly investing in yourself and any of these experiences you get. I think it's actually true that you actually don't learn as much from failure as you'd like. But if you're able to just balance the amount of time that you spend failing, then actually you learn quite a lot. But I think that when it comes to the startup, for Stripe, I don't think that I grew up with a passion for payments. That was not a thing that I dreamed about as a child.
[0:12:46] But I got really excited about it while I was there and while I was working. And I still am. I think that it's actually a really impactful problem. And I think that maybe there is some sort of passion that's really important to keep you coming into work every day and to push that extra mile for having an impact, for actually moving the needle for users. I think one thing that people do get wrong is that they focus too much on the technology and too little on the impact of it. And I think that's a way to build something that's kind of fun technically, but I think it's not as
[0:13:15] If you think of the number of companies started on the basis of technological innovation, it's very, very small. And I think it's ironic maybe as someone who's in the AI technology business. But I think that AI is actually a little bit special, I think, in terms of how this calculus plays out. So let's dive into that a bit more. Could you highlight some differences between what it was like being in the early days of Stripe and being in the early days of OpenAI?
[0:13:42] Yeah, so for Stripe, you know, we did the classic startup playbook. And I guess part of why it's classic is because, you know, PG wrote a lot of essays on the things that we did. But there we really started with just like, what's a pain point, right? And actually, the funny thing is that...
[0:13:57] I think that just even discovering this pain point of payments was not an easy thing. When I joined, there were still some previous side project that Patrick and John had been working on. It was kind of there that it was going to be like, we're going to launch this app and that we're going to do payments for it to kind of show off the payments thing. But it just turned out no one wanted that thing and everyone wanted payments. And a lot of it was like, go out and find your first users.
[0:14:21] You know, talked to a friend who was doing some startup and we were like, hey, can we do your payments? He's like, sure. And so he integrated the API super quickly and charged some cards. He's like, great, how do I get my money? And we were like, huh, yeah, we should build that, huh? And recurring billing was, again, something where we set up the APIs for it. And we were like, great, we've got one month to implement the backend. You've got people billing on a monthly basis. The funny thing is we actually later learned that everyone wants to charge up front for their monthly.
[0:14:49] I was just going to ask, I think, what you were about to move on to. Yeah. OpenAI, we started with no customers.
[0:15:17] There was no one to ask questions to. Our customer is sort of the AI technology. It's almost this thing that we're building. It's sort of this breakthrough we're trying to create. And so I think it's the very big contrast. And we actually started OpenAI without even thinking of it as a company. We started as a nonprofit. I think we have a pretty ambitious mission to build AGI that benefits everyone. But we really didn't know exactly what that meant. And so I think we had to just even figure out
[0:15:45] What differentiated us? We started with a bunch of people who had just graduated from their PhDs, and I think everyone was very excited about doing things differently and felt like there were all these problems they'd run into academia or seen at other labs. But the question of what made us us, that was the kind of thing that we had to really spend our time on. Yeah, it really felt like, obviously, being in AI, I was keeping up to date with the Hide and Seek, the OpenAI 5, all the other cool stuff that you guys have been working on. And then after GPT, it really felt like...
[0:16:15] You guys had started to deviate away from a researcher's perspective, like innovating directly in AI models to building things which are more general artificial intelligence-like. So there was always this fundamental theme of building products which solve cool and interesting problems. And now I can, you know...
[0:16:35] Now we're starting to see things which are definitely commercially applicable because I don't see how hide and seek could have actually been played out, how somebody could use your hide and seek model, if you know what I mean. Yes. So just on this point, and I know this is slightly controversial, one of OpenAI's original philosophies was to release the research that they actually make and have it available to the general public.
[0:16:58] Now, some of these more powerful models, such as DALI, you're not releasing. Sorry, the lights turn off. I'll turn them on in a second. You're not actually releasing the model waits for. Have you deviated away from this original mission statement or kind of how do you balance that contrast between, hey, we want to democratize some stuff, but still some stuff might be potentially too dangerous and dangerous how, if so? Yeah, I think it's a very important question. I think that a lot of people have very strongly held opinions about what we should be doing.
[0:17:28] Which I actually think is great because, you know, I think that the fundamental theme is the mission, like if you read that first sentence of our initial launch blog post, you know, that we say that we want to advance digital intelligence in a way most likely to benefit all of humanity, unconstrained by a need to generate financial return. And I actually think that, you know, that question for us, right, how do we actually advance AI, like for real, like actually build it, get it rolled out into society, make it be something that's beneficial for humans.
[0:17:58] Like that question, like we shouldn't have expected in 2016 that we knew all the mechanisms. And I think that there's like a lot in that blog post that we got wrong. I think that, you know, specifically like, you know, open source everything. I think that that strategy turns out to be something that was just like not actually the right mechanism. And, you know, you can really see it with, you know, GPT-2 was definitely the first time we were like, oh, we've got something. And like, you know, this model itself may not be the.
[0:18:24] The thing that's going to be the most impactful, but it's so clear. We're going to scale it up over upcoming years. It's going to get better and better, and it's going to be able to be used for things that are not the kinds of things that advance the OpenAI mission. And so what do you do? How do you handle that? I think we had a real sort of moment of like, well, there's no playbook for this. Yeah, really the first time doing something at that scale and releasing it to other people.
[0:18:50] The open source thing is kind of interesting because Hugging Face and Segment both, well, I guess Hugging Face still is open source, but Segment as well started as an open source project and they found traction through this open source stuff, which eventually led to monetization. Furthermore, technical companies out there, how much do you rate this as a strategy to build a startup where you build a following through open source and then release it or try to monetize it?
[0:19:16] Yeah, no, I mean, I definitely think the kind of like open plus model is something that's pretty well trodden at this point. You know, there's like the, you know, the MongoDBs of the world are even a good example there. And I think it's actually very exciting because, you know, a decade ago, 20 years ago, like this kind of thing would just be impossible. And I think it's just like...
[0:19:35] You know, sort of if you think about just how to get free just distribution. I mean, they're literally you just end up in all these companies and all these users who actually care and that you kind of just get to focus on building your project. So I think it's actually like a really great strategy to leverage the community. There's a lot of smart people out there who are going to think of things to do with it that you would never think of.
[0:19:55] And if you even look at where we've ended up, with GPT-2, we were just like, well, we're not really sure what to do. We'll release the small models. We'll not release the big ones. And this caused a big hoopla because everyone's like, oh, too dangerous to release, which was not a statement we actually said. It was a headline that some editor wrote. And then everyone's like, OpenAI, why are you saying that? Yeah, where's the open gone? Yeah. Exactly. But where we ended up was...
[0:20:18] So we actually still get that developer base of people who are just probing these models in ways that we would just never think of using them for applications we would never consider. For example, there's people who are using GPT-3 to pour through tax documents to find tax deductions for you. Again, that's not a problem that I ever would have considered building an AI for, but there's someone out there who's like, this thing you built actually is a good match for this problem. And I think that that across...
[0:20:46] You know, I think that the API approach is great. I think that, you know, open source and just let people, you know, sort of get in the innards of your technology and modify it for you is great. And just these days, like there's just so much sort of, you know, like you basically end up with these companies who have a lot of capital to pay you and that they're like, well, we could pay a bunch of engineers to try to figure this thing out, or we could just, you know, pay a support contract. And it's actually just like economically a win-win for them to do it. Yeah, I think it's a fantastic business model as well. I'm a big user of GPT and they're,
[0:21:15] After using it, I've definitely seen the advantages that even other AI startups can have by generating data through GPT or something along those lines. Sorry, just to make this more about startups than the actual technology itself. Obviously, people here who know of OpenAI, they know that you've had absolutely huge names backing the funding of the company when it first started. Now, a lot of newer AI startups really don't have the same kind of opportunity or access to talent.
[0:21:45] That you guys obviously had. What would you recommend to these startups who also have big visions, but don't have that same kind of advantage that OpenAI has had? Yep. I mean, I think you always got to start in the same place. You've got to try to be building something that people want. And I think that, you know, if you are starting from a place with less resources and fewer people, like you still got yourself and you've still got a laptop and that actually with that, you actually have just like a pretty huge amount of opportunity relative to.
[0:22:14] Where anyone was, you know, 10 years ago. And so I think that that starting point is already pretty good. And, you know, they think that the question of like, you know, I think that, you know, for us, like, you know, maybe it looks like we just like started out in some, you know, sort of, you know, king made position or something like that. But I think for each of us, like, you know, I was a college dropout, you know, just a couple years prior when I went to go work on Stripe. And so I think that a lot of these are that it is possible in, you know, today's
[0:22:43] Very internet first age to pull yourself up by your bootstraps. And I think that there is just like a lot of opportunity that is available. And that I think that, you know, there's like one thing that I found very surprising about when I came out to Silicon Valley was the degree to which it felt like there was this like sort of there was this network out here of people who are just willing to help.
[0:23:04] And, you know, I think that it does like, that there is like some barrier to entry. It's not all just like, you know, you just show up in Silicon Valley and everyone's like knocking on your door to give you advice. I think it's just very much like you, you know, you get like one person to kind of believe in you and then they introduce you to more people or, you know, that kind of thing. But, you know, I know lots of people who got started by just going to meetups. They didn't know anyone and now are building some of the most interesting startups that I know. And so I think that that approach of just like, there's like a friendly ecosystem. I mean, honestly, I know that...
[0:23:34] Maybe this is controversial today, but I still believe like come out to Silicon Valley, like even if you can't live here, just like spend time here and get to know people. Because I think this is still just like the biggest concentration of tech talent people who have done it before that exists in the world. Yeah, and I remember watching some of the Blitzscaling videos and they make this point about Silicon Valley as well. The network effect is, you can't find that kind of network effect in any other countries yet. And there's already this established culture and ecosystem of that there as well.
[0:24:02] I think that's some good advice. But to kind of almost flip that question on its head, with these AI startups, I know you're an angel investor, but do you see these, when you take a look at startups, what common pitfalls do you see these early stage startups actually making? Yeah, I mean, I think that it's very easy to get too focused. You know, we talk a lot about technology versus users, and I think it is very easy to get too focused on the tech and to just not.
[0:24:28] I think that maybe one thing in AI startups in particular is that it's really easy to not actually know what your system's doing. You had some model, you trained it, you see your loss, you see what the metrics say, and then you're like, great, I have a great model, and then you roll it out, and it turns out that you just totally broke some use case for users, or there's some bias in there that's causing problems for some subset of users. I think that the most important thing is really having your finger on the pulse. A lot of the things that we did in the early days of Stripe, which were actually...
[0:24:55] Ported over when we released our API is just that very tight loop with customers. And there's very simple tactics you can do. Like, you know, at this point, most people are in Slack. So it's actually very easy to get shared Slack channels set up with your customers and your customers love it. And their whole team's in there and you love it because you can just, you know, ask them how it's going or whenever there's a bug, it's not gonna be like a huge barrier to entry. Like, you know, no one has to file tickets. Like it's just like super, super fluid. So I think that these kinds of things where it's like,
[0:25:24] Just break the, you know, don't treat this whole thing as a waterfall process. Just treat it as like your goal is to really understand how your users are being impacted and to be just like, you know, I think the thing that matters most in startups is definitely the OODA loop, you know, the ability to orient and to respond to new information. And I think that there's a lot of people who I think just kind of, you know, again, just want to focus too much on just themselves.
[0:25:51] What they're doing and not really worry about actually like how it's impacting their users. Yeah, it's a really good advice, actually. And like as an investor, is this one of the key points that you look for when investing in a company? Like how tightly coupled is the other founders or the executives to users? And also about what else what else would you also look for in the companies? Yeah, definitely. I would say I think one of my favorite investments is I these I this.
[0:26:17] Pair of founders who operate a company called Wave that does payments in Africa. And that I can't really think of any words to describe them besides force of nature. And really what they do is that they just went out and they figured out for them that they have to go and build these relationships with these banks across various African countries. There's lots of different regulatory regimes and lots of different just sort of understanding.
[0:26:44] People in these villages and the different cities that they're operating in who are just very, very diverse and very different kind of sort of desires and use cases. And I think that they're both from the US and that they just moved out there, right? Kind of a contrast to a lot of people I've seen in the Bitcoin world who are saying that we want to get payments into various, various different countries and then just try to do it from San Francisco. And so I think that that just like you got to really.
[0:27:14] Put your sort of life and your work where your goal is and to really think about what is it going to take for me to really understand what people need. So I think that that kind of approach is something I really respect and admire. So yeah, to put a word in there, I'd probably call that like the hustle of the founders, like how much work are they willing to put in and get their hands dirty, basically.
[0:27:38] There is one thing that I think is important. I think Hustle, for me, it's got two kind of meanings to it. There's the put the work in, like you said, but there's also the cut the corners. And I think that cutting the corners is actually just a mistake. This is something that we saw, too, in the early days of Stripe, that we had these competitors who just would sort of bypass the regulatory regime. There's some rules about how you have to identify your users.
[0:28:06] One of our competitors just didn't require it. We did. And we were like, we're building a worse product, right? Because they're circumventing all the rules. And that it took a real amount of like self-restraint and sort of reflection to not follow them in a race to the bottom there. And at the end of the day, they failed and the stripe has not yet failed. And I think that that just shows you that like the culture that you bake in early, you can't undo it.
[0:28:30] That at some point, you're going to have to pay the piper. And so I think that the hustle, you've got to make sure that you are on the right side of what that word can mean. Yeah. So just on that point, it also seems like when you are manually faxing over things and doing a lot of things by hand in the early days, that could also be, by some definition, cutting the corners in the sense that you don't have, like, to a user, you show, hey, look, here's this automated system, which is actually going on.
[0:28:54] On the back end, you guys are just doing it by hand. So I think that's an effective use of cutting the corners. But obviously, don't break rules and don't break laws and regulatory issues. So being close to the user is obviously one thing. Is there potentially anything else that you would look for, characteristics of the founders-wise?
[0:29:20] I was going to say, I mean, I think for me personally, you know, so I definitely do angel investing most as a hobby, and I usually tend to look for people who I think can use the support. I just think doing a startup is really hard, and that I like to be able to figure out how to just, you know, sort of enable more people to do it successfully. And, you know, these days I think that I try to be cast a fairly wide net when it comes to the actual technology, but I usually like to just have...
[0:29:45] Technology that I am excited about or that I'm like, yeah, I just get it. There's some companies that I just feel like they're for someone else, they're not for me. And not even in the sense that I would be a user, but just in the sense that I understand it. And that there's one company that I'm really excited about is a fertility benefits company that I'm just like, I just kind of see how much just overhead there is and how many people are just lost by the system or just not well served by it. So I think that just doing
[0:30:14] Sort of, you know, solving problems that can help people, maybe is the simplest way to put it. Okay, yeah, cool. So philanthropic in nature, I guess. Well, that one's not philanthropic. That one is like, they're going to make a lot of money. Also help people at the same time. Exactly, yeah. And that's the great thing about startups is that, you know, that I think that you can think like, there's a lot of things that people don't like about capitalism, but I think that fundamentally the goal is to align the creation of...
[0:30:43] You only get it if you create value. And so you really got to focus on something that's really, really useful and that people want. Cool. So speaking about technology, AI has been dubbed as one of the five disruptive technologies alongside blockchain, 3D printing, XR, which you touched on briefly, and IoT as well. Is AI the one which excites you the most? And why is that? And also, which of these other technologies do you have a personal interest in? Yeah.
[0:31:13] Well, I just think it's funny because it's all acronyms, all buzzwords, which is always a thing to set up everyone's red flags. And for sure, I think that AI is the technology that most excites me. I mean, like I said, putting my time where my mouth is here, this is what I've been working on for six years and hope to work on for a very long time. And I think that AI, for me, the reason that I got excited about initially is the idea.
[0:31:43] of a computer being able to solve problems that I can't. Like that was just mind blowing, right? You know, you think about programming where you just think super hard about a problem and you write down all the rules in this very literal way and the computer does it and like, you know, it's pretty cool, right? You can build a website, you can build a game, you can build these sort of, you know, all these like objects that existed in your head.
[0:32:02] But the idea of like, that the object didn't even exist in my head, I just kind of had a vague idea of like, that's the goal that I want. And you know, there's problems that humans aren't going to be able to solve. Like, I don't think we're on very good trajectory right now on climate change. I don't think that we're on very good trajectory for giving everyone like super low cost healthcare. I think universal education is also just like, you know, there's too few teachers and like, you know, that we have a demographic time bomb coming up where it's just like that we're gonna have this inversion of like the number of people working versus number of people retired. All of these things are just like, yeah, these are not good problems.
[0:32:32] Do you ever come across people who say, oh, so what you're doing is going to replace my job and I'm not going to be able to work and earn a living?
[0:32:58] Yeah, I think this part is actually very interesting because, of course, this is, you know, when people, maybe remind a couple of years that whenever I talk to people about AI or AGI that, yeah, immediately it's going to take my job. Is it going to be a Terminator? And that's kind of what you get, right? And I think that...
[0:33:14] First of all, we're seeing this is just not how the technology is playing out. I think that there's so much vision that people had for decades about what a smart machine would be that actually just doesn't comport with the technology we're building. You look at like Codex or Copilot. You know, I think there's probably a lot of people in this audience who are users or played with GitHub Copilot, which is powered by our models. And, you know, that's a pair programmer that sits in your editor. You're typing things and it just suggests.
[0:33:38] What you might want to type next. And it saves people a lot of time. It saves people time on boilerplate. It saves them time on refactors. It saves them time on, here's a library you're using that you're not really familiar with, and it just knows how to use it. And it's contextual. It really understands what's going on in your program, and it does the right stuff, and it gives you good variable names. I don't think that anyone looks at that and says, ah, programming's over. It's just what a machine does. I have actually met one or two people who had not liked it.
[0:34:04] I think what you just said is actually very important, right? It's like we lower the barriers and I'm like, man, like the more you can lower the barriers to entry for developers, the better, right? That's just going to create so much opportunity for people that you think about how much useful stuff has been created.
[0:34:30] Through programming to date. And if you could just 10x that, like there's so much more demand for programmers than we can fill. So there's that whole arc. But I think that basically what's going on is that the codex effect, the copilot effect.
[0:34:39] It's happening across a lot of different industries. And it's like that the water level is rising of what these AIs can do, but they can't do a full job. Because if you think about what a full job is, it's doing what we're doing right now. It's like sitting down with users and talking to them and really understanding deeply what's going on. And like, you know, it's not about like the, you know, you do got to type on a keyboard at some point, but that's like 1% of the overall picture. I think even like, you know, people think about building like fast food restaurant bots. And the thing is like, you got to...
[0:35:04] Work with customers there too, right? It's interesting you mentioned that because when I first got access to GPT, I don't even think the Instruct series had been released yet. I was pretending that GPT was a business investor and I was asking me questions. I was trying to have a dialogue back and forth, seeing what kind of questions would actually come out. But surely this is a matter of time where, and speaking of time.
[0:35:26] Like, where do you see this technology affecting society within the next five to ten years? Because obviously we've seen huge, huge shifts since Transformers were released about five years ago. And, you know, technological evolution is exponential. So what do you kind of see happening in the next five to ten years in society with respect to technology? Yeah, well, first of all, I think anyone who has great certainty about what the future looks like is either deluded or lying. I think that, you know, rewind just a couple of years and.
[0:35:54] If anyone could predict the events of 2020 and 2021 and now 2022, I think that they would have, I have no idea how anyone could have done that. So I think that the same is kind of true for how this technology will play out. But there are some things we can say. Like, I think that we know that, you know, that vision we had when we built GPT-2 of we're just going to make this better and better. That's still true. And that's not going to peter out anytime soon. So, you know, GPT-3, like the funny thing is, I think it's just not a very good model. I think that I'm like, yeah, like, you know.
[0:36:24] I'm very happy that it's useful for a lot of people, but we know how to make this way, way better, and that's what we're working on, and that's what's going to happen. And so you think about what happens as you get to your GPT-7, right, that can, you know, is, like, actually really smart. You can ask it for problems, like, you can have this thing sitting over your shoulder as you're, like, trying to learn something, and it asks, you know, it just presents you exactly the right kinds of Socratic questions at the right time for you to, like, really absorb material. You can talk to it and it understands what you're doing.
[0:36:50] Where your gaps are when you're trying to just go about your day. So I think that there's this kind of deep integration that's going to be possible with these systems that you can kind of see already today. And so I think it's just almost like, think about smartphones, right? When those first came out, they were actually kind of maybe not that good. Or maybe your palm paddle is not very good. And now you can't live without it.
[0:37:15] And I think that we're going to be somewhere on that transition and exactly where and exactly how deep and exactly which fields. I think that is still just up for debate. Yeah, it's interesting because back when they came out, sorry, biggest pain of having a meeting in the office. Yeah, back when they came out, they were a game changer.
[0:37:35] That day's technology, they were incredibly good. And I would say the same thing is there with GPT-3. As somebody in AI, I do notice some of the pitfalls of GPT. But yeah, I can definitely imagine with number four, number five, you'll slowly start to address these and the technology will improve and eventually open up to potentially regular users or regular people as well who aren't doing research.
[0:37:57] So last question from me before I ask one of the many questions that the audience has passed through. So thank you for that. Please do keep them coming. Which five companies at whatever stage, but please do mention what stage they are, irregardless of their technology, are you a big fan of and interested in and why is that? Yeah. So I think I'd say, look, there's some obvious ones. Like I think that Tesla and SpaceX, which I'm not sure exactly what stage to say those are in, early stage in some ways still.
[0:38:27] I think are really inspiring. I think that the fact of just doing things in physical space, I think is just something I've just always had a little bit of inferiority complex about. All I do is arrange text on a screen. Cars and rockets, come on, what are we doing with our lives? But I also think that maybe to get to ones that I think maybe are a little bit more unique, I think that Quizlet is actually an example of a company that's just a really quiet success that
[0:38:56] It's just used by a huge fraction of students out there. And they're online flashcards. I don't know whether... Small for the UK, I've never heard of Quizlet. Yeah, it's very big in the US. It's something like 50% of US high school students use it. But it's just flashcards. It's like you can do online flashcards and then everyone's used it. So there's flashcards for any subject you want. And for me, it's just such an empowering tool. It's just like, you can now just learn better.
[0:39:26] And I just really like that, you know, it started as a dorm room project, wasn't intended to be a company, and it became a very, very, very, very nice, successful business. Coinbase, I think, is also an interesting one. Like, I'm actually a little bit of a, I don't know whether a crypto skeptic is the right term, but, you know, it's something I dug very deeply into while at Stripe and kind of concluded it just wasn't for us. Despite then helping launch a cryptocurrency while at Stripe. Exactly right.
[0:39:55] Yes. But Coinbase, I think, is just, again, interesting because it's just they had a conviction on this technology, right? It's a new thing that's coming and they're just like, this is something we want everyone to have and we're going to support it. We're going to integrate deeply with it and make it accessible to people. And so I just really admire that kind of vision and approach. I think the vision was an important part of why Coinbase is a big success today because we had GDAX back in the day. No, not GDAX, sorry, that's Coinbase.
[0:40:20] I can't remember. MTGOX or something like that. Yeah, MTGOX. Yeah, yeah, yeah. And that was very prominent before Coinbase. And I think that was just ended up the founders of it where they just took money and ran. So it was really good for Coinbase to stick around, believe that crypto would actually be a thing and carry on working towards that. Yep, yep. Okay, cool. So let's open it up to the audience. I don't know how Chris managed to get...
[0:40:49] These to display on the screen. So I would just read them out. So Linus asks, what is your view on ML in combination with robotics? And is OpenAI still actively doing research in this space? Yeah, so we had a robotics team that we put a lot of effort into getting off the ground. I remember when we looked at each other, we were like, you know, who would say that OpenAI has the world's best?
[0:41:12] Or sorry, the world's best robotics research. No one. I mean, come on. And so we were really starting from scratch there. And we ended up actually getting some results that just the field has been trying to get for decades unsuccessfully. And we got it. For example, being able to control dexterous manipulation with a robot hand. So we have a video of we used reinforcement learning to train a robotic hand to manipulate a Rubik's Cube, which is super cool. You can just watch it actually do the solve. And then we disbanded the team.
[0:41:41] And really what happened there is we realized that just the virtual space is moving so much faster than the physical one, that the robotic space, part of the problem is the hardware. It's somewhat a software problem, but it's really hard to get much data off your robots. That robot hand would run for like 20 hours and then break because it'd been around for two decades, but no one was able to use it. So it was still just like a research prototype almost. And so I think that that is kind of where things are at is that the hardware is actually a big bottleneck.
[0:42:10] And that you can work around this to some extent by having a simulator and running things in simulation. But there's some amount of like, well, why take that pain if you could just download the internet and use that? That being said, I think that robotics is going to end up being super impactful. There's going to be a lot of it. And I think that the technology, like the software side is basically ready at this point. Like we have so many great techniques, but they're not hitting, you don't have your robot Butler walking around just yet because no one has the software, no one has the hardware for it.
[0:42:39] So as a follow-up to that, is OpenAI going to go back into robotics in the near future? Is this on the roadmap or are you leaving that up to, well, obviously you don't know the future. I think, yeah, probably not going to thrash back anytime soon, but I mean, I think that we always update with new information. So maybe if someone comes out with a super great physical robot that we would be like, oh, we can write software for that. Cool. There are a couple of questions here about risks and ethical considerations. I think they're all from slightly different angles. So I'll try to,
[0:43:09] Phrase these questions just so when I say them, I can get to the points that I think you're making. So Max asks, what is OpenAI doing to address the risks of AGI? And Fisayo asks a similar question, which is how often do ethical considerations come up with the AI industry as a whole and with OpenAI specifically? Yeah, so I think that today ethical considerations come up all the time, right? That you're rolling out these systems that...
[0:43:38] Sort of have an encoding of society in them. And that means you come with society's biases if you're not careful. And there's a question of, well, what do you want the model to say? Like, what is acceptable speech? And some of these things, you know, we don't have great answers for as a society.
[0:43:54] And we're sort of forced to encounter them. And there's some things that everyone can agree, just like the model shouldn't say, like various offensive or very biased things. And those ones are still things that the technology, we're just starting to figure out how to really work through them. And that we found that you can basically take feedback from humans and feed it back into the model. And then the model really starts to respect that. And that's something that I think has really gone a long way. But I think that what we're going to see is that the...
[0:44:21] The stakes that we're encountering are going to be just increasing over time. Today, it's kind of thinking about this bias speech coming out of the model that once you start really rolling out our code generation models, you got to start thinking about, well, you're going to be running some code that just came out of the system. You probably wouldn't run code that just came out of some arbitrary human. You probably want to make sure that it doesn't do anything malicious or buggy or cause downstream problems. And then once you have systems that are more integrated into running
[0:44:49] Cities and power grids and talking to lots of people across the internet, then you really are going to start having to think about society-wide effects. And so I think that we are on this ramp of increasing technology capability, but it comes with increasing technological risk. And figuring out how to mitigate that, I think, is the core goal of OpenAI. We think that we can build great technology that will benefit lots of people, but we also think it is incumbent upon us.
[0:45:15] It's not something we can push off to someone else to make sure that those risks are mitigated. And so part of what we do is release through an API so that we actually see everything that happens, that we have the ability to set the rules of the road, that we can kick people off the platform, all sorts of things that back when the vision was, let's just open source it and let people do what they will, that that was something that would have never been in our power.
[0:45:36] Yeah, I know you've actually done this personally because I have a friend here who works on a startup which is detecting hate speech. So he was using GPT to actually generate hate speech and then he got an email saying, you're generating too much hateful stuff, so please be careful. Yes, yes, yep, yep. So George kind of asks a similar question about the security aspects of this. So I'm just going to enhance this question a little bit after reading out his question. So I see there are security concerns with AI. What is your opinion of this and what impact would this make?
[0:46:03] So I think there are two aspects of this. One is AI being like self-learning and being a danger to itself when it's integrated into more dangerous systems. But I think the other interesting aspect is also how malicious actors can use this AI. So yeah, if you could just speak about that. Yeah. Well, I think that to this first question of we're going to have these like increasingly powerful and responsible systems and what if they go wrong? You know, that is a great example of this, I think. Yes. Yep. And I think that
[0:46:33] You know, this is a question that, you know, in the 90s, you know, people started really thinking hard about and realized that it just sounds like a really hard problem. And like, how do you make sure that the AI is aligned with, you know, even knows what humans want and that kind of thing? And can we even write down the like value function of humanity and those kinds of problems? And I think that basically like what we're finding is that, first of all, you know, today we're already encountering these problems and they're going to get harder and harder over time, but we don't have to encounter the full version yet.
[0:46:59] And so I think that one answer is just try to make the deployment be as continuous as possible. You actually want to see the failures. And this was one very deliberate choice with GPT-3. We had a choice of how narrow to make the rules of the road versus just how much to just encounter different use cases. And the strategy we ended up taking was, let's just encounter a wide variety of different stuff and just learn. And just make sure that the scale of it happening is just not too big. And then actually, you get so much value back that it just is totally worth it.
[0:47:25] And so I think this strategy is actually pretty good for us as a society, is just sort of lean into the fact that this is difficult and that there are going to be problems, and then just make sure that we're being very responsive and continuing to have that loop. All the things I was just talking about that startups should be doing. Yeah, I think it's quite different from text on the screen to when you're actually interacting with physical cities and, I don't know, controlling traffic lights, for example, where you can't just build these, you can't just deploy these systems. I mean, maybe you can build a whole city or have a city dedicated for research, but, you know, it's very hard to, once something is in production,
[0:47:55] Take that back after it's already done something negative. Yep. But I mean, I think that we're in this world already with, say, self-driving cars, right? I think that you can't have the self-driving robo-taxi from day one. You can't just be like, nothing. And then it's just like, OK, who wants to try this thing first? And that that is not just a danger to the people in it. It's a danger to people around it. And so I think that you're starting to see this years-long.
[0:48:20] Rollout process that's been happening, right? Validation and sort of people starting to get a sense of what they can do. And, you know, at this point, Cruise is starting to operate fully driverless taxis in San Francisco. But before then it was driverless taxis where there's a person sitting behind the wheel in case something goes wrong. And before then, you know, that it was just like, you know, out in the stretches of desert or, you know, in abandoned places. And so I think that the same thing has got to be true for any of these systems we deploy is like, you don't want the city management robot to just appear one day. I think that you want to have it be a lot smaller.
[0:48:48] Smaller things. And if you can't do that, then I think that you do just have an intractable problem. So last question directly related to the ethics. So not quite Terminator or Skynet, but along similar lines, what happens if state actors had access to AI? Yep.
[0:49:07] Well, I think that today, I mean, you know, if you, you know, deep fakes are already starting to kind of creep a little bit into this zone of being used for political purposes. And, you know, the nice thing about where AI is today is like, despite how great it's going to be and how impactful it's going to be, I think that just like right now, it's just not very good. And that, you know, get into this whole incremental rollout and that, you know, that there's always a game of cat and mouse where the...
[0:49:34] The offense gets a new tool and then the defenders have to figure out how to mitigate that tool. I think that's all starting to happen, but this is not the like, you know, sort of frontline, like, you know, sort of main tool in the arsenal of any country yet. Just simply because I think that, you know, that the models are just not capable of it yet. But that will change. And I think that the important thing is that, you know, I think it's just same answer as before. So I think it's just important for society that this is all as incremental as possible. The big shocks to the system are the things that I think can cause.
[0:50:03] Real disruption and sort of lots of those negative effects that just come unchecked to the surface. Yeah, good to stay up to date with the times and see exactly how the current state is affecting it and where that can go in the future. Yes. Tej asks a two-part question. The first is, can you talk about the Foundation API model as a service? You guys have released GPT-3 and Codex. Are you planning to release DALI and other Foundation models?
[0:50:29] Yep. So definitely, definitely we are. You should expect to see more offerings coming from us. And, you know, I think we're kind of in a world now, which is super exciting where you train a model and it's actually useful to people. Like, you know, at the beginning of the deep learning revolution, everyone was like, we're going to build computer vision APIs. And it just turned out no one wants a computer vision API. Certainly not a, you know, computer vision API that can classify.
[0:50:52] Like the models that 2012 could. But it turns out that people actually do want a GPT-3. And I think that people do want a DALI. And people do want these other models that are coming simply because they can solve problems in a pretty broad general way that you just can't do otherwise. So we're really excited to keep releasing this stuff and let people build on top of it. Yeah, obviously the advantages that OpenAI models have is that they're trained on so much data that they can handle out of distribution stuff, which are regular.
[0:51:19] So the second part of the question is touching on the unit economics of the foundation models. And how do you think about energy costs and energy consumption when you're running these extremely large scale models? Yep. I mean, I think basically the thing we've been finding is just that people do just want the big models and that, you know, I think everyone's afraid that like, okay, well, what about the GPU costs and things like that? But just that the economic return per flop invested,
[0:51:49] It's just totally there. And so it actually, I think, is the case that, yeah, these foundation models are just worth it and that people want them. Energy cost, I think, is a very misunderstood thing in this field. I think that it's definitely the case that we care a lot about doing our part on keeping emissions low. And basically, every cloud is carbon neutral.
[0:52:12] We run in Microsoft's data centers, and they're on trajectory to be totally 100% on renewables, I think, this decade. And if you look at the total amount of energy output from all of the AI models, it's a tiny sliver of global energy usage. And so I think that actually the ROI for society
[0:52:33] On what we're doing is actually totally there. And, you know, I think that again, it's important that, you know, we're always trying to think about how to make things more efficient. And partly because that gives you better performance, but it also, you know, sort of gives you just better performance per energy or per dollar invested. But I think that overall, we should expect these technologies at some point to be huge and massive and deployed everywhere and to be using lots and lots of the world's resources. But it should only do that if it's actually generating far more value than it's consuming.
[0:53:02] That's kind of interesting because I was thinking like the next big thing in AI research is going to be about how to make models smaller. Now that we're validated that models can actually learn to the capabilities of what GPT and DALI can actually do, how we can make them so they can maybe train on a single GPU instead of on, it was either half a million or $12 million. I can't remember how much it cost. Right, right, right. I think that's super valuable too because people are going to want those models as well. But I think it's just like this whole direction is just so...
[0:53:30] You know, can handle so many different use cases that I think you're going to want models of every shape and size that can plug into just, you know, you want something on your phone, right? Like, you know, there are models running on your phone right now. It would be great if those were way smarter. But you also want a model that you can go to and ask, like, you know, hey, can you help me solve this? Like, you know, help me design a cure for this disease and that it just like goes off and thinks and comes back with an answer. And that one, I think you don't care if it's, you know, a one gigabit model or a giant one.
[0:53:58] So Hamza asks a couple of questions. I'll get back to that in a second. I do have one other interesting ethics question, which is by Dawson Walters. Is it ethical to use AI to invest in the stock market? For example, if algorithms investing in the tobacco industry, it might result as people becoming more unhealthy as a whole. Yeah. I mean, I think that today it is...
[0:54:24] Is the question specifically for, is it good for algorithmic trading to exist? Yes, I imagine that's the line of questioning. Well, I mean, it does exist and lots of people are doing it. And so, you know, whether or not we like it may be a different story. I definitely think that like the question of, okay, so is it good to pursue things that can just return lots of money without regard to externalities? I think that is just like definitely not good.
[0:54:50] And I think one thing that's really important as AI is being deployed and impacting the world is that you're not just saying like, ah, here's a value function of just make lots of money and play out. But I don't think that's unique to AI, right? I think that you can almost think of AI as like, you know, I think the best analogy for what an AGI might be is a company or a country, right? You know, something that's like bigger than any individual and pursues goals, you know, sort of almost autonomously. And if unchecked can do sort of things that are not very aligned with human values.
[0:55:19] And I think that one thing that's really interesting about companies is they are made up of humans and sometimes do really terrible things. And figuring out how to make sure that you're just constantly putting checks and balances on the system in order to make sure that it is the case that there's lots of regulation on tobacco companies that prevent them from being too successful. Because we know that, yes, they can make lots of money and capitalism is supposed to correct for this, but it actually doesn't because it almost hacks the reward model.
[0:55:43] Yeah, I think that's a really interesting point, just in the context of if you were to train a large AI model on human history, you'd see that humans cause war most of the time. And obviously, this is definitely not something that we want to embark into our AGI models. So I do have a couple more questions about that, but I think that's a separate discussion. Regarding what Hamza has asked, so what data skills will be commoditized due to AI and become less important thanks to AI?
[0:56:12] And therefore, what are your views on the future of coding? Yeah, I mean, I think that basically the mechanical skills are just in general going to become just sort of much less, you know, sort of unique. I think it will be the case that you'll just have better and better tools. I mean, it's kind of like, you know, thinking about if you go from not having NumPy or Pandas to having NumPy and Pandas, like, you know, maybe parsing big CSV files or something and, you know, figuring out a format on structured data.
[0:56:39] Suddenly stops being this like super unique skill. But man, that was not something that anyone really, you know, no one as a child is like, I want to like format spreadsheets. And I think that that is kind of where we're going. So I think that what is going to be valuable, increasingly valuable is vision, right? Is just thinking about what do you want the system to really do? That having an idea for, okay, like.
[0:56:59] What is a data source that you even want to go and process and parse and put into whatever model you're in? And these kinds of executive decisions and just really thinking through the problem, understanding the space very deeply. This is where our models struggle. But this is also the thing that humans love. And so it's actually a very good match. Cool. And what would you recommend? So I know you went through this journey very recently, but what would you recommend to people who wanted to learn AI if they were starting today? Yep.
[0:57:28] I think number one thing is just start trying to build stuff. So for me, I usually get started by just read a book and then start trying to build something. So that's how I started learning to program. That's how I got to AI. So I started to do Kaggle competitions. And you can actually find my repo. It's funny. I think maybe my most starred GitHub repo where I just kind of like Kaggle solutions and just kind of talking a little bit about.
[0:57:54] How I was solving the problems. It was really just me just like trying to explore and get a sense of what this thing can do. I think it's just like that. Get your hands dirty. There's so many papers. Reimplementing things out of papers is also a really, really great way to get started. But yeah, I think it is just like, you know, the stuff is very accessible now. There's so much that's out there. And so just get started. So that's an interesting point, which I think we can generalize a little bit. So as a CTO, you came into OpenAI, which is fundamentally an AI company.
[0:58:23] As somebody who didn't have the technical expertise to understand the core algorithms that were going on at some point in time, was that a challenge for you to communicate with some of your AI engineers? And once you learned AI, what kind of opportunities opened up now that you did understand their language?
[0:58:44] You know, honestly, I thought when we started OpenAI that my technical skills would not be very useful. Like, I thought that I was just going to basically just sort of pave the way for the researchers to do their research. And that's what I thought my job would be. And so, you know, day one, people were like debating something and they wanted to write something up on a whiteboard. We didn't have a whiteboard. I was like, I'll go get the whiteboard. And I was excited about that. That's what I thought was the best way to contribute. But it was really only a few weeks in that we started to build OpenAI Gym, which is a piece of software that was the first thing we released.
[0:59:14] Actually turns out to be just like a pretty traditional engineering problem. And a lot of how I approached it was by just like, you know, you talk to your user and they tell you kind of what they want and you go and you try to implement it. But it was very difficult because I definitely didn't understand a lot of the design considerations and sort of the domain very deeply. And so what I would usually do is I'd like come up with five different ways of doing something. And I'd show this to one of our researchers and he'd be like, these four are bad.
[0:59:36] And I'd be like, great, that's exactly what I wanted, right? So you need to have this real humility, right? This real sense of like, look, I'm not the domain expert here. I'm going to do my best, but like, I just need to be so responsive to what my customer is telling me. And that algorithm, I think, worked really well. So if you were to rebuild Jim today, would you have needed to interact with customers as much as you did?
[0:59:58] Definitely not nearly as much. I mean, you know, if you're building something that you do want to be used by people, like, you know, if you can use it yourself, that's actually really, really good. And that if you can be that customer, but you definitely need to have some customer that is your touch point. And I think that a lot of how I build stuff is that I usually try to get the bare minimum thing that could possibly work to be used by one person. And that person can be me, but sometimes it's someone else. And, you know, even today, it's still sometimes like, I'm just, you know, building some.
[1:00:25] Scoreboard or something for someone else's consumption. And then it's just like, you got to get it up as soon as possible. Cool. I think this brings us to 5.15. So the end of our conversation, which is a shame. I've definitely enjoyed talking to you, Greg. And I hope the audience has enjoyed tuning in and listening to this talk as well. So Greg, thank you very much for joining us. And I wish you the best of success for both yourself and OpenAI in the future. Thank you. Really appreciate it. No problem. Bye.