{"url": "https://anchor.fm/s/8c1524bc/podcast/play/49445511/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2022-2-22%2F255295552-44100-2-a29439ec62cd8376.mp3", "page_url": "https://podcasters.spotify.com/pod/show/ycombinator/episodes/45---Building-Dota-Bots-That-Beat-Pros---OpenAIs-Greg-Brockman--Szymon-Sidor--and-Sam-Altman-e1g3f67", "title": "#45 - Building Dota Bots That Beat Pros - OpenAI's Greg Brockman, Szymon Sidor, and Sam Altman", "show_or_channel": "Y Combinator Startup Podcast", "date": "2017-11-08", "duration_s": 3637, "type": "podcast", "speakers": ["Sam Altman", "Greg Brockman", "Szymon Sidor"], "confidence": "high", "notes": "OpenAI Five / Dota. YouTube: https://www.youtube.com/watch?v=UdIPveR__jw", "people": ["sam-altman"], "id": "2017-11-08_y-combinator-startup-podc-45-building-dota-bots-that-beat-pros", "audio_duration_s": 3637.2898125, "worker": "jc-w07", "transcribe_s": 171.7, "model": "large-v3", "segments": [{"start": 0.0, "end": 23.97, "text": "Hey, this is Craig Cannon, and you're listening to Y Combinators podcast. Today's guests are Greg Brockman, Shimon Sador, and Sam Allman. Greg is the CTO and co-founder of OpenAI. Shimon is also at OpenAI. He's a research scientist there. And before we get going, if you haven't yet subscribed or reviewed the podcast yet, that would be awesome if you did. All right, here we go. Now, if you look forward to what's going to happen."}, {"start": 23.97, "end": 50.43, "text": "Over upcoming years is the hardware for these applications for running neural nets really, really quickly are going to get faster than people expect. And I think that what that's going to unlock is you're going to be able to scale up these models and you're going to see qualitatively different behaviors from what you've seen so far. At OpenAI, we see this sometimes. For example, we had a paper on this unsupervised learning where you train a language model."}, {"start": 50.43, "end": 78.03, "text": "You train a model to predict the next character in Amazon reviews. And just by learning to predict the next character in Amazon reviews, somehow it learned a state-of-the-art sentiment analysis classifier. And so it's kind of crazy if you think about it, right? You just were told, hey, predict the next character. If you were told to do this, well, the first thing you'd do is you'd learn the spelling of words and you'd learn punctuation. The next thing you do is you start to learn semantics, right, if you have extra capacity there. And that this effect goes away if you use a slightly smaller model."}, {"start": 78.03, "end": 107.28, "text": "And what happens if you have a slightly larger model? Well, we don't know because we can't run those models yet. But in upcoming years, we'll be able to. What do you guys think are the most promising underexplored areas in AI if we're trying to make it come faster? What should people be working on that they're not? Yeah, so there are many areas of AI that we already developed by quite a bit. There's some basic research in just classification deep learning and reinforcement learning."}, {"start": 107.28, "end": 135.54, "text": "And what people do is people kind of try to invent problems such as solving some complicated games of hierarchical structure and they try to add kind of extra features to their models to combat those problems. But I think there's very little research happening on actually understanding the existing methods and their limits. So for example, it was a..."}, {"start": 135.54, "end": 159.06, "text": "Long had believed in deep learning that kind of to parallelize your computation, you need to cram as small batches as possible on every device. And in fact, Baidu did this impressive engineering feat where they took recurrent neural networks and they implemented the..."}, {"start": 159.06, "end": 182.46, "text": "Kind of GPU assembly code to make sure that you can fit like batch size one RNNs on every GPU. And, you know, despite like all those smart people working on those problems, like only very recently did Facebook kind of took a cold, quiet look at just like very basic problem of classification."}, {"start": 182.46, "end": 210.99, "text": "Great paper called Image That In One Hour, they showed that if you actually take a code that does image classification and if you fix all the bugs, you actually can get away with much larger batch size and therefore finish the classification probably much faster. And, you know, it's not the kind of sexy research that people want to see where you have like some hierarchy of big RNN, but it actually, this kind of research, I think at this point will advance field the most."}, {"start": 210.99, "end": 239.94, "text": "So Greg, you mentioned hardware in your initial answer. In the near term, what are the actual innovations that you foresee happening? So the big change is that the kinds of computers that we've been trying to make really fast are general purpose computers that are built on the von Neumann architecture. You basically have a processor, you have a big memory, and you have some bottleneck between the two. With the applications that we're starting to do now, suddenly..."}, {"start": 239.94, "end": 263.74, "text": "You can start making use of massively parallel compute that the architectures that these models can run on sort of the fastest are going to look kind of like the brain where, you know, the brain is basically you have a bunch of neurons that all have their own memory right near to them and that they all talk to their neighbors and maybe there's some kind of longer range skip connections and that just no one's really had incentive to develop."}, {"start": 263.74, "end": 292.27, "text": "And so what we've seen is that, well, you move your neural networks from running on a CPU to a GPU, and now suddenly you have 1,000 CUDA cores running in parallel, and that you can get massive performance boosts there. Now if you move to specialized hardware that is sort of much more brain-like and that sort of runs in parallel with a bunch of tiny little cores, that you're going to be able to run these models sort of insanely faster."}, {"start": 292.27, "end": 318.54, "text": "Okay, so I think one of the most common questions or threads of question that were asked on Twitter and Facebook were generally how to get into AI. Could you guys give us just a primer of where someone should start if they're just a CS major in college? Yeah, absolutely. So it really depends on the nature of the project that you would like to do."}, {"start": 318.54, "end": 345.95, "text": "Tell you a bit about our project, which is essentially developing large-scale reinforcement learning for Dota 2. And the majority of the work is actually engineering. And essentially taking the algorithms that we have already implemented and trying to scale them up is usually..."}, {"start": 345.95, "end": 362.34, "text": "The fastest way to get improvement in our experiments. So essentially becoming a good engineer for our team is much more valuable than for example people spending..."}, {"start": 362.34, "end": 387.63, "text": "So just to echo this, because I hear this come up all the time. People say it's like my dream to work at OpenAI, but I got to go get an AI PhD. So I'll see you in like five or seven years. If people are just a really solid engineer, but no experience at all with AI, how long does it take someone like that to become productive for the kind of work at OpenAI that we're looking for? So someone like that can actually become productive from day one."}, {"start": 387.63, "end": 415.39, "text": "And that there's, with different engineers who show up at OpenAI, there's a spectrum of where they end up specializing. There are some people who focus on building out infrastructure and that actually this infrastructure can range from, well, we have a big Kubernetes deployment that we run on top of a cloud platform and building tooling and monitoring and sort of managing this underlying layer. It actually looks quite a bit like running a startup."}, {"start": 415.39, "end": 425.81, "text": "And that a lot of the people who are most successful at that have quite a bit of running at large scale in a startup or production environment."}, {"start": 425.81, "end": 453.62, "text": "There's kind of a next level of getting closer to the actual machine learning, where if you think of how machine learning systems look, that they tend to be this like magical black box of machine learning, this core. And you actually try to make that core be as small as possible because machine learning is really hard, eats a lot of compute. It's really hard to tell what's going on there. And so you want it to be as simple as possible. But then you surround it by as much engineering as you possibly can. So what percent of the work on the Dota 2 project would you guys say was what people would..."}, {"start": 453.62, "end": 482.22, "text": "Really think of as like machine learning science versus engineering. Essentially, as far as day-to-day work goes, this kind of work was almost non-existent. There was maybe a few personal weeks spent on that compared to personal spent on engineering. And I think maybe placing some good bets was one of it. Good bets on the machine learning side?"}, {"start": 482.22, "end": 508.75, "text": "On the machine learning side, yeah. And they're often more about what not to do rather than what to do. So at the very beginning of the project, we knew we wanted to solve a game, a hard game. We didn't know exactly which one we wanted to do because these are great testbeds for pushing the limits of our algorithms. And one of the great things about it too... And just to be clear, you guys are two of the key people, but the entire team was like 10 people? About 10 people. That's right."}, {"start": 508.75, "end": 532.62, "text": "And, you know, these things, good test beds for algorithms, see what the limits are to really push the limit of what's possible. And you know for sure that when you've done it, that you've done it. It's very binary testable. And so actually the way that we selected the game was we went on Twitch and just looked down the list of most popular games in the world and started, you know, number one is League of Legends. The thing about League of Legends is it doesn't run on Linux."}, {"start": 532.62, "end": 562.11, "text": "And it doesn't have a game API. And little things like that actually are the biggest barrier to making AI progress in a lot of ways. And so looking down the list, Dota actually was the first one that kind of had all the right properties. Runs on Linux. That it has a big community around replay parsing. That there's a built-in Lua API. It was actually, this API was meant for script, for building mods rather than for building bots. And we were like, but we could probably use it to build bots."}, {"start": 562.11, "end": 588.14, "text": "And one of the great things about Valve as a company is that they're very into having these open hackable games where people can go and do a bunch of custom things. And so kind of philosophically, it was very much the right kind of company to be working with. So we actually did this initial selection back in November, and we were working on some other projects at the time, and so it didn't really get started until late in December."}, {"start": 588.14, "end": 617.78, "text": "One of the funny things is that by total coincidence, in mid-December, Valve released a new bot-focused API and that they were saying, hey, our bots are famously bad, that maybe the community can solve this problem, so we'll actually build an API specific for people to do this. And that was just one of those coincidences of the universe that just worked out extremely well. So we were kind of in close contact with the developer of this API and kind of all throughout."}, {"start": 617.78, "end": 647.68, "text": "At the very beginning of the project, well, what are you going to do? The first thing was we had to become very familiar with this game API to make sure we understood all the little semantics and all of the different corner cases, and also to make sure that we could run this thing on a large scale and to turn it into a pleasant development environment. At the time, it was just two of us. One person was working with the bot API, building a scripted bot. Basically, this is the learn all the game rules, think really hard about how it works. The particular person who wrote it, Raffle,"}, {"start": 647.68, "end": 674.0, "text": "has played about three or four games of Dota in his life, but he's watched over a thousand hours of Dota gameplay and has now written the best Dota scripted bot in the world. And so that sort of a lot of just writing this thing in Lua, getting very intimately familiar with all those details. In the meanwhile, what I was working on was trying to figure out how do you turn this thing into a..."}, {"start": 674.0, "end": 678.1, "text": "Docker container. And so they had this whole build process."}, {"start": 678.1, "end": 707.15, "text": "Turns out that Steam can only be in offline mode for two weeks at a time, that they push new patches all the time. So you needed to go from this like, you know, sort of manually download the game and whatever to actually have an automated repeatable process. It turns out that the full game files are about 17 gigabytes and that our Docker registry can only support five gigabyte layers. And so I had to write a thing to chunk up things into five gigabyte tarballs and put those in S3 and slurp them back down. So a bunch of sort of like things there where it's really just about figuring out what the right workflow is, what the right abstractions are."}, {"start": 707.15, "end": 716.4, "text": "And then the next step was, well, we know we want to be writing our bots in TensorFlow and Python. How do you get that? Why was that? Well, because so machine learning."}, {"start": 716.4, "end": 744.22, "text": "It's actually quite interesting that a lot of the highest order bit on progress, just like having the game API is a higher order bit, it's also can you use tools that are familiar and sort of easy to iterate with before the world of kind of modern machine learning frameworks ever write their code in MATLAB. If you had a new idea, it would take you two months to do it. Good luck making progress. And so it was really all about iteration speed. And so if you can get into the Python world, well, we have..."}, {"start": 744.22, "end": 754.56, "text": "These large code bases that we built up of high quality algorithms that there's just so much tooling built around it that that's like the optimal experience. And so the next step was."}, {"start": 754.56, "end": 783.86, "text": "to port the scripted bot into Python. And so the way I did that was I literally just renamed all of the .lua files to .py, commented out the code, and then started uncommenting function by function. And then you run the function, you get an exception, you then go and uncomment whatever code it depends on. And as mechanically as possible, I tried to be like a human transpiler. And that Lua is one index, Python zero index, so you have to do that. That Lua doesn't distinguish between an array type and a dictionary type, and so you kind of have to disambiguate those two. But for the most part, I did something that could have"}, {"start": 783.86, "end": 809.92, "text": "And so I ended up with a file with a bunch of dummy calls, and I knew exactly which calls I needed."}, {"start": 809.92, "end": 839.68, "text": "And then implemented on top of gRPC, a protobuf-based protocol where on every tick, the game would dump the full game state, send the diff over the wire, reassemble that into an in-memory state object, Python. And then all of these API methods would be implemented in Python. And so at the end of this, it sounds like a bit of a Frankenstein process, but it actually worked really well. And in the end, we had something that looked just like a typical OpenAI gym environment. And so all you have to do is you say, gym.make this."}, {"start": 839.68, "end": 854.72, "text": "And suddenly you're playing Dota and that your Python code just has to call into some object that implements the Lua API. And suddenly these characters are running around the screen doing what you want."}, {"start": 854.72, "end": 883.47, "text": "This was like a lot of the kind of thing that I was working on in the pure engineering side. And actually, as time went on, so Shimon and Jakob and Jay and others joined the project. And most people were building on top of this API and really didn't have to dig into any of the underlying implementation details. So personally, my one machine learning contribution to the project, I'll tell you about that. Because my background is primarily startup engineering, building large infrastructure, not..."}, {"start": 883.47, "end": 904.66, "text": "Sort of machine learning, definitely not a machine learning PhD. I didn't even finish college. So I kind of reached a point where I'd gotten the infrastructure into a pretty stable point that I felt like, all right, I don't have to be fighting the fires there very constantly. I have some time to actually focus on digging into some of the machine learning. One particular piece that we were interested in doing was behavioral cloning. So we had..."}, {"start": 904.66, "end": 929.33, "text": "One of the systems that we had built was to go and download all of the replays that are published each day. And so the way this game works is that there's about 1.5 million replays that are available for public download. Valve clears them out after two weeks. And so you have to have some discovery process. You have to stick them in S3 somewhere. Originally we were downloading all of them every day and realized that was about two terabytes worth of data a day."}, {"start": 929.33, "end": 957.62, "text": "That adds up quite quickly. So we ended up filtering down to the sort of most expert players. But we wanted to actually take this data, parse it, and use it to clone the behavior for a bot. And so I spent a lot of time with like sort of, you know, it's basically you need this whole pipeline to download the replays, to parse them, to, you know, kind of iterate on that, to then take it, train a model, and try to predict what the behavior would be. And, you know, at first, it's just like..."}, {"start": 957.62, "end": 984.59, "text": "One thing I find very interesting is the sort of different workflow that you end up with when doing machine learning. There are a bunch of things where when software engineers join OpenAI that are just very surprising. For example, if you look at a typical research workflow, you'll see a lot of files named like experiment. Whatever the name of the experiment is. One, two, three, four. And you look at them and they're just like slight forks of the same thing. And you're like, is this what version control is for? Why are you doing that?"}, {"start": 984.59, "end": 1010.99, "text": "After doing this cloning project, I learned exactly why. Because the thing is, if you have a new idea for, okay, well, I've kind of got this thing working, and now I'm going to try something slightly different. As you're doing the new thing, well, machine learning is, to some extent, very binary. At the start, it just doesn't work at all, and you don't know why. Or it kind of works, but it has some weird performance, and you're not sure exactly, is it a bug? Is it just how this data set works? You just don't know. And so if you've gotten it working at all,"}, {"start": 1010.99, "end": 1040.72, "text": "And then you make a change. You're always going to want to go back and kind of compare to the previous thing you had running. And so you actually do want the new thing running side by side with the old thing. And if you're constantly stashing and unstashing and checking out and whatever, then you're just going to be sad. And there are a lot of like kind of workflow issues like that, that you just, you got to bang your head against the wall. And then you see like, I've been enlightened. So before we progress further on the story, can you just explain the basics of training a bot in a game? Like how are you actually giving it the feedback?"}, {"start": 1040.72, "end": 1069.41, "text": "Oh, yeah, so on a higher level, we are using reinforcement learning with self-play. So what that means, I mean, it's not rocket science, even though reinforcement learning sounds all fancy. Essentially what's happening is we have a bot which observes some state in the environment and performs some actions based on that state. And based on those actions,"}, {"start": 1069.41, "end": 1095.78, "text": "And then the bot gets feedback on whether it's doing good or not, and then tries to select the actions that yield to high, that positive feedback to high reward."}, {"start": 1095.78, "end": 1124.02, "text": "And to give us a sense for how well that works, so the bot plays against itself to get better. Once you had everything working, how good would a bot from day N do against a bot from day N-1? So I guess we have a story that kind of illustrates what to expect from those techniques. So when we started this project, our goal wasn't to really do research. I mean, at some high level it was, but..."}, {"start": 1124.02, "end": 1148.0, "text": "But we are very goal-oriented. All we wanted to do is we wanted to solve problem. Right? We want to solve .fi v5, and we had our my-solve.onv1. And the way it started, it was like early days when there was just Greg and Rafa, and Rafa was implementing scripted bots. So he just, like, literally write the logic. I think this is what bots should do. When he sees a creep, he needs to attack it, yada, yada. And he spent like three months of his time."}, {"start": 1148.0, "end": 1171.55, "text": "Trafo is actually a really good engineer, so we had a really good scripted bot. So what happened then, he got it to the point where he didn't think he could improve it much more, so we tried, okay, let's try some reinforcement learning. I was actually on vacation at the time, but there was another engineer, Jakub, who..."}, {"start": 1171.55, "end": 1201.22, "text": "Throughout my vacation, which I found super surprising, I leave, there is nothing, I come back, there is this reinforcement learning bot. And actually, it's beating our scripted bot after a week worth of engineering effort. Possibly it was two weeks, but it was something very miniature compared to the development of scripted bots. So actually, our bot, which didn't have any assumptions baked about the game, figured out the underlying game structure well enough to beat anything that we could code by hand, which was pretty amazing to see. And at what point do you decide to compete?"}, {"start": 1201.22, "end": 1215.14, "text": "Well, so maybe I should finish up my story. Sorry if it's running a bit long, but it'll get good shortly. So just finish up my machine learning contribution. So I basically spent about a month."}, {"start": 1215.14, "end": 1240.98, "text": "Thank you so much."}, {"start": 1240.98, "end": 1270.78, "text": "And anyway, I got this to the point where it was actually creep blocking pretty reliably, pretty well. And then at that point, I turned it over to Jay, who's also working on the project. And he used reinforcement learning to fine tune that. And so suddenly it went from only understanding the actions rather than the intent to suddenly it really knew what it was doing and kind of has the best creep block that anyone has seen. And so that was my one machine learning contribution to the project. So time went on. And one of the most important parts of the project was having a scoreboard. So we had a metric on the wall."}, {"start": 1270.78, "end": 1279.82, "text": "Which was the true skill of our best bot. So true skill is basically an evil-like rating that measures the win rate of your bot versus others."}, {"start": 1279.89, "end": 1305.23, "text": "And you put that on the wall and each week people just try all the ideas and some of them work. Some of them improve the performance. And that we actually ended up with this very smooth, almost linear curve. So we posted it in a blog post. And that that really means kind of like exponential increase in the strength of this bot over time. And that part of that is sometimes these data points where you just train the same experiment for longer. Typically our experiments last for maybe up to two weeks."}, {"start": 1305.3, "end": 1332.1, "text": "But also a lot of those were, well, we had a new idea. We tried something else. We made this tweak. We added this feature, removed this other component that wasn't necessary. And so we chose the goal of 1v1. I don't recall exactly when, but it must have been in the spring or maybe even early summer. But we really didn't know, are we actually going to be able to make it? And unlike normally when you're building an engineering system, you think really hard about all the components."}, {"start": 1332.1, "end": 1345.33, "text": "It's like, well, you decompose it into this subsystem, that subsystem, that subsystem, and you can measure your progress as what percent of the components are built. Here, you really have ideas that you need to try out and that..."}, {"start": 1345.33, "end": 1372.51, "text": "It's sort of unpredictable in some sense. And actually, one of the most important changes to the project in terms of making progress was initially the way that the project management was happening was that each week, well, so we'd written down our milestones of let's beat this person by this date. Let's beat this other person by this date. Let's be able to do kind of these outcome-based milestones on kind of a weekly or biweekly basis that those things would come and go and you wouldn't have them."}, {"start": 1372.51, "end": 1399.01, "text": "And then what are you supposed to do?"}, {"start": 1399.01, "end": 1427.6, "text": "And at that point, like two weeks out, how good was the bot? Oh, it was barely sometimes winning with professionals that we had testing. But not even always?"}, {"start": 1427.6, "end": 1442.96, "text": "No, no, no. It sometimes happens. So to be specific, I'm just spooling this back in. So July 8th is when we had our first win against our semi-protester. A sequence of losses."}, {"start": 1442.96, "end": 1472.61, "text": "Sequence of losses. And then we were kind of more consistent with it. And then he went on vacation. And so he was on some laptop somewhere that was not very good. And then we were consistently beating him. But that was not very reliable data. This was the week before the international. And so we didn't really know how good we were getting. We knew that the true skill was going up. When was the last time that an OpenAI employee beat the bot? How far out was that from the profession? I think like a month or two before TI."}, {"start": 1472.61, "end": 1500.32, "text": "We're not very good at Dota. Okay, but so like a month or two out, it could beat all the OpenAI people. Two weeks out, it could one time beat a semi-pro. I'd say four weeks was the first time that it beat the semi-pro. And then two weeks out, we don't know. We still can't really find out. I mean, I guess we could rerun that bot, but we really didn't know how good it was at that time. We just knew, hey, we're able to beat our semi-pro occasionally."}, {"start": 1500.32, "end": 1526.3, "text": "We, going into the international, figured that, hey, there's a 50-50 shot. And I think we were telling Sam the whole way, like, you know, with these things, you never really trust the probabilities. You just trust the trend of the probabilities. Even that was just swinging wildly. You guys would text me every night. It would be, oh, we're going to, you know, no chance. We're not going. Oh, we're definitely going to win every game. Yep. Yep. And so it was very clear that our own estimates of what was going to happen were miscalibrated."}, {"start": 1526.3, "end": 1535.84, "text": "And throughout the week of TI, actually, we still didn't know. And what was happening is we... You guys all went to Seattle for this week."}, {"start": 1535.84, "end": 1565.14, "text": "Most of the team went there. So you were holed up in a hotel or a conference center or something in Seattle? Well, actually, the reality of it was that we were holed up near the stadium when the event was happening. Let me describe how we were holed up. So we were given a locker room in the basement of Key Arena. So we all had production badges, and so you feel very special as you walk in. You're just like, oh, yeah. I just get to skip the line and go to the backstage area. But it was literally a locker room."}, {"start": 1565.14, "end": 1592.9, "text": "And then we'd play against the pros. And we had a partition that we set up, which was just like a black cloth, basically, between the whole team sitting there being like, are we going to be able to beat this pro? Maybe? And trying to be as quiet as possible. And these pros who were playing."}, {"start": 1592.9, "end": 1620.93, "text": "You know, on Monday, they brought three or I think two pros and like one very high ranked analyst by. And we had our first game and, you know, we really didn't know what was going to happen. And we beat this person 3-0. And, you know, this was actually a very exciting thing for everyone at OpenAI. So at the time, I was kind of live slacking the updates as the game is like this person, you know, just said this and like that. And like, you know, now it's this many last hits. And yeah, were you winning by a large margin?"}, {"start": 1620.93, "end": 1647.92, "text": "So, yeah, do you remember the details of that one? Which game specifically? This was Blitz. Oh, Blitz. I think we won every game. Yeah, we did 3-0. I don't know exactly what the margin was. We have all the data, but Valve brought in the second pro, this professional named Pycat, and he played the bot, and we beat him once, we beat him twice, and then he beat us. Oh, okay. And looking at the game, we knew exactly what had happened."}, {"start": 1647.92, "end": 1677.62, "text": "Yeah, essentially what happened is he accumulated a bunch of want charges, right? There's this item that accumulates charges, and he accumulated more charges than our bot has ever seen in-game. Because just our bots don't... It turns out that there was a small... I think it's safe to say kind of a bug in a setup. In Dota? In our setup. Oh, okay. So basically it passed some threshold that your bot was not ready for."}, {"start": 1677.62, "end": 1703.5, "text": "Well, I'd say very specifically that kind of the root cause here was that he had gone for an item, an early wand build. And we had just never done item early wand build. And so it's just like our bot had just never seen this particular item build before. And so it had never had a chance to really explore what does it mean? And so it had never learned to save up stick charges and to use them and whatever. And so that it would do is very good at calculating like who's going to win a fight. Wild."}, {"start": 1703.5, "end": 1726.78, "text": "Because I kind of recognize that he's like, I wonder what happens if I push on this axis. And sure enough, it was an axis the bot hadn't seen. And so then we played a third match against another pro, went 3-0 on that. And it's actually very interesting getting the pros' reactions because we also didn't really know, are they going to have fun? Are they going to be cool? Are they going to hate it? And we got a mix of reactions. Some of the pros were like, this is the coolest thing ever. I want to learn more about it."}, {"start": 1726.78, "end": 1748.93, "text": "One of the pros was like, this thing's stupid, I would never use it. But apparently, after the pros left that night, they spent four hours just talking about the bot and what it meant. The players were highly emotional in their reactions to the bot. They were never beaten by the computer. So it's kind of unbelievable. So for example..."}, {"start": 1748.93, "end": 1777.73, "text": "One of the players actually managed to eventually beat the bot. He was like, okay, this bot is effing useless. I never want to see it. And then he kind of called down. And after five or ten minutes, he's like, okay, this is actually great. This is going to improve my practice a lot. And so after your bot, it lost that first time. Did they start talking about counterintuitive strategies to beat it? Well, so I think at that point that, you know, I think that... Actually, I don't know. Maybe you can answer that particular question."}, {"start": 1777.73, "end": 1805.5, "text": "Yeah, so I don't think pro players are interested in that. The pro players are mostly interested in the aspect where it lets them get better at the game, which means that... But there was a point after the event where we set up this big LAN party where we had like 50 computers running the bot. We kind of unleashed this swarm of humans at our bot. And they found all the exploits. And we kind of expected them to be there because..."}, {"start": 1805.5, "end": 1826.53, "text": "The bot can only learn as well as the environment in which it plays allows it to. So there are some things that it's just never seen. And of course, those ones will be exploitable. And we are kind of excited about our next step, which is 5v5, because 5v5 is..."}, {"start": 1826.53, "end": 1851.3, "text": "One giant exploit. Essentially, it's about exploiting the other team, being where they don't expect you to be, doing other distribution things. Naturally, we know we will have to solve those problems head-on for 5v5. One thing I think was pretty interesting about the training process is that a lot of our job while we were doing this was seeing what the exploits were and then making a small tweak that fixes them."}, {"start": 1851.3, "end": 1877.65, "text": "And, like, the way that I now think about machine learning systems is that they're really a way to make the leverage of human programmers go way up. Okay. Right? Because, again, normally, like, when you're building a system, you build component one, component two, component three. And that kind of your marginal return on, like, building component four is, like, similar to your marginal return on component one. Whereas here, a lot of the early stuff that we did is just, like, your thing goes from being, like, crappy to, like, slightly less crappy."}, {"start": 1877.65, "end": 1905.28, "text": "Once we were at the International and we had this loss to PyCat, we knew, okay, well, the root cause here is just it's never seen this item build before. Well, all we had to do was make a tweak to add that to our list of item builds. And then it played out this scenario for the next, you know, however long. Can you walk me through actually how that tweak works on the technical side? Because my impression is kind of what you guys have been saying. It's just been in a million games. So it kind of has learned all this stuff."}, {"start": 1905.28, "end": 1928.9, "text": "And some people talk about these networks as just very gray, and they don't actually know how to manipulate what. How are you guys getting in there and changing things? Yes, so it's kind of funny. In some sense, on a high level, you can compare this person to teaching a human. You see a kid doing maths, and it's kind of confusing."}, {"start": 1928.9, "end": 1955.23, "text": "Addition with subtraction, suppose, right? And you're like, look here, this symbol. This is what you're not seeing clearly, right? And the same with those tweaks to our bot. So clearly our bot has never seen this want build that Greg mentioned. And all we had to do is we had to say that when the bot plays games and chooses..."}, {"start": 1955.23, "end": 1978.51, "text": "What items to purchase, we just need to add some probability of sampling that specific build that it has never seen. And when it plays a couple of games against opponents that use that build, when it uses this build a couple of times itself, then it becomes more comfortable with the idea of what happens, what are the in-game consequences of that build."}, {"start": 1978.51, "end": 2001.2, "text": "So I have a couple different levels to that answer that I think are pretty interesting. So one is at a very object level. So the way that these models work is you basically do have a black box which takes in some list of numbers and outputs a list of numbers. And it's very smart in how it does that mapping. But that's what you get."}, {"start": 2001.2, "end": 2026.78, "text": "And then you think of this as this is my primitive. Now what do I build on top of that so that as little work as possible has to be done inside of the learning here? And a lot of your job is, well, one thing that we noticed that we'd forgotten as well on Monday was, well, it wasn't that we'd forgotten, we just hadn't gotten around to it, was the passing in data that corresponds to the person was passing in the visibility of a teleport. So as a human, you can see when someone's teleporting out."}, {"start": 2026.78, "end": 2048.99, "text": "Our bot just did not have that feature. The list of numbers passed in did not have that feature. And so, well, one of the things you need to do is you need to add it. And so that kind of goes from your feature is, you know, your feature vector is however long it was. And now it's got one more feature on it. And the bot wasn't recognizing that as an on-screen thing? So it doesn't see the screen. It's passed data from the bot API. Oh, okay. And so it really is given whatever data we give it."}, {"start": 2048.99, "end": 2072.3, "text": "And so it's kind of on us to do some of this feature engineering. And you want to do as much as you can to make it as easy as possible so that it has to do as little work inside as possible. So it can spend, you think of it as you've got some fixed capacity. Do you want to spend that on learning the strategy? Do you want to spend it on learning how to map, choose which creep you want to hit? Do you want to spend that on trying to parse pixels?"}, {"start": 2072.3, "end": 2099.18, "text": "At the end of the day, I think a lot of our job as the system designers here is to push as much of that model capacity and as much of the learning towards the interesting parts of the problem that you can't script, that you can't possibly do any processing for. And so that's kind of one level, is that a lot of the work ends up being identifying which features aren't there or kind of engineering the observation and action spaces in an appropriate way."}, {"start": 2099.18, "end": 2119.73, "text": "Another thing is, I think, is like another level where you zoom out is like the way that this actually happened was, so, you know, we're there on Monday and people got dinner and then Shimon and Jakob and Raffel and Saiho and I think, you know, maybe one or two others stayed up all night to do surgery on our running experiment. And so..."}, {"start": 2119.73, "end": 2149.04, "text": "It was very much like you've got your production outage and everyone's there, all hands on deck, trying to go and make the improvements. So specifically, to kind of zoom in and to give you a bit of a feel, what it felt like working on the bot. This was a very tiring week. Every day, the day was just meeting with the pros and watching our bot getting excited. And the nights were kind of coding up the next version of the experiment. Because actually, it's a little known fact, but..."}, {"start": 2149.04, "end": 2165.17, "text": "How are you discerning that?"}, {"start": 2165.17, "end": 2182.08, "text": "Well, this was, again, something of almost a coincidence. I mean, there might be something a little bit deeper. But so, you know, kind of the full story of the week was we did the Monday play. And, you know, there we lost to PyCat. And so just to clarify, are you guys in the competition or not in the competition?"}, {"start": 2182.08, "end": 2202.94, "text": "The thing that we did was we did a special event to play against Dendy, who's one of the best players of all time. And while we were there, we were also like, well, let's test this out against all these other pros because they're physically here right now. And let's see how we do. Got it. All right. So Monday happens. You start training it. Yep. And so actually, yeah, so this experiment we've kicked off."}, {"start": 2202.94, "end": 2232.51, "text": "You know, maybe sometime the prior week. Yep. Three weeks before, I think. Yeah, something like that. And we've been running this experiment for a while. And our infrastructure is really meant for you run an experiment from scratch. You know, you start from complete randomness and then you run it. And then you, two weeks later, go and see how it does. We didn't have two weeks anymore. And so we had to do this surgery and this very careful, like, you know, read every single character of your commit to make sure that you're not going to have any bugs. Because if you mess it up."}, {"start": 2232.51, "end": 2246.61, "text": "We're out of time. There's nothing you can do. And it's not one of those things like if you're just a little bit more clever that you can go and do a hot patch and have everything be good. It's just literally the case that you got to let this thing sit here and it's got to bake. And so Monday."}, {"start": 2246.61, "end": 2274.96, "text": "Came and went, we were running this experiment that we perform surgery on. And the next day we got a little bit of reprieve where we just played against some kind of lower ranked players who are kind of commentators and popular in the community, but we're not pushing the limit of our bot. On Wednesday at 1 p.m., our contact from Valve came by and said, hey, I'm going to get you Arteezy and Sumail, who are basically the top players in the world."}, {"start": 2275.15, "end": 2296.05, "text": "And I was like, could we push them off to Thursday maybe? And he was like, their schedule is booked. You're going to get them when you get them. And we were scheduled to get them at 4 p.m. So we looked at our bot to see how it was doing. And we've kind of been along the way gauging it. We tested it against our semi-pro player. And he said this bot is completely broken."}, {"start": 2296.94, "end": 2325.57, "text": "Oh, no. And, you know, kind of pictures of maybe we had a bug during the surgery, like went through our head. And he showed us the issue. And he said, look, first wave, the spot takes a bunch of damage. It doesn't have to take. There's no advantage to that. I'm going to run and I'm going to go kill it. I'll show you how easy it is. He ran in to kill it and he lost. Okay. But don't jump ahead. Explain what happened. So he played it five times and he lost each time until he finally did figure out how to exploit it. And we realized what was going on."}, {"start": 2325.57, "end": 2350.85, "text": "And we did not at all expect that it was like..."}, {"start": 2350.85, "end": 2372.08, "text": "One of the major examples of the things that we kind of didn't have explicit incentive for and yet the bot actually learned them. And yeah, essentially, I mean, it was kind of funny because, of course, when the bot..."}, {"start": 2372.08, "end": 2398.66, "text": "Played against its other versions. It was just like good baiting strategy that was kind of out of distribution. But it had a very interesting psychological effect on humans because optimal strategy was not to fall for the bait. It was kind of to wait it out a little bit because the bot already is at a disadvantage. But he's like, okay, look how stupid this bot is. I'm going to go for a kill. So it kind of had interesting psychological effect on humans, which I thought was like..."}, {"start": 2398.66, "end": 2423.23, "text": "It almost knows it's a bot. It knows how it's attacked. It's funny to see a bot which kind of seems like it's playing with emotions. Of course it was not what actually happened, but it seemed this way. So now we were faced with a dilemma. It's 1pm on Wednesday. These best players are going to be showing up at 4pm. We have a broken bot. What are we going to do? And we know that our Monday bot is not going to be good enough. We know it's not going to cut it."}, {"start": 2423.23, "end": 2448.64, "text": "And so the first thing we do is we're like, well, Monday bot, it is pretty good at the first wave. This new bot is a super bot thereafter. So can we stitch the two together? So we had some code for doing something similar. So we kind of revived that. And then in the three hours, Jay spent his time doing a very careful stitch where you run the first bot and then you cut over at the right time to the second bot."}, {"start": 2448.64, "end": 2472.03, "text": "And this is literally just like bot one plays the first X amount of time and then bot two takes over. Literally just that. And he finished it 20 minutes before the thing. We ran it by our semi-pro. Semi-pro was like, this is great. So we at least, we got that done in the nick of time. But the other question was, how do we actually fix this bot? I mean, actually just to finish your story, there's like one aspect because we are also..."}, {"start": 2472.03, "end": 2498.9, "text": "Kind of uncertain what happens when you switch over from one bot to the other. So I was actually standing by the pro who was playing it, and I was looking at the timer at the moment when it was switching. I was like... I kind of distracted the guy in case something stupid happened. Just to try to distract him for once. And of course, it was probably completely unnecessary, but we weren't sure what had happened there. So I didn't know about that part of the story. So..."}, {"start": 2498.9, "end": 2514.94, "text": "The question of how do you actually fix it? So there was a little bit of debate of like, maybe we should abandon ship on this, switch back to our old experiment, run that one for longer. And I forget who suggested it, but someone was like, I think we just have to let it run for longer because you learn a strategy of baiting well."}, {"start": 2514.94, "end": 2543.65, "text": "The counter strategy for that is just don't bait. Play well the whole time. And so we got that run for the additional three hours. And so we first played our TZ who showed up on our switch bot, kind of the Franken bot. And that beat him three times. And we're like, all right, let's try out this other bot and just see what happened with the additional three hours of training. Because our semi-protester at least validated that it looks like it's fixed. And so in that three hours of training, how many games is it actually playing simultaneously?"}, {"start": 2543.65, "end": 2570.86, "text": "That's a good question. Quite a bit. And so we played this new bot against RTZ. Didn't know how it was going to do. And sure enough, it beats him. And he loved it. He was having a lot of fun. He ended up playing 11 games that day. Or maybe it was 10. But I think that he was just like, oh, this is so cool. We were supposed to have Sumail that day as well. But due to a scheduling snafu, he had to be at some panel. And so timing didn't work out."}, {"start": 2570.86, "end": 2591.71, "text": "But Arteezy and his coach, who also coaches Sumail, both said, Sumail's going to beat this bot. It's going to happen. Maybe he'll have a little bit of trouble to figure it out for the first game, but after that, you're in trouble. And so you're like, all right, we've got one more day to figure out what to do. And so what did we do?"}, {"start": 2591.71, "end": 2621.12, "text": "I don't know. It's kind of like some nice dinner. Kind of what did you say? We kind of went for some nice dinner. We kind of rested a lot. We kind of chatted, you know, like, you know, slacked with some people at home. And then in the morning we download the new parameters of the network and just let it play. So we literally just hung out and just let it go. Just let it play. It's the exact opposite of how I'm used to engineering deadlines happening. Yeah. Normally it's your work right up until the minute."}, {"start": 2621.12, "end": 2638.02, "text": "So you guys were getting like full nights of sleep, nice and relaxed. Oh, no, no, absolutely not. Okay, so to make this clear, the night before the night, like two nights before the day where we got the rest and relaxation, the night looked something like the following."}, {"start": 2638.02, "end": 2666.22, "text": "We had full day of dealing with the problems and kind of like emotional highs and so on. So it's absolutely not good. Come midnight, we start working. Okay, we need to make all those changes. Like the one thing that we talked about around midnight, we start with four people. And we are also tired that like, you know, we look through all the comments that we are going to add to the experiments. There were actually two people looking at them because we didn't trust a single person given how tired we all were. So they were like looking at those comments."}, {"start": 2666.22, "end": 2693.84, "text": "I was doing this, like, updating the model, which is a lot of nasty, like, off-by-one indexing things. So even though it was a short call, it took me, like, six hours to do. Somewhere around 3 a.m., we had, like, a phone call with Azure. Because it turns out that with a certain number of machines, you start exceeding some limits. So we tried to make them raise the limits."}, {"start": 2693.84, "end": 2722.11, "text": "Around 6 a.m. we were okay. We were ready to deploy this. Deploying is just like a one-man job. Jakub was just clicking the deploy and fixing all the issues that came up. I was staying around just exclusively to make sure that Jakub doesn't fall asleep. And eventually at 11 a.m. the experiment was running and we kind of..."}, {"start": 2722.11, "end": 2750.77, "text": "So it had over 24 hours to train? Sorry, just to repeat the timeline. So this was Monday, it was when we played the first set of games, had the loss, did the surgery that night. We played it, I guess, starting at 11 on Tuesday. Then Wednesday, 4 p.m. is when we played Arteezy."}, {"start": 2750.77, "end": 2775.78, "text": "And then train for longer. I don't think we made any changes after that. Maybe we made some small ones. But then on Thursday is when we played Sumail. Okay. So I think Tuesday to Wednesday was the night where we made the last changes. And there was quite a bit of different work going on that all kind of came together at once. One thing I think is really important was..."}, {"start": 2775.78, "end": 2789.92, "text": "One of our team members, his handle is Cyho, who's a very well-known programming competition competitor, was spending a lot of time just watching the bot play and seeing why does it..."}, {"start": 2789.92, "end": 2819.66, "text": "What are all the weird tweaks and really getting intuitions for, oh, because we're representing this feature in this way, and so if we change it to this other thing, then it's going to work in a different way. And I think that it's almost this very human-like process of watching this expert playing the game and trying to figure out what are all the little micro decisions that are going into this macro choice. And it's kind of interesting starting to have this very different relationship to the system you build."}, {"start": 2819.66, "end": 2844.54, "text": "Because normally the way that you do it is, well, your goal is to have everything be very observable. And so, yeah, you want to put metrics on everything. And if something's not understandable, add more logging. That's how you design the systems. Whereas here, you do have that for the surrounding bits, but for this machine learning core, that there you really do have to understand it at more of a sort of behavioral level."}, {"start": 2844.54, "end": 2872.14, "text": "Was it ever stumping you where you're just like, oh, it's being creative in a way that we didn't expect it to and it may be even working, but you don't know why or how it decided to make that choice? Yeah, I think the debating story that we shared. Debating is the main one? Is the main story like this. We kind of viewed small ones like where, you know, there was like some early days of the project where like we have professionals like playing the next edition of the voting. It's like, hmm."}, {"start": 2872.14, "end": 2891.06, "text": "Elbot is really good at crippling. And we are like, oh, what is crippling? I'd say that there is also one other part of the story that I think is interesting. And then I think probably we can wrap up this part. So to see how well... Our semi-protester had played hundreds of games against this bot over the past..."}, {"start": 2891.06, "end": 2919.98, "text": "And so we wanted to see just how does he benchmark relative to Arteezy. And so we had him play against Arteezy and Arteezy was up the whole game. It was just like beating him to the last hit by like 500 milliseconds every single time. And so our semi-pro was like, all right, I've got one last-ditch effort to go try this strategy that the bot always does to me. And it's like some strategy where you do something complicated and then you triple wave your opponent. You get him out of the tower. You have regen. You go in for the kill."}, {"start": 2919.98, "end": 2947.1, "text": "And he did it and it worked. And this was the bot had like taught him the strategy that you could use against a human. And I think that was like very interesting and a good example of the kinds of things that you can get out of these systems. That they can discover these very sort of, you know, non-obvious strategies that can actually be taught to humans. And how did it go with Sumail? So with Sumail, we went undefeated. And I think it was 5-0 that day."}, {"start": 2947.1, "end": 2952.37, "text": "One thing that's actually interesting, so we'll probably blog about this in upcoming weeks, but we've actually been playing against..."}, {"start": 2952.37, "end": 2978.69, "text": "A bunch of pros since then. So our bot has been in very high demand. And some of these pros have been live streaming it. And so we've gotten a better sense of kind of watching as humans go from just being completely unable to beat it to if you play against it for long enough, you can actually get pretty good. And so there's actually a very interesting set of stats there that we'll be kind of pulling and analyzing in a bit."}, {"start": 2978.69, "end": 3003.44, "text": "Are there humans that consistently beat the bot today? Yeah, so I think there's one who has like a 20% win rate or something. I think it might be actually 30. All right. And that player played hundreds of games. And just finds strategies to exploit. No, actually. It becomes essentially as good as the bot. Really? At what the bot is doing. You should find it extremely surprising, but it turns out that..."}, {"start": 3003.44, "end": 3017.31, "text": "He played hundreds of games with it, so it's actually... And is he a top player? Like, does he beat most humans? These are all professionals. It's not just some random kid who's good at beating the bot. That's right. The way to think about this is that, like, yeah, I mean..."}, {"start": 3017.31, "end": 3040.85, "text": "Being a professional video game player is a pretty high bar. I think everyone wants to be a professional video game player who play these games and the number of pros is very small. And there are some who have really like, when you're playing hundreds of games against it, you're going to get very, very good at the things that it does. And so talking to Arteezy, I was asking him, has it changed your play style at all? And he said, he thinks that..."}, {"start": 3040.85, "end": 3069.9, "text": "The thing that it's done for him is it's helped him focus more because while you're just there in lane last hitting, now suddenly that's just so rote because you just have been doing it so much. You've gotten so good at it. And I think that one really interesting thing to see is going to be how can you improve? Can you improve human play style? Can you change human play style? And I think that we're starting to see some positive answers in that direction. So I know we're almost out of time. I could do like a little lightning round and just quickly go through some of these questions, guys."}, {"start": 3069.9, "end": 3092.88, "text": "Actually, to the question of what kind of skills you need to work at OpenAI, could we have a very small... That was going to be the first lightning round question. So a specific list of things that we found very useful, at least in the Dota team, is some knowledge of distributed systems, because we build a lot of those and those are..."}, {"start": 3092.88, "end": 3119.2, "text": "Easy to not do properly. And another thing that we found very important is actually writing back free code. Essentially, I know it's kind of taken for granted in computer science community that everybody makes bugs and so on, but here it's even more important than other projects that you minimize it because they're very hard to debug. Specifically..."}, {"start": 3119.2, "end": 3148.22, "text": "Many bugs manifest in kind of lower training performance where to get that number it takes a day and in like a spree of hundreds of comments it's really easy to miss. And primary way of debugging this is actually reading the code. So every bug has very high cost associated with it. So actually writing this correct bug-free code is quite important to us."}, {"start": 3148.22, "end": 3176.3, "text": "We sometimes actually kind of sacrifice good engineering habits, good kind of code modularity to make our code shorter and simpler and kind of having less, essentially less lines where we can make bugs. And I guess lastly, as we mentioned, primary skill is good engineering. But if somebody really feels like..."}, {"start": 3176.43, "end": 3204.74, "text": "Gosh, I really need to brush up my maths. I really need to go in there and feel comfortable, not have somebody ask me questions about maths. I think mostly getting good basics in linear algebra and in basic statistics. Especially when doing experiments, it's easy to make elementary statistics mistakes."}, {"start": 3204.74, "end": 3230.7, "text": "So I wanted to talk about some non-technical skills that I think are really important. So one is that..."}, {"start": 3230.77, "end": 3260.64, "text": "I think that there's a real humility that's required if you're coming from an engineering background like I am and working in these projects where you're no longer the technical expert in the way that you're used to. I think that if you go and you build, let's say you want to build a product for doctors. I think you can talk to 10 doctors and honestly, whatever thing you're going to build is probably going to be a pretty valuable addition to their workflow because doctors can't really build their own software tools. Some can, but as a general rule, no. Whereas with machine learning research,"}, {"start": 3260.64, "end": 3275.52, "text": "Everyone that you're working with is very technical, can build their own tools. But if you inject engineering discipline in the right place, if you build the right tool at the right time, if you kind of look at the workflow and think about, oh, we could do it in this other way, that's where you can really add a bunch of value."}, {"start": 3275.52, "end": 3305.36, "text": "Knowing when to inject the engineering discipline, but also knowing when not to. And being, you know, to Shimon's point, sometimes, well, we really just want the really short code because we're really terrified of bugs. And so that can yield different choices than you might expect for something that's just a pure production system. Who writes the least bugs at all of OpenAI? That's a contentious question. What's the question? Who writes the least bugs per line of code at all of OpenAI? I'm definitely not going to say me."}, {"start": 3306.83, "end": 3326.62, "text": "Possibly, Jakub? Yeah. It's hard to say. Yeah, it's really hard. It could be Greg. I rate a lot of bugs. I caught Jakub the least amount of times on bugs. It's more okay to have bugs that are going to cause exceptions. And my bugs usually cause exceptions."}, {"start": 3326.62, "end": 3352.98, "text": "That's fine. That's fine. Yeah. It's what you don't want is the things that cause correctness issues where it gets 10% worse. Yeah. So there was another question related to skills, but this is for non-technical people. So Tim Biko asks, how can non-technical people be helpful to AI startups? Well, I was going to say, I think one important thing is that for AI generally right now, I think there's a lot of noise."}, {"start": 3353.1, "end": 3379.54, "text": "And I think it can be hard to distinguish what is real from what's not. I think just like simply educating yourself, I think is like a pretty important thing. Like I think it's very clear that AI is going to have a pretty big impact. And that's just look at what's already been created and extrapolate that without any new technology development, any new research. And it's pretty clear there's going to be baked into lots of different systems. There are a lot of ethical issues to work through. And I think that being..."}, {"start": 3379.54, "end": 3408.5, "text": "Kind of a voice in those conversations and educating yourself, I think, is a really important thing. And then you look to, well, what are we going to be able to develop next? And I think that that's where the really transformative stuff is going to come. Okay. I once saw a post of Greg's rescue time report and was pretty shocked. Do you have any advice for working such long, focused hours? I think it's not a good goal. I would not have a goal of trying to maximize the number of hours you sit at your computer. For me, I do it because I love it."}, {"start": 3408.5, "end": 3438.56, "text": "The thing that the activity that I love most in the world is when you're in the zone writing code, producing it for something that's meaningful and worthwhile. And so I think that as a second order effect, it can be good. But I wouldn't say that like that is the way to have an impact. I will also say more specifically, the only way I've ever seen people be super productive is if they're doing something they love. There is nothing else that will sustain you over a long enough period of time. Okay. Is the term AI overused by many startups just to look good in the press? Yes."}, {"start": 3438.56, "end": 3466.74, "text": "Indeed. Okay. What is the last job that will remain as AI starts to do everything else? The last human job. What is going to be the hardest thing for AI to do? It is a hard question to answer in general. Because I think it's actually not AI researcher. The AI researcher will kind of go before. It's actually very interesting when you ask people this question. I think that everyone tends to say..."}, {"start": 3466.74, "end": 3495.41, "text": "Whatever their job is as the hardest one. But I actually think that AI research is going to be one that you're going to want to make these systems very good at doing. I think the last question, maybe this is obvious, is can you just connect the dots between how playing video games is relevant to building AGI? Yeah. It's actually maybe one of the most surprising things to me, the degree to which games end up being used for AI research. And the real thing that you want, right, is you really want to have..."}, {"start": 3495.41, "end": 3522.8, "text": "Algorithms that are operating in complex environments where they can learn skills and that you want to increase the complexity of those skills that they learn. And that's either you push the environment, you push the complexity of the algorithms, you scale these things up, and that that's really the path that you want to take to building really powerful systems. So games are great because they are a prepackaged environment that some other humans have spent time sort of making, first of all,"}, {"start": 3522.8, "end": 3552.3, "text": "Putting in a lot of complexity, making sure that there's actual intellectual things to solve there. Or not even just intellectual, but interesting mechanical challenges that you can get human-level baselines on them so you know exactly how hard they are. They're very nice, unlike something like robotics, where you can just run them entirely virtually. And that means you can scale them up and you can run many copies of them. And so they're a very convenient testbed. And I think what you're going to see is that there's a lot of work that's going to be done in games. But the goal is, of course..."}, {"start": 3552.3, "end": 3578.96, "text": "Bring it out of the game and actually use it to solve problems in the real world and to actually be able to interact with humans and do useful things there. So I think they're a very good sort of starter and a very good place to... I think one thing that I really like about this Dota project and bringing it to all these pros is that we're all going to be interacting with super advanced AI systems in the future. And right now, I think we don't really have good intuitions as to how they operate, where they fail, what it's like to interact with them."}, {"start": 3578.96, "end": 3609.06, "text": "And this is a very low stakes way of having your first interaction with very advanced AI technology. Cool. If someone wants to get involved with OpenAI, what should they do? Well, we have a job posting at our website. I guess the tips that are given about how to get a job at OpenAI are very geared towards the specific job posting that we have there, which is a large scale reinforcement learning engineer. Cool. And in general, we look for people who are..."}, {"start": 3609.06, "end": 3637.29, "text": "Very good at whatever technical access they specialize in. And we can use lots of different specialties. Great. Thanks, guys. Just to echo that, everyone thinks they have to be an AI PhD. Not true. Neither of these guys are. All right. Thanks a lot. Cool. Thanks. Thank you. All right. Thanks for listening. So as always, the video and transcript are at blog.ycombinator.com. And if you have a second, please subscribe and review the show. All right. See you next week."}]}