{"url": "https://www.youtube.com/watch?v=vQXAsdMa_8A", "page_url": "https://www.youtube.com/watch?v=vQXAsdMa_8A", "title": "Demis Hassabis: Towards General Artificial Intelligence", "show_or_channel": "MITCBMM", "date": "2016-07-11", "duration_s": 4050, "type": "talk", "speakers": ["Demis Hassabis"], "confidence": "high", "notes": "MIT CBMM lecture 'Towards General Artificial Intelligence' (2016).", "people": ["demis-hassabis"], "id": "2016-07-11_mitcbmm-demis-hassabis-towards-general-artifici", "audio_duration_s": 4049.8619375, "worker": "jc-w13", "transcribe_s": 102.4, "model": "large-v3", "segments": [{"start": 0.05, "end": 24.27, "text": "I'm Tommaso Poggio. I am the director of the Center for Brains, Minds, and Machines, which is a center between MIT and Harvard, located in BCS in building 46. And I have the pleasure of hosting Demis."}, {"start": 24.4, "end": 51.68, "text": "I don't need to say much about him. If you look on Wikipedia or Financial Times, there's a very good caricature of Demis. And you can find him everywhere. He was a chess child prodigy, studied computer science in Cambridge, started a couple or more of successful computer games companies. Then he became a neuroscientist."}, {"start": 51.68, "end": 79.6, "text": "Got a PhD at UCL in London. And then I was lucky enough that I can put on my CV that he was a postdoc of mine for a brief period between 2009 and 2010, I think. And then we saw each other a couple of times once he came to speak at one of the symposia for the MIT 150."}, {"start": 79.6, "end": 108.94, "text": "This was 2011. And we had one session of that symposium, which was called Brains, Minds, and Machines. One session was titled The Marketplace for Intelligence. And you spoke about DeepMind that you had just started. And so DeepMind is an amazing achievement. Demis managed to put together a company, sell it to Google."}, {"start": 108.94, "end": 136.34, "text": "The company is also a great research lab, I would say the best one in AI these days, with high-impact papers in Nature and so on, and achievements like AlphaGo winning against what is arguably the best player in the world, Lee Sedol. I was in Seoul for the last game, the fifth game, and it was exciting and historic."}, {"start": 136.34, "end": 162.77, "text": "And it's great to have Demis here kind of telling us about what went on and what was the background of it. Demis. Thanks, Tommy. Thanks, Tommy, for that very generous introduction. Thank you all for coming. It's great being back at MIT. I always love coming back here and seeing and catching up with old friends."}, {"start": 162.77, "end": 186.93, "text": "So today I'm going to split my talk into two. The first half of it is I'm going to give you a kind of whirlwind overview of how we're approaching AI development at DeepMind and the kind of philosophy behind our approaches. And then the second half of the talk will be all about AlphaGo and the sort of culmination of our work there and what we're going to do with it going forwards."}, {"start": 188.05, "end": 210.53, "text": "So DeepMind, first of all, it was founded in 2010 and we joined forces with Google in early part of 2014. So we've been there for just over two years now. One of the ways we think about DeepMind and one of the ways I describe it is as a kind of Apollo program for AI, Apollo program effort for AI. Currently, we have more than 200 research scientists and engineers."}, {"start": 210.53, "end": 240.54, "text": "So it's a pretty large team now and we're growing all the time. So obviously there's a lot of work going on and I'm only going to be able to touch on a small fraction of it today. So apart from experimenting on AI, which is obviously the main purpose of DeepMind, at least half of my job and half of my time is spent on thinking about how to organize the endeavor of science. And what we try to do at DeepMind is try to create an optimal environment for research to flourish in."}, {"start": 240.54, "end": 270.32, "text": "And the way, I mean, that would be a whole talk in itself, but just to sort of give you a one-line summary, what we try to do is fuse the best from Silicon Valley startup culture with the best from academia. And so, you know, we've tried to combine the kind of blue-sky thinking that you get and interdisciplinary research you get in the best academic places with the focus and energy and resources and pace of a top startup. And I think this fusion has worked really well."}, {"start": 270.32, "end": 301.74, "text": "So our mission, as some of you will have heard me state, the way I kind of articulate that is in two steps. So step one, try and fundamentally solve intelligence. And then if we were to do that, I think step two kind of follows naturally. Try and use that technology to solve everything else. Certainly that's why I've always been obsessed with working on AI since I can remember. Because I truly believe that it's one of the most important things that mankind could be working on and will end up being one of the most powerful technologies we ever invent. So more prosaically,"}, {"start": 301.74, "end": 332.27, "text": "What we're trying to do at DeepMind, what we're interested in doing is trying to build what we call general purpose learning algorithms. So the algorithms we create and develop at DeepMind, we're only interested in algorithms that can learn automatically for themselves from raw inputs and raw experience. And they're not handcrafted or pre-programmed in any way. The second important point is this idea of generality. So the idea that a single set of algorithms or single system can operate out of the box."}, {"start": 332.27, "end": 360.48, "text": "across a wide range of tasks. In fact, this sort of connects with our operational definition of intelligence. I know that's kind of a big debate, and there isn't really a kind of consensus around what intelligence is. But operationally, we regard it as the ability to perform well across a wide range of tasks. So we really emphasize this flexibility and generality. So we call this type of AI artificial general intelligence."}, {"start": 360.48, "end": 390.38, "text": "Internally at DeepMind. And the hallmark of this kind of AI is that it's flexible and adaptive and possibly you could argue inventive. And I'm going to come back to that at the end once we've covered AlphaGo. And the key thing about it is that it's built from the ground up to deal with the unexpected and to flexibly deal with things that it's never potentially seen before. So by contrast, obviously AI is a huge buzzword at the moment and it's hugely popular."}, {"start": 390.38, "end": 418.22, "text": "Both in academia and industry. But still a lot of the AI that we find around us or that's labeled AI is of this kind of narrow, what I would call narrow AI. And that's really software that's been handcrafted for a particular purpose and a special case for that purpose. And often the problem with those kinds of systems is that they're hugely brittle. As soon as the users interact with those systems in ways that the teams of programmers didn't expect, then obviously they just catastrophically fail."}, {"start": 419.79, "end": 439.26, "text": "Probably still the most famous example of that kind of system is Deep Blue. And obviously, that was a hugely impressive engineering feat back in the late 90s when it beat Garry Kasparov at chess. But Deep Blue, it's arguable whether it really exhibited intelligence in the sense that"}, {"start": 439.26, "end": 470.78, "text": "It wasn't able to do anything else at all, not even play strictly simpler games like tic-tac-toe. It would have to be pre-programmed again from scratch with expert knowledge. So the way we think about AI and intelligence is actually through the prism of reinforcement learning. And most of you will probably be familiar with reinforcement learning, but I'm just going to cover it quickly here in this cartoon diagram for those of you who don't know what it is. So you start off with an agent."}, {"start": 470.78, "end": 498.43, "text": "or an avatar. It finds itself in some kind of environment trying to achieve a goal in that environment. That environment can be obviously the real world in which case the agent will be a robot or it could be a virtual environment which is what we mostly use in which case it's a kind of avatar of some sort. Now the agent only interacts with the environment in two ways. Firstly it gets observations through its sensory apparatus and reward signals and we mostly use vision."}, {"start": 498.43, "end": 523.34, "text": "But we're looking to use other modalities pretty soon. And the job of the agent system is kind of twofold. Firstly, it's got to try and build as accurate a statistical model as it can of the environment out there based on these noisy, incomplete observations that it's getting in real time. And once it's built the best model it can, then it has to decide what action to take."}, {"start": 523.34, "end": 545.46, "text": "From the set of actions that are available to it at that moment in time, to best get it incrementally towards its goal. So reinforcement learning, that's basically the essence of reinforcement learning. And this diagram is very simple, but of course this hides huge complexities and difficulties and challenges that would need to be solved to fully solve what's in this diagram."}, {"start": 545.46, "end": 572.06, "text": "But we know that if we could solve all the issues and challenges behind this framework, then that would be enough for general intelligence, human-level general intelligence. And we know that because many animal systems, including humans, use reinforcement learning as part of their learning apparatus. In fact, the dopamine neurons in the brain implement a form of TD learning."}, {"start": 572.06, "end": 589.57, "text": "What we were committed to philosophically in terms of our approach at DeepMind at the beginning was this idea of grounded cognition. And this is the notion that a true thinking machine has to be grounded in a rich sensory motor reality. But that doesn't mean it needs to be a physical robot."}, {"start": 589.57, "end": 614.29, "text": "As long as you're strict with the inputs, you can use virtual worlds and treat them, these avatars and these agents in these virtual worlds, like virtual robots in the sense that the only access they have to the game state is via their sensory apparatus. So there's no cheating in terms of accessing the internal game code or game states underlying the game."}, {"start": 614.93, "end": 622.13, "text": "And we think if you treat games in that way, then they can be the perfect platform for developing and testing AI algorithms."}, {"start": 622.35, "end": 646.54, "text": "And that's for many reasons. Firstly, you can create unlimited training data. There's no testing bias in the sense that I think one of the challenges of AI is actually creating the right benchmarks. And very often, this sort of turns out to be an afterthought for an AI lab to build the benchmarks. And we think actually crafting the right benchmarks is just as difficult, maybe even more difficult, than coming up with the algorithms."}, {"start": 646.54, "end": 665.55, "text": "And games, of course, have been built for other purposes to entertain and challenge human players. And they've been built by games designers. So they weren't built for testing AI programs. So in that sense, they're really independent in terms of a testing training ground for our AI ideas."}, {"start": 666.1, "end": 690.13, "text": "Obviously, you can run millions of agents in parallel. And we do that on the Google Cloud. And most games have scores. So this is a convenient way to incrementally measure your progress and improvement of your AI algorithms. And I think that's very important when you're setting off on a very ambitious goal or mission like we have, which may be multi-decades. It's important to have good incremental measures that you're going in the right direction."}, {"start": 691.98, "end": 710.45, "text": "So this kind of commitment then leads to this idea of end-to-end learning agents. And this notion of going from starting with, say, raw pixels and going all the way to deciding on an action. And at DeepMind, we're interested in that entire stack of problems from perception to action."}, {"start": 711.89, "end": 740.69, "text": "And I think we've, over the sort of five years that DeepMind's been going, have pioneered this use of games for AI research. And I see many other research organizations now and industrial groups starting to use games themselves for their own AI development. So I guess the first big breakthrough that we had at DeepMind was really starting this field, this new field of deep reinforcement learning."}, {"start": 741.3, "end": 769.38, "text": "And this is the idea of kind of combining deep learning with reinforcement learning. And this allows reinforcement learning to really work at scale and tackle challenging problems. Until we sort of came up with this idea of deep reinforcement learning, RL, of course, as a field has been going for more than 30 years. But generally speaking, up till then, they've only been applied to sort of toy problems, little grid world problems, and nothing really challenging or impressive."}, {"start": 769.38, "end": 799.41, "text": "had been done with RL research. So we wanted to kind of take that further and apply it to a really challenging domain. So initially, we picked Atari 2600 platform, which is really the first iconic games platform from the 80s. And conveniently, there's a nice emulator, an open source emulator, which we took and improved. And then there are hundreds of different classic Atari games available on this emulator."}, {"start": 800.05, "end": 825.9, "text": "I'm just going to run you one video in a second showing how the agent performs in this Atari environment. But before I do, just to sort of confirm with you what you're going to see, the agents here only get the raw pixels as inputs. So the Atari screens are 200 by 150 pixels in size. It's about 30,000 pixels per frame. And the goal here is simply to maximize the score. Everything else is learned from scratch."}, {"start": 825.9, "end": 857.2, "text": "So the system is not told anything about the rules, what they're controlling, or even the fact that pixels and video streams next to each other are correlated in time. It has to find all that structure for itself. And then there's this notion again of generality, one system able to play all the different Atari games out of the box. So we call this system DQN, and we think it really is a kind of general Atari player. So this is a little medley of the same"}, {"start": 857.2, "end": 882.06, "text": "Out of the box, the same hyperparameters playing all these very different games, very different rule sets, very different objectives, very different visuals, out of the box with the same settings and the same architecture. And it performs better than top human players on more than half of the Atari games. And since our Nature paper, we've now increased that to about 95% of the Atari games."}, {"start": 882.67, "end": 908.24, "text": "And here's just ending with a boxing where it's the red boxer here. And it does a bit of sparring with the inbuilt AI and then eventually corners it and just racks up an infinite number of points. So if you want to know more about that work, you can see our Nature paper from last year. And the actual code is freely available as well, linked from the Nature site. So you can play around with the DQN algorithm yourselves."}, {"start": 909.62, "end": 932.75, "text": "So that's sort of two planks of our philosophy is grounded cognition and reinforcement learning. A third sort of pillar, if you like, of our approach is the use of systems neuroscience. And as a neuroscientist myself, I think this is going to play a very important part of understanding what intelligence is and then trying to recreate that artificially."}, {"start": 932.82, "end": 962.13, "text": "But when I talk about neuroscience, I really want to stress I'm talking about systems neuroscience. And what we mean by that is really the algorithms, the representations, and the architectures the brain uses, rather than the actual low-level, say, synaptic details of how the neural substrate works. So we're really talking about this kind of high level, this computational level, if you like, of how the brain functions. Now, I haven't got time to really go into all the areas that..."}, {"start": 962.13, "end": 988.0, "text": "We're sort of using neuroscience inspiration for. But suffice to say, it's some of the key areas that we're working on. Memory, attention, concepts, planning, navigation, imagination. All these areas that we're pushing hard on now to going beyond the work we did for Atari. And actually, the area of the brain that I studied for my PhD, the hippocampus, which is the center part of the brain here in pink."}, {"start": 988.0, "end": 1013.66, "text": "is actually implicated in many of these sort of capabilities. So it seems like perhaps the notion of creating an artificial hippocampus of some sort which mimics the functionality of the hippocampus might be a good plan. So I haven't got time to go through all of these different areas and the work we're doing here, but I'll just touch on a couple of the most interesting ones."}, {"start": 1013.66, "end": 1043.65, "text": "So one big push that we have at the moment is adding memory to neural networks. And what we really want to do is add very large amounts of controllable memory. So what we've done is created this system which we're sort of dubbing the neural Turing machine. And what effectively it is, is you take a classical computer, you train a recurrent neural network on it from input-output examples, and that recurrent neural network you can think of as like the CPU effectively."}, {"start": 1043.65, "end": 1072.26, "text": "And what we give this recurrent neural network is a huge memory store, a kind of KNN memory store that it can learn to access and control. And this whole system is differentiable from end to end. So the recurrent neural network can learn what to do through gradient descent. And really, that is then all the components of a kind of von Neumann machine that you need, except here, it's all neural and it's all been learned."}, {"start": 1072.26, "end": 1102.67, "text": "So that's why we call it the neural Turing machine, because it has all the aspects you need for a true Turing machine. So here's a little cartoon diagram of what the Turing machine does. And you can think of this kind of input tape and then the CPU, which is this recurrent neural network that actually has LSTMs as part of it. And then it's trying to produce the right output. And then it has this huge memory store to the side that it can learn to read and write elements to, vectors to."}, {"start": 1103.47, "end": 1131.01, "text": "Now, with this kind of system, we can start moving towards symbolic reasoning using these kinds of neural systems, which is really one of the big holy grails of what we want to do. And, of course, there's a classic problem in AI, many unsolved kind of classic problems. One of the problems we apply this neural tree machine to has been inspired by the Schr\u00f6del class of problems, which are these block worlds from the 70s and 80s."}, {"start": 1131.01, "end": 1149.71, "text": "And the idea here is to manipulate the blocks in some way and answer questions about the scene. You know, like put the red pyramid on the green cube or what's next to the blue square. And both manipulate this world and also answer question and answer about it."}, {"start": 1149.78, "end": 1175.1, "text": "Now, we're not ready yet to, neural tuning machines can't scale to the full complexity of the full Shrewdiloo problem. But we have kind of cut it down to a 2D version, Blocksworld version, where we can kind of solve some quite interesting things. So we call this kind of mini Shrewdiloo. And it has aspects of Tower of Hanoi and other problems in it. And the idea here is that you've got this little Blocksworld that you're looking side on."}, {"start": 1175.1, "end": 1203.74, "text": "And all these different colored blocks. And you're given the start configuration here on the left-hand side and the goal configuration you want to reach. And what the system can do is lift one block from one column and put it down on the top of another column. That's the only moves you're allowed to do. And it gets trained through seeing many starting examples and end examples and doing trial and error with reinforcement learning."}, {"start": 1203.74, "end": 1228.67, "text": "And improving itself over time. And then once it's done its training, we then test it on new start positions and goal positions that it's never seen before. And it has to try and solve these problems in an optimal number of moves. So I'm just going to run this little video, which will show you going from that start position on the left to end up on the goal position."}, {"start": 1228.67, "end": 1254.5, "text": "I think this one's about 12 moves. It's actually a pretty hard task to do in an optimal number of moves. It's really hard even for humans to do this. And so now it's solving pretty interesting kind of logic puzzles. Also, what we've been using your true machines to do recently is solve graph problems, which, as you all know, are kind of quite a general class of problems."}, {"start": 1254.5, "end": 1274.03, "text": "And we'll be publishing something pretty impressive, I think, in the later part of this year on this topic to add to our archive paper that we already published last year. Now, we're also experimenting with language as well. And we've incorporated a kind of..."}, {"start": 1274.03, "end": 1302.26, "text": "Cut down version of language into these shroolooloo tasks. And here, the neural Turing machine is reading a set of constraints that are given to it in code that you can see at the bottom of the screen. So here, each of the blocks are numbered. And there are some constraints that you want to satisfy with the goal configuration. So in this case, block three should be down from block five, four up from two, one up from four, and six down from three."}, {"start": 1303.15, "end": 1331.54, "text": "And so it reads this in character by character, remembers these instructions, and then starts executing the actions. And then it solves the puzzle, and this is the end goal, the end position that satisfies all those constraints. Another thing we're moving to now is"}, {"start": 1331.54, "end": 1359.39, "text": "There's still challenges to overcome in Atari, but we're also starting to move towards 3D environments. So we've repurposed the Quake 3 engine and added modifications to it. We call it Labyrinth. And we're starting to tackle all kinds of navigation problems and interesting 3D vision problems within this kind of Labyrinth-like environment. So I'll just roll the video of this agent."}, {"start": 1359.39, "end": 1380.19, "text": "Finding its way through the 3D environment, picking up these green apples, which are rewarding, and then trying to find its way to the exit point. And again, all of this behavior is learned just through the only inputs are the pixel inputs. And it has to learn how to control itself in this 3D environment and find its way around and build maps of the world."}, {"start": 1380.19, "end": 1402.91, "text": "So here, for an agent like that, we're starting to integrate some of these different things together, deep reinforcement learning with memory and 3D vision perception. So as we take this forward, we're kind of thinking as one of our goals over this next year is to kind of create a rat-level AI, an AI agent that's capable of doing all the things a rat can do."}, {"start": 1402.91, "end": 1434.96, "text": "And rats are pretty smart, so they can do quite a lot of things. So we're looking at the rat literature actually for experimental ideas, experimental tests that we can test our AI agent on. So now I want to switch to AlphaGo, which is also part of these big pushes that we're doing into going beyond the Atari work. So one of the reasons we took on AlphaGo is we wanted to see how well these neural network approaches could be meshed with planning approaches."}, {"start": 1435.34, "end": 1461.54, "text": "And Go is really the perfect game to test that out with. So this is the game of Go, for those of you who don't play. This is what a board looks like. It's 19 by 19 grid. And there's two sides, black and white, taking turns. And you can place your stone, your piece, which is called a stone, anywhere on an empty vertex on the board. Now, the history of Go has got a long and storied tradition in Asia."}, {"start": 1461.54, "end": 1488.82, "text": "It's more than 3,000 years old. Confucius wrote about it 2,000 years ago. And he actually talked about Go being one of the four arts you need to master to be a true scholar. So it's really regarded in Asia up there with poetry and calligraphy and art forms. There's 40 million active players today and more than 2,000 professionals."}, {"start": 1488.82, "end": 1506.67, "text": "Who start going to Go school before they're teenagers from around the age of 8, 9 or 10. They go to special Go schools instead of normal schools. And although the rules of Go are incredibly simple, in fact I'm going to teach you how to play Go in two slides in a minute."}, {"start": 1506.67, "end": 1537.54, "text": "It actually leads to profound complexity. One way of just quickly illustrating that is that there are more than 10 to the power 170 possible board configurations. So that's more than there are atoms in the universe by a large margin. So the two rules are, rule one, the capture rule. Stones are captured when they have no free vertices around them. And these free vertices are called liberties. So let's take a look at kind of..."}, {"start": 1537.54, "end": 1563.09, "text": "A position from a mid-game or early part of a Go game. And let's zoom into the bottom right of the board to just illustrate this first rule. So here, you can see this white stone that's surrounded by the three black stones only has one remaining free vertex, one remaining free liberty. So if black was to play there, it would totally surround that white stone and that white stone would be captured and removed from the board."}, {"start": 1563.47, "end": 1593.22, "text": "And actually big groups of stones can be captured in this way, not just one at a time. Whole large groups can be captured if you surround all of their empty vertices. So that's the first rule. The second rule is called the co-rule, and that states that repeated board position is not allowed. So let's imagine we're in this position now, and it's white to play. Now white could capture that black stone by playing here and taking that black stone off the board. So now it's black's move."}, {"start": 1593.22, "end": 1622.26, "text": "And you might be wondering, well, can't black just capture back by replacing that stone and taking white? So what happens if black was to play this? And this is not allowed. Because if black was to play back there and remove the white stone, now you'll see that this position we're in now is identical to the position we started with. So that's not allowed. So that black move is not allowed. Black would have to play somewhere else first to break this symmetry and then can go back and recapture that stone."}, {"start": 1622.83, "end": 1655.26, "text": "And that's it. That's the rules of Go. And the idea of Go is that you obviously want to take your opponent's pieces by surrounding it, but actually the main thing you're trying to do is wall off parts of empty territory on the board. And then at the end of the game, when both players pass, they don't think they can improve their positions any further, you count up the number of territory you've got, and you add the prisoners that you've taken from your opponent, and the person with the most points wins the game."}, {"start": 1655.26, "end": 1679.12, "text": "The rules of Go are simple, but it's pretty much the most profound and elegant game I think that mankind has ever devised. And I say that as a chess player. I think Go is really the pinnacle of perfect information games. It's definitely the most complex game that certainly humans have spent a significant amount of time mastering and play at a very high professional level today."}, {"start": 1680.34, "end": 1705.36, "text": "And because of this huge complexity of Go, it's been an outstanding grand challenge for AI for more than 20 years, especially since the Deep Blue match. And the other interesting thing for us is that, and I'm going to come back to this more in a minute, that if you ask top Go players, they'll tell you that they rely on their intuition a lot to play Go. So Go really requires both intuition and calculation to play well."}, {"start": 1706.16, "end": 1734.26, "text": "And we thought that mastering it, therefore, would involve combining kind of pattern recognition techniques with planning. So why is Go hard for computers to play? Well, the huge complexity means that brute force search is not tractable. And really, that breaks down into two main challenges. Firstly, the search base is really huge. There's a branching factor of more than 200 in an average position in Go."}, {"start": 1734.51, "end": 1764.75, "text": "And the second problem, which is probably an even bigger problem, is that it was thought to be impossible to write an evaluation function to tell the computer system who is winning in a mid-game position. And without that evaluation function, it's very difficult to do efficient search. So I'm just going to unpack these by comparing Go to chess, and you'll see the kind of difference. So in chess, in an average position, there are about 20 possible moves. So the branching factor in chess is 20."}, {"start": 1765.55, "end": 1796.06, "text": "In Go, by contrast, as I just mentioned, it's more like 200. So there's an order of magnitude larger branching factor. Plus, Go games tend to last two to three times longer than chess games. The evaluation function, so why is this so difficult for Go? Well, we still believe, actually, that it's impossible to handcraft a set of rules to tell the system who is winning. So you can't really create a kind of expert system for Go."}, {"start": 1796.06, "end": 1822.53, "text": "for evaluating a Go position. And the reasons are there's no concept of materiality in Go. So in chess, as a first approximation, you can just count up the value of the pieces on each side, and that'll tell you roughly who's winning. You can't do that in Go, because obviously all the pieces are the same. Secondly, Go is a constructive game. So the board starts completely empty, and you build up the position, move by move."}, {"start": 1822.53, "end": 1845.15, "text": "So if you're going to try and evaluate a position halfway through or the beginning of the game, it's very difficult because it involves a huge amount of prediction about what might happen in the future. If you contrast that with chess, which is a kind of destructive game, all the pieces start on the board and actually the game gets simplified as you move towards the end game. The other issue with Go is that..."}, {"start": 1845.15, "end": 1869.3, "text": "It's very susceptible to local changes, very small local changes. So even moving one piece around out of this mass of pieces can actually completely change the evaluation of the position. So Go is really primarily a game about intuition actually rather than calculation. And because the possibility is so huge,"}, {"start": 1869.3, "end": 1894.34, "text": "I think it's kind of at the limit of what humans can actually cope with and deal with and master. And, you know, I've talked to a lot of top Go players now, and when you ask them about when they play a brilliant move, why they played it, they'll just tell you actually quite often that it felt right. And they'll use those words. If you ask a chess grandmaster why they played a particular move, they'll usually be able to tell you exactly the reasons behind that move."}, {"start": 1894.34, "end": 1922.99, "text": "I played this move because I was expecting this. And if that happens, then I'm going to do this. And they'll be able to give you a very explicit plan of why that move was good. And you can see that Go definitely has a history and tradition of being intuitive rather than calculating because it has notions of things like the idea of a divine move. And actually, there are some famous games in history that get names."}, {"start": 1922.99, "end": 1941.06, "text": "And within those games, there are famous moves. And those moves are sometimes named as well. And if you talk to a top Go player, they dream about at one point in their career playing one of these kind of divine moves. A move so profound, it's almost as if it was divinely inspired."}, {"start": 1941.06, "end": 1967.76, "text": "And you can look that up online. There's some really interesting stories from the Edo period in Japan of these kind of incredible games played in front of the Shogun and these divine moves being played, ghost moves. So how did we decide to tackle this kind of intuitive aspect of Go? Well, we turned to deep neural networks. And in fact, what we did is we used actually two deep neural networks."}, {"start": 1968.46, "end": 1995.15, "text": "So I'm just going to take you through the training pipeline here. We started off with human expert data that we downloaded about 100,000 games from internet Go servers of strong amateurs playing each other. And we first of all trained through supervised learning what we called a policy network. And this new deep neural network, what it was trained to do was to mimic the human players. So we gave it a position."}, {"start": 1995.15, "end": 2018.61, "text": "from one of those games. And obviously, we know what the human player played. And we train this network to predict and play the move the human player played. And after a whole bunch of training, we can get pretty reasonably accurate. We can get to about 60% accuracy in terms of predicting the move that the human would play. But obviously, we don't want to just mimic."}, {"start": 2018.61, "end": 2047.46, "text": "How human players play, especially not just amateur players. We want to get better than the human players. So this is where reinforcement learning comes in, where we then iterate through self-play this policy network many millions of times, playing against itself and incrementally improving the weights in that network to slowly increase its win rate. So after millions of games of self-play, this sort of new policy network"}, {"start": 2047.46, "end": 2068.45, "text": "Has about an 80% win rate against the original supervised learnt policy network. Then we freeze that network and we play that network against itself 30 million times. And that generates our new kind of we call gold data set. And we take a kind of position."}, {"start": 2068.45, "end": 2093.3, "text": "Form each of those 30 million games. And obviously, we have the position. And we also know the outcome of the game. We know who finally won, black or white. And then with that much data, we were finally able to crack the holy grail of creating an evaluation function. So we created this second neural network, the value network, which is a learned evaluation function. So it learned to take in"}, {"start": 2093.3, "end": 2116.53, "text": "Board positions and try and actually predict who is winning and by how much. So after all of this training, which is a lot of compute power and training on that, we end up finally with two neural networks. The policy network, which takes a board position coded as a trinary vector."}, {"start": 2116.53, "end": 2142.1, "text": "As an input and the output is a probability distribution over the likelihood of each of the moves in that position. So the green bars here and the height of the green bars on the green board represent the kind of probability mass associated with each of the moves possible from that position. And then the second network is we get this value network here in pink."}, {"start": 2142.1, "end": 2174.56, "text": "And again, you take the board position as an input, but here the output of the network is just a single real number between 0 and 1. And that indicates whether white or black is winning and by how much. So if it was 0, that means white would be completely winning, and 1, black would be totally winning, and 0.5, the position would be about equal. So we take those forwards, but the neural networks are not enough on their own. We also need something to do the planning."}, {"start": 2174.56, "end": 2199.82, "text": "And for that, we turn to Monte Carlo Tree Search to stitch this all together. And it uses the neural networks to make the search more efficient. So I'm just going to show you how the search works here. So imagine that we're in the middle of pondering what to do in a particular position. And imagine that position is at the root node of this tree, represented by the little mini go board here."}, {"start": 2200.11, "end": 2229.42, "text": "And perhaps we've done a few minutes or a few seconds of planning already. So we've already looked at a few different moves represented by the other sort of leaf nodes here. And what you do is you've got two important numbers here. Q is really the current action value of the move, the estimate of how good the move is. And P is this sort of probability of the move from the policy network in terms of how likely it is a human would play that move."}, {"start": 2229.74, "end": 2257.68, "text": "And let's imagine we're following the most promising path at the moment that we've found so far, the bold arrows here that are coming down. And we end up at a node, a position that we haven't looked at so far. So what happens here is we expand the tree. And we do that by first calling the policy network to find out which moves are most probable in this position. So instead of having to look at 200 possible moves,"}, {"start": 2257.68, "end": 2284.54, "text": "You know, all the different possible moves in this position. We just look at the top three or four that the policy network tells us are most likely. And so that expands the tree there. And then once we've expanded the tree, we evaluate the desirability of that path in two ways. One is that we call the value network and that gives us an instant estimate of the desirability of that position."}, {"start": 2284.54, "end": 2309.17, "text": "And we also do a second evaluation routine using Monte Carlo rollouts. So we roll out maybe a few thousand games to the end of the games, and then we back up the statistics of that back to this node. And what we do, what we found is that by combining these two evaluation strategies, we can get a really accurate evaluation of how desirable that position is."}, {"start": 2309.55, "end": 2337.33, "text": "And then, and of course, that's one of the parameters we experiment with is the mixing ratio between the rollouts, what the rollouts are telling us, and what the value network is telling us. And as we improved AlphaGo, we trusted the value networks more and more. So I think now, you know, the Lambda parameter is about 0.8 in favor of trusting the value network. And we started, when we started on this around last summer, we got to, it was about 0.5."}, {"start": 2338.42, "end": 2367.39, "text": "So then once you have that, you back the Q value up the tree. And then once you've run out of time or you allocated time, you basically pick the move that has the highest Q value associated with it. So if we think about what these neural networks are doing then for us in terms of the search, you can think of it in this way. Imagine that this is the search tree from the current position. It's totally intractable. It's really huge."}, {"start": 2367.39, "end": 2398.42, "text": "What we do is we call the policy network to really cut down the width of that search, to narrow that down. And the value network really cuts the depth of the search. So instead of having to search all the way to the end of the game and collect millions of statistics like that to be even reasonably accurate, we can truncate that depth search at any point we like and call the value network. So once we built the AlphaGo system, it was time to evaluate how strong it was and test it out."}, {"start": 2398.42, "end": 2409.15, "text": "So the first thing we did was play it against the kind of commercially best available Go programs out there. The two best ones are Crazy Stone and Zen."}, {"start": 2409.15, "end": 2436.94, "text": "They've won all the recent computer Go competitions in the last few years. And they've reached to about strong amateur levels. So in Go, the ratings are, you start off as this thing called Q, K-Y-U, and you go down in score as an amateur. And then as you get better as a strong amateur, you get a Dan rating, which goes from one Dan to about six or seven Dan. And then finally, you can become professional, and then the Dan ratings start again from one to nine."}, {"start": 2436.94, "end": 2460.05, "text": "So really, these programs were about the strength of a strong amateur, a strong club player. And AlphaGo did incredibly well against them. So it won in the 495 matches we tried, it won all but one, and it could do a 75% win rate against these other programs, even when they were given a four-move head start, which is huge in Go. It's called a four-stone handicap."}, {"start": 2460.05, "end": 2472.58, "text": "And this graph here that I'm showing you is just the single machine version of AlphaGo. And it was even stronger on the distributed version. And these rankings are"}, {"start": 2472.58, "end": 2499.22, "text": "Quite subjective, these Go rankings. So we actually created a numerical ranking, ELO ranking, that's on the y-axis on the left-hand side, which is based on chess ELO ratings and is purely statistical in terms of the win rates of the different programs. And what we found is that a gap of about 200 ELO points or 250 ELO points translates to about an 80% win rate."}, {"start": 2499.22, "end": 2521.84, "text": "And AlphaGo was more than 1,000 ELO points better than the other best programs. And so this was back in October. So this is not the most recent version. And we beat all of these other programs. So it was time to test ourselves against some of the world's top human players. So what we did back in October was challenge"}, {"start": 2521.84, "end": 2550.32, "text": "This lovely guy called Fan Hui, who's now based in France, but was born and grew up in China. He's the current reigning three times European champion. He's a two-down professional, started playing Go at seven, and turned professional in China at age 16. It's very difficult to turn professional in China. So he was a top, top player before moving to France, and now he coaches the national French team. And we challenged him in October, and this is what happened."}, {"start": 2550.32, "end": 2579.22, "text": "After first game, maybe he don't like fight. He like play slowly. So it's why begin the second game, I fight. Do mistakes sometimes. I say, maybe I'm right. It's why for another game, I fight all the time. Now it's complete. Now it's complete. But I lose all my game."}, {"start": 2581.81, "end": 2611.18, "text": "So AlphaGo won 5-0, and much to our kind of surprise, and became the first program to ever be a professional at Go. And if you ask AI experts, even the top programmers of these other programs, even until sort of a year before, they were predicting this moment would be at least another decade away. So it was about a decade earlier than the top experts in the field expected, and certainly a decade earlier than the Go world."}, {"start": 2611.18, "end": 2633.46, "text": "This was the distributed version. And this story ends well, though. He looks distraught here, but we ended up hiring him as a consultant on our team after this, and he joined the side of the programs afterwards. Actually, but one interesting point about this, which is interesting, is that..."}, {"start": 2633.46, "end": 2647.57, "text": "He then came into the office for about a week every month to make sure he was part of our, making sure we weren't overfitting in our self-play by carrying on pitting our wits against him. And he felt that his..."}, {"start": 2647.57, "end": 2672.21, "text": "Play had improved by playing against AlphaGo. And actually, he went from ranked about 600 in the world at the time in October to in January, February, like three, four months later, being ranked 300 in the world. And he'll tell you that it really opened his mind. He said, it freed my mind from the constraints of this 3,000 years of tradition to think in a different way about the game. So it was very interesting."}, {"start": 2672.21, "end": 2695.57, "text": "So again, if you want to read the technical details of this, this is another Nature paper from Cover that was a couple of months ago. And I think it's caused a really big storm in the AI world and the Go world. So then it was time to take our ultimate challenge, which was just a few weeks ago now. We decided to challenge Lisa Dole, who is an absolute legend of the game."}, {"start": 2695.57, "end": 2721.26, "text": "I call him like the Roger Federer of Go. And he's been undisputedly the best player of the past decade, and he's won 18 world titles. And he's also famed for his creative style and creative brilliance. So he was the perfect player for us to pit our wits against. And we played him in early March for a million dollar first prize in Korea."}, {"start": 2721.39, "end": 2736.24, "text": "Now, just before I go to the results, I just want to do a side note on compute power here, which I always get asked about. So we use roughly the same compute power for this match as we did for the FanHui match. So it's around about 50, 60 GPUs worth of compute."}, {"start": 2736.24, "end": 2762.51, "text": "And you might ask, well, why don't we just use more? Well, actually, there's quite a lot of diminishing. There's a sort of asymptotes quite quickly, actually, the strength of the program with more compute power. And one of the reasons is it's actually quite hard to paralyze MCTS algorithms. They work much better and more efficiently if you do them sequentially. And if you batch them across lots and lots of GPUs, you don't actually get that much more effectiveness out of it."}, {"start": 2762.61, "end": 2780.62, "text": "And one measure of that is that the distributed version, surprisingly, probably, I think, maybe to many of you, only wins about 75% of the time against the single machine version. So we played the match, and many of you have seen it. We actually won 4-1. And it was pretty astounding to us because..."}, {"start": 2780.62, "end": 2809.66, "text": "Even the day before the match, they interviewed Lee Sedol, and he was saying he was confident he was going to win 5-0, and the whole Go world thought there was no chance we could win. Obviously, they were looking at the Fan Hui matches and trying to estimate maybe we'd improve 10-20% since then, and that would stand no chance against Lee Sedol. But actually, in the five months that we had between the two matches, the new version of AlphaGo could beat the old version of AlphaGo 99.9% of the time."}, {"start": 2809.66, "end": 2839.33, "text": "So it's pretty astoundingly much stronger. And actually, it was an amazing experience out there, and I'll talk about the cultural significance in a second. But one very nice thing is the president of the Korean Go Association, in the middle there, awarded us an AlphaGo with an honorary 9 Dan certificate. So it was really beautiful. We're going to have that framed up on the wall for its creative play. And I just want to touch on those kind of themes, actually,"}, {"start": 2839.33, "end": 2868.64, "text": "About the creativity and intuition. And that's one of the reasons I explain to you how to play Go. Because I want to just try and explain to you some of the significance of what AlphaGo did. Now chess is really my main game that I play. But I play Go well enough to be able to appreciate what's going on. Now probably the best move that AlphaGo played in the whole five game series. And maybe the Go world will decide to name this move. Is move 37 in game two. And this is the position."}, {"start": 2868.64, "end": 2892.66, "text": "AlphaGo was black, and AlphaGo decided to play here. It's called a shoulder hit move, this move here in red. And I'm going to try and explain to you why this is so amazing, this move, by telling you a little bit about Go. So there's two key lines in Go, the third and the fourth line of the board. That's the critical lines in Go. So here's the third line."}, {"start": 2892.66, "end": 2920.74, "text": "Now, if you play a stone on the third line, what you're really trying to do, you're telling your opponent is, I'm interested in taking territory on the side of the board. That's what a third line move means. A fourth line move, by contrast, is the fourth line. What that means, if you play on the fourth line, is I'm trying to take influence and power into the center of the board. So you're going to try and influence the center of the board and radiate that influence across the board. So that's towards the center."}, {"start": 2920.74, "end": 2950.99, "text": "And the beauty of Go, and I think one of the reasons why it ended up evolving to a 19 by 19 board, is the playing on the third and fourth lines and going for territory or influence is considered to be about perfectly balanced, right? So the territory that you get for playing the third line is about equal to what the opponent gets by playing on the fourth line and getting power and influence into the center. The idea is that that influence that you get and power you get, you store it up for later, and eventually that will give you territory somewhere else in the board."}, {"start": 2951.09, "end": 2966.82, "text": "So that's the classic 3,000 years of history of Go. And yet, AlphaGo played on the fifth line to take influence towards the center of the board. And so this is kind of astounding. It goes against 3,000 years of..."}, {"start": 2966.82, "end": 2996.22, "text": "History of Go. And then just to show you how astounding that was to the Go fraternity, I just want to show you a clip from the commentary, the live commentary. And so we had commentating live. There were lots of commentary channels. There was actually 14 live channels in China. It's all three national TV stations in Korea. But also, we had an English channel via YouTube. And we had this fantastic commentator called Michael Redmond, who's the only Westerner ever to get to Nine Down."}, {"start": 2996.22, "end": 3023.95, "text": "So he's the only English-speaking person to ever get to 9-ban. And look at his reaction to this move 37. So, oh, just to show you, so what turned out was about 50 moves after this move, that move here influenced the fight over in the bottom left corner, right? About 50 moves later. So you can't calculate that because there's too many possibilities. That was the influence of the power of that move."}, {"start": 3023.95, "end": 3063.7, "text": "So this is Michael Redmond seeing this move. He doesn't even know where it is. So I think he means a misclick as opposed to a click miss. So he was thinking that our operator..."}, {"start": 3063.7, "end": 3094.7, "text": "The person actually playing the moves for AlphaGo, Aja Huang, was the lead programmer, had actually entered the move wrong into the machine because it was that surprising a move. And this is what Lisa Dole thought of it. He disappeared to the bathroom for 15 minutes. So that's his empty seat there. No one knew what happened to him. So he just disappeared for 15 minutes. So, you know, maybe it will be called the face-washing move or something because they're usually named after something that happened."}, {"start": 3094.7, "end": 3120.61, "text": "And actually, later when we investigated the statistics behind this, we found that the policy network gave the prior probability of this move as less than 1 in 10,000. So AlphaGo overcame. So it's not just repeating what it's seen in these professional games, because it would never have thought to play this move. Later on, some of the nine damn pros commented, it's not a human move. No human would ever have played this move."}, {"start": 3120.61, "end": 3150.46, "text": "So this is a really kind of original move, if you like. And one thing that we think is going on here is that... No, we can't yet. We need to build more visualization tools to actually do that. We're building that at the moment. It's pretty hard to explain why it's done that for us. So here, in terms of these kind of surprising moves..."}, {"start": 3150.46, "end": 3180.05, "text": "I think this actually shows some kind of originality. And what it might mean is, when I talked to Michael Redman about this, he said that AlphaGo has this very light touch. It doesn't commit to territory early. And what we think is going on is that AlphaGo really likes influence in the center of the board. And it likes it so much, and it's so good at ultimately making that influence pay later on in the game, that it actually thinks fifth line influence is good enough. So this may cause a whole rethink."}, {"start": 3180.05, "end": 3206.19, "text": "in the game of Go as to what's an acceptable trade. Then I must say the other really spectacular move was played by Lee Sedol in Game 4. So we won the first three games, and then Lee Sedol came back strongly because he's an incredible games player. And I've met many of the best games players in the world, Garry Kasparov and others, but I put Lee Sedol at the top of all the games players I've met in terms of his creativity and fighting spirit. And in Game 4..."}, {"start": 3206.19, "end": 3232.4, "text": "He won game four, and he did it by playing this incredible move, move 78 here. I haven't got time to go into why this is so special, but basically, when we looked at the data on this as well, we found that AlphaGo thought the probability of this move was also less than 1 in 10,000. So it was totally unexpected for AlphaGo, and that meant all the pondering and search it had done up to the moves prior to this ended up having to be thrown away."}, {"start": 3232.4, "end": 3255.52, "text": "So it basically had to start again as soon as this move happened. And for some reason, this caused some mis-evaluation in the value net, which we're still investigating what happened there. So cultural impact of this match was huge. We had 280 million viewers. That's more than the Super Bowl. And 60 million viewers just in China for the first game."}, {"start": 3255.52, "end": 3277.71, "text": "We were being stopped in the streets in Korea, and it was pretty crazy. And 35,000 press articles literally every day. It was the front page of all the newspapers in Korea. And the thing I liked most, actually, was that it popularized Go in the West. There was a worldwide shortage of Go boards for the last few weeks after."}, {"start": 3277.71, "end": 3304.37, "text": "It's still now, I think. If you're trying to order a Go board, you might have trouble because of this game, which is fantastic to see. The press coverage was just insane. These are pictures of the press room. It was just like a scrum, a number of live TV cameras, like 50 live TV cameras at the back. It was on all the national TV stations, jumbo screens in the shopping districts. It was pretty crazy. It was amazing to see."}, {"start": 3304.56, "end": 3326.24, "text": "I think for Korea, it was the perfect matchup of they love technology, they love AI, and they love Go. So for them, it was the perfect storm. And this is one interesting thing I want to show is the rate of progress of AlphaGo. So we started this project only about just over 18 months ago. And the progress has been relentless from the beginning."}, {"start": 3326.24, "end": 3350.13, "text": "We found that these techniques, which can improve themselves and you can kind of create more data and then train new versions and then that can create more better, higher quality data. That kind of virtuous cycle has delivered about a one rank improvement per month, which is pretty astounding. And the interesting thing is we haven't really seen any asymptote yet. So we're quite interested to see how..."}, {"start": 3350.13, "end": 3371.18, "text": "How far this can go and what is the sort of optimal play or get near to optimal play in Go? How much further is there to go? And actually, I think most of the Go professionals are really interested in this question as well. And I'm pretty sure that just like with FanHui, when we ultimately hopefully release AlphaGo in some way to the public,"}, {"start": 3371.18, "end": 3404.54, "text": "I think it will improve the standard of Go and bring in whole new ideas. So after the heat of battle, I had a great dinner catch-up with Lisa Doll, who's also an amazing and lovely guy. And we talked about the match, and he told me that it was one of the greatest experiences of his life. And the fact that it had totally, just the five games he played, had made him totally rejuvenated for his passion for Go and the ideas and creativity about what could be done. It's paying a few thousand a day."}, {"start": 3404.54, "end": 3438.62, "text": "It depends on how many machines we use. Potentially. I mean, these pros play several thousand games a year, probably about 1,000, 2,000 when they're training. So it's quite a lot. Plus, they read a lot about all the ancient games. I don't think so, because there are three different schools of Go, the Japanese, the Koreans, and the Chinese. And they're very competitive against each other. And they approach the game differently."}, {"start": 3438.62, "end": 3467.18, "text": "And I think that creative tension has forced them out of local Maxima, I would say. So just to compare Deep Blue with AlphaGo, just to be clear again about the differences. So Deep Blue, again, not to take any away from the immense achievement that it was for its time, absolutely incredible, but it used handcrafted chess knowledge. By contrast, AlphaGo has no handcrafted knowledge. All the knowledge it has is learned from expert games and through self-play."}, {"start": 3467.38, "end": 3489.58, "text": "Deep Blue did full-width search, pretty much looked at all the alternatives, and that's why it needed to crunch 200 million positions per second. By contrast, AlphaGo uses these two neural networks to guide the search in a highly selective manner, and that means we only need to look at 100,000 positions per second to deliver this kind of performance."}, {"start": 3490.9, "end": 3519.97, "text": "So I just want to finish by a couple of words on intuition and creativity. And this may be a little bit controversial, so I'm not saying this is kind of like the full truth of the matter or even fully encompasses on everything to do with intuition and creativity. But I think these are interesting thoughts. So we have to sort of define a little bit what do we mean by intuition. And one way I'd like to, at least the way I think about it for Go, is it's kind of this implicit knowledge that humans have acquired through experience, obviously of playing Go."}, {"start": 3519.97, "end": 3540.02, "text": "But it's not consciously accessible or expressible. So certainly not to communicate to someone else, but not even to themselves. But we know this knowledge is there, and we know it's of very high quality because we can test the knowledge and verify it behaviorally. Obviously, it's the output of the moves that the player plays."}, {"start": 3540.91, "end": 3569.84, "text": "Secondly, what is creativity? And I'm sure everyone in this room has their own pet definition. But I think, again, it definitely encompasses sort of the ability to synthesize the knowledge you have and use that knowledge that you've accumulated to produce novel or original ideas. And I think, certainly, at least within, albeit the constrained domain of Go, I think AlphaGo has pretty clearly demonstrated these two abilities. And obviously, while playing games is..."}, {"start": 3569.84, "end": 3598.86, "text": "A lot of fun and I believe the most efficient way to go about AI development. Obviously that's not the end goal for us. We want to apply the technologies that we've built here as part of AlphaGo that we believe are pretty general purpose. Extend them, use components of them and apply them to have impact on big challenging real world problems. And we're looking at all sorts of areas at the moment like healthcare, robotics and personal assistants."}, {"start": 3599.98, "end": 3626.43, "text": "So I just want to thank the amazing AlphaGo team who did all this incredible work, really incredible engineering and research effort. And also, again, I just want to stress all this work I've just shown you today is really less than a tenth probably of the work that we're doing at DeepMind. And if you're interested in seeing all our publications, they're all on our website, and there's sort of about 70, 80 publications there now of all of our latest work."}, {"start": 3626.43, "end": 3683.49, "text": "And of course, I must mention, if you want to get involved, we are hiring both research scientists and software engineers. Thanks for listening. So, yeah. Yeah, go on. Do you want to use this? Well, for a second. Thank you. So let's have a couple of questions. Anybody? Yeah. Okay. I think the question was, can groups of players together beat AlphaGo?"}, {"start": 3683.49, "end": 3716.83, "text": "Maybe. So that's something that we might play in the future, actually, is a group of top professionals versus AlphaGo. And it'd be quite interesting to see, because it's known that some of these top players are really good at opening or middle game or end game, and you could kind of switch between them, and I'm sure they'd be a lot stronger together. So maybe we'll do that towards the end of the year or next year. Yeah. It's behind you? Yeah. Yeah. Yeah. So it's a huge..."}, {"start": 3716.83, "end": 3735.47, "text": "So, yes, so the question was using visualizations to understand better how AlphaGo works. So we think this, I mean, this is a huge issue with the whole deep learning field, actually, is that how can we better understand these black boxes that are, you know, doing these amazing things, but quite opaquely."}, {"start": 3735.47, "end": 3764.9, "text": "And I think what we need is a whole new suite of analysis tools and statistical tools and visualization tools to do that. And again, I look to my neuroscience background for inspiration. For those of you who do fMRI or that kind of analysis, I think we need the equivalent of kind of SPM for virtual brains. So we actually have a project called Virtual Brain Analytics, which is around building these kinds of tools so that we can better understand what representations these networks are building."}, {"start": 3764.9, "end": 3815.42, "text": "So hopefully in the next year or so, we'll have something much more to say about that. Yeah. Yeah. Yeah. Yeah, so it's a really good question, actually. The question is, can we do away with the supervised learning part and just go all the way from literally random using reinforcement learning up to expert? We plan to do this experiment, actually. So we think it will be fine, but it will take a lot longer to train, obviously, without bootstrapping with the human expert play."}, {"start": 3815.42, "end": 3838.13, "text": "So until now, we've been just concentrating on trying to build the strongest program we can in the fastest time. So we haven't had time to experiment with that. But there are a number of experiments like that that we want to go back to and try. I will say that a very smart master's student from Imperial College in London did do this for chess from scratch, and they got to kind of international master standard."}, {"start": 3838.13, "end": 3878.8, "text": "So it seems like this is definitely sort of possible. And actually, we've hired him now. And so he may be the person that would end up, Matthew Lye, he's called, that may end up looking at this as well. So maybe someone from near the back. Yeah. Sorry? Yeah. Yes. Yes, potentially. So we're thinking about adding learning into that part too. And also..."}, {"start": 3878.8, "end": 3941.09, "text": "You know, maybe there are ways of doing away with some of that Monte Carlo tree search, too. There are other ways of doing that search, kind of more like imagination-based planning. So we're thinking about that as well. Maybe back there? Yeah. So I think the question, if I understand it correctly, is that if agents play games well, is that AI? Is that what you're asking? Well, I mean, obviously, that's our thesis, is that this will work. But I think you have to be careful how you build the AI."}, {"start": 3941.09, "end": 3968.64, "text": "There are many ways you could build AI for games that would not be generalizable. And I think that's been the history. That is what generally for commercial games, which I've also helped make lots of commercial games which have AI in them. And usually the inbuilt AI is special case. Usually it's finite state machines or something for the game. And it utilizes all kinds of game state information that if you were just using perception, you wouldn't have."}, {"start": 3968.64, "end": 3986.08, "text": "So I think you have to be careful that you use games in the right way and you treat the agent really as a virtual robot with all of that entails in terms of what it has access to. And I think as long as you're careful with that, then it's fine."}, {"start": 3986.08, "end": 4015.95, "text": "And one way we enforce this is we have a whole separate team, an evaluation team of amazing programmers, most of them are ex-games programmers, who build the environments and the APIs to the environments and so on. And they're entirely separate from the algorithm development teams. And the only way the AIs can interface with the games is via these very thin APIs. So we know there's no way, even if a researcher was to be lax with this or lax with this, they could access things they're not supposed to, the agents."}, {"start": 4018.67, "end": 4049.81, "text": "So we'll just go around in questions. Yeah, here. Well, we're doing both, actually. So there's self-training in terms of there's self-training producing high-quality data. It's tweaking itself through this deep reinforcement learning. And we're also actively doing tons of research in terms of new architectures or parameters and other things. So it's all of the above. So we really threw everything at it."}]}