{"url": "https://www.youtube.com/watch?v=4cBkGaj1S3Q", "page_url": "https://www.youtube.com/watch?v=4cBkGaj1S3Q", "title": "\uc778\uacf5\uc9c0\ub2a5\uacfc \ubbf8\ub798 \uc54c\ud30c\uace0 \uac1c\ubc1c\uc790 \ub370\ubbf8\uc2a4 \ud558\uc0ac\ube44\uc2a4 \uac15\uc5f0 (Demis Hassabis : Artificial intelligence & Futrue )", "show_or_channel": "irena peabody (KAIST lecture repost)", "date": "2016-03-11", "duration_s": 5351, "type": "talk", "speakers": ["Demis Hassabis"], "confidence": "medium", "notes": "KAIST lecture \"Artificial Intelligence and the Future\" during AlphaGo vs Lee Sedol match, Daejeon; repost by third party.", "people": ["demis-hassabis"], "id": "2016-03-11_irena-peabody-kaist-lect-demis-hass", "audio_duration_s": 5350.1794375, "worker": "jc-w21", "transcribe_s": 234.7, "model": "large-v3", "segments": [{"start": 17.2, "end": 39.95, "text": "So thanks, everyone. It's amazing to be here. And thanks to my friend from MIT inviting me. And I've been wanting to come visit Keist for a long time because I've heard of so much great work that goes on here and all the brilliant students and all the work that you do. So it's very exciting to be here."}, {"start": 39.95, "end": 69.34, "text": "I'm also very excited to be in South Korea in general because of seeing the excitement and energy for AlphaGo and AI research and for the game of Go. So it's a real pleasure and honor to be here. So I wasn't expecting quite so many people. I was expecting it to be a normal academic lecture. So I'm going to talk to you today about really an overview of"}, {"start": 69.34, "end": 98.66, "text": "What DeepMind does. So of course I will talk about AlphaGo and give you some technical details and you can ask me some questions about that. But then also I want to talk to you about the rest of the work that we're doing because AlphaGo is very important. We have an amazing team on AlphaGo of about 15 people. But DeepMind is actually more than 200 people, so scientists. So actually it's only about one-tenth of the work we're doing at DeepMind."}, {"start": 98.66, "end": 129.17, "text": "So let me start by saying, of course, AI is the science of trying to make machines smart. And I'm sure many of you are involved in AI and machine learning here. So DeepMind, as Sang Wang said, was founded in 2010. And we joined forces with Google in 2014. And we did that to try and turbocharge our mission to solve artificial intelligence. So how do we think about DeepMind?"}, {"start": 129.17, "end": 157.81, "text": "Well, we think about it as kind of like an Apollo program effort for AI. So, as I said, there's more than 200 research scientists now. We're based in London, and we're growing all the time. But the other thing we're experimenting with at DeepMind is, of course, we're experimenting on AI and trying to advance AI, but the other big experiment we're doing is on organization itself. How can we organize science?"}, {"start": 157.81, "end": 185.9, "text": "More efficiently and more productively, but still make it very fun and collaborative. So what we've tried to do at DeepMind is try to combine the best of academia and how that should work in an ideal world with the best from the top Silicon Valley startups and how they operate. So from academia, we're taking blue sky thinking, ambitious thinking, interdisciplinary work, which I know is how KAIST works too."}, {"start": 185.9, "end": 217.25, "text": "And then combine that with the energy and the focus and the buzz of the best Silicon Valley startups. And I think we've done a reasonably good job of that so far. And I think that's what gives us some of our unique edge. So DeepMind, what is our mission? So we articulate it normally in two steps. So step one, fundamentally solve intelligence. And then step two, use it to try and solve everything else."}, {"start": 217.25, "end": 248.34, "text": "Step two kind of follows on from step one in terms of what we can use general AI for. So how do we plan to do this? Well, we're interested in building general purpose learning algorithms. So these are algorithms that learn automatically from raw inputs. They're not pre-programmed or handcrafted in any way. So we're only interested in algorithms, building algorithms at DeepMind that can learn for themselves directly from raw data and raw experience. Secondly,"}, {"start": 248.34, "end": 279.81, "text": "We're interested in the idea of generality. So this is one system, a single system or a single set of algorithms that can operate across a wide range of tasks out of the box. Of course, we have an example of general purpose learning algorithms. It's up here. And the brain is the only example we have of that. But we know this is possible. So at DeepMind, we talk about this kind of AI as artificial general intelligence, AGI."}, {"start": 279.81, "end": 309.09, "text": "And the hallmarks of AGI are that it's flexible and adaptive and potentially even inventive. And it's built from the ground up to deal gracefully with the unexpected. If we contrast that with what is mostly called AI these days, which we call narrow AI, usually that type of AI is handcrafted, so the solutions are directly programmed into the machine, and they're special case for a single task."}, {"start": 309.09, "end": 339.22, "text": "And the problem with these kinds of systems is they're very brittle. So if something unexpected happens, they just break. They catastrophically fail. They have no way of dealing with something that the programmers didn't already cater for. So, of course, probably the most famous example still of this kind of narrow AI system is Deep Blue. As you all know, famously beat Garry Kasparov in 1997. But the thing about..."}, {"start": 339.22, "end": 366.46, "text": "Deep Blue was is that it couldn't do anything else. The systems and the program behind Deep Blue couldn't even be used to play a strictly simpler game. Like in the UK, we call it Noughts and Crosses, a very simple game you teach children. And even though Deep Blue had mastered chess, none of the programs and none of the knowledge it had would be useful for any other game, let alone any other type of domain outside of games."}, {"start": 366.46, "end": 396.06, "text": "So I actually came away from this match. I remember this match very well. I was studying at Cambridge. And obviously with my chess background, I was following the match very closely. I actually came away more impressed by Garry Kasparov's mind than with the machine. Because obviously Kasparov could play chess at the same level as the machine, but also do many, many other things that we can all do. So Deep Blue was an immense technical achievement. Of course, it was a huge watershed moment in AI."}, {"start": 396.06, "end": 427.28, "text": "But what we want to try and do is go beyond that. So the way we think about AI at DeepMind is through the glasses or the prism of reinforcement learning. And I know lots of you in the audience will know what reinforcement learning is, but for those of you who don't, I'll just explain it very quickly with this diagram. So if we think about intelligence with this kind of framework. So you start off with the agent system here on the left. And the agent finds itself in some kind of environment."}, {"start": 427.28, "end": 452.32, "text": "which is trying to achieve a goal. And that environment could be real-world environment, in which case the agent is a robot, or it could be a virtual environment, like a game, in which case the agent would be an avatar. And the agent only interacts with the environment in two ways. One way is that it gets observations about the environment through its sensory apparatus."}, {"start": 452.32, "end": 479.55, "text": "So we mostly use vision for the moment, but we could use other sensory modalities and we plan to in the future. And the key thing is that about the system is it only gets incomplete and noisy observations. It doesn't have perfect information about the environment out there. And the job of the agent system is twofold. One is to build as accurate a model as possible of the environment out there."}, {"start": 479.55, "end": 503.84, "text": "Based on these noisy and incomplete observations and update that in real time. And once it has that kind of model of the world, to then use that model to try and select the best action from the set of actions that are available to it at that moment in time, that will get it best towards its goal, its overall goal. And once it's decided what to do, it outputs the action."}, {"start": 503.84, "end": 533.82, "text": "The action gets executed, and then that may or may not make a change to the environment, and then that drives a new observation. So all of this happens in sort of real time. Now, although this diagram looks very simple, it hides a huge amount of technical complexity, of course. And if we could solve everything to do with this diagram, then we know this would be enough for general intelligence. And the reason we know that is because this is how biological systems learn."}, {"start": 533.82, "end": 563.63, "text": "So animals learn using reinforcement learning, and also in our human brains, we also use reinforcement learning as one of the learning techniques our own brains use. And we know this because some very famous experiments in the 90s, actually on monkeys, recording from single neurons, showed that the dopamine neurons actually code for prediction error. So some of the neurons in our brain basically implement a form of reinforcement learning called TD learning."}, {"start": 566.42, "end": 596.59, "text": "The second big principle we had at DeepMind was the idea of grounded cognition. So sometimes this is called embodied cognition. And this is the proposal that a true thinking machine has to be grounded ultimately in a rich sensory motor reality. So it's the only way you can get true grounded thinking. Now normally when people subscribe to this point of view, and maybe many of you here also agree with this,"}, {"start": 596.59, "end": 618.99, "text": "You start working on physical robots because of course physical robots are truly grounded in the real world and in real data stream. But actually the problem is, and I know there's a lot of robotics goes on here and we love robots as an application, but they're quite fiddly to use. They're expensive and they can be quite tricky on the hardware."}, {"start": 618.99, "end": 647.6, "text": "So you can end up spending a lot of your time fixing the motors and the hardware and making the hardware work, and then less time you have to spend on the intelligence side, the software and the intelligence algorithms. And we wanted to focus, we were a small company when we started, and we wanted to focus on the intelligence question and the software side. And we didn't want to be distracted by hardware. So instead of that, partly because of my background making video games, we thought..."}, {"start": 647.6, "end": 676.13, "text": "It would be a really good idea to use games as a platform for developing and testing AI algorithms. Now, games, we've come to believe, are a perfect platform for this for many reasons. Firstly, you can create unlimited training data by just running the game for longer. There's no testing bias. So one of the problems with AI development in the past has been that the people coming up with AI algorithms have also been the people designing the tests."}, {"start": 676.13, "end": 699.3, "text": "And even if you try and be as objective as possible, you end up subconsciously designing tests that your algorithms are kind of well suited to. Of course, here, these games are being created by game designers and for other purposes, obviously to entertain people, not for testing AI algorithms. So we have to deal with real challenging problems. Obviously, in software,"}, {"start": 699.3, "end": 729.23, "text": "Software agents, we can run many thousands, even millions of these agent systems in parallel to develop them. And games, often games have scores, which is quite a convenient way of measuring if your agent system is making incremental progress and becoming smarter. So we like using games as a testing platform, and it's one of the core things we do at DeepMind. So this comes to then the sort of"}, {"start": 729.23, "end": 759.23, "text": "The idea of end-to-end learning agents. And what we mean by that is starting with the raw inputs, so usually that's the pixels on the screen, and going all the way to deciding on an action, which is the whole intelligence stack. And we're interested in that entire problem set, all the way from perception to action. And in fact, we work on all of these different areas."}, {"start": 759.23, "end": 789.04, "text": "Our big innovation, we call it deep reinforcement learning, is combining all of this together. So we want to combine deep learning with reinforcement learning. And this allows reinforcement learning to work at scale. So the first thing we tried this on was Atari games, and old classic Atari games from the 80s. So the reason we chose Atari is because it's the first"}, {"start": 789.04, "end": 817.76, "text": "A sort of gaming platform that was really popular and that has hundreds of different games available for it. Now what we combine the deep learning with the reinforcement learning to do is the deep learning does the perceptual processing and then the reinforcement learning effectively does the decision making. So I'm going to roll a few videos now and before I do I just want to explain to you what it is you're going to see."}, {"start": 817.76, "end": 845.36, "text": "The agents only get the raw pixels as inputs. So there's 30,000 numbers coming in per frame because the Atari screens are 200 by 150 pixels in size. And the goal here is simply to maximize the score. The agents learn everything else from scratch. They're not told what they're controlling, what gets them scores, what the rules of the game. All of that they have to learn from first principles."}, {"start": 845.36, "end": 878.82, "text": "So they don't even know, for example, that pixels next to each other are correlated in time in a video stream. It has to learn all that structure for itself. And then there's the idea of generality again, which is a single system to play all the different games. The same single system with the same hyperparameter settings. So the first video I'm going to show you is the iconic game Space Invaders. So if you're as old as me, you'll remember."}, {"start": 878.82, "end": 903.47, "text": "This game from the 80s. And initially, I'm just going to roll the video now. At the beginning, before the agent has had any training, it's just trying random things. So it's controlling the green rocket at the bottom of the screen, and it's losing its three lives immediately, because obviously it has no idea even what it's controlling, let alone what it should be doing. But then if you leave the program playing the game overnight,"}, {"start": 903.47, "end": 932.16, "text": "For about 20 hours, you come back the next day, and now it's superhuman level at playing the game. So every bullet it fires will hit one of the space invaders. You'll see it's learned the pink mothership that comes across at the top of the screen in a minute is worth the most points. So it does these amazingly accurate shots to do that. And if you remember, space invaders, they get faster as there's less of them. So just watch very carefully the last shot."}, {"start": 932.16, "end": 959.57, "text": "And you'll see that it's predicting where the space invader will be. So it's very cool. So you can see it's built a very accurate model of this data stream that it's seeing. So I'm going to show you a second video now, which is Breakout. It's my favorite Atari video. And I'm going to show you more gradations of the agent getting slowly better over time."}, {"start": 960.14, "end": 986.67, "text": "So in this game, you control the bat at the bottom of the screen, the bat and ball, and you're trying to break through this rainbow-colored wall brick by brick. So you can see the agent here is, after 100 games, it's not very good. It's missing the ball, but it's starting to get the idea it needs to move the bat towards the ball. So now after 300 games, now the agent is getting the ball back almost all the time, even very fast angles, which better than humans can do this."}, {"start": 986.67, "end": 1016.3, "text": "So then we thought, that's pretty good, but let's leave it for another 200 games. So we played 500 games now. And then it discovered, to our surprise, the optimal strategy was to dig a tunnel around the side and then put the ball to the back. It's just amazing accuracy. And so this is the optimal strategy that it invented. And what's very funny is that the designers, the researchers and engineers, they're brilliant engineers and researchers who built this system."}, {"start": 1016.3, "end": 1043.07, "text": "called DQN, but they're not so good at playing Atari games. So they didn't even know that strategy. So they were taught something by the agent system. So this agent, which we call DQN, Deep Q Network, is really a general agent for Atari. Of course, only for Atari games, but within the domain of Atari games, it really is a general artificial intelligence."}, {"start": 1043.07, "end": 1067.3, "text": "So here I'm going to run a medley of different games just very quickly. And you'll see the same agent can play all these very different games, fighter pilot games, one of the very early 3D games. And you can see all of these games have very different rules. Of course, the classic Pong, it's controlling the green bat. It always wins 21-0. Then, you know, submarine games."}, {"start": 1067.3, "end": 1097.97, "text": "Many, many different games, really different ones. And the boxing controls the left-hand boxer. It does some sparring, and then it gets the inbuilt AI to the side, and then it just gets infinite points to win. So it's just worked out how to exploit the system. So this is the first time, really, that there's been any kind of general agent that can play, start with raw experience, raw data, and for itself learn,"}, {"start": 1097.97, "end": 1122.62, "text": "Through trial and error, learn from its mistakes and master quite a challenging set of problems on these Atari games. And if you're interested in this work, you can see in our first Nature paper last year, front cover, that we also released the code. So you can play around with DQN yourselves if you want and apply it to your problems. It's all on the Nature site. So then we moved on from..."}, {"start": 1122.62, "end": 1148.42, "text": "Atari games and 2D games. And now we're working on 3D games. We work on robot simulators. Eventually we'll work on real robots. And of course, as you know, we've been working on Go as well, which I'll talk about in a bit. So here is the same system now working on some 3D games. So we tried it in a driving game. And again, just from the raw pixel inputs, it's learned how to drive around a racetrack."}, {"start": 1148.42, "end": 1162.27, "text": "And overtake cars, drive around 200 kilometers an hour, and do some quite cool overtaking maneuvers that we'll see in a second. And again, just from the pixels on the screen, no other input."}, {"start": 1162.27, "end": 1191.07, "text": "It learns how to drive cars. It can also, we're now up to the games in the late 90s, so we're using the Quake engine and we use it to create maze problems, 3D maze problems. And our agent can find rewards like these apples in the maze and then remember how the maze works and find its way out of the maze to the exit. All just from the pixel inputs."}, {"start": 1191.07, "end": 1220.88, "text": "So of course now we're making more and more complex and sophisticated 3D environments and we'll be publishing more work on that later this year. So I just wanted to say a little word about neuroscience. I don't have time to talk about this too much but my PhD was in neuroscience, my undergrad was in computer science and we try and combine both those two areas at DeepMind. So we're very interested in systems neuroscience and we're quite interested in"}, {"start": 1220.88, "end": 1237.18, "text": "Systems level, so that means the algorithms and the representations and the architectures the brain uses, rather than the specific implementation details of synapses and cortical columns. So it's this kind of higher computational level that we're very interested in."}, {"start": 1237.18, "end": 1260.18, "text": "And the reason we're interested in systems neuroscience is that we think it's very important to look at the brain as a source of inspiration for new algorithms. But probably how those algorithms are going to be implemented in a computer relative to the brain will be different because, of course, silicon is different from wetware carbon. So we don't think that actually..."}, {"start": 1260.18, "end": 1287.23, "text": "Exactly copying the specifics of the implementation details is the right way to go if you're interested in building AI. So the kinds of things we're using neuroscience inspired ideas for is memory, attention, concepts, so kind of abstract knowledge, planning, navigation, and imagination. All these different areas that we would like to imbue our computer systems with the ability to do. And in fact,"}, {"start": 1287.23, "end": 1316.38, "text": "One of the brain areas called the hippocampus that's here in pink in the center of your brain, this is what I studied for my PhD, turns out to be quite critical for a lot of these functions, for memory, for navigation, for concepts and for imagination. So actually, and the hippocampus, obviously the neuroscientists here will know, but those of you who aren't, the hippocampus is quite a different structure from the cortex. It's three-layered instead of six-layered. It's a much more ancient structure. All mammals have..."}, {"start": 1316.38, "end": 1335.97, "text": "And it's very interesting to think about what functions the hippocampus actually provides for the whole brain. So you can think of one of the projects we have going on at DeepMind is to create a kind of artificial hippocampus. So now I'm going to talk about, of course, AlphaGo."}, {"start": 1335.97, "end": 1360.08, "text": "Which is obviously now right in the news. The reason we chose to work on AlphaGo, which I'll explain in a little bit, is that we wanted to see if these kinds of deep reinforcement learning techniques, you know, deep learning and reinforcement learning, could be put together with planning, long-term planning. And Go is actually the perfect testing ground to do that."}, {"start": 1360.08, "end": 1390.03, "text": "Because obviously it's a very intuitive game, but you have to do calculation and planning as well. So I'll assume pretty much because I'm in Korea, all of you know how to play Go. So I'm not going to explain that. And obviously here's the Go board. Of course, as I've talked about many times, and I'm sure you've seen me talk about this in press conferences the last couple of days, it has a hugely rich tradition and history with 40 million players today and 2,000 professionals."}, {"start": 1390.03, "end": 1419.04, "text": "And it's probably one of the most studied games that humans have ever studied, which is fantastic. So that means that the level of challenge is very great. And obviously, you know, just one figure showing how complicated Go is, is that the number of possible board configurations is 10 to the power 170, which is more than the atoms in the universe. And I've been told that the number of possible games that you can play is 10 to the power 700."}, {"start": 1419.04, "end": 1442.67, "text": "So, you know, there's immense numbers. So why go? Why did we decide to take on this challenge a couple of years ago? Well, as I said, probably the most complex game that mankind has ever devised. Certainly that's played professionally and is a perfect information game. It's been an outstanding grand challenge for AI for several decades."}, {"start": 1442.67, "end": 1472.56, "text": "Especially since Deep Blue beat Kasparov. So there have been many projects and efforts on trying to crack the game of Go. Of course, we were very attracted to this because any solution and playing this game, humans require intuition and calculation. So any solution we would come up with would involve combining pattern recognition with planning. So for us, that was the next stage after the Atari games."}, {"start": 1474.32, "end": 1496.35, "text": "So why is Go so hard for computers to play? Well, this immense complexity, this 10 to the power 170, makes brute force search techniques intractable. So you can't just search out everything. And there are two main challenges, really, to writing a good Go program. One is reducing this huge search base to something manageable."}, {"start": 1496.35, "end": 1508.62, "text": "And the second thing is that it was thought to be impossible to write an evaluation function to tell the machine which side is winning in a particular position."}, {"start": 1508.62, "end": 1535.04, "text": "Having been here last couple of days watching the nine Dan professionals commentate on the game and like yesterday's game was obviously very exciting and I'll talk about that in a little bit. Actually a lot of the nine Dan professionals were divided in their opinion about who was winning through most of the game and I didn't realize that even at that level for a close game that's very complicated evaluating who is winning is very tough."}, {"start": 1535.04, "end": 1560.99, "text": "And what was interesting is that AlphaGo was pretty confident, even though we as the team were very nervous, that it was winning. It was doing better from around halfway through the game and increasingly more and more confident. And it's interesting to hear Lee Sedol in the press interview afterwards saying that he also felt that he was behind in the game for most of the game and couldn't catch up, which is..."}, {"start": 1560.99, "end": 1588.88, "text": "Also what AlphaGo thought, but not what all the other nine damn professionals who were commentating thought. So this is very interesting. So let me just illustrate the complexity of this compared to the next most complicated perfect information game, chess. So in chess, which is the one I know well, in an average position, there are maybe around an average of 20 possible moves. So the branching factor in chess is around 20."}, {"start": 1589.17, "end": 1612.8, "text": "In Go, by contrast, in an average position, there's more like 200 moves. So the branching factor is one order of magnitude bigger. So also Go games last for much longer. They're much longer games. So the complexity of the search tree is just really huge. Secondly, there's this issue of intuition again versus calculation."}, {"start": 1612.8, "end": 1642.26, "text": "So I play Go. I'm not very strong, but I play it well enough to understand and appreciate its beauty. And, you know, Go is primarily a game about intuition rather than brute calculation. So even when you ask some very strong players, why did they make this move? They will tell you often that it just felt right. And so their subconscious intuition was telling this is right. So in effect, what they're using is all of their lifelong learning and experience to bring to bear on it."}, {"start": 1642.26, "end": 1663.22, "text": "But they haven't consciously calculated it out in a logical sense in the way, say, a chess player would tell you. They'd be able to tell you exactly why they made the move and the reasons, the calculations they made. Of course, Go also requires calculation. But I think what separates the weaker players from the stronger players is their intuitive ability."}, {"start": 1664.02, "end": 1693.95, "text": "And probably the reason for this, and I had dinner with Lisa Dole before the matches started, and we had a great discussion about games playing and games playing ability. And he confirmed to me that this was the case about intuition, and I think he's a great example of a fantastically talented, intuitive player. You know, the best in the world. And he agreed that because there's so many possibilities in the game of Go, that that's why even top players have to rely on their intuition."}, {"start": 1693.95, "end": 1724.75, "text": "Because the myriad of possibilities is just too large. The other really difficult thing about writing evaluation function compared to chess is that in Go, there's no concept of materiality. So in chess, the queen is worth nine points, the rook is five, the knight is three. So as a first approximation, you can just add up the points on each side and roughly tell who's winning. Of course, you need to do a lot more than that to make a really strong chess program, but that will already get you started."}, {"start": 1724.75, "end": 1743.31, "text": "In Go, this is impossible because obviously all of the pieces are worth the same and it's very unclear in the middle game whose territory is whose. So how do we overcome these two challenges? Well, we turn to deep neural networks and deep learning. So we actually built two neural networks."}, {"start": 1743.31, "end": 1771.38, "text": "So the first one, we started off with about 100,000 games that we downloaded from the internet, Go servers, and of human expert data. But it was a mixture of kind of strong amateur, it was probably like strong amateur level games. And what we used, we trained our first neural network, which we call the policy network, and we trained it in a supervised learning fashion. And what we trained it to do was we took positions out of these games,"}, {"start": 1771.38, "end": 1801.44, "text": "And obviously, we knew what the human expert player did, which move they played next. And we trained this policy network to predict and mimic what the human player did in that position. Then once that got reasonably good, we got the supervised learning policy network to play itself many thousands, even millions of times. And we use reinforcement learning now to, instead of just mimic the human experts,"}, {"start": 1801.44, "end": 1828.58, "text": "Actually, what we wanted to do was play better than the human experts, certainly the amateur level players. And so what we did was we optimized during this self-play for win rate, which is actually ultimately what you want a computer program to do. You're not interested in mimicking the right moves. You actually want it to increase its winning rate. And so this sort of refinement of the policy network with reinforcement learning"}, {"start": 1828.58, "end": 1853.58, "text": "We created a new policy network that could beat the original one 80% of the time. So it was a lot stronger. Then we froze that network, and we got that to play itself 30 million more times. And that generated our final data set, which we called our gold data set, of 30 million positions, one position from each game."}, {"start": 1853.58, "end": 1882.26, "text": "And of course now we had this database of more games than probably ever been played. And we were able to now try and attempt to create this fabled evaluation function. And what we did is we had 30 million positions and we also knew the end results of those games. So we used that as the training set to train this new neural network, the value network here in pink, to actually make a prediction about who was winning."}, {"start": 1882.26, "end": 1894.61, "text": "Who is going to ultimately win from any position? So after all of that training, we end up with two neural networks. The policy network here in green that"}, {"start": 1894.61, "end": 1923.04, "text": "It takes a board position as an input, and the output is a probability distribution over the likelihood of the moves from that position. So the green board that you can see here on the top, the size of the green bars is the probability that AlphaGo assigns to the likelihood of that particular move at that particular board position. So the higher the green bar, the more probable. And the second network we get is the value network, which also takes a board position in as an input."}, {"start": 1923.04, "end": 1943.57, "text": "And then the output here is a single number between 0 and 1, a real number, and 0 indicating white is winning completely, 1 would be indicating black is winning completely, and 0.5 would mean even game. And this is really the key to the power of the strength of AlphaGo."}, {"start": 1944.18, "end": 1974.66, "text": "So now we have those two neural networks. We can combine that with Monte Carlo tree search and also a rollout policy to create the final AlphaGo system. So I'm just going to explain very quickly with this diagram how that works. So let's imagine that AlphaGo is thinking about making a move. So imagine it's in that top board position represented by the node there, the small little Go board. And it's done. Maybe we're halfway through the search."}, {"start": 1974.66, "end": 1995.1, "text": "And it's done a bit of searching. And you can see here that the Q value is the estimated value of that particular move from that position. And we want to follow down the most promising path, so where the Q values are highest. And you can see that the alpha go will follow the tree."}, {"start": 1995.1, "end": 2022.29, "text": "Down to a position where it's never considered it so far from this particular position. So you can see that here is the left-hand arrow at the bottom, the black arrow. Now, it now has to create a new part of the tree. So it calls the policy network to expand the tree at this point. So the green network I was describing. But we don't add in all the possible moves."}, {"start": 2022.29, "end": 2049.36, "text": "From that position, we only add in the top three or four most probable moves that the policy network is saying are most likely. So that really cuts down the size of the search tree. So we expand the search tree, this node in that fashion. Then once we've expanded the search tree, we now call the second neural network, the value network, to evaluate that new position in the tree, that new node."}, {"start": 2050.13, "end": 2078.86, "text": "We also combine it with a second way of evaluating the position, the more traditional way, which is called the rollout policy, which is actually to roll out games all the way to the end of the game and then collect statistics of how many wins and losses there were for each side. So it's kind of two different ways of evaluating the position. The value net, which instantaneously evaluates the position, and then the rollout policy that does a kind of statistical analysis of the position."}, {"start": 2078.86, "end": 2105.89, "text": "And actually we found AlphaGo is stronger if you combine those two different evaluations together. And we mix that together to give a final evaluation. And once we've done that, we then back up these new Q values, these new evaluations, back up the tree. And then when AlphaGo has run out of time and it needs to make a move, then it basically picks the move."}, {"start": 2105.89, "end": 2134.9, "text": "That has the most promising Q value associated with it. So let me just explain now with this diagram how that helps with this combinatorial problem that Go has. So imagine that this is the search tree from a position in a Go game. Obviously, it's this huge search tree of possibilities. And what you can think of is the policy network."}, {"start": 2134.9, "end": 2164.8, "text": "Actually cuts down the breadth of the search. So instead of searching all these 200 moves from each position, we just search the top three or four. And then you can think of the value network as reducing the depth of the search. So instead of having to search all the way to the end of the game to find out who won and doing that millions of times, at any stage, we can truncate the search and call the value network to get an estimate of how well AlphaGo is doing in that situation."}, {"start": 2164.8, "end": 2193.1, "text": "So now you can see with these two neural networks, we've completely reduced the search space down to something much more tractable. So how did we evaluate AlphaGo once we built it? Well, sort of last summer, actually last spring, we started evaluating it against the other top programs that are available. And actually the best programs, commercial programs, are called Crazy Stone and Zen."}, {"start": 2193.1, "end": 2220.83, "text": "And they're both around five Dan, six Dan amateur level. And when we ran AlphaGo against these programs, AlphaGo won 494 out of 495 games. And in fact, it has a 75% winning rate against Crazy Stone and Zen, even if it gives them a four stone handicap, which is pretty huge. And obviously, AlphaGo is even stronger on distributed machine."}, {"start": 2220.83, "end": 2249.55, "text": "So the gap between AlphaGo and the next best programs is more than 1,000 ELO points. Actually, this is a lot larger now because this is the AlphaGo version from October that I'm talking about here. So then, as you all know, we beat all the other programs very convincingly. So now it was time to test against professional human players."}, {"start": 2249.55, "end": 2278.18, "text": "So we called up our esteemed opponent, who has now become a friend of ours since the match, Fan Hui, who is the European champion of the last three years. And he's written books about Go and he coaches the French national team. He grew up in China and became professional there. And he's Tu Dam Pro. And in October, we played Fan Hui in a match and we won, surprisingly to us, 5-0."}, {"start": 2278.18, "end": 2307.89, "text": "So AlphaGo became the first program ever to beat a professional player in an even game under tournament conditions. And this was very surprising to the machine learning and AI world, and I think the Go world. It was more than a decade earlier than many experts, even expert programmers from programming other Go programs were expecting. So again, this was a front cover Nature paper."}, {"start": 2307.89, "end": 2336.03, "text": "A couple of months ago. So if you're interested in the details of how it all works, it's all detailed out there in the paper. And it created quite a big stir in the academic circles in terms of downloads and other things. So now, as you all know, we're here. It's the final kind of ultimate challenge. I think playing Lisa Doll,"}, {"start": 2336.03, "end": 2360.64, "text": "And in this five-game match. Now, of course, we're currently 2-0 up, but the games have been very close, very interesting, and I think anything can happen still. That was my prediction before we came in it. We knew that AlphaGo was quite strong because in our internal tests, we test the programs against older versions of it itself."}, {"start": 2360.64, "end": 2377.6, "text": "So the new version of AlphaGo has played hundreds and thousands of games against many different older versions of AlphaGo. And because of that, we can estimate how strong the new one is because we know how strong the October one was because we calibrated that against FanHui."}, {"start": 2377.6, "end": 2394.78, "text": "And we know how much better Lisa Doll is than Fan Hui. And we know how much better our current program is, better than October. So we are quite confident about the intersection of that. But the problem is, the reason there's high uncertainty for us is that..."}, {"start": 2394.78, "end": 2423.73, "text": "Obviously, there can be some problem of overfitting because they're all different versions of AlphaGo. So we try and keep diversity in the tournament pool, but ultimately they're still based off of the same kind of techniques, even though they have different hyperparameter settings and other things. We also try to, for example, make sure that our old versions are kept in the pool and they're given high handicaps. So it's sort of many challenges for the new version of AlphaGo."}, {"start": 2423.73, "end": 2450.5, "text": "So we can't be sure until we play these matches that there wasn't some whole area or some type of position that for some reason AlphaGo never gets into when it plays itself. And newer versions can beat the older versions, but they're still not experiencing this particular type of position. So we don't think there are any holes like that, but that's why we wanted to play someone as amazing as Lisa Dole, who is..."}, {"start": 2450.5, "end": 2470.13, "text": "Fame for his creativity and fighting style. And so if anyone can find the weaknesses, it will be him. And we wanted to know if there was some big weaknesses that we didn't know about. Because of course, we can't evaluate it anymore. It's way too strong. It's been way too strong for us for more than a year."}, {"start": 2470.13, "end": 2489.09, "text": "In fact, we have one strong Go player on our team, Aja Huang, who is sixth down amateur, but it was able to beat him from, you know, more than about a year ago. So we have no real way of evaluating this ourselves. I've spent many nights..."}, {"start": 2489.09, "end": 2518.64, "text": "Looking through AlphaGo games against AlphaGo. And it looks amazing to me. But some unbelievably beautiful games. But I'm not an expert player, so I can't say for sure. So it's been a real honor and privilege to be able to play Lee Sedol and get to know him. And I wish him the best of luck in the next three games. No, I mean that."}, {"start": 2518.86, "end": 2544.85, "text": "And so I want to just mention about compute power that we're using because I think I got quite a lot of questions on the internet about this. We're actually using roughly the same compute power as we use for the Fan Hui match. So it isn't some crazy amount of compute power. It's about the equivalent of sort of 50 GPUs, 100 GPUs, that kind of level."}, {"start": 2544.85, "end": 2553.62, "text": "But I want to just say that the reason that we're not using more computer power is because you actually get a lot of diminishing returns with compute power."}, {"start": 2553.62, "end": 2580.5, "text": "Because it's, for example, very hard to paralyze Monte Carlo tree search. It's known to be very hard to paralyze. So you get really diminishing returns for each extra GPUs you add. And actually, we drew those graphs. If you look in the supplementary materials of our Nature paper, you'll see that we drew the performance curves of adding more GPUs. How much does it improve? And actually, you basically asymptote."}, {"start": 2580.72, "end": 2599.65, "text": "So it really is an algorithmic advance, not a hardware issue. And just probably quite surprising to most people, which is why I want to say it more clearly, is that the distributed version of AlphaGo is only 70% win rate against the single machine version of AlphaGo, right?"}, {"start": 2599.65, "end": 2629.34, "text": "So it's only about 200 ELO point difference. So it's maybe one or two Dan rankings weaker. So it's still very, very strong. Of course, for a match like this, we want to use the strongest version we have, which is the distributed version. But the single machine version is also incredibly strong. So how does AlphaGo differ from Deep Blue?"}, {"start": 2629.34, "end": 2652.64, "text": "You know, I talked about Deep Blue at the beginning and the immense achievement that that was. But AlphaGo, really, for us, you know, we're very pleased with the performance of AlphaGo and we're very proud of how it's worked. And I'm super proud of the team, amazing team that built it. But for us, it's not so important just the achievement of winning."}, {"start": 2652.64, "end": 2674.18, "text": "Against the top professional. It's also actually more important to us is how we did it, what the methods that we used. And here's the sort of differences. Deep Blue used handcrafted chess knowledge distilled by the programmers from talking to chess grandmasters. AlphaGo, by contrast, is knowledge learned from expert games and self-play."}, {"start": 2674.18, "end": 2701.78, "text": "Deep Blue used full-width search. AlphaGo obviously uses the neural networks to do very selective, highly selective search. And you can see that in this quite surprising figure, I think, is that even though AlphaGo is, you know, we're 20 years on from Deep Blue, so obviously machines have got a lot faster, but Deep Blue was analyzing and looking at 200 million positions per second, whereas AlphaGo..."}, {"start": 2701.78, "end": 2724.27, "text": "Only looks at 100,000 positions per second. Much, much less. And that's because it's more human-like in the way that it tries to play. So, of course, human players, top players, search much, much less than this as well. Maybe a few hundred moves. But they use their enormous knowledge."}, {"start": 2724.27, "end": 2755.12, "text": "And kind of experience, which is coded in the neural networks of their minds to actually do good judgment and intuition about what moves are worth looking at and what kinds of positions are good. And AlphaGo, whilst definitely not as efficient as a human, is more towards that end of the spectrum. So I'm just going to end now by talking about a few bigger picture things. And of course, playing games is extremely fun."}, {"start": 2755.12, "end": 2775.15, "text": "It's great for development of the algorithms, and it's very exciting, amazingly exciting when we have matches like this. But the motivation for us at DeepMind is not just to play games, of course. We want to use these techniques we've developed and apply them to all kinds of important real-world problems."}, {"start": 2775.15, "end": 2803.5, "text": "So the things we're looking at are healthcare applications, robotics and control of robots, and smartphone assistance, making them more intelligent and smarter on your mobile phone. And we hope that we're going to start working on these things soon, and we hope that the technologies and the methods we've been developing for AlphaGo, when we extend them in the future, they'll be used in all these kinds of areas."}, {"start": 2804.88, "end": 2830.83, "text": "So let me just talk about the philosophical level of why I've worked on AI for more than 20 years, and I believe it's one of the most important problems we can work on. I think that two of the big issues facing us as a society today are information overload. We're just deluged with data, both as users and as scientists, all the way from important things like genomics and physics."}, {"start": 2830.83, "end": 2844.35, "text": "So much data to things like entertainment where there are more TV channels than we can comprehend. So all these things mean that they have too much information for us to process."}, {"start": 2844.35, "end": 2872.26, "text": "Then there's also system complexity. So the systems, the kind of systems we would like to understand better and master as a society are incredibly complex themselves. So climate, disease, energy, macroeconomics, all of these things are incredibly complicated systems. And so I think of AI as a kind of meta solution to all these other problems. If we can make AI work in this kind of general way, then we should be able to apply it to help"}, {"start": 2872.26, "end": 2902.02, "text": "Human scientists and clinicians and researchers to advance their areas more quickly. So for me, the dream is to use this kind of AI we're developing to make AI scientists or AI-assisted science possible. And just a quick word about ethics and responsibility. As with all powerful new technologies, and AI is no exception, they must be used ethically and responsibly. I think that technology is kind of neutral in and of itself."}, {"start": 2902.02, "end": 2930.32, "text": "I think that trying to build AI in this neuroscience-inspired way, in this general way, and then comparing it to the human mind, will maybe allow us to better understand some of the mysteries of our own minds. Things like what dreaming is, what the nature of creativity is, and perhaps even start getting towards being able to answer the question of what is consciousness. So I just want to end by thanking my amazing AlphaGo team."}, {"start": 2930.32, "end": 2986.83, "text": "You know, all the people responsible for that. And then to also mention that we're, of course, hiring. And if you want to work on these kinds of things, please feel free to apply. Well, as I said, probably the most complex game that mankind has ever devised. Certainly that's played professionally and is a perfect information game. It's been an outstanding grand challenge for AI for several decades."}, {"start": 2986.83, "end": 3016.72, "text": "Especially since Deep Blue beat Kasparov. So there'd be many projects and efforts on trying to crack the game of Go. Of course, we were very attracted to this because any solution and playing this game, humans require intuition and calculation. So any solution we would come up with would involve combining pattern recognition with planning. So for us, that was the next stage after the Atari games."}, {"start": 3018.45, "end": 3040.51, "text": "So why is Go so hard for computers to play? Well, this immense complexity, this 10 to the power 170, makes brute force search techniques intractable. So you can't just search out everything. And there are two main challenges, really, to writing a good Go program. One is reducing this huge search base to something manageable."}, {"start": 3040.51, "end": 3052.78, "text": "And the second thing is that it was thought to be impossible to write an evaluation function to tell the machine which side is winning in a particular position."}, {"start": 3052.78, "end": 3079.2, "text": "Having been here last couple of days, watching the nine Dan professionals commentate on the game, and like yesterday's game was obviously very exciting, and I'm going to talk about that in a little bit. Actually, a lot of the nine Dan professionals were divided in their opinion about who was winning through most of the game. And I didn't realize that even at that level, for a close game that's very complicated, evaluating who is winning is very tough."}, {"start": 3079.2, "end": 3105.15, "text": "And what was interesting is that AlphaGo was pretty confident, even though we as the team were very nervous, that it was winning. It was doing better from around halfway through the game and increasingly more and more confident. And it's interesting to hear Lee Sedol in the press interview afterwards saying that he also felt that he was behind in the game for most of the game and couldn't catch up, which is..."}, {"start": 3105.15, "end": 3133.04, "text": "Also what AlphaGo thought, but not what all the other nine damn professionals who were commentating thought. So this is very interesting. So let me just illustrate the complexity of this compared to the next most complicated perfect information game, chess. So in chess, which is the one I know well, in an average position, there are maybe around an average of 20 possible moves. So the branching factor in chess is around 20."}, {"start": 3133.33, "end": 3156.93, "text": "In Go, by contrast, in an average position, there's more like 200 moves. So the branching factor is one order of magnitude bigger. So also Go games last for much longer. They're much longer games. So the complexity of the search tree is just really huge. Secondly, there's this issue of intuition again versus calculation."}, {"start": 3156.93, "end": 3177.31, "text": "So I play Go. I'm not very strong, but I play it well enough to understand and appreciate its beauty. And, you know, Go is primarily a game about intuition rather than brute calculation. So even when you ask some very strong players, why did they make this move? They will tell you often that it just felt right."}, {"start": 3177.31, "end": 3207.38, "text": "And so their subconscious intuition was telling them this is right. So in effect, what they're using is all of their lifelong learning and experience to bring to bear on it. But they haven't consciously calculated it out in a logical sense in the way, say, a chess player would tell you. They'd be able to tell you exactly why they made the move and the reasons, the calculations they made. Of course, Go also requires calculation. But I think what separates the weaker players from the stronger players is their intuitive ability."}, {"start": 3208.18, "end": 3238.1, "text": "And probably the reason for this, and I had dinner with Lisa Dole before the matches started, and we had a great discussion about games playing and games playing ability. And he confirmed to me that this was the case about intuition, and I think he's a great example of a fantastically talented, intuitive player. You know, the best in the world. And he agreed that because there's so many possibilities in the game of Go, that that's why even top players have to rely on their intuition."}, {"start": 3238.1, "end": 3268.96, "text": "Because the myriad of possibilities is just too large. The other really difficult thing about writing evaluation function compared to chess is that in Go, there's no concept of materiality. So in chess, the queen is worth nine points, the rook is five, the knight is three. So as a first approximation, you can just add up the points on each side and roughly tell who's winning. Of course, you need to do a lot more than that to make a really strong chess program, but that will already get you started."}, {"start": 3268.96, "end": 3287.46, "text": "In Go, this is impossible because obviously all of the pieces are worth the same and it's very unclear in the middle game whose territory is whose. So how do we overcome these two challenges? Well, we turn to deep neural networks and deep learning. So we actually built two neural networks."}, {"start": 3287.46, "end": 3315.54, "text": "So the first one, we started off with about 100,000 games that we downloaded from the internet, Go servers, and of human expert data. But it was a mixture of kind of strong amateur, it was probably like strong amateur level games. And what we used, we trained our first neural network, which we call the policy network, and we trained it in a supervised learning fashion. And what we trained it to do was we took positions out of these games,"}, {"start": 3315.54, "end": 3336.5, "text": "And obviously, we knew what the human expert player did, which move they played next. And we trained this policy network to predict and mimic what the human player did in that position. Then once that got reasonably good, we got the supervised learning policy network to play itself."}, {"start": 3336.5, "end": 3357.97, "text": "Many thousands, even millions of times. And we use reinforcement learning now to, instead of just mimic the human experts, actually what we wanted to do was play better than the human experts, certainly the amateur level players. And so what we did was we optimized during this self-play for win rate."}, {"start": 3357.97, "end": 3387.39, "text": "Ultimately, which is actually ultimately what you want a computer program to do. You're not interested in mimicking the right moves. You actually want it to increase its winning rate. And so this sort of refinement of the policy network with reinforcement learning created a new policy network that could beat the original one 80% of the time. So it was a lot stronger. Then we froze that network and we got that to play itself 30 million more times."}, {"start": 3387.39, "end": 3410.45, "text": "And that generated our final data set, which we called our goal data set, of 30 million positions, one position from each game. And, of course, now we had this database of more games than, you know, probably ever been played. And we were able to now try and attempt to create this fabled evaluation function."}, {"start": 3410.45, "end": 3438.75, "text": "And what we did is we had 30 million positions, and we also knew the end results of those games. So we used that as the training set to train this new neural network, the value network, here in pink, to actually make a prediction about who was winning, who was going to ultimately win from any position. So after all of that training, we end up with two neural networks. The policy network here in green that..."}, {"start": 3438.75, "end": 3468.08, "text": "It takes a board position as an input, and the output is a probability distribution over the likelihood of the moves from that position. So a model is possible of the environment out there based on these noisy and incomplete observations and update that in real time. And once it has that kind of model of the world, to then use that model to try and select the best action from the set of actions that are available to it at that moment in time that will get it"}, {"start": 3468.08, "end": 3495.92, "text": "Best towards its goal, its overall goal. And once it's decided what to do, it outputs the action, the action gets executed, and then that may or may not make a change to the environment, and then that drives a new observation. So all of this happens in sort of real time. Now, although this diagram looks very simple, it hides a huge amount of technical complexity, of course. And if we could solve..."}, {"start": 3495.92, "end": 3516.03, "text": "Everything to do with this diagram, then we know this would be enough for general intelligence. And the reason we know that is because this is how biological systems learn. So animals learn using reinforcement learning and also in our human brains, we also use reinforcement learning as one of the learning techniques our own brains use."}, {"start": 3516.03, "end": 3549.17, "text": "And we know this because some very famous experiments in the 90s, actually on monkeys, recording from single neurons, showed that the dopamine neurons actually code for prediction error. So some of the neurons in our brain basically implement a form of reinforcement learning called TD learning. The second big principle we had at DeepMind was the idea of grounded cognition. So sometimes this is called embodied cognition."}, {"start": 3549.17, "end": 3578.38, "text": "proposal that a true thinking machine has to be grounded ultimately in a rich sensory motor reality. So it's the only way you can get true grounded thinking. Now, normally, when people subscribe to this point of view, and maybe many of you here also agree with this, you start working on physical robots, because of course, physical robots are truly grounded in the real world and in real data stream."}, {"start": 3578.38, "end": 3591.15, "text": "But actually, the problem is, and I know there's a lot of robotics goes on here, and we love robots as an application, but they're quite fiddly to use. They're expensive, and they can be quite tricky on the hardware."}, {"start": 3591.15, "end": 3619.74, "text": "So you can end up spending a lot of your time fixing the motors and the hardware and making the hardware work, and then less time you have to spend on the intelligence side, the software and the intelligence algorithms. And we wanted to focus, we were a small company when we started, and we wanted to focus on the intelligence question and the software side. And we didn't want to be distracted by hardware. So instead of that, partly because of my background making video games, we thought..."}, {"start": 3619.74, "end": 3648.29, "text": "It would be a really good idea to use games as a platform for developing and testing AI algorithms. Now, games, we've come to believe, are a perfect platform for this for many reasons. Firstly, you can create unlimited training data by just running the game for longer. There's no testing bias. So one of the problems with AI development in the past has been that the people coming up with AI algorithms have also been the people designing the tests."}, {"start": 3648.29, "end": 3671.42, "text": "And even if you try and be as objective as possible, you end up subconsciously designing tests that your algorithms are kind of well suited to. Of course, here, these games have been created by game designers and for other purposes, obviously to entertain people, not for testing AI algorithms. So we have to deal with real challenging problems. Obviously, in software,"}, {"start": 3671.42, "end": 3701.36, "text": "Software agents, we can run many thousands, even millions of these agent systems in parallel to develop them. And games, often games have scores, which is quite a convenient way of measuring if your agent system is making incremental progress and becoming smarter. So we like using games as a testing platform, and it's one of the core things we do at DeepMind. So this comes to then the sort of"}, {"start": 3701.36, "end": 3731.38, "text": "The idea of end-to-end learning agents. And what we mean by that is starting with the raw inputs, so usually that's the pixels on the screen, and going all the way to deciding on an action, which is the whole intelligence stack. And we're interested in that entire problem set, all the way from perception to action. And in fact, we work on all of these different areas."}, {"start": 3731.38, "end": 3761.22, "text": "Our big innovation, we call it deep reinforcement learning, is combining all of this together. So we want to combine deep learning with reinforcement learning. And this allows reinforcement learning to work at scale. So the first thing we tried this on was Atari games, and old classic Atari games from the 80s. So the reason we chose Atari is because it's the first"}, {"start": 3761.22, "end": 3789.89, "text": "A sort of gaming platform that was really popular and that has hundreds of different games available for it. Now, what we combine the deep learning with the reinforcement learning to do is the deep learning does the perceptual processing and then the reinforcement learning effectively does the decision making. So I'm going to roll a few videos now and before I do, I just want to explain to you what it is you're going to see."}, {"start": 3789.89, "end": 3817.52, "text": "The agents only get the raw pixels as inputs. So there's 30,000 numbers coming in per frame because the Atari screens are 200 by 150 pixels in size. And the goal here is simply to maximize the score. The agents learn everything else from scratch. They're not told what they're controlling, what gets them scores, what the rules of the game. All of that they have to learn from first principles."}, {"start": 3817.52, "end": 3850.98, "text": "So they don't even know, for example, that pixels next to each other are correlated in time in a video stream. It has to learn all that structure for itself. And then there's the idea of generality again, which is a single system to play all the different games. The same single system with the same hyperparameter settings. So the first video I'm going to show you is the iconic game Space Invaders. So if you're as old as me, you'll remember."}, {"start": 3850.98, "end": 3875.63, "text": "This game from the 80s. And initially, I'm just going to roll the video now. At the beginning, before the agent has had any training, it's just trying random things. So it's controlling the green rocket at the bottom of the screen, and it's losing its three lives immediately, because obviously it has no idea even what it's controlling, let alone what it should be doing. But then if you leave the program playing the game overnight,"}, {"start": 3875.63, "end": 3904.34, "text": "For about 20 hours, you come back the next day, and now it's superhuman level at playing the game. So every bullet it fires will hit one of the space invaders. You'll see it's learned the pink mothership that comes across at the top of the screen in a minute is worth the most points. So it does these amazingly accurate shots to do that. And if you remember, space invaders, they get faster as there's less of them. So just watch very carefully the last shot."}, {"start": 3904.34, "end": 3950.35, "text": "And you'll see that it's predicting where the space invader will be. So it's very cool. So you can see it's built a very accurate model of this data stream that it's seeing. So I'm going to show you a second video now, which is Breakout. So thanks, everyone. It's amazing to be here. And thanks to Sang Wan, my friend from MIT."}, {"start": 3950.35, "end": 3978.14, "text": "Thank you for inviting me and I've been wanting to come visit KAIST for a long time because I've heard of so much great work that goes on here and all the brilliant students and all the work that you do. So it's very exciting to be here. I'm also very excited to be in South Korea in general because of seeing the excitement and energy for AlphaGo and AI research and for the game of Go. So it's a real pleasure and honor to be here."}, {"start": 3978.14, "end": 4007.26, "text": "So I wasn't expecting quite so many people. I was expecting it to be a normal academic lecture. So I'm going to talk to you today about really an overview of what DeepMind does. So of course I will talk about AlphaGo and give you some technical details and you can ask me some questions about that. But then also I want to talk to you about the rest of the work that we're doing because"}, {"start": 4007.26, "end": 4033.68, "text": "AlphaGo, it's very important. We have an amazing team on AlphaGo of about 15 people. But DeepMind is actually more than 200 people, so scientists. So actually, it's only about one-tenth of the work we're doing at DeepMind. So let me start by saying, of course, AI is the science of trying to make machines smart. And I'm sure many of you are involved in AI and machine learning here."}, {"start": 4034.67, "end": 4059.39, "text": "So DeepMind, as Sang Wang said, was founded 2010, and we joined forces with Google in 2014. And we did that to try and turbocharge our mission to solve artificial intelligence. So how do we think about DeepMind? Well, we think about it as kind of like an Apollo program effort for AI."}, {"start": 4059.39, "end": 4087.09, "text": "So, as I said, there's more than 200 research scientists now. We're based in London, and we're growing all the time. But the other thing we're experimenting with at DeepMind is, of course, we're experimenting on AI and trying to advance AI, but the other big experiment we're doing is on organization itself. How can we organize science more efficiently and more productively, but still make it very fun and collaborative?"}, {"start": 4087.09, "end": 4110.14, "text": "So what we've tried to do at DeepMind is try to combine the best of academia and how that should work in an ideal world with the best from the top Silicon Valley startups and how they operate. So from academia, we're taking blue sky thinking, ambitious thinking, interdisciplinary work, which I know is how KAIST works too."}, {"start": 4110.14, "end": 4140.0, "text": "And then combine that with the energy and the focus and the buzz of the best Silicon Valley startups. And I think we've done a reasonably good job of that so far. And I think that's what gives us some of our unique edge. So DeepMind, what is our mission? So we articulate it normally in two steps. So step one, fundamentally solve intelligence. And then step two, use it to try and solve everything else."}, {"start": 4140.0, "end": 4170.86, "text": "You know, we think that step two kind of follows on from step one in terms of what we can use general AI for. So how do we plan to do this? Well, we're interested in building general purpose learning algorithms. So these are algorithms that learn automatically from raw inputs. They're not pre-programmed or handcrafted in any way. So we're only interested in algorithms, building algorithms at DeepMind that can learn for themselves directly from raw data and raw experience."}, {"start": 4171.38, "end": 4198.64, "text": "Secondly, we're interested in the idea of generality. So this is one system, a single system or a single set of algorithms that can operate across a wide range of tasks out of the box. Of course, we have an example of general purpose learning algorithms. It's up here. And the brain is the only example we have of that. But we know this is possible. So at DeepMind,"}, {"start": 4198.64, "end": 4226.75, "text": "We talk about this kind of AI as artificial general intelligence, AGI. And the hallmarks of AGI are that it's flexible and adaptive and potentially even inventive. And it's built from the ground up to deal gracefully with the unexpected. If we contrast that with what is mostly called AI these days, which we call narrow AI, usually that type of AI is handcrafted."}, {"start": 4226.75, "end": 4256.4, "text": "So the solutions are directly programmed into the machine, and they're special case for a single task. And the problem with these kinds of systems is they're very brittle. So if something unexpected happens, they just break. They catastrophically fail. They have no way of dealing with something that the programmers didn't already cater for. So of course, probably the most famous example still of this kind of narrow AI system is Deep Blue."}, {"start": 4256.4, "end": 4281.42, "text": "As you all know, famously beat Garry Kasparov in 1997. But the thing about Deep Blue was that it couldn't do anything else. The systems and the program behind Deep Blue couldn't even be used to play a strictly simpler game. Like in the UK, we call it Noughts and Crosses, a very simple game you teach children. And even though Deep Blue had mastered chess,"}, {"start": 4281.42, "end": 4307.49, "text": "None of the programs and none of the knowledge it had would be useful for any other game, let alone any other type of domain outside of games. So I actually came away from this match. I remember this match very well. I was studying at Cambridge. And obviously with my chess background, I was following the match very closely. I actually came away more impressed by Garry Kasparov's mind than with the machine. Because obviously Kasparov..."}, {"start": 4307.49, "end": 4333.2, "text": "could play chess at the same level as the machine, but also do many, many other things that we can all do. So Deep Blue was an immense technical achievement. Of course, it was a huge watershed moment in AI. But what we want to try and do is go beyond that. So the way we think about AI at DeepMind is through the glasses or the prism of reinforcement learning."}, {"start": 4333.39, "end": 4363.7, "text": "And for those of you, I know lots of you in the audience will know what reinforcement learning is, but for those of you who don't, I'll just explain it very quickly with this diagram. So if we think about intelligence with this kind of framework, so you start off with the agent system here on the left, and the agent finds itself in some kind of environment which is trying to achieve a goal. And that environment could be real world environment, in which case the agent is a robot, or it could be a virtual environment, like a game."}, {"start": 4363.7, "end": 4395.06, "text": "In which case the agent would be an avatar. And the agent only interacts with the environment in two ways. One way is that it gets observations about the environment through its sensory apparatus. So we mostly use vision for the moment, but we could use other sensory modalities and we plan to in the future. And the key thing is that about the system is it only gets incomplete and noisy observations. It doesn't have perfect information about the environment out there."}, {"start": 4395.5, "end": 4425.55, "text": "And the job of the agent system is twofold. One is to build as accurate as my favorite Atari video. And I'm going to show you more gradations of the agent getting slowly better over time. So in this game, you control the bat at the bottom of the screen, the bat and ball, and you're trying to break through this rainbow-colored wall brick by brick. So you can see the agent here is, after 100 games, it's not very good. It's missing the ball, but it's starting to get the idea it needs to move the bat towards the ball."}, {"start": 4425.68, "end": 4455.66, "text": "So now after 300 games, now the agent is getting the ball back almost all the time, even very fast angles, which better than humans can do this. So then we thought, that's pretty good, but let's leave it for another 200 games. So we played 500 games now. And then it discovered, to our surprise, the optimal strategy was to dig a tunnel around the side and then put the ball to the back. It's just amazing accuracy. And so this is the optimal strategy that it invented."}, {"start": 4455.66, "end": 4483.18, "text": "And what's very funny is that the designers, the researchers and engineers, they're brilliant engineers and researchers who built this system called DQN, but they're not so good at playing Atari games. So they didn't even know that strategy. So they were taught something by the agent system. So this agent, which we call DQN, Deep Q Network, is really a general agent."}, {"start": 4483.18, "end": 4512.3, "text": "For Atari, of course, only for Atari games, but within the domain of Atari games, it really is a general artificial intelligence. So here I'm going to run a medley of different games just very quickly, and you'll see the same agent can play all these very different games, fighter pilot games, one of the very early 3D games, and you can see all of these games have very different rules. Of course, the classic Pong, it's controlling the green bat, it always wins 21-0."}, {"start": 4512.3, "end": 4535.7, "text": "Submarine games. Many, many different games. Really different ones. And the boxing controls the left-hand boxer. It does some sparring. And then it gets the inbuilt AI to the side. And then it just gets infinite points to win. So it's just worked out how to exploit the system. So this is the first time really..."}, {"start": 4535.7, "end": 4553.82, "text": "There's been any kind of general agent that can play, start with raw experience, raw data, and for itself learn through trial and error, learn from its mistakes, and master quite a challenging set of problems on these Atari games."}, {"start": 4553.82, "end": 4583.7, "text": "And if you're interested in this work, you can see in our first Nature paper last year, front cover, that we also released the code. So you can play around with DQN yourselves if you want and apply it to your problems. It's all on the Nature site. So then we moved on from Atari games and 2D games. And now we're working on 3D games. We work on robot simulators. Eventually we'll work on real robots. And of course, as you know, we've been working on Go as well, which I'll talk about in a bit."}, {"start": 4583.76, "end": 4610.43, "text": "So here is the same system now working on some 3D games. So we tried it in a driving game. And again, just from the raw pixel inputs, it's learned how to drive around a racetrack and overtake cars, drive around 200 kilometers an hour, and do some quite cool overtaking maneuvers that we'll see in a second. And again, just from the pixels on the screen, there's no other input."}, {"start": 4610.43, "end": 4639.23, "text": "It learns how to drive cars. It can also, we're now up to the games in the late 90s, so we're using the Quake engine and we use it to create maze problems, 3D maze problems. And our agent can find rewards like these apples in the maze and then remember how the maze works and find its way out of the maze to the exit. All just from the pixel inputs."}, {"start": 4639.23, "end": 4669.06, "text": "So of course now we're making more and more complex and sophisticated 3D environments and we'll be publishing more work on that later this year. So I just wanted to say a little word about neuroscience. I don't have time to talk about this too much but my PhD was in neuroscience, my undergrad was in computer science and we try and combine both those two areas at DeepMind. So we're very interested in systems neuroscience and we're quite interested in"}, {"start": 4669.06, "end": 4685.33, "text": "Systems level, so that means the algorithms and the representations and the architectures the brain uses, rather than the specific implementation details of synapses and cortical columns. So it's this kind of higher computational level that we're very interested in."}, {"start": 4685.33, "end": 4708.34, "text": "And the reason we're interested in systems neuroscience is that we think it's very important to look at the brain as a source of inspiration for new algorithms. But probably how those algorithms are going to be implemented in a computer relative to the brain will be different because, of course, silicon is different from wetware carbon. So we don't think that actually..."}, {"start": 4708.34, "end": 4735.39, "text": "Exactly copying the specifics of the implementation details is the right way to go if you're interested in building AI. So the kinds of things we're using neuroscience-inspired ideas for is memory, attention, concepts, so kind of abstract knowledge, planning, navigation, and imagination. All these different areas that we would like to imbue our computer systems with the ability to do."}, {"start": 4735.39, "end": 4764.56, "text": "One of the brain areas called the hippocampus that's here in pink in the center of your brain, this is what I studied for my PhD, turns out to be quite critical for a lot of these functions, for memory, for navigation, for concepts and for imagination. So actually, and the hippocampus, obviously the neuroscientists here will know, but those of you who aren't, the hippocampus is quite a different structure from the cortex. It's three-layered instead of six-layered. It's a much more ancient structure. All mammals have..."}, {"start": 4764.56, "end": 4784.14, "text": "And it's very interesting to think about what functions the hippocampus actually provides for the whole brain. So you can think of one of the projects we have going on at DeepMind is to create a kind of artificial hippocampus. So now I'm going to talk about, of course, AlphaGo."}, {"start": 4784.14, "end": 4806.29, "text": "Which is obviously now right in the news. The reason we chose to work on AlphaGo, which I'll explain in a little bit, is that we wanted to see if these kinds of deep reinforcement learning techniques, you know, deep learning and reinforcement learning, could be put together with planning, long-term planning. And Go is actually the perfect..."}, {"start": 4806.29, "end": 4833.18, "text": "Testing Ground to do that because obviously it's a very intuitive game and but you have to do calculation and planning as well. So I'll assume pretty much because I'm in Korea all of you know how to play Go so I'm not going to explain that and obviously here's the Go board. Of course as I've talked about many times in and I'm sure you've seen me talk about this in press conferences last couple of days it has a hugely rich tradition and history."}, {"start": 4833.18, "end": 4852.42, "text": "With 40 million players today and 2,000 professionals. And it's probably one of the most studied games that humans have ever studied, which is fantastic. So that means that the level of challenge is very great. And obviously, just one figure showing how..."}, {"start": 4852.42, "end": 4872.59, "text": "So, you know, these immense numbers."}, {"start": 4872.59, "end": 4895.2, "text": "Why Go? Why did we decide to take on this challenge a couple of years ago? Green board that you can see here on the top, the size of the green bars is the probability that AlphaGo assigns to the likelihood of that particular move at that particular board position. So the higher the green bar, the more probable. And the second network we get is the value network, which also takes a board position in as an input."}, {"start": 4895.2, "end": 4920.83, "text": "And then the output here is a single number between 0 and 1, a real number, and 0 indicating white is winning completely, 1 would be indicating black is winning completely, and 0.5 would mean even game. And this is really the key to the power of the strength of AlphaGo. So now we have those two neural networks. We can combine that with..."}, {"start": 4920.83, "end": 4946.78, "text": "Monte Carlo tree search and also a rollout policy to create the final AlphaGo system. So I'm just going to explain very quickly with this diagram how that works. So let's imagine that AlphaGo is thinking about making a move. So imagine it's in that top board position represented by the node there, the small little Go board. And it's done, maybe we're halfway through the search."}, {"start": 4946.78, "end": 4963.2, "text": "And it's done a bit of searching. And you can see here that the Q value is the estimated value of that particular move from that position. And we want to follow down the most promising path, so where the Q values are highest."}, {"start": 4963.2, "end": 4992.94, "text": "And you can see that the alpha go will follow the tree down to a position where it's never considered it so far from this particular position. So you can see that here is the left-hand arrow at the bottom, the black arrow. Now, it now has to create a new part of the tree. So it calls the policy network to expand the tree at this point. So the green network I was describing. But we don't add in."}, {"start": 4992.94, "end": 5021.52, "text": "All the possible moves from that position. We only add in the top three or four most probable moves that the policy network is saying are most likely. So that really cuts down the size of the search tree. So we expand the search tree, this node in that fashion. Then once we've expanded the search tree, we now call the second neural network, the value network, to evaluate that new position in the tree, that new node."}, {"start": 5022.26, "end": 5051.01, "text": "We also combine it with a second way of evaluating the position, the more traditional way, which is called the rollout policy, which is actually to roll out games all the way to the end of the game and then collect statistics of how many wins and losses there were for each side. So it's kind of two different ways of evaluating the position. The value net, which instantaneously evaluates the position, and then the rollout policy that does a kind of statistical analysis of the position."}, {"start": 5051.01, "end": 5069.52, "text": "And actually, we found alpha goes stronger if you combine those two different evaluations together. And we mix that together to give a final evaluation. And once we've done that, we then back up these new Q values, these new evaluations, back up the tree."}, {"start": 5069.81, "end": 5097.94, "text": "And then when AlphaGo has run out of time and it needs to make a move, then it basically picks the move that has the most promising Q value associated with it. So let me just explain now with this diagram how that helps with this combinatorial problem that Go has. So imagine that this is the search tree from a position in a Go game."}, {"start": 5097.94, "end": 5121.47, "text": "Obviously, it's this huge search tree of possibilities. And what you can think of is that the policy network actually cuts down the breadth of the search. So instead of searching all these 200 moves from each position, we just search the top three or four. And then you can think of the value network as reducing the depth of the search."}, {"start": 5121.47, "end": 5150.03, "text": "So instead of having to search all the way to the end of the game to find out who won and doing that millions of times, at any stage, we can truncate the search and call the value network to get an estimate of how well AlphaGo is doing in that situation. So now you can see with these two neural networks, we've completely reduced the search space down to something much more tractable. So how did we evaluate AlphaGo once we built it?"}, {"start": 5150.03, "end": 5177.97, "text": "Well, sort of last summer, actually last spring, we started evaluating it against the other top programs that are available. And actually, the best programs, commercial programs, are called Crazy Stone and Zen. And they're both around five-dan, six-dan amateur level. And when we ran AlphaGo against these programs, AlphaGo won 494 out of 495 games."}, {"start": 5177.97, "end": 5209.33, "text": "And in fact, it has a 75% winning rate against Crazy Stone and Zen, even if it gives them a four stone handicap, which is pretty huge. And obviously, AlphaGo is even stronger on distributed machine. So the gap between AlphaGo and the next best programs is more than 1,000 ELO points. Actually, this is a lot larger now because this is the AlphaGo version from October that I'm talking about here."}, {"start": 5209.33, "end": 5229.36, "text": "So then, as you all know, we beat all the other programs very convincingly. So now it was time to test against professional human players. So we called up our esteemed opponent, who has now become a friend of ours since the match, Fan Hui."}, {"start": 5229.36, "end": 5258.83, "text": "Who is the European champion of the last three years. And he's written books about Go. And he coaches the French national team. He grew up in China and became professional there. And he's Tu Dam Pro. And in October, we played at Fan Hui in a match. And we won, surprisingly to us, 5-0. So AlphaGo became the first program ever to beat a professional player in an even game under tournament conditions."}, {"start": 5258.83, "end": 5289.63, "text": "And this was very surprising to the machine learning and AI world, and I think the Go world. It was more than a decade earlier than many experts, even expert programmers from programming other Go programs were expecting. So again, this was a front cover Nature paper a couple of months ago. So if you're interested in the details of how it all works, it's all detailed out there in the paper."}, {"start": 5289.63, "end": 5316.11, "text": "And it created quite a big stir in the academic circles in terms of downloads and other things. So now, as you all know, we're here. It's the final kind of ultimate challenge. I think playing Lisa Doll in this five-game match. Now, of course, we're currently 2-0 up."}, {"start": 5316.11, "end": 5332.77, "text": "But the games have been very close, very interesting, and I think anything can happen still. That was my prediction before we came in it. We knew that AlphaGo was quite strong because in our internal tests, we test the programs against older versions of itself."}, {"start": 5332.77, "end": 5350.18, "text": "So the new version of AlphaGo has played, you know, hundreds and thousands of games against many different older versions of AlphaGo. And because of that, we can estimate how strong the new one is because we know how strong the October one was because we calibrated that against FanHui, right?"}]}