[0:00:05] Well, can I say welcome, everybody? A particular welcome to everyone who's come as part of the Cambridge Science Festival.
[0:00:15] It's great to see you all here. For those of you for whom this is the first CSAR event that you've attended, I'm David Wallace. I'm president of Cambridge Society for the Application of Research. I've got some preliminaries which you will have seen on the screen, and that is no tests of any kind are planned to my knowledge.
[0:00:39] If the alarm goes off, then we have to go out and this number, and we cannot have people sitting in the aisle, I'm afraid. Standing at the back, I guess we're going to get away with. If the alarm goes off, exits are here and at the back. We are recording now, as you will have seen, so it should be mobile phones off. And since we've got microphones going around with questions, actually better that they're actually off rather than silent because they will...
[0:01:05] The microphones will pick up your incoming signal and you'll be permanently on the recording. So mobile phones off, please. Demis, what can I say? You've given us a wonderful headache. This is a record for us. It's hugely stretched our organization. We realized that this was going to be a popular event.
[0:01:30] When more than 700 posters for the talk were downloaded from our website. And we knew this theater has a maximum of 300. We've got the overspill and this is it folks. And I'm slightly nervous. We mustn't allow anybody else in now. We must stop please at the back.
[0:01:48] I want to thank, in particular, it's been a huge team effort here, but I wanted to thank Valerie Anderson, Edward Briffa, and Andrew Shepard, who've done a huge job in trying to deal with the extra numbers that we have tonight. So I don't know if they're here. Valerie, Edward, Andrew, I think you should stand up and take a round of applause because your presence here is due to them.
[0:02:16] In fact, I realize that Valerie's not here because she sacrificed her seat to go into the overspill in the Bevan room. Now, the fact that we had so many downloads, there are so many here, the interaction for Demis is really easy. I don't have to say anything. You know exactly who he is and why he's here. If you don't, then you Google Demis Hassabis or a slight play on words, Google DeepMind. So, Demis, please, and thank you.
[0:02:54] Well, thanks very much for that very warm welcome. And thanks, everyone, for coming. And it's amazing to see such a big crowd. I always love coming back to Cambridge and seeing old friends and lecturing here at Cambridge. So today, I'm going to talk about artificial intelligence and what we've been doing at DeepMind, especially over the last sort of year, year and a half. And I've titled this talk Explorations at the Edge of Knowledge. And I hope that becomes clear to you why I've done that towards the end of the talk.
[0:03:23] So for those of you who don't know about DeepMind, we founded the company in 2010 in London. And in 2014, we joined forces with Google to try and accelerate our mission to solve intelligence. So one way we think about DeepMind is as an Apollo program for AI.
[0:03:42] And what I mean by that is that the vision behind the company was to bring together the best researchers in the world, give them all the compute resources and engineering resources they could use, and leave them free to make as much progress as possible towards this one goal.
[0:04:00] And in order to facilitate that, we've created a kind of unique research culture at DeepMind. It's sort of been designed to combine the best from academia, or at least the way academia should work in an ideal way, and combine that with the best from startups, the best from the sort of Silicon Valley startups. So take the kind of blue sky research that happens in academia and cross-disciplinary work, and combine that with the buzz and focus and energy you get in the best startups.
[0:04:28] So that's the way I kind of describe the culture at DeepMind. And I'm happy to answer questions about that. That would be a whole sort of talk in itself, how we organize science. But we try to do it to make the pace of innovation and scientific innovation faster.
[0:04:43] So what about our mission? Well, I mentioned solving intelligence, and that's sort of step one of our mission. And then once we've solved intelligence, fundamentally understand intelligence, then we believe you could use it to solve literally everything else. So that's our kind of two-step mission statement.
[0:05:03] And it's sort of slightly tongue-in-cheek, but actually we really believe that if you solved intelligence in a very general way like we're trying to do at DeepMind, then step two would naturally follow on. So more prosaically, how are we going to achieve this? So we talk about at DeepMind trying to create the world's first general-purpose learning machine. And the two key words there are learning and general. So the kind of algorithms we work on at DeepMind, all of them...
[0:05:33] Learn automatically from raw inputs or raw experience. There's no pre-programming or set of heuristics or rules that we program into the system. The system has to learn for itself from first principles, directly from raw data. The second thing is this notion of generality. So the same system, single system or single set of algorithms,
[0:05:55] Operating can operate across a wide range of tasks, potentially straight out of the box, even tasks that are completely novel that it's never seen before. So you might ask, how can a system do this? Well, we have evidence of such a system. It's sitting on our shoulders in our brains. We know this is possible. And the question is just figuring out the right sorts of algorithms and architectures and representations.
[0:06:21] So we call this kind of AI, this sort of general learning system, artificial general intelligence, or AGI, to sort of really distinguish it from the normal sort of AI that mostly is worked on today. So we call that kind of AI narrow AI. So this is this kind of expert system, heuristics-based, symbolic form of AI that's really been the pervasive form of AI for the last sort of 40, 50 years.
[0:06:47] And the pinnacle of narrow AI still is when famously IBM's computer Deep Blue beat Garry Kasparov at chess in the late 90s. And that was obviously a huge watershed moment for AI and a huge advance and a huge technical feat. But really, when you ask the question, where does the intelligence of the system reside? It's not really in the machine or the algorithm. It's actually in the minds of the team of brilliant programmers that created Deep Blue.
[0:07:16] What they did was is they talked to an equally brilliant team of chess grandmasters, human chess grandmasters, and they picked apart the knowledge that they had about chess and they codified it into rules and heuristics and then combined that with brute force search in order to beat Garry Kasparov. But nothing in that system would allow it to play any other game, not even something strictly simpler like noughts and crosses. You would have to start again from the beginning and reprogram a whole new set of heuristics.
[0:07:46] So in some sense, this leaves us a little bit unsatisfied if the question is to get to general intelligence. In a system like Deep Blue, we have this intuitive feeling that really it's not an intelligent system in the way we would normally regard intelligence. And we think what's missing is this idea of generality and this idea of learning. So instead of that, we think about intelligence within the framework of what's called reinforcement learning.
[0:08:12] And I'm just going to quickly explain what reinforcement learning is. I'm sure many of you will be familiar with it, but for those of you who aren't. So you can think of, first of all, on the left-hand side here, the agent or the system. So internally at DeepMind, we call our AI systems agents. And this agent finds itself in some kind of environment that it's trying to achieve a goal in. Now, that environment could be the real world, in which case the agent would be a robot, or it could be a virtual environment like a game, in which case the agent would be an avatar.
[0:08:42] Now, the agent only interacts with the environment in two ways. Firstly, it gets observations and rewards through its sensory apparatus. Now, mostly we use vision, but you could use other modalities like audition and touch. And in fact, we're expanding to multi-modalities right now. And based on these observations, which are coming in real time, they're incomplete, they're noisy, they're coming through the agent's sensors. One of the jobs of the agent is to build...
[0:09:10] The best statistical model it can, most accurate statistical model it can of the environment out there based on these noisy, incomplete observations. And then once it has that model, the second job of the system is to select which is the best action from the set of actions available to it at that moment that will best get it towards its goal. And this may involve imagination and planning into the future and trying hypothesis testing and all sorts of quite exotic action planning techniques.
[0:09:40] Now, this agent is usually embedded in a real-time situation or scenario. So when it runs out of time to make these plans, it has to execute and output the best action it's found so far. That action gets executed. It may or may not drive a change in the environment, and then that drives a new observation. And then the agent will update its model in real-time in response to what has just happened. So in essence, this is the whole...
[0:10:09] And, you know, although this diagram is very simple, this actually hides huge technical complexities and research challenges behind this.
[0:10:19] But we know that if we could solve all those challenges behind this diagram, that would be enough for general intelligence. And we know that for two reasons. One is that mathematical frameworks of intelligence have been built up by also my co-founder during his PhD. He did work on a framework called AIXI, which was a kind of mathematical proof of what, if you had infinite compute time and memory, what would be required for general intelligence.
[0:10:44] And secondly, we know from biology, this is the way that animals learn, including humans. So in humans, it's the dopamine system in the brain that implements a form of reinforcement learning. So we know both from mathematics and from biology that reinforcement learning is a sufficient solution to the intelligence question. So we took this further, and our first big result, which I'm sure a lot of you will be familiar with back a few years ago now, was applying deep learning.
[0:11:16] Which is hierarchical neural networks to process real perception and then we combine this with reinforcement learning which did the planning and selected the goals and got towards achieving whatever goal the system had.
[0:11:30] And we tested this deep reinforcement learning system called DQN on the iconic Atari games from the 70s, which most of you will be too young probably in here to remember. But here are some of those games. They're some of the most iconic games ever. Space Invaders, Pong, these kinds of things, Pac-Man. And what we did is we built a system that only took the pixels.
[0:11:53] On the screen as inputs. So it got about 30,000 numbers per frame, per 50th of a second. So Atari screens are around 200 by 150 pixels in size. So 30,000 numbers. And we told the system that the goal was to maximize the score. That's it.
[0:12:11] So it was told nothing else about the game it was playing. So it didn't know what the controls did. It didn't know what it was to get points. It didn't even know that it was a video stream. So it didn't know that pixels next to each other are correlated in time. So it has to find all this structure for itself through experience, by playing the game, sometimes many hundreds or even thousands of times, and then figuring out the rules and its controls and the structure of that world for itself and first principles.
[0:12:39] And the amazing thing was all these systems worked. And for the first time around 2013, we showed that this system could learn any Atari game, and there's hundreds of different ones, at least as good as the best human experts. So this was a real surprise for the whole field and gave a real boost to this idea of general learning systems that there may be within our reach.
[0:13:01] So this was our first big Nature paper and Nature front cover. So you can read all the details of this in that article. And also, we released the code open source. So you can actually play around with this algorithm yourselves if you want to. So I'm going to spend most of my time today talking about our newest and maybe kind of most famous, if you like, advance, which happened last year. It's called AlphaGo, our new program called AlphaGo, extending the techniques we used to crack the Atari games into the ancient game of Go.
[0:13:32] Now, I hope because I'm in Cambridge and lots of you have math backgrounds, a lot of you will know and love this game, Go. And if you haven't played it, I really recommend you do. It's an absolutely amazing game. But this is what a Go board looks like. So you play Go between two players, play the black stones and the white stones. The board starts empty. It's a 19 by 19 grid. And you take turns placing the stones on the vertices of the grid. Now, the aim of the game is to surround your opponent's stones with your stones.
[0:14:01] Or to wall off empty areas of the board as territory. And there's only two rules in the game. So I could teach you it in less than five minutes. But obviously, it's been going on for thousands of years. And we still haven't fully mastered all the intricacies of the game.
[0:14:18] So this is what a game of Go might look like at the end of a game. So here, this is the end of a game, the end position, and the board is mostly filled up now. And what you do is you count up the amount of empty squares that your stones have surrounded. And the person that has the most amount of territory wins the game, including any captured stones that you've captured during the game. So in this case, it's actually a very close game, and in fact, white wins by a single point.
[0:14:46] So this is the aim of the game. And it's quite actually hard to even understand what the aim is at the end because it's not like in chess where there's a checkmate. It's very clear objective. In Go, it's actually quite intuitive even to decide when the end has come. So even deciding that is quite a tricky task for a beginner. So the history of Go is long and storied. So it's over 3,000 years old.
[0:15:11] And it was originated in China. And in Asia, in China and Japan and Korea, this is what they play instead of chess. But it has even sort of deeper cultural meaning for them. So Go, you should think of as something more akin to an art than it is to just a game. And in fact, Confucius wrote about Go as one of the four great art forms that any true scholar would have to master.
[0:15:35] So it's something more like art or poetry. That's how it's regarded by those cultures and also the professionals who play the game.
[0:15:43] Today, it's more popular than ever. There's 40 million active players, over 2,000 professionals. The 2,000 professionals, what happens in any of these countries is if you show promise when you're around five or six years old in your normal school, in kind of after-school go club, you're taken out of normal school and you're put into special go schools where you study go and play go 12 hours a day, seven days a week from the age of six. And you actually live.
[0:16:08] With your Go master along with a whole bunch of other promising prodigies with you. So that's how, you know, it's phenomenally seriously taken this endeavor. And so, you know, these players devote their whole lives to mastering and learning about this game.
[0:16:26] And I would say Go is probably the most elegant game mankind has ever devised. As I mentioned, there's only two rules, sort of beautifully simple, and yet the complexity that comes out of it is immense. One easy way of illustrating that is that there are...
[0:16:40] 10 to the power 170 possible board positions. So that's way more than there are atoms in the observable universe, which is estimated to be about 10 to the 80. So there's no way you could brute force a solution to Go, even if you took all the world's computers and ran them for a million years. So we have to do something much cleverer here than a kind of brute force search. So you might ask, well, why is Go so hard for computers to play? Since 97, when Deep Blue beat Kasparov,
[0:17:09] Go has been one of the holy grails for AI research. Can we get a program to be the world champion at the game of Go? And there are two main challenges, one I've already alluded to, which is that the search space is really enormous. So just to give you another example of that is, in Go, on average, there are around 200 possible moves. In chess, in an average position, there are around 20 moves. So Go has an order of magnitude bigger branching factor than chess has.
[0:17:37] But actually harder than the sheer amount of possibilities is that until AlphaGo came along, it was thought to be impossible to directly write what's called an evaluation function to tell the system whether black or white was winning and by how much.
[0:17:53] And this is critical. An evaluation function is critical to these game systems working. If they're going to make choices through this incredibly complex, dense tree of possibilities about which choices are good for it and which choices are good for the opponent, it has to be able to evaluate the current situation and future situations in terms of their desirability.
[0:18:13] Now for chess, writing an evaluation function directly is quite easy because chess has a lot, it's obviously a simpler game, but it also has materiality. So as a first approximation, you can just count up the value of the pieces on each side, you know, queens, nine points, rookies, five, knighties, three points, and so on. And that will tell you to a reasonable first approximation who's winning.
[0:18:34] And you can codify all sorts of other things, pawn structure, mobility of the pieces, all these other things that chess grandmasters do to determine whether position is good or bad for their side. Now, all of that is impossible in Go. You don't have all the pieces are the same. The idea is that even a small...
[0:18:57] Change in the position of one piece can totally change the outlook of the position. So they have huge influence, each of these pieces exactly where they are. But perhaps the hardest thing is that chess is what I would call a destructive game in the sense that you start with all the pieces on the board and the game gets simpler as you take pieces off. Go is a constructive game in the sense that the board starts empty and you start filling it up.
[0:19:20] So if you're going to judge a complicated middle game position in chess, you can just look at the current situation that tells you everything about that position. In Go, in a middle game position, you actually have to project into the future what might happen in order to evaluate the current situation. So it's much, much harder to understand whether you're winning or not in a complex mid-game position.
[0:19:42] So many people are tempted to apply the deep blue type of techniques to Go. And no program until AlphaGo came along. So after 20 years of research, very active research in Asia and other places, had beaten a professional at all, let alone anyone close to world champion standard. So you might ask, well, if it's so hard for computers to play, how do human professionals cope with this immense complexity?
[0:20:12] And actually, the way they do it is they rely on their intuition and instinct and creativity more than their calculating abilities. So it's very different from the game of chess. So again, if you ask a chess grandmaster why they made a particular move, they'll usually be able to tell you, you know, I played A because I was expecting B to happen and then I was planning C. Now, that plan may turn out not to be good, but they usually have an explicit plan in mind for the reason why they did something. They can explain to you.
[0:20:40] In Go, it's very different. If you ask a top player in a complex position why they made a particular move against someone, one of their peers, they'll often tell you it felt right, and they'll really mean that. They won't be able to actually explain explicitly what it is about that particular move and why they've done it, but they'll just have this feeling. So they'll be using their intuition and pattern matching in order to do that.
[0:21:03] So we overcame these problems by turning to our neural network systems and reinforcement learning algorithms. And what we did was we tried to mimic this human intuition component by using deep neural networks. So we actually trained two neural networks we required to master Go. So the first one was a policy network here in green on the left-hand side.
[0:21:29] And what we did, first of all, is we downloaded around 100,000 games from Internet Go servers of amateur players, actually fairly sort of middling club players. And what we did initially was get this first neural network system through supervised learning to mimic the moves that the human players were making.
[0:21:47] So what we did is we would take a random ball position, say in blue here, sort of indicated in blue at the bottom here, and we would train the system to predict the move that the human player would make.
[0:22:00] And what that gave us, once it was fully trained, was a policy network that could give you the top five or top 10 most probable moves in that particular position. And that's what the bars on the green board indicate, is the probability of a human player playing in that position, playing in that place in that current position.
[0:22:22] So that obviously narrows down the breadth of the search you need to make. You can now just look at the top five moves rather than all 200 from that position. And then what we did is once we had that, we played that system against itself millions of times and we improved it through reinforcement learning. So it learned from its own errors.
[0:22:44] So as it played against itself, different versions of itself, it would win or lose. And then if it won a game, it would reinforce the decisions it makes. So next time in similar situations, it would make those more likely to make those decisions. And conversely, if it lost the game, it would weaken the strength of those decisions so that next time in that position, it would be less likely to play those same moves. And what that did is we created our own database of millions of games of the system playing against itself.
[0:23:12] And then we were able to use those games to train a second neural network called the value net. And what we did here is we took a random position from each of these games, and obviously we also know the final result of the game, you know, who won, which side won or lost. And we trained this neural network to predict that itself.
[0:23:30] So take a board position and then predict with some confidence level who is going to win from that board position. So for the value net, it output a single real number between 0 and 1. 0 meaning white was sort of 100% likely to win, 1 meaning black was 100% likely to win, and 0.5 meaning the game was exactly even.
[0:23:51] And the amazing thing was, is that combining these two systems together, so the policy network to narrow down the breadth of the search and the value network, which we could call at any point in the depth of the search, to give us an approximate evaluation of what was happening. And these two networks working in tandem with Monte Carlo tree search, which we use for the searching algorithm, would allow us to make this sort of seemingly intractable problem tractable. So we beat all the kind of...
[0:24:21] Handcrafted Go programs pretty early on in this project. Then we played the European champion who is a two-down professional and three times European champion behind closed doors. And we won that match 5-0. And then that was a sort of huge sort of first breakthrough, which then resulted in our second Nature paper where we detailed how the system worked.
[0:24:44] And then we felt we were ready for the ultimate test, which was to take on the world champion in Go.
[0:24:51] And we set up a $1 million challenge match in Korea against Lee Sedol, who is the South Korean genius grandmaster. He's a legend of the game. The way I like describing him is like the Roger Federer of Go. So he's won 18 world titles, kind of like Grand Slams, and he's acknowledged to be the greatest player by far of the last decade, but he's still at the top of his game.
[0:25:16] So we played him, and he's also fated for being incredibly creative. So in some ways, he's probably the hardest opponent we could have picked. And before the match, everyone was predicting, including himself, that he would win 5-0 easily, and it would be a whitewash. But in fact, what happened is that AlphaGo won 4-1. And this was proclaimed by experts both in the Go field and AI field as about a decade before people were expecting.
[0:25:43] And for those of you who know that, you know, who work in technology or research, you'll know that almost never happens, right? Usually we're a decade late. So this is kind of a fairly strange event for us. And it was obviously an amazing once-in-a-lifetime event. You know, the games were watched by 200 million people.
[0:26:02] More than 200 million people across Asia. And it's like 35,000 press articles. It's a huge event. And the whole of Korea kind of came standstill for that week. It's a really amazing once-in-a-lifetime experience. Now, the key thing is, obviously, we won the match, which was great. But it wasn't just the fact that we won. It was how AlphaGo won that was important.
[0:26:24] So what astounded people was how creative the program was. So not just the fact that it learned to mimic other human expert players and regurgitate that on command. It actually created what seemed to be new ideas. And I just want to take you through one of those ideas to try and convince you, if I can, that these were truly original thoughts, ideas that AlphaGo was coming up with.
[0:26:51] So here's a position from game two and move 37, which is my favorite move from the whole series of matches. And I think we'll go down in Go history. And in this position, AlphaGo is playing black and LisaDoll is playing white. And AlphaGo played this move on the right-hand side here, middle of the right-hand side of the board that I've indicated here with the white triangle. That was move 37.
[0:27:15] Now, that move is called a shoulder hit because you're diagonally next to an opponent's stone. Now, to explain why that is such an astounding move for all the professionals in the room, all the commentators, that we had rooms filled of Korean commentators and English language commentators, and they were all shocked by this. And the reason is because there are two important lines in Go. So it's a 19 by 19 board, but there are two critical lines in tradition. The third line...
[0:27:41] Here, that I've marked with the line here. Now, if you play on the third line, what you're saying to your opponent is, I'm going to take territory to the side of the board. I'm going to wall off territory to the side of the board. And I'm going to bank those points. Instead of that, so here's the side of the board. That's what you're doing when you play on the third line. If you play on the fourth line instead, what you're saying to your opponent is, I'm actually going to take power and influence into the center of the board.
[0:28:06] And the idea behind playing on the fourth line is that later on, that power and influence will get you territory somewhere else on the board that will be equal to the territory you gave up locally on the third line. So for 3,000 years, the received wisdom has been that playing on the third line and your opponent playing on the fourth line is an equal trade, or vice versa. So basically, generally speaking, if you do that, you've got an equal trade and things should proceed as normal.
[0:28:34] But what you'll notice is that AlphaGo played on the fifth line. So it played on the fifth line, one line higher up, to take power and influence into the center of the board. And the fifth line is obviously giving away, you're sort of selling your opponent. You can take territory from the fourth line. So you're giving away a whole extra line of territory towards the side of the board, which is huge in Go. So what we think this means is that for 3,000 years, humans may have been underestimating
[0:29:03] The value of power and influence to the center of the board, despite 3,000 years of professional contemplation and play, which is kind of incredible if you think about that. And obviously, the interesting thing is, so Go is like art. That's how it's considered. In fact, if you talk to top Go players, they'll tell you it embodies all the complexities of the universe. That's how they think of it. But it's interesting because it's objective art.
[0:29:27] So any one of us in this room could come up with an original move by playing something random. But it wouldn't mean it was any good. It might be original, but it wouldn't mean it was any good. So the key thing about this move is that it turned out to be pivotal to winning the game. And what happened was that these two stones here in the bottom left-hand side that are surrounded by the white stones and seem to be in trouble that I've indicated with the white triangles there.
[0:29:54] 50 moves later, around 50 moves later, the fighting from the bottom left-hand corner spilled out into the center of the board and went all the way to the right-hand side of the board. And that move 37 stone happened to be in the perfect place to decide that running fight. So it was as if somehow AlphaGo had been prescient about what was going to happen and then sort of positioned this piece in exactly the right place to decide that decisive battle over on the left-hand side.
[0:30:23] So I've told you that this is a wonderfully creative move and the professionals were astounded, but I'm a fairly weak Go player, so you can't just take my word for that. So I want to show you a clip of...
[0:30:35] One of the top professionals in the world commentating on this move live. So just before I roll the video, what you're seeing is the two professional Go players. But the guy on the right is called Michael Redmond. And he's a 9 Dan professional. 9 Dan is like the highest. It's like kung fu. He's sort of the top level black belt, right? So that's the highest level you can get to. And he's the only Westerner ever to reach that level, 9 Dan. So he's the strongest Westerner of all time.
[0:31:01] And what they're doing is they've got a demo board. They're talking about this in front of a whole room the size of this, full of journalists. And they're watching the game room live on the laptop in front of him at this Move 37 moment. But just listen to how he reacts to that. The Google team was talking about is this kind of evaluation. That's a very surprising move. I thought it was.
[0:31:32] It was a mistake. So you can see here, he was so surprised by the move. He initially misplaced the stone. He put it in a different place because he couldn't believe that it had gone there. And then what his colleague said, Chris Garlock, was afterwards to me, he said he thought that our computer operator had misclicked on the monitor. That's how unthinkable that move was, that move 37 was, to any trained top professional.
[0:31:58] So that was pretty revolutionary, and there were lots of examples of that during the game. And then I must mention Lisa Dole's incredible genius too. So he went down, we actually won the first three games, and it was incredibly tense for us, especially game one, because what we'd done was we'd internally tested our system against itself.
[0:32:20] And we had ELO ratings, which are like a Bayesian analysis of the rating, the strength of the program. And we'd beaten the European champion. And in theory, we knew how the distance between the European champion and the world champion. And in theory, our own internal system was higher than that bar. But the problem was, when you're training your system against itself, you don't know how much is overfitting to just beating its own versions of itself. So you don't actually know if your statistics are true or not until the first game.
[0:32:49] So the first game, we were incredibly worried. I mean, if our statistics had been wrong and we'd overfitted, then we might have just lost all five games quite embarrassingly. But once we won the first game, that anchored. We knew we were going to win the match because that anchored our entire rating analysis. And we knew that we could extrapolate that correctly. But then he came back in game four.
[0:33:11] And maybe the pressure was off a bit. And he did this incredible move himself, move 78 in game four, which I'm not going to go through. But this incredible move in the center of the board, this wedge move, which I think will also go down in Go history. And what happened is that it actually confused AlphaGo into mis-evaluating the whole position. So it confused that value net, that pink network I showed earlier on, to mis-evaluate what was going on here. And the interesting thing about this was that
[0:33:38] After the match, so actually some of the Chinese commentators who were watching this, they called it a God move. So in the sense of it was divinely inspired. And this just shows you how much sort of spiritual philosophy there is in Go in the sense of people spend their whole careers, the top players, trying to play what's called a divine move once in their lives, which means a move so beautiful and so kind of un-understandable that it must have been inspired by God. So this is maybe Lisa Dole's move of that. And the thing was is that...
[0:34:08] AlphaGo knew this was a very unusual move because we looked in the log files, obviously, straight after this match to see what happened. And we asked AlphaGo what it thought the probability was of a top human player playing that move. And it gave a probability of 0.007%. So it knew this was like a 1 in 10,000 chance move from a top, top player.
[0:34:31] So somehow, and obviously this is interesting too because it explains why AlphaGo had not seen that kind of move before because AlphaGo would not have played that move, right? So somehow, so how would it have ever explored that during self-play, which is quite interesting. And obviously the only way to overcome that is to have Lisa Doll in your testing room 24-7, which we don't have, right? So none of us are good enough at Go, you know, very few, there's hardly anyone else in the world who would be good enough to...
[0:34:57] Be able to live with AlphaGo for that long in order to be able to play this incredibly masterful move themselves. So it's very interesting that AlphaGo kind of knew this was super unusual. And it kind of threw out its search during those two minutes and sort of had to recalculate everything from scratch during that move.
[0:35:16] Now, I mentioned about the impact of the match, you know, 280 million viewers, 35,000 press articles. Also, the thing I'm most proud of is that Go boards were sold out online for two months in the West. So, you know, I've heard of whole new clubs at MIT and elsewhere, like, sort of, you know, order of magnitude more people joining those clubs, which is great to see. Now, I've talked a lot about intuition and creativity, and I just wanted to, you know, they're quite...
[0:35:41] Loosely defined words in science. So I wanted to just talk a little bit about how I operationally think about them. I'm not saying this encompasses all of intuition and creativity. I just think it's an interesting way to think about it. So intuition, you could regard, I think, as implicit knowledge that's been acquired through experience, but you can't consciously access or express. I think that's really what we call when we say this sort of folk word, intuition. I think that's what we mean.
[0:36:09] Now, I ask, well, how do we know that knowledge is there? Well, we can test the existence and the quality of that knowledge. We can verify it behaviorally. So obviously, in Go, that's very easy. We can give someone a Go position, and we can evaluate the quality of the move that they pick. So I think AlphaGo did mimic this function of intuition that humans use to play Go.
[0:36:35] So what about creativity then? Well, I think, again, I'm not claiming this is all of creativity, but I think this is one way to operationalise it at least, is it's the ability to synthesise knowledge you already have to produce a novel or original idea in the service of some goal. And again, by this, maybe you would argue, slightly narrower definition of creativity, I think AlphaGo clearly demonstrated both these abilities. Again, caveated, albeit in a constrained domain of the game of Go.
[0:37:06] So what's next? What about exploring the edge of knowledge? Which I feel like that's what we're doing at DeepMind. That's what I set DeepMind up to do, is to really explore the absolute boundary of knowledge in every sense of that word, of that mission.
[0:37:21] And we're doing that with AlphaGo, because you could consider, we beat LisaDoll last March, and yet we've continued to develop AlphaGo. But why? Because we've already beaten the world's best player, we've published the algorithm, there are actually some Chinese and Japanese companies who have cloned AlphaGo now, and they're very strong. But we didn't feel it was quite finished yet. We felt there was more science to be done. So firstly, we're perfectionists, and we wanted to fix the knowledge gaps.
[0:37:51] That Lisa Doll exposed in game four. And one of the ways we can do that, so we had lots of ideas of doing that, and we've actually solved that now, is creating a version of AlphaGo, branching it from the main version, and having that version.
[0:38:06] Its goal is to create delusions and confusions in the main AlphaGo. So it's not rewarded for winning. It's rewarded for confusing the main AlphaGo, right? So in some ways, it's sort of built to be a specialized adversarial opponent to AlphaGo in order to, if you think about it as the knowledge landscape, to pull AlphaGo into areas of the knowledge landscape that it had not explored on its own through self-play.
[0:38:33] So lots of really interesting things there. We also optimize the performance. So it used to take three months to train a whole version of AlphaGo from scratch. Now we can do it in one week, which is pretty incredible.
[0:38:43] The second thing we wanted to do was understand the representations AlphaGo used and created. So we would like to interpret the knowledge that AlphaGo has created. What motifs had it found in Go? Do they match the kinds of things humans think about? And actually, we're conducting an fMRI experiment on some of these Go players. We're comparing the way their brain reacts and the representations their brain has about certain ball positions. And we're seeing if that tells us anything when we compare that to the AlphaGo.
[0:39:13] AlphaGo representations that we're using. So in essence, we wanted to create perfection. And so we took a big step towards that with our new system. We tested it online on the top Go servers in the world against all the world's top 20 who, once they heard AlphaGo was playing online. We're playing under a pseudonym, but it became clear after our results after a while, this must be AlphaGo or something strange was going on, some alien. And then all the top players played, and we won 60-0.
[0:39:41] So which is completely sort of, you know, obviously if you do some sort of Bayesian analysis of that, you'll see that the level difference you would require to win to get a perfect score like that. And no delusions. So none of those situations happened that we found in game four.
[0:39:56] And on top of that, there were tons of new innovations. So here, playing in this really small corner as Black is an incredible new innovation. Crawling along the second line, which is thought to be really inefficient, was another new innovation. There's dozens of these. And in fact, the Korean Go team has now compiled a book of these games. And that's what they're mostly slaving over and studying to try and mine the new insights from it.
[0:40:24] He's the Chinese number one and the current world number one. He's only 19 years old. He was one of the players playing this, AlphaGo. And afterwards, he said, humanity has played Go for thousands of years, and yet, as AI has shown us, we have not yet even scratched the surface. And he said, the union of human and computer players will usher in a new era. Together, man and AI can find the truth of Go. And that's the words here. That's how they use it. They talk about the truth of Go like we would talk about, as scientists, the truth of science.
[0:40:53] And this is not without precedent. This has happened twice before in Go. Once in the 1600s in Japan, and again in the early 1900s, 1930s to 40s, with this incredible player called Go Saigen. And he came up with whole new theories that no one had ever seen before, and it took Go to a whole new level. Go improved by a whole new level based on his theories.
[0:41:21] And they're talking about AlphaGo as being the third revolution. And I just want to mention why I think this will happen in Go and improve human play more than it did in chess.
[0:41:33] So if you look at Magnus Carlsen today, who's the world chess champion, he's roughly the same level as Garry Kasparov was. They're both incredible, brilliant champions, but they're roughly the same level. And the reason is, and so why is that when we have computer programs that are way, way stronger than the top human player now? And the reason is, I think, is because chess programs are better tactically.
[0:41:58] Whereas I think AlphaGo is better strategically. So tactically, what I mean by that is that chess programs, top chess programs, never make tactical mistakes anymore. Now, you can't really improve as a human on that. It's not like these human grandmasters are trying to make mistakes. They know they shouldn't be making mistakes. So it's no good telling them, try harder so you make less mistakes. It doesn't help. Secondly,
[0:42:20] Chess programs have got massive opening databases. Obviously, they've got infallible memory. They've also got, interestingly, endgame databases. So for example, any situation now which has got nine pieces or less on the board left is solved.
[0:42:34] And I mean solved in the mathematical sense, solved. So obviously there's been billions of CPU cycles based on calculating all the possible brute force trajectories out whether you have nine pieces or less. So if you're in an end game against a computer and there's eight pieces or less left, you've got no chance. And you can't improve by that because you'd have to memorize all these.
[0:42:56] Billions of positions and moves. So there's sort of no real way, the dimension that chess programs are better on, there's no real way for humans to improve with that. Whereas you contrast that with AlphaGo, it's coming up with new ideas. Playing on the fifth line is maybe okay. Playing in the small corner is all right. Crawling on the second line is acceptable in some situations. And this has opened up whole new, these are ideas that top human players can incorporate into their own game.
[0:43:23] And improve themselves. And I think it's actually inspired a whole new level of creativity because as the European champion said to me after he played AlphaGo, he said that his mind had felt it freed his mind from the shackles of tradition to think the unthinkable. So you've got to understand for 30 years while these guys are training in these Go schools, they're told not to do certain things or certain things are not received wisdom, these sort of Chinese proverbs. And now it's acceptable to try all these creative things.
[0:43:52] So I hope then, when I think about things more generally, that this is a sign of things to come. So I'm really inspired by and excited by the idea of man and machine working together. Humans and machines collaborated together to achieve amazing things. And I think that our true potential of human ingenuity will be unlocked and augmented by these AI tools. And we will sort of unlock our true potential.
[0:44:21] One way you can think of what we did with AlphaGo, it's a bit like building the Hubble telescope for astrophysicists and astronomers. We've built this amazing tool that Go players can now use to explore their own universe and sort of open their eyes up to the possibilities of, as Kaji was saying, the truth of Go. Now, what I imagine is that, of course, we didn't...
[0:44:46] We're not interested in beating games in an end in itself as the ultimate end goal. That's great as a waypost, but we want to build, we use it as a very efficient development platform and testing platform for testing our algorithmic ideas and developing our algorithmic ideas very efficiently. But what we want to do is build these general purpose algorithms and we want to apply them once we train them in game situations, we want to apply them to real world applications, which is obviously the purpose of this society.
[0:45:15] And there's a couple of things where I can imagine the sorts of things AlphaGo-like techniques might be useful for will be areas where there's huge combinatorial explosion, but there's a clear objective function, and even better, maybe there's some kind of simulation of that real-world system. And I can imagine areas like drug discovery or material design as being potentially quite amenable to these kinds of intuitive search processes.
[0:45:43] We're also applying this technology to the real world right now. We have a great
[0:45:49] Pilot program with the NHS to try and help with medical diagnosis, things like cancer imaging, and all these kinds of things that we think could be made more efficient and improved by having an AI tool working together with the clinicians. Smartphone assistance. Also, I think there's a big opportunity in personalized education. All sorts of areas I think this kind of technology is going to end up touching.
[0:46:18] One thing we did straight after AlphaGo as a proof of concept was we applied variants of the AlphaGo algorithm to data center optimization, optimizing Google's data centers. And what we did is we hooked it up to the cooling system of these huge data centers that Google has.
[0:46:36] And we found out that we actually could save 40% of the energy the cooling systems used, which translated to 15% energy saving across the whole data center. So we rolled that across all of Google's data centers, and it's saving hundreds of millions of dollars a year in energy, as well as obviously being great for the environment.
[0:46:54] And the really amazing thing is that Google already has the best data center engineers and the most efficient data centers. So when we suggested this to the data center engineers, they said, good luck. Try it. But you won't find anything. And we tried it. We switched it on. And one week later, we were getting this 40% saving. So now we're thinking about
[0:47:15] With them, the data center engineers, designing new data centers that have more controls in them and more fine-grained control than a human operator could cope with. And we think we could even save even more.
[0:47:28] I'm just going to finish now by just going back to that step two. Hopefully, I've convinced you there isn't quite as fanciful as it might first have seemed at the beginning. And I like thinking about meta solutions. And I think they're more efficient to work on, obviously, than just solutions. And I see, as today, the biggest challenges facing us as a society are information overload and system complexity of everything around us. We're sort of deluged by information and data. And the question is, what do we do about it?
[0:47:57] How do we find the structure and the insights in that data? And then what about the systems we want to solve, like from disease to macroeconomics to climate? They're such hugely complex systems. I think we're getting to the point now where the systems we would like to master as a society are so complex, it's difficult for even the smartest humans or even a group of the smartest humans to fully master these systems completely unaided.
[0:48:24] So I think solving AI in the kind of way we're proposing is potentially a meta solution to many of these other problems. Obviously, one word of caution that I always talk about is, as with any powerful new technology, and AI is no different, it has to be used ethically and responsibly. So the technology itself is neutral, but it depends on how we decide to deploy it and how we decide to distribute the proceeds and the benefits that come from technology. It's got to be for the benefit of everyone.
[0:48:55] And then I'll just end by saying that my personal dream has always been, and the reason I've worked on AI my entire career, is I wanted to make AI scientists or AI-assisted science possible, science and medicine possible, to really accelerate the pace of scientific innovation in the world. Thanks for listening.