[0:00:10] So I'm going to talk about optimality and a little bit about intuition and creativity in respect to AI, which are sort of three topics that I've been thinking about a lot over the last year. But before I start, I just want to put up this diagram, mostly for the benefit of Jan. And so at DeepMind, we think reinforcement learning is a really great and general sort of way to think about intelligence.
[0:00:39] In the sort of very high level sense of an agent that builds a model of the world around it based on its experiences and observations, including any rewards it gets. And then it uses that model to imagine the future and make plans and action sequence plans about how to get towards whatever goal it might have in mind. And I think within that,
[0:00:59] If we can solve all the problems within that, we'll have solved intelligence. And indeed, this is the way that natural intelligence works. Now, of course, there's obviously lots of issues with credit assignment and data efficiency and many, many issues that are known about reinforcement learning. But we're working on, and many other people are working on, many solutions to those problems, including imitation learning, hierarchical planning, and so on, transfer learning that can get around some of these data efficiency issues.
[0:01:25] But with that, there's lots of things I'd like to talk about, but I've only got 20 minutes. So I'm actually going to focus most of this talk on AlphaGo and what's happened in the last year, but even actually as recently as the last couple of weeks, and the kinds of general things that we've taken from that in terms of progress towards AI. So you can think of this as a proxy, if you like, for the progress we've made over the last year or two towards AGI.
[0:01:51] So AlphaGo, hopefully most of you know now how to play Go by now with AlphaGo out there, but for those of you who don't, this is a Go board. Go is played on a 19 by 19 board. It starts off totally empty. Black and white take turns to put stones on the board. They can put them anywhere on the board, but once they're placed, they stay there, so they don't move on the board. Now, the aim of the game is very simple.
[0:02:15] It's to surround empty spaces of territory or also surround your opponent's stones to capture them. And the player that surrounds the most amount of territory wins the game. Now, Go itself is originated in China 3,000 years ago. It's more popular than ever now today. It's 40 million players across Asia. And there's 2,000 professionals who make their living just off of Go.
[0:02:42] And they spent their entire lives studying Go. And in fact, they've been professionals in Go for many hundreds of years now. And in terms of the complexity of the game, I think it's the most elegant game that mankind has ever devised. So there are really only two rules. If I had an extra five minutes, I could teach you those rules. But those two rules end up creating an immense amount of complexity.
[0:03:03] And one way to think about that is there are 10 to the power 170 possible positions in Go, possible board positions, which is more than the atoms in the observable universe. So you can't brute force sort of solve Go, which is one of the reasons why it's been so hard for computers to play.
[0:03:21] So if we compare this to chess, obviously its closest neighbor, in Go there's a branching factor of 200. There are 200 possible moves in an average position in Go, whereas in chess there's about 20. So it's an order of magnitude higher branching factor. But actually the more difficult problem is how to write an evaluation function that will tell the system who is winning in a particular position, whether it's black or white. And it's very difficult and it was thought to be impossible for many years to create this kind of evaluation function.
[0:03:51] So let's take a look at why that's hard. Well, I think it actually is impossible to probably create a handcrafted set of logical rules to tell the system who's winning. And that's how chess evaluation functions generally work, chess computers. So they distill a bunch of knowledge from human grandmasters, and they codify that up in databases of rules, and those end up giving an equation which the system can then use to determine whether a position is good or not.
[0:04:22] And the reason that's so hard in Go is for multiple reasons. So one is that there's no concept of materiality in Go. So in chess, as a first approximation, you can just add up the pieces on each side of the board. Queen is nine points, rook is five points, and so on. And whoever's got more points is generally winning. In Go, all the pieces are worth the same. Also, Go is a constructive game.
[0:04:44] And in this sense, chess is a destructive game. And what I mean by that is in chess, all the pieces start on the board and they get removed as the game goes on. So the game gets simplified as the game goes on. In Go, it's a constructive game. So the board starts empty and you put pieces on the board. So that means if you're trying to evaluate a mid-game position, a complex mid-game position, the information isn't just on the board. You also have to make a prediction into the future about what might happen.
[0:05:12] Whereas with a chess position in the mid-game, you have all the information there that you need to make that evaluation. Also in Go, very small change to even like one stone by one position, one square, can totally change the evaluation of the position. So it's very brittle to changes. Because of all this, the way humans get around playing this to very high level is they generally use their intuition rather than
[0:05:40] Calculating abilities like you would in chess. So if you ask a great Go player why they've made a particular move in a very complex game, they'll often tell you it felt right. They'll use those words. Whereas a chess grandmaster will never really say that. They'll tell you, you know, I thought A was going to happen, then I was going to plan B, and then I thought C would happen. Now that plan may not turn out to be very good, but they'll usually have an explicit plan in mind. With Go, it's much more implicit.
[0:06:06] So what we did to get around both of these problems is turn to deep neural networks. And in fact, we trained two neural networks to get around both of these two challenges. So we created a policy network that basically learned to public distribution over moves, which moves were more likely to be valuable to assess. And this was initially bootstrapped from human data, actually amateur games from an online Go server, and then improved through reinforcement learning.
[0:06:34] So the program played against itself and improved those predictions until it could beat the human trained version of the policy net. The second network we created was a value network, which took all of these games, self-play games that AlphaGo played against different versions of itself and created a huge database of millions of positions where you had the position as the input, but you also knew the final result of the game.
[0:06:57] And that allowed us to create this fabled evaluation network. We call it the value net, which could give you a probability of who is going to win from that position based on the configuration of the pieces. So the input would be the board, and the output would be a value between 0 and 1, a real number between 0 and 1, telling you whether white or black was winning with 0.5 as meaning an even position.
[0:07:18] And we combine this together with Monte Carlo Tree Search to create AlphaGo. I haven't got time to go into the technical details of this, but if you're interested, you can check out our Nature paper on AlphaGo, which has actually came out in...
[0:07:32] February of this year, actually February, March of this year. And since then, Chinese company Tencent have created an AlphaGo clone, which is now very powerful. And also a Japanese project in this has also created a program called Deep Zen. So they've used these techniques to create their own Go programs.
[0:07:52] Now, as many of you know, we took this program forwards to AlphaGo for our ultimate challenge was to take on Lisa Doll, who's a sort of legend of the game, acknowledged as the greatest player of the past decade and has 18 world titles. And surprisingly, we won the match in Seoul 4-1, and it was sort of acclaimed by many of the experts in this field, both in Go and in AI computer Go, that it was about a decade before people were expecting this breakthrough to happen.
[0:08:23] Now, it was important, obviously, that we won the match, but actually what was more interesting was how AlphaGo played and the kinds of moves that it came up with. So I'm just going to quickly show you one special position from Game 2, which was probably the most famous thing that AlphaGo has done. And this was the position in Game 2, and AlphaGo here is black, and it's on Move 37, so it's relatively early in the game, and Lisa Doll is playing the white stones.
[0:08:52] And AlphaGo played this move here on the right. Hopefully you can see that at the back there, which is called a shoulder hit move right next to the white stone here. And this is considered to be an incredible move and totally unthinkable for professionals to do. And I'll just try and briefly explain why this move was so astounding to the Go world. So there are two important lines in Go. There's the third line, which is marked here. Now, if you play a stone on the third line, what you're saying to your opponent is that you're interested in taking territory.
[0:09:21] Surrounding empty space to the side of the board. So obviously, if you play on the third line, you're trying to take territory. Instead of that, if you play on the fourth line, what you're saying to your opponent is, you can have some territory on the side of the board, but I want power and influence towards the center of the board, inwards looking. And the idea of playing on the fourth line is that later on in the game, you can use that power and influence to give you points somewhere else on the board much, much later in the game.
[0:09:50] Now, the perceived wisdom of 3,000 years of playing Go is that playing on the third and fourth lines is about equally balanced. So the trade-off you get from the influence from playing on the fourth line, from giving your territory away to your opponent on the third line is about equal. So that's why this move is so astounding because AlphaGo here is telling Lisa Dolls, well, in fact, it's signaling that it's worth taking influence to the center of the board from the fifth line, so one line further in.
[0:10:17] Giving away territory from the fourth line. So that much more territory. So that's considered to be very suboptimal in Go. And hence, this move is incredibly surprising for any kind of Go professional to eat. And they wouldn't even consider moving somewhere like that. So it turned out that this move ended up being key to AlphaGo winning the game. And the reason that happened is around 50 moves later, these two stones I've marked here in the bottom left of the board
[0:10:46] The fighting from the left-hand corner ended up spilling out into the center of the board, and ultimately, these black stones connected with this move 37 stone on the right, around 50, 60 moves later. And that's what ultimately, that motif is what won the game. So interestingly in Go, Go is considered to be a sort of art form in Asia, more akin to poetry or something like that than it is to just a game.
[0:11:11] But the interesting thing about Go is sort of like objective art in the sense that any one of us here could come up with a random move that would be new or novel. But the question is, is it any good? And in Go, of course, we can put this to the test. So this move 37 has gone down in history not because it's novel, but also because it was critical to the win here. So there's some new idea here that's worth studying.
[0:11:34] Now, obviously, Lee Sedol came back in game four, and he won his match in game four, which he said was the most valuable win of his career, with his own incredible inventive move on move 78. Now, I haven't got time to go into why this move was amazing, but it was interesting that his creative qualities were able to come up with something like this that fooled that version of AlphaGo.
[0:11:57] So I only want to mention a little bit about intuition and creativity. This whole game is about intuition and creativity. That's the way humans play it. So, you know, has AlphaGo sort of displayed these attributes? Well, I think to talk about that, we need to sort of give some definitions to these two terms that I've used. So, you know, one, I think we're all about intuition. One way of operationally sort of defining it is that it's sort of implicit knowledge that's acquired through experience, but it's not consciously expressible or even accessible.
[0:12:26] So we can't access that knowledge consciously ourselves and certainly not in a way that we can express to others. So you might ask, well, how do we know that knowledge exists and the quality of that knowledge? Well, obviously, we can verify this behaviorally by testing it. In Go, that's very easy by testing what the system would do in a particular position. So what about creativity?
[0:12:47] Well, one operational definition of creativity, and I'm not saying this by any means this covers all of creativity, is the ability to synthesize knowledge you already have in such a way to produce a novel original idea in the service of some goal. And I think at least by these two definitions of intuition and creativity, I think AlphaGo has clearly demonstrated these abilities, albeit in a constrained domain like Go. So the impact of the game was pretty huge. 280 million people watched the match.
[0:13:19] So actually just over the new year, the Christmas break, we unleashed our new version of AlphaGo on the internet and on the online Go servers, the two most popular ones in Asia. And actually AlphaGo won its match of 60-0.
[0:13:41] against all the world's top players. So actually all the top 20 players in the world played against it in the end, including we beat the world number one, KJ, 3-0. So it does seem like we fixed those knowledge gap issues. And what was really interesting is that these games were...
[0:14:00] Massively reported across Asia and thousands of people watching the online games even though we didn't officially announce them as AlphaGo matches until after the matches because we wanted a clean test on these.
[0:14:12] These 60 games have now become a treasure trove for the Go playing world. And AlphaGo has come up with many innovations that now the professionals are analyzing and combing over and thinking about incorporating into their own games. So here's two here. So here's AlphaGo, in both cases, black.
[0:14:31] And here in one position against the number 20 player in the world, AlphaGo plays in the bottom right-hand corner here on what's called the 3-3 point. And this is considered, if you were a novice and you played this, your teacher would basically slap your hand because this corner is way too small, considered to be way too small to play this early in the game. And yet AlphaGo has shown that this is a plausible strategy. Here's another example on the right-hand side, I think against the number 10 in the world.
[0:15:00] Where AlphaGo has crawled what's called crawling along the second line. Now that's considered to be also a complete unthinkable thing to do because you're getting so little territory for your moves. You're just getting one line of territory on the side of the board. And yet AlphaGo here shows that this can be a very extremely strong strategy in certain conditions.
[0:15:20] So these things are completely overturning 3,000 years of received wisdoms, these sort of ancient Chinese proverbs, if you like, that have been handed down through generations. So Gu Li, so you're going to have to take my word for it, this is what some of the top players in the world said about AlphaGo after they played these matches over the Christmas break. So Gu Li is a legend in China. He's won the most world titles of anyone in the world. And he said afterwards, together humans and AI will soon uncover the deeper mysteries of Go.
[0:15:51] Kuji, the world number one, the current world number one, he said after the matches, humanity has played Go for thousands of years, and yet, as AI has shown us, we have not yet even scratched the surface. The union of human and computer players will usher in a new era. Together, man and AI can find out the truth of Go. So this is really, for them, this is hard for us to understand in the West, but for them, this is kind of very profound exploration for them. It's a bit like the way we would explore science. So I think this is kind of an interesting...
[0:16:19] A paradigm for us to think about AGI in general in more topics as we improve AI techniques and they become more general. The idea of humans and machines collaborating together cooperatively I think will achieve amazing things. I think of AI and human ingenuity augmented by AI tools as unlocking our true potential. So I think of AlphaGo as
[0:16:41] For the Go players, at least, a bit like the Hubble telescope is for astrophysicists. You know, kind of ultimate tool for them to explore their particular universe. In this case, the game of Go. So that made me think about optimality and discovering optimality and sort of, you know, these ideas of thinking how far off optimal are we? And in Go, this is a natural topic of conversation. They talk about is how many handicap stones, how many...
[0:17:08] What head start stones would they need in order to be able to compete against God or perfect play? And the assumption was that the top players were around two stones away from perfect play. So that means if they got two stone head start, they'd be able to play, be a perfect player.
[0:17:27] And what we've shown is that after 3,000 years of study, this is not nearly sufficient enough to find optimal play. And we're actually a long way away from that. But the interesting thing is that unlike chess programs, I think AlphaGo is going to help humans get better at Go. Because this happened in chess, obviously, 20 years ago with Deep Blue and the following programs. But chess programs are generally better than human players because they're better tactically.
[0:17:53] They don't make any mistakes and they have huge databases of both openings and end games that are solved. So for example, in end games, I think the latest is any nine piece or less position is now being solved.
[0:18:06] Offline by huge brute force. So obviously, no human player, you can tell a human player, you know, make less mistakes. That doesn't help them. Or you can say to them, you know, in this eight-piece position, we know that white is winning. But that also doesn't help. You know, they can't memorize millions of 200-move sequences. But AlphaGo is better strategically.
[0:18:26] And I think what is actually shown in these sort of games we played is that these are qualitative new ideas, qualitative new ideas that players can potentially absorb and use in their own games. And this has already been happening. So I think the strength of human play is going to see huge improvement as we release more and more games of AlphaGo and eventually maybe the program itself. So I think...
[0:18:48] What's happened here is that we've shown human players that are actually being stuck in a local Maxima, and we've shown them that this can be broken out of.
[0:18:58] And maybe this is going to usher in a new era of Go. Many people are talking about this as in history of Go, there's been two big giant leaps of progress. One several hundreds of years ago with more modern Go techniques, and then one again in 1930s with this legendary player called Go Saigen, who was in his prime in the 30s and 40s. And it's been thought that human players improved by two stones during that period.
[0:19:23] When Go, Cygain introduced all these new ideas. People are talking about this AlphaGo era as maybe a new era in Go like that. So AlphaGo has enabled these fascinating mysteries to be explored, but there are countless other domains that also suffer from this combinatorial explosion. Obviously, we're not doing this just to play Go. We want to use these general techniques to use in other areas. We think of AGI as the ultimate tool for us to investigate and explore the universe around us.
[0:19:53] And I think there are many, many areas of science I think we can apply this to. And here are two examples I think that would be particularly amenable to this sort of approach. Material design, like superconductor design, and perhaps drug discovery, I think might be two areas where we can apply these kinds of techniques.
[0:20:11] And we're already applying it in areas of healthcare. We're looking at robotics. And also just earlier this year, last year, looking at optimizing data centers too. So we used variations of the Africa algorithms to optimize the Google data centers and the power they used. And we saved 15% of the power using these data centers. And we're now rolling that out across the fleet.
[0:20:33] So what are the outstanding challenges? So we've made a lot of progress. There's been huge excitement over the last five years, but there's plenty of challenges yet to have been solved that we're going to need if we want to get anywhere close to AGI. So here's just a list of ones that we are working very hard on. Imagination-based planning with generative models, hierarchical planning, unsupervised learning we've heard about, memory and one-shot learning.
[0:21:00] So sort of having episodic memory, which is our subject of our recent Nature paper. Learning abstract concepts, continual and transfer learning. And we think these things are all key topics, and we're working very hard on these, and hopefully we'll have some work to announce on this in the next year or so. So in terms of solutions and AI, I think everyone in this room probably agrees with these sentiments. We're confronted at the moment with huge information overload and system complexity on the sort of domains that we would like to master.
[0:21:30] Whether that's climate or disease or macroeconomics. And I think these are huge challenges for humans to tackle unaided, even the world's top human experts. So I think solving AI is potentially a meta solution to many of these other problems. And that's why we work so hard on AI at DeepMind.
[0:21:50] But obviously, the caveat and again why we're all here is that all these sort of powerful new technologies and AI is certainly no different, must be used ethically and responsibly for the benefit of all. And that's why we've been helping with efforts like the partnership on AI, and I know there's many other people in the room who've worked hard on that, as a first sort of baby step towards coordinating ethical use of this kind of technology.
[0:22:15] The dream, certainly for me and many at DeepMind, is to use this kind of AI that we're talking about to create AI scientists or AI-assisted science and to make AI-assisted science and medicine possible. Thanks for listening.