[0:00:05] So I'm going to talk about artificial intelligence and the work we're doing at DeepMind, especially concentrating on our most recent work over the last sort of year. So I've titled this talk, Exploring the Frontiers of Knowledge, which is what I think we've been doing in a number of domains now in the last 12 to 18 months. So for those of you who don't know, DeepMind is a company founded in 2010 in London, and we think of the company as a kind of Apollo program effort.
[0:00:34] for artificial intelligence. So we have over 400 research engineers and scientists, more than 250 PhDs. And I think it's the biggest collection of scientific sort of brain power anywhere in the world focused on this topic. And what we try to do as well is create a new way of organizing science. So you can think of the environment at DeepMind as a kind of hybrid between the best of academia,
[0:01:02] and Academic Labs combined with the intensity and the focus of the best startups, the best Silicon Valley startups. So we tried to fuse those two cultures together and try and take the best from both. Now, in 2014, we joined Google to turbocharge our mission. And the way we articulate our mission is in a two-step process. So step one, try and fundamentally solve intelligence, both understand natural intelligence and recreate it artificially.
[0:01:33] And then once you have that technology, step two we think naturally follows. Use it to solve absolutely everything else. Now that may seem a little bit fanciful to you, but I hope by the end of this talk you'll be convinced that this is at least plausible once we've created this kind of technology. So more prosaically, the way we think about our mission is to build the world's first general purpose learning system.
[0:01:59] So we really stress the two words, general and learning. So all the algorithms and the systems we build at DeepMind are learning, have learning at their heart. So they learn how to master tasks directly from raw experience or raw data. They're not told anything about the environment they're put in. They have to learn everything from first principles.
[0:02:20] And the second thing is this idea of generality. The idea that one single system with one set of algorithms out of the box can operate across a wide range of environments. So it can transfer their knowledge from one domain to another domain seamlessly. Of course, we have an example of such a general purpose learning system. It's the human brain.
[0:02:40] And that's why at DeepMind, we also have a whole group that looks at neuroscience to try and be inspired by neuroscience and the way the brain works to think of new architectures and algorithms that might work for machine learning. So our first big breakthrough happened about two or three years ago now. And what we did is, at DeepMind, we use virtual environments like games as a training ground for testing our AI algorithms on.
[0:03:06] Games are the perfect training ground for this because, in a way, they're kind of microcosms of real life. What we're interested in is artificial agents that can achieve goals and process their perception inputs around them and build models of the world they find themselves in to achieve a certain task or a certain goal. So games are very good for this. And we started off with
[0:03:30] The iconic games from the 80s, the Atari games. And actually our first big piece of work which was featured in Nature was to build a system that could play any Atari game directly from the pixel inputs. Only receiving the pixel inputs as the only data it would receive. So it has to figure out for itself what the rules of the game are and how to score points. And it does this by a process of trial and error and actually experiencing the game for itself.
[0:04:00] But today, I'm going to talk about our more recent work on a system called AlphaGo. Now, AlphaGo is a program designed to play the ancient board game of Go. Now, for those of you who don't know what Go is, this is what the board looks like. Go is played on a 19 by 19 grid, and there's two players, black and white, and they take turns to play stones on the board. Once a stone is placed, it's not moved anymore. It just stays where it is.
[0:04:27] And the board starts off empty and gradually fills up. Now, the aim of the game is to surround your opponent's stones or surround and wall off empty areas of territory. And the end of the game might look something like this board configuration. And then you count up the areas that you surrounded and you add up to the number of stones that you've captured from your opponent. And the player that has the most points wins the game.
[0:04:53] Now, the thing about Go is that it only has two rules. I could teach you the game in five minutes, but it leads to incredible complexity. It's probably the most elegant game that mankind has ever devised. And in fact, the history of Go is a long and storied one. It's over 3,000 years old and originated in China. And it's what they play in Asia instead of chess. In fact, it's more popular than chess, and it's more deeply ingrained in the culture.
[0:05:21] So today, there's more popular than ever. There's 40 million active players and over 2,000 professionals, people who make their living only from playing Go. And those 2,000 professionals come from a whole series of Go schools.
[0:05:38] What happens in Asia, in Korea, in Japan or in China is if you show promise in the game of Go at the age of sort of five or six or seven, you get taken out of normal school and you get put into Go school where you study Go 12 hours a day, seven days a week with your peers who are also trying to become professional Go players. This is taken really seriously and it's been like this for hundreds of years.
[0:06:03] Now, one way to illustrate the complexity of the game is that there are more board configurations in the game of Go than there are atoms in the universe. So there's 10 to the power 170 possible board configurations. So there's no way that you can solve this game through brute force calculation. It's much too complex. Even if you took all the compute power in the world and you ran it for a million years, that wouldn't be enough compute power to calculate all the variations in Go.
[0:06:32] And that's one of the reasons that makes Go so hard for computers to play. It's taken 20 years since Deep Blue famously beat Garry Kasparov in 1997 at the game of chess. And ever since then, cracking the game of Go has been one of the holy grails of AI research.
[0:06:50] And there's two reasons it's been so hard. One is this complexity I've just talked about. So the search space in Go is really huge. What we refer to is the branching factor of a game. And in Go, in an average position, there are 200 possible moves. If you compare that to chess, in an average position in a chess game, there are only 20 possible moves. So there's an order of magnitude more possibilities every move in the game of Go.
[0:07:15] The second big challenge, even bigger challenge, is that it was thought impossible until AlphaGo came along to write down an evaluation function to tell the system who is winning, whether it's black or white, and by how much. Now, an evaluation function is critical to a computer system to play any game in order for it to decide which amongst the many possibilities is the best one for it and the worst one for its opponent.
[0:07:40] So in Go, for 20 or 30 years now, people have been trying to write handcrafted, pre-programmed evaluation functions like they do in chess to describe to the system who's winning. But in Go, because there's no concept of materiality, all the pieces are worth the same. There's just this concept of a stone. It's very difficult to write a function directly that tells the system who's winning. So how do humans, human professionals, and the best human players cope with this enormous complexity in these two problems?
[0:08:12] Well, actually, they rely on their intuition and their instinct, much more so than chess players do. So chess is really a game about calculation. If you ask a professional chess player why they made a certain move, they'll usually tell you, I was thinking A would happen, then I was going to do B, and then I thought C would happen. So they usually have an explicit plan. Now, that plan may not turn out to be good at the end of the day, but they usually can explain to you explicitly why they've made a move.
[0:08:38] Now, in Go, it's very different. If you ask a top professional player in a complex game why they played a move, they might just tell you it felt right, and they'll use those words. So they're really using their intuition and instinct rather than calculation. So how did we overcome this problem in our system AlphaGo? Well, we turned to deep neural networks to kind of capture this intuitive aspect of Go.
[0:09:03] We trained two neural networks actually, one called a policy network that took a board position in and output a probability distribution describing the likelihood of each of the possible moves on the board. And what that means is instead of having to search out 200 possible moves every position, you can just look at the top five or six most probable ones. And the second thing we did is
[0:09:24] We improved the system by having the system play against itself millions of times and over a matter of months, and each time learning incrementally from its mistakes through a process of trial and error called reinforcement learning, and incrementally getting better with each practice game, much like humans do.
[0:09:44] And through that process, we were able to build a second neural network called a value net, which took a board position in, this board position in blue at the bottom here, and output a single number, a real number between 0 and 1, which was the system's prediction about who was winning the game. 0 being white, completely winning, 1 being black, completely winning, and 0.5 meaning an equal game.
[0:10:08] So this value net, this pink neural network here on the left, is the fabled evaluation function. But instead of programming it directly, we got the system to learn it for itself. So I haven't got time to go into the technical details of this, but if you're interested, you can read our Nature front cover article on this that came out earlier this year, sorry, last year. So we took this system and we wanted to test it, bring it to the ultimate sort of challenge.
[0:10:39] And in March of last year, we took on a legend of the game, the South Korean grandmaster Lee Sedol. He's won 18 world titles and is widely acknowledged to be the greatest player of the past decade. And we played this match in Seoul, five-game match for a million dollars first prize, and the whole of Korea basically came to standstill. At one point, every national TV broadcaster was broadcasting the match live, all of the different TV channels.
[0:11:08] So we played this game. It was an amazing once-in-a-lifetime experience. And we managed to win, to everyone's surprise, the match 4-1. And this was described and proclaimed as a decade before its time from the AI experts and the Go experts. And that's very unusual for all of you involved in technology for something to happen a decade before it was predicted in technology and engineering. But more importantly than the fact that AlphaGo won this match was how it won.
[0:11:38] And I just want to take a moment to try and explain to you some of the innovations that AlphaGo came up with. So in the game of Go, there are every now and again, every sort of 20 years, there's famous games are played where a move so astounding is played. It's recorded for all history. It's written about in the Go books. And it's sort of regarded as a sort of mystical move. And I think in this five-game match, there were several moves like that. But this is my favorite one.
[0:12:05] This is a position from game two and this is move 37. AlphaGo is black and LisaDoll is white. And AlphaGo played this move here on the right hand side marked by the white triangle. So it's called a shoulder hit because it's off to the side of that white stone. Now this move was totally unthinkable to all the professionals in the Go world. And I just want to take a moment to try and explain to you why this move was so creative and so unthinkable.
[0:12:35] In the game of Go, there are two important lines on this 19 by 19 board. There's the third line here. Now, if you play a stone on the third line, what you're saying to your opponent is you're trying to surround territory to the side of the board. So this is how you get points at the end of the game. So you're trying to claim territory. Instead of that, if you play on the fourth line, what you're saying to your opponent is you can take territory on the third line, but I'm going to take power and influence into the center of the board.
[0:13:02] And the idea of playing on the fourth line is that later on in the game, that power and influence will translate into points and territory somewhere else on the board. Now, for 3,000 years of human study and contemplating this game, the third and fourth lines were considered to be equal. So if you play on the third line, your opponent plays on the fourth line, that's more or less an equal exchange. But instead of that, AlphaGo played on the fifth line.
[0:13:28] And this is completely thought to be an unthinkable idea because if you play on the fifth line, you're giving your opponent territory from the fourth line. So they're getting much more territory for your influence. But AlphaGo seems to like this move and moves like this. So it may be that for 3,000 years, humans have been underestimating the power and the use of this influence towards the center of the board and sort of underrating its value.
[0:13:58] So later on, the interesting thing about Go is that it's considered to be more than just a game in China and Japan and Korea. It's considered to be more like an art form, something more akin to poetry. In fact, Confucius wrote about Go as one of the four arts that had to be mastered by any true scholar.
[0:14:17] And the nice thing about Go is not only is it art, it's actually objective art. Because any one of us here could come up with some random move which would be creative in the sense that no professional would have thought of it. But what makes this move truly creative is the fact that it ended up working. It ended up winning the game, this game two, this critical game two, because of this move.
[0:14:39] So what happened is that around 50 moves later, these two stones I've marked on the left-hand corner here with the white triangles ended up spilling into the center of the board, the fighting from the left-hand corner. And around 50 moves later, that fighting ended up joining up with the stone, that move 37 stone on the right-hand side. And that stone was perfectly positioned to tip the balance of that fight in favor of AlphaGo. And that's why it won the match.
[0:15:07] Of course, Lee Sedol himself was inspired to come up with his own incredible move in game four, which is the game that he won on move 78. I haven't got time to explain to you why this was so incredible, but also this move will probably go down in history too. Now, the interesting thing about AlphaGo's moves is that move I just showed you, that move 37, AlphaGo gave that move a probability of 1 in 10,000. So in some sense, the AlphaGo system knew that this move was very unusual.
[0:15:35] Because it gave the probability that a human would play that move as 1 in 10,000. So it wasn't just learning and copying what humans do. It was actually innovating. So the impact of the AlphaGo match was huge. 280 million people watched the five games across that week, 35,000 press articles. And my favorite statistic was the sales of online Go boards were sold out across the world for the next month as people picked up this great game.
[0:16:06] And I just want to take a moment to discuss intuition and creativity, which I think is what AlphaGo has demonstrated here. Now, I sort of use the word intuition quite a lot, but what do I mean by it? Well, I think one operational definition, one useful way to think about intuition is that it's implicit knowledge that has been acquired through experience, but it can't be consciously expressed or communicated. Now, you might ask, well, if you can't express it or even access it consciously, how do you know that knowledge is there?
[0:16:37] Well, we can test implicit knowledge, and neuroscientists know this, by testing it behaviorally. It can verify and test it behaviorally. And with Go, of course, this is very easy. You can give somebody a Go position, and you can test what move would they come up with and verify the quality of that move. So what about creativity, then? So creativity, one operational definition of that would be the ability to synthesize knowledge that you have, you've picked up through experience, to produce a novel or original idea in the service of some goal.
[0:17:06] Now, I'm not saying that encompasses all of creativity, but I think that captures some of the essence of creativity. And at least by these two operational definitions, I feel that AlphaGo has clearly demonstrated these abilities throughout these games, albeit, of course, in a constrained domain of the board game Go. So you might think, well, we beat the world's best player. What else was there left to do with Go? Well, actually, we continue to work on this.
[0:17:34] system, AlphaGo, for the next nine months. And we're still working on it now. And we're working on it because we wanted to explore the limits of knowledge, this idea of exploring the frontiers of knowledge. We first wanted to fix the knowledge gaps that we saw in game four that caused us to lose that game against LisaDoll. We wanted to optimize the performance of the system.
[0:17:54] We wanted to kind of give back to the Go community with AlphaGo and help improve human play through them being able to explore and test themselves against AlphaGo. But in short, what we were after was the scientific idea of trying to create perfection and understand what that was. And here we had a domain where we could potentially try and do that.
[0:18:18] And earlier, actually just over Christmas and early in the new year, just a couple of weeks ago, I'm pleased to say that we released a new version of AlphaGo online, and it won 60-nil against all the world's top 20 players on the two most popular online Go servers, including the current world number one, KJ, who we beat 3-nil, which is really astounding. And it's caused huge waves in Asia again. Not only, again, did it win 60-nil,
[0:18:48] But those games are now being studied in all these professional Go schools. And new books are being written. And old books are being rewritten about theory about Go.
[0:18:58] Because of these new innovations that AlphaGo has come up with. And here are two more, playing as black here in the bottom right-hand corner, another unthinkable move, or here on the right-hand side against the world number two, crawling along the second line, which is both of these things were considered to be extremely bad ideas in Go theory. And AlphaGo has shown that they're very plausible in certain types of positions.
[0:19:22] And then, interestingly, the reaction of the Go world afterwards, there were some really interesting quotes from these top Go players. So this is Ke Jie, the Chinese in world number one. And he said, humanity has played Go for thousands of years, and yet, as AI has shown us, we have not yet even scratched the surface. And he said, the union of human and computer players will usher in a new era. Together, mankind and AI can finally find the truth of Go.
[0:19:50] And that's how they very much see Go, the game of Go. It's kind of almost like the way scientists would see their discovery of the natural world. They feel they're discovering the universe through this game. So people are talking about this as a new era in Go. And there is precedent for this. In the 1930s and 40s, this incredible Go player called Go Saigen changed the face of Go and improved people by several notches through his new theories and new ideas.
[0:20:21] And there's been two eras in Go like this, Go Saigen being the most recent one. And people have been thinking and saying this is now the third era of Go with AlphaGo's new innovations.
[0:20:32] And I think that AlphaGo will actually make human players better. And you can sort of ask, well, why didn't chess programs, when they beat Garry Kasparov, really make chess players materially better? Magnus Carlsen, who now is the current world champion, is a fantastic player, but probably about the same level as Garry Kasparov was at his height. They're the same rating. And I think the reason that is is that chess programs are better tactically.
[0:20:56] So they calculate more. They don't make mistakes. They have things like opening and end game databases. So these are all things that the computer is better at than the human. And really, humans can't learn from that. We can't just calculate more or be told, don't make mistakes. Obviously, the chess players are trying not to make mistakes. So that doesn't really help.
[0:21:14] Whereas AlphaGo is actually better strategically and thematically. It actually has new ideas about the game. And these ideas can be analyzed and absorbed and potentially used by the best human players. And we're already seeing that even a couple of weeks after these 60 matches. All these new innovations, proliferation of new innovations. And I actually think this has opened up the human Go professional world's eyes that actually maybe humans were stuck in a local Maxima for 3,000 years.
[0:21:44] And we've only just realized that now. And actually, some of the Go players have been saying their minds, having seen AlphaGo, their minds have been freed. They feel like they've been freed from this cage of tradition. So they can actually go and explore for themselves all sorts of new ideas. I think this is really interesting in future when we think about humans and machines together.
[0:22:08] And I think that humans and machines collaborating together in a complementary way will allow us to achieve really amazing things. And I think of AI as this incredibly powerful tool that will augment human ingenuity and unlock our true potential. In fact, one way you can think about AI, and indeed AlphaGo, is like, I think of it as analogous to the Hubble telescope, a kind of ultimate tool to explore the universe.
[0:22:37] Of course for Go players, AlphaGo was allowing them to explore their universe of the game of Go. And I think there are many other domains in the real world that suffer from this kind of combinatorial explosion that Go has.
[0:22:53] Now, obviously, as I said at the beginning, we test our systems on games because they're the most convenient way to develop our AI algorithms. But obviously, ultimately, we're not interested in just being good at games. We want to translate those algorithms into the real world and be useful and make huge impact on real world situations.
[0:23:11] And one reason we believe we can do that is because we're building general purpose learning systems. They're not being handcrafted for, say, chess. We've actually built, we believe, general purpose algorithms that could be taken from the games that we test them on and applied to real world.
[0:23:29] So here are two areas that we're looking at that have this kind of iterative optimization process, drug discovery, material design. But there are many, many other domains as well that could benefit from this type of algorithmic exploration. And we're applying these to all sorts of other areas, health care, robotics, and even optimizing data centers.
[0:23:52] So a variation of AlphaGo we took over the last summer, and we applied it to Google's data centers. And we managed to save 15% of the power that was used in those data centers by controlling the cooling systems more efficiently.
[0:24:07] 40% more efficiently. So obviously, this is great. It saves tens of millions of dollars in terms of power usage. And of course, it's great for the environment. And we think that we could use these same algorithms, perhaps, on entire national energy grids, which are, in some ways, much bigger versions of these data centers. So it's just one exciting area that we think we can apply these kinds of algorithms to.
[0:24:33] And I'm just going to finish by coming back to that point two of our mission statement and the idea of what I call is meta solutions. So I think in the world today, we're faced with some really complex situations. All these systems we would like to master, from disease to macroeconomics to particle physics, contain enormous amounts of data now. We're kind of deluged by information.
[0:24:56] And the complexity of these systems is so high, and in some cases bordering on chaotic systems, that we have to confront the idea that maybe they're too complex for humans to understand unaided. And this presents huge challenges for us today as a society. And I think solving AI and having this incredible system as a tool that we can use and deploy is potentially a kind of meta solution to many of these other problems.
[0:25:25] Now, obviously, AI is a very powerful technology. And like all powerful new technologies, it must be used ethically and responsibly. And we're in the vanguard of trying to do that. And along with several other big companies, we created earlier last year the partnership on AI for the beneficial use of AI for all of society so that everyone benefits from it. And there's a lot more to come on that front.
[0:25:50] And my personal dream is that once we've cracked AI, what I'd like to do is use it to create AI scientists or AI assisted science to help drive science and medicine breakthroughs as fast as possible. Thanks for listening.