{"url": "https://www.youtube.com/watch?v=FMBVBYnBYmQ", "page_url": "https://www.youtube.com/watch?v=FMBVBYnBYmQ", "title": "Learning from First Principles - Demis Hassabis (DeepMind) @ NIPS2017", "show_or_channel": "imaginary reality (NIPS 2017 repost)", "date": null, "duration_s": 2149, "type": "talk", "speakers": ["Demis Hassabis"], "confidence": "medium", "notes": "NIPS 2017 (Long Beach, Dec 2017) talk \"Learning from First Principles\" (AlphaZero); third-party upload.", "people": ["demis-hassabis"], "id": "0000_imaginary-reality-nips-2-learning-from-first-principles-demis-h", "audio_duration_s": 2148.774625, "worker": "jc-w12", "transcribe_s": 55.3, "model": "large-v3", "segments": [{"start": 9.07, "end": 38.11, "text": "Thanks. I hope you can all hear me well. Sorry for my croaky voice, and I have to bear with them coming down from the cold, so I just have to bear with my coughing. So I'm going to talk today about learning from first principles, and I'm going to cover quite a few different things, and talk a little bit about AlphaZero, our latest work at the end. So our mission is to firstly try and solve fundamentally solved intelligence."}, {"start": 38.11, "end": 68.08, "text": "And then it's our belief that if you're able to do that, you should be able to use it to pretty much solve everything else, or at least it should affect almost every aspect of our lives. And when we started DeepMind back in 2010 now, we had a sort of philosophy behind our approach. How are we going to approach this sort of big, ambitious mission? And really, we split intelligence along a number of dimensions. And I'm just going to mention four of the key axes."}, {"start": 68.08, "end": 97.1, "text": "That we use to sort of inform our philosophy about how we're going to approach building general intelligence. So the first and most important is committing to this idea of learning versus hand crafting solutions. So we wanted to build systems that learn for themselves the first principles directly from the raw data rather than being spoon fed or hand crafted with solutions that the programmers themselves had come up with."}, {"start": 97.1, "end": 126.5, "text": "Secondly, there's this notion of generality. So we wanted a system that was able to work across a wide range of environments and tasks and potentially out of the box, so including potentially novel tasks that it's never seen before. And that's versus specifically kind of handcrafting a system to deal just with one problem or one task. Third was this notion of groundiness versus logic-based systems."}, {"start": 126.5, "end": 156.14, "text": "So we're firm believers that a true thinking system or true cognitive system needs to be fully grounded in the sensory mode to reality. So you need to be able to trace the origins of the knowledge that it's created all the way back to the raw basic inputs. Rather than in contrast logic based systems which are sort of these ethereal systems where the symbolic logic floating kind of unconnected to the actual data and real input."}, {"start": 156.37, "end": 179.87, "text": "We all want input. Then the fourth thing is this notion of active versus passive learning. So a lot of systems, a lot of AI systems are what I would call passive. So for example, a classifier gets an input and then it outputs a decision about that input. But what we're after is agent-based systems that are active participants in their own learning."}, {"start": 179.87, "end": 208.98, "text": "Much like Alison was mentioning earlier about how children learn. And indeed, of course, as Lisa was mentioning, about how animals learn too. So these are the sort of four axes we were thinking about, and we committed to be, we thought the right approach was to be at the left-hand extreme here on all these four axes. And taken together then, that gives you the idea of a system that's a general purpose learning agent."}, {"start": 208.98, "end": 236.37, "text": "And that's what we internally call a deep mind, artificial general intelligence. And that's what we're trying to pursue. Now, in order to do this, we actually look to neuroscience, system neuroscience, and cognitive psychology ideas, including child development and animal intelligence, for inspirations for algorithmic ideas, representation ideas, and architectural ideas that we can borrow and build our systems around."}, {"start": 237.81, "end": 266.13, "text": "Our first attempt at building this type of system was a system called DQN, and we basically put together deep learning and reinforcement learning at scale to form deep reinforcement learning, which is the subject we now call it, and to try and do something impressive with it in terms of how far we can push the system. So we built an end-to-end learning system, an end-to-end learning agent, that just takes more inputs"}, {"start": 266.13, "end": 292.18, "text": "and a reward signal, for example the score from a game, and then has to figure out everything else for itself. We used games and simulations as efficient platforms for testing and developing these AI algorithms because we could run millions of versions of these at once in the cloud versus something like using real robots. And our first big landmark result, as many of you know, was applying this to Atari games."}, {"start": 292.18, "end": 316.82, "text": "Classic Atari games from the 80s. And building one system that could master all the dozens and dozens of very different Atari games and play them better than humans could. So we trained up our neural network system, which we call DQN, DQQ network. And basically it's a very large convolutional net that uses the reward signal to modify its rates."}, {"start": 318.13, "end": 346.51, "text": "And I've successfully played many of these dozens of Atari games. I don't want to spend very long on this work, because I want to spend most of the time talking about our next piece of work, AlphaGo. So AlphaGo is our system to play the game of Go. So for those of you who don't know, this is what the Go board looks like. It's playing on a 19 by 19 board, and black and white take turns to play stones on the vertices of the board."}, {"start": 346.77, "end": 372.72, "text": "Now the aim of the game in Go is to surround your opponent's pieces with more pieces, or to saw off empty areas of territory. When you get to the end of the game, so this is a position at the end of the game, you basically count up the amount of territory that you've surrounded and compare it to your opponent's territory, whoever has the most points wins the game. So in this case, White wins by just a single point, it's a very close game."}, {"start": 372.98, "end": 402.4, "text": "Now Go is an extremely simple game to learn, there's only two rules really, but it's incredibly elegant because from that simplicity comes incredible complexity. And one easy way to illustrate that is the fact that there are 10 to the power of 170 possible war positions in Go, which is way more than there are atoms in the observable universe. So that's one of the reasons why Go has been so hard for computers to play."}, {"start": 402.4, "end": 432.13, "text": "Obviously, in the late 90s, IBM's Deep Blue famously beat Garry Kasparov at chess. And since then, people for the next 20 years have been trying to apply those same techniques to the game of Go. But really, they were never able to get further than strong amateur level, let alone kind of world champion level. So one of these problems is this enormous search space. So there's 10 to the power of 170 possible positions. But actually, an even more interesting and harder problem"}, {"start": 432.13, "end": 462.8, "text": "is the problem of trying to create an evaluation function. So it was thought impossible. In fact, it may still be possible to directly write down an evaluation function for Go that will allow the computer system to determine whether the current position is good or not for it. And that's obviously very critical to the sorts of traditional techniques like alpha, beta search. The reason that's so difficult in Go, and I'll come back to this at the end when I talk about chess, is that there's no concept of materiality in Go."}, {"start": 462.8, "end": 489.89, "text": "All the pieces are the same. So there's no easy heuristic to tell you who's currently winning. Another problem with Go is that Go is what I call a constructive game, whereas chess is a disruptive game. What I mean by that is that in chess you start off with all the pieces on the board, and as the game goes on, pieces are removed. So the game gets simpler. In Go, the board starts empty and you fill it up. So if you're in a mid-game position in both chess and Go, in chess..."}, {"start": 489.89, "end": 519.97, "text": "All the information is there right in front of you in the current war position. But in Go, although it's a perfect information game, you also have to project forwards in time and think about what might happen in order to figure out who's going to get what territory. It's not obvious from the current position because there's still a lot of construction to go. So you might ask, well, how do humans deal with this level of complexity and the professional Go players? How do they do that? Well, if you talk to the top Go players,"}, {"start": 519.97, "end": 545.23, "text": "They'll tell you that the game is primarily one about intuition. That's what they do. They rely on their instincts and intuition rather than calculation, which is what a chess grandmaster would do. They would explicitly calculate out particular planets, whereas Go is much more about feel, that this move feels right in this particular situation. So we thought this was a great challenge for us and our learning systems."}, {"start": 545.23, "end": 573.81, "text": "We know that the traditional brute force systems don't work with Go, and so that's why we chose Go as a good target for us for the types of learning systems we wanted to build. So we dealt with these two problems by using two neural networks. So the first neural network, which we call the policy network here on the left in green, we trained on 100,000, just over 100,000 human games, human amateur games that we downloaded from the internet."}, {"start": 573.81, "end": 602.66, "text": "And what we did with the policy now is we tried to train it to predict what the move was that the human was going to play. And it got reasonably good at doing this to the extent that if you give it any random ball position here in blue, then it will output a probability distribution of how likely it thinks humans are going to play certain moves in the current position. So that means you can basically look at the top three or five most likely moves"}, {"start": 602.66, "end": 629.66, "text": "Rather than looking at all the 200 possible moves there are on average in a particular go position. So this narrows down the width of the search. And the second year, most importantly, intriguingly I would say, was we trained a second year network on the data produced by this first network playing against itself many millions of times. And we took positions from those games and we also knew which side ended up winning and losing, ultimately."}, {"start": 629.66, "end": 654.0, "text": "And we trained the second neural network, the value network, here in pink, to predict from the current position who was likely to be the winner and what was its confidence level over that. So the value network turns a real number between 0 and 1, 0 meaning 100% confident whites will win, 1, 100% confident blacks will win, and 0.5 that currently predicts to lose again."}, {"start": 654.99, "end": 676.48, "text": "We took this first version of AlphaGo and we put it together with Monte Carlo Tree Search, which is a very important component of the overall system, which cooled these two neural networks. And then we challenged one of the legends of the game of Go, the South Korean genius called Lee Sedol. He's a legend of the game. He has 18 world titles. You can think of them as like grand sands."}, {"start": 676.48, "end": 704.78, "text": "And I call him Roger Federer of GO, because effectively that's the same sort of echelon he occupies. So he's acknowledged as the greatest player in the past decade. We had this $1 million challenge match nearly two years ago now in South Korea. And, you know, it was a huge event and really kind of an amazing experience for us, once in a lifetime experience. And over 200 million people watched the matches online or on TV. And to everyone's surprise, including Lisa Donalds,"}, {"start": 704.78, "end": 730.83, "text": "AlphaGo won 4-1. And this is sort of proclaimed to be about a decade before its time. So when I spoke to several of the world experts in ComputerGo a couple of years before we built AlphaGo, they were all predicting, and in their interviews, that they were at least 10 years away from using the current methods that they were using, traditional methods, to be able to get to anywhere close to world champion level."}, {"start": 731.82, "end": 759.36, "text": "So this was pretty surprising for everyone. But also what was more interesting was not just the fact that AlphaGo won, but how it won. What kinds of moves it made and what sorts of ideas it had. So because it was mostly trained through this notion of self-play, it actually came up with new ideas and new motifs that it discovered through that self-play. And here's one famous example from game two. This is the second game, AlphaGo is Black."}, {"start": 759.36, "end": 785.92, "text": "And on move 37, AlphaGo played this stone here on the right-hand side. You can see indicated at the middle of the board with the white triangle. And this was totally shocking for the commentators viewing this match. And there's several funny YouTube clips where they literally fall off their chairs. And the reason is because this move is being played on the fifth line. So there's 19..."}, {"start": 785.92, "end": 813.01, "text": "By 19 this board, and this move 37 was on the 5th line. Now in the opening part of the game, you generally only play on the 3rd and 4th lines. And playing on the 5th line gives away territory on the side of the board from the 4th line, which is quite a lot of territory. So it's quite a lot of points at the end of the game. So it's basically inconceivable for a professional to play a move like this. And if you're an amateur and you play the move like this, your teacher will quickly correct you."}, {"start": 813.14, "end": 842.85, "text": "So this is an original kind of unthinkable move, but yet in Go, Go is in China considered to be a sort of art form, and in Asia it's considered to be kind of an art form. But it's interesting because it's like objective art. So, you know, any one of us in this auditorium can come up with a novel move, like a random move in this position. But the key thing is whether it's a sort of beautiful move and considered to be a great move, is whether it actually affected the outcome and really worked."}, {"start": 842.85, "end": 870.13, "text": "And that's why this move 37 is considered so beautiful, because about 100 moves later, these two stones here in the bottom left-hand corner are marked with white triangles, which are in a bit of trouble because they're surrounded by white stones. That fighting in the bottom left-hand corner, 100 moves later, spills out into the middle of the board and perfectly connects up with move 37, that stone. And that stone is positioned perfectly to decide that fight decisively. And that's what ends up allowing AlphaGo to win the game."}, {"start": 870.51, "end": 887.15, "text": "So it's almost as if it's thought presciently ahead of time that a few dozen moves later, that stone would be in the critical position, even though it looks like a bad move locally. Of course, I should mention that Lisa Gold herself"}, {"start": 887.15, "end": 911.6, "text": "He was inspired to do his own genius move on move 78 in game 4. I haven't got time to cover where that is, but there is actually a documentary now about AfriGo, which was directed by, there's a more brilliant director, brilliant guy called Greg Coase, and now it's available on all sorts of streaming devices, so I recommend you check that out if you want to know more."}, {"start": 914.03, "end": 943.7, "text": "So I've talked a little bit about intuition and a little bit about creativity, but what do I mean by those terms? Well, I'm just going to caveat this first of all by saying I don't think this encompasses all the complexity of what these terms mean, but I think these are at least reasonable operational definitions that we can think about. So I think I'm using intuition as simply implicit knowledge that we have and that's been acquired through experience, of course, but it's not consciously expressible or accessible."}, {"start": 943.7, "end": 960.5, "text": "So we can't access it consciously ourselves, and we certainly can't express it to anyone else. The second thing is, you might think, well, if it's that kind of knowledge, how do we know that it's there? Well, of course we can test it behaviourally."}, {"start": 960.5, "end": 990.7, "text": "So we can verify the existence of this knowledge and the quality of it by testing it behaviourally. So for Go, it's very easy. You just give the entity or the person a Go position and you see that you can evaluate the quality of the decision that they make. Secondly, what about creativity? So again, there's many dimensions of creativity, but I think one reasonable operational definition is the ability to synthesise knowledge that you already have in a new way to produce a novel or original idea in the service of someone."}, {"start": 990.7, "end": 1018.83, "text": "And I think within the domain, albeit constrained domain of Go, I think AlphaGo clearly demonstrated these abilities and most professional Go players will agree. So we continued with the development of AlphaGo because our mission was not to just create a program that was good at playing Go, but we wanted to make progress towards this idea of a general learning algorithm."}, {"start": 1018.96, "end": 1045.04, "text": "And so our approach at DeepMind is to try and make something work in as general a way as possible, and then look at it and analyze it after it's reached a certain performance, and to try and extract things from it, simplify it, and actually make it more and more general. So that's what we did with AlphaGo Zero, which is the latest and final version of AlphaGo. We wanted to remove this part of the bootstrapping from human data."}, {"start": 1045.04, "end": 1067.18, "text": "Because often you don't have any human data, and it's also specific to that particular domain. So we started in this case completely from random play with zero knowledge, and had AlphaGo zero play against itself millions of times. We only implemented it for its mistakes. And what we ended up with was an even stronger, more efficient, and more general version of AlphaGo."}, {"start": 1067.18, "end": 1098.51, "text": "So we changed the architecture a little bit. We created a dual-headed architecture. So AlphaGo Zero only has one neural network that tries to predict two things, the policy net and the value net. But it's only one neural network. And we think this is one of the reasons why AlphaGo Zero is stronger, because this results in better regularization. And then it's actually quite a simple algorithm now. So AlphaGo plays about 25,000 games in one batch against itself at the current full strength that it has."}, {"start": 1098.51, "end": 1126.24, "text": "Then we take all that new data and we train a new policy network to predict that version of AlphaGo's moves, what that version would do. And we also train a new value network that predicts the winners from those games, from mid-game positions. Then we put that together into a new iteration of AlphaGo's train automatically, and then we pitch that new version"}, {"start": 1126.24, "end": 1156.56, "text": "Played a few hundred games against the old version. And if it wins 55% or more, significantly better, then it replaces the old version as the new now generator of the next level of data. If it's not significantly better, we continue with the old version and collect more data. And then try to retrain again after another 25,000 games. So this is the curve of improvement. Performance is on the y-axis. Days and time is on the x-axis."}, {"start": 1156.56, "end": 1185.33, "text": "And you can see that just after three days, AlphaGo, AlphaGo Zero, it surpasses the version of AlphaGo that we played against each other. So it's already world champion level, human world champion level after three days. After 21 days, it beats the best version of AlphaGo that we ever created called AlphaGo Master, which beats a lot of the other top professional players. And then you'll notice after 40 days,"}, {"start": 1185.33, "end": 1215.47, "text": "We stopped running the program, but it still has an asymptote. So we never actually found the asymptote of this self-improving process. It's got to something kind of pretty crazy, like 5,000 ELO, but it's still improving. So who knows actually what the final optimal play and asymptote would be. And then we went back and looked through the different plays that the machine was doing after GoZero was playing at different time scales."}, {"start": 1215.47, "end": 1241.41, "text": "And we found that, you know, at the beginning, as you would expect, it's playing like a total rank amateur, over-concentrating stones, playing in stupid places like the corner of the board. And then over time, it starts rediscovering the best human strategies. So after, you know, around 36 hours, 48 hours, it's now playing quite common, what's called Jodekis, which are the best plays that humans have found. And then the frequency of those plays dropped off again."}, {"start": 1241.41, "end": 1270.0, "text": "As it gets to 72 hours. And we think that's because AlphaGo Zero has decided that there are actually some better types of approaches than these Jozeckis and starts inventing its own ones. And these ones now are being analysed and utilised by human players and professionals around the world. So in a way you can think of AlphaGo Zero as rediscovering 3,000 years of human knowledge. So Go has been around for 3,000 years in 72 hours."}, {"start": 1272.18, "end": 1302.34, "text": "So you can read more details about this in our two papers that we published last year and this year. And then finally, I want to talk a little bit about our latest creation, AlphaZero, which is now even more general. So now we've removed the word Go from this, and it's just AlphaZero, a generic games play algorithm. And several people were skeptical, I think, about even AlphaGo Zero, that could it play even other games, like chess?"}, {"start": 1302.34, "end": 1329.07, "text": "And Shogi. So we were pretty sure it could, but we decided to actually test this thesis out. And we used AlphaZero and we trained it on three major perfect information games. Chess, Shogi, and Go. Shogi is a Japanese chess, which is a very fun game in itself for those of you who don't know, but I haven't got time to talk about that in this talk. So I'm just going to focus on AlphaZero's play in chess here, which is the game where I know best."}, {"start": 1329.07, "end": 1357.41, "text": "Now, before we start this off, I talked to a few of my friends from the chess world, including some ex-world champions, and I basically asked the question that how far off did they think from optimal play were the current chess engines? So as you all know, the current chess engines are far beyond human world champion overall, far beyond where people use, and they've got really good now. They're sort of 3,400, you know, they're kind of unbeatable by human players."}, {"start": 1357.41, "end": 1383.9, "text": "And actually as a chess player myself, I figured that they were probably pretty close to optimal, maybe one or two, 200 points off optimal. And so it wasn't clear that a learning system could be as good as these handcrafted systems, given the amount of effort that had gone into building these systems over the last 20, 30 years. So computer chess is probably the most"}, {"start": 1383.9, "end": 1407.12, "text": "One of the most, if not the most, stunning domain in the history of AI. So its birth of computer chess was really at the birth and the dawn of computer science. So people like Turing, Shannon, even Babbage were thinking about computer chess in the early days of computer science. And current chess engines are highly specialized systems that have been obviously very successful in playing chess."}, {"start": 1407.12, "end": 1433.46, "text": "And they use an alpha beta search and a whole bag of handcrafted heuristics that have been distilled from human grandmasters. So if we look at Stockfish, which was the program that won the 2016 World Championship, this is the list of special case heuristics and extensions that Stockfish has. So a gigantic list of..."}, {"start": 1433.46, "end": 1460.34, "text": "You know, specific types of techniques to help it play chess and rules to help it play chess. So instead of that, we just replace all of that in hand engineering with self-play reinforcement learning and self-play multi-card research. So there's no opening book in AlphaZero. There's no in-game database. There's no heuristics of any kind. It just starts completely from random play."}, {"start": 1460.34, "end": 1483.5, "text": "And I wasn't actually sure, as a chess player, that this would work, given how important openings and end games are. And, you know, a program like Stockfish plays the opening and endings nearly perfectly. Oh, and the other thing was we wanted, obviously, again, just a notion to generality, the same algorithm, the same type of parameters for all three games."}, {"start": 1484.34, "end": 1510.53, "text": "And much to our pleasant surprise that, you know, almost the first time we tried to run this, that we found that AlphaZero beat the state-of-the-art chess programs in all three games. So in chess, we beat Stockfish 28-0, so we didn't lose one game out of 100 with 72 draws, including some wins with black. And in Shoki, it beat the world champion Elmo 90-8."}, {"start": 1510.53, "end": 1537.87, "text": "And in Go, we beat AlphaZero, beat AlphaGo Zero 60-40 after three days, comparing them both at three days of training. So AlphaZero was able to surpass NotFish's ELO strength in four hours, ELMO in two hours, and AlphaGo, what we call AlphaGo Lead, the least evolved version, in eight hours. So this whole thing, all three games, in less than 24 hours."}, {"start": 1538.19, "end": 1567.82, "text": "The other thing is you might be thinking, well, is it a question of compute power? We honestly don't think it is. I mean, we use Google's Tensor Processing Unit, all of those in the cloud, and that's a quite reasonable amount of power. But actually, one way to measure this, an easy way to measure this, is how many moves are searched per decision made for AlphaZero. And for human grandmasters, they search about tens of moves per decision."}, {"start": 1567.82, "end": 1594.03, "text": "State-of-the-art chess engines like Stockfish search tens of millions of moves for each decision they make. So Stockfish has about 70 million moves per decision. And AlphaGo Zero is right in the middle of the spectrum in that it's using about tens of thousands of moves per decision. So it's right in the middle, so it's three orders of magnitude less than the state-of-the-art chess engines."}, {"start": 1594.03, "end": 1613.09, "text": "Obviously, it's still free on its magnitude more than humans do, but it's moving towards the right direction in terms of being more human-like. We did a quick analysis. We're going to do more analysis in our full paper, as opposed to the sort of preliminary archive paper, is we looked at the openings that were discovered by AlphaZero."}, {"start": 1613.09, "end": 1642.86, "text": "And I'm pleased to say that my favourite opening, the English opening, here in the top left, survived this eight hours of training. Whereas surprisingly, some really popular openings, like in the bottom left here, the King's Indian Defence, was not really favoured by AlphaZero at all, at any stage. And if you look at AlphaZero style, it's not, like, I wouldn't say it doesn't play like a human, but it also doesn't play like computer engines. It sort of plays in a third, almost alien way."}, {"start": 1642.86, "end": 1668.0, "text": "And what AlphaZero loves, and we've got to analyze this a little bit more, but we have a few chess pro worlds that look at this, including on our team, and AlphaZero loves long-term positional sacrifices. And that isn't really even a thing in chess, because sacrifices are mostly about tactics. So the idea of sacrifice is, you know, you give your queen away because you think you've calculated out that you can checkmate your opponent, you know, in a few moves later."}, {"start": 1668.0, "end": 1693.65, "text": "So these are tactical sacrifices. They're quite unusual, but they're played. And there's famous world champions like Li Ka-tao, who's famous for doing those kinds of sacrifices. But what AlphaZero does is it makes sacrifices for some very long-term positional gain, which I was surprised that chess had the capacity to allow for things like long-term positional sacrifices. So it appears that this is an actual thing in chess."}, {"start": 1693.65, "end": 1723.46, "text": "And I think the reason my initial take on that is that because it doesn't have a concept of materiality, so it doesn't have a rule telling it that rooks are five points, knights are three points, pawns are one point, it doesn't know anything about that. So instead of that, everything's kind of contextual. It sort of thinks, in this kind of position, these assets I have, this rook that can move like this, isn't that useful? Whereas if my opponent's pawn or knight is really powerful in this position, so I'm going to swap them."}, {"start": 1723.46, "end": 1752.37, "text": "And it doesn't have to overcome some rigid rule it has in its system that tells it that it's going to lose two points by swapping the book for the knight. Now, obviously, a stock fish and those kind of chess engines can do sacrifices, but they have to calculate enough to see some endpoint that they will get that material back or there will be some other compensation for it. Whereas AlphaZero doesn't need to do that. It can be patient about recovering that material at some point down the line."}, {"start": 1752.59, "end": 1773.17, "text": "So I think the key thing here is that rigid rules, like the traditional chess engines have, can't be dynamically rebalanced in a situation. You can't just say, these systems can't have another if-then rule saying, in this specific kind of position, this rule is not worth five points. It's worth three points. It's always worth five points in its hard-coded databases."}, {"start": 1773.39, "end": 1802.27, "text": "So the initial conception from my chess friends have been telling me is that maybe our conception of chess has been too limited. So this could be a very exciting moment for chess. And I think humans can learn from this because these ideas are strategic rather than tactical. So we can actually graph that into our own play. I just want to, near the end now, just talk quickly about, show you a couple of, for those of you who know chess, and as you probably know, I love games. So there's quite a lot of game theory in this talk."}, {"start": 1802.27, "end": 1824.48, "text": "But here's a position I want to show, a couple of positions showing these kinds of long-term sacrifices. So alpha zero is white here, and basically what it's done is sacrificed its bishop, you can see here, it's just taken a pawn, and it's going to give away its bishop. So now it's a bishop down in this position, you can see the black bishop there in the top left corner on b7."}, {"start": 1824.48, "end": 1847.74, "text": "But it's completely calm about this. And I think what's happened is, it's decided that that bishop there that Black has, who is hemmed in by the pawns, isn't worth very much. So it's definitely not worth three points. So it's calm, and he's even allowed his queen to be swapped off. This is kind of un-Roboth, if you're a bishop now. I mean, this looks ridiculous in some sense. But it's just calm. I mean, it's only got one pawn for it."}, {"start": 1847.74, "end": 1874.75, "text": "So, you know, it just can't be carried on playing. And then eventually, it just really, like, you know, like, Stockfish is sort of thrashing around with his bishop, but it can't do anything. And so, and then eventually, you know, it just marches its king in, and then, you know, Stockfish just dies. So, Stockfish is kind of amazing. And then there's another one, because I like this one. It's Mid-Game Night Sacrifice. So this is sort of the middle of the game, and..."}, {"start": 1874.75, "end": 1899.18, "text": "AlphaZero has just played this knight move to the corner of the board to check the king. And it basically gives up this knight, incredibly. So it just allows Stockfish to take the knight. And it just calmly goes and develops his pieces. And then it does this brilliant maneuver, which I love, and I showed one of the grandmasters this, and he just couldn't believe it. So it takes his pawn here, and then it moves its queen into the corner."}, {"start": 1899.18, "end": 1918.48, "text": "I mean, I haven't got time to explain to you how just kind of alien and incredible this maneuver is. Now it's lined up with a bishop to attack that critical pawn on c6, right? But what a maneuver. So, you know, I just think, you know, there's a lot more here to explore, but I just think this is kind of a chess from another dimension, I would call it. And so..."}, {"start": 1918.67, "end": 1944.53, "text": "Derek Asparta himself, actually, even before he saw the AlphaZero results, he was talking about Deep Blue was an end of some kind of era, and he felt AlphaGo, because it's a learning system, he calls it a Type B system versus a Type A system, is some kind of beginning. So I just want to end by talking, and I know that Gary Mark is going to be talking later, and I wanted to preempt some of his comments. So there's some limitations, obviously, with this system."}, {"start": 1944.53, "end": 1970.91, "text": "Where I think of it is intelligence search through a huge combinatorial space. So what you have to have is a clear objective functioning and you have to have either large amounts of data or an efficient simulator in this case to generate your own data. And this system still lacks a lot of things after zero. It can't generate hypotheses or design its own experiments and it hasn't got any kind of abstract knowledge or causal reasoning abilities. So we're interested in all those things."}, {"start": 1970.91, "end": 1991.92, "text": "But I do want to say, a caveat to all of that, a few years ago, I think pretty much everyone in this room would have thought that these results were kind of unthinkable. And, you know, the system is very general. You know, it is the general games-playing machine that I always dreamed about making 20 years ago, and some of my colleagues did. And, you know, I think we have it now, for at least the perfect information against."}, {"start": 1991.92, "end": 2016.64, "text": "Obviously, we're not claiming this is solved in AI in any shape or form. There's loads of huge challenges that remain. Learning the model, learning the rules, dealing with partial information, uncertainty, there's the question of efficiency, to be able to do one-shot learning and abstract knowledge. I think Josh Tellerman's going to talk a lot about that. But one point I'd make, all these things that we're working on, all those projects, and we think that they're very important, clearly, for general intelligence."}, {"start": 2016.64, "end": 2041.73, "text": "But there's one thing about human efficiency I just want to mention, which is that, obviously, humans, it's not quite right to say that, you know, let's take AlphaGo. Xero had 30 million games, and we worked out that someone like Mr. Dog probably played about 30,000 games in their lifetime. And that seems like a huge discrepancy. But don't forget that humans are coming to this with some evolutionary priorities already, and also a lot of transfer learning, and most importantly, apprenticeship learning."}, {"start": 2041.73, "end": 2058.69, "text": "So, obviously, Lisa Gores read Go books, which are themselves distillations of many other humans' knowledge on the game. And they've also been taught by their Go masters. So, I think in the future, it'd be quite interesting to actually test the... You could actually maybe think about an information test."}, {"start": 2058.69, "end": 2085.97, "text": "On a book, in the sense of you could have one version of AlphaGo Zero that learns purely through self-play, and another version that once it's able to read books, it reads a book and then does self-play. And then you can kind of compare how many games of self-play was that book worth. So I kind of look forward to the time when we're able to do that. And what we're after really, I think, and I discuss this a lot with my colleagues here, is that the minimum set of massively general products is what we're after."}, {"start": 2085.97, "end": 2103.82, "text": "And obviously, just to end, we're not looking to just play games with our systems. That's just our proving ground and our testing ground, albeit it's a very fun one. And really what we're trying to do is apply these ideas to the real world. We've really started doing that in areas of healthcare, to energy optimization, data centers, and all sorts of other things."}, {"start": 2103.82, "end": 2118.29, "text": "And for me, the most exciting thing will be to try and apply this in areas of science. And although it's still quite limited in the ways I've just mentioned, I think there are many areas of science that may be quite amenable to this type of approach, like drug design, material design, and some aspects of biotech."}, {"start": 2118.29, "end": 2138.7, "text": "And I think ultimately this is the era I'm looking forward to as I work on AI, is the power of humans and machines together. You know, humans and us and our smart tools figuring out some complex scientific questions. I think human ingenuity augmented by AI will unlock our true potential. Thank you."}]}