{"url": "https://www.youtube.com/watch?v=DXNqYSNvnjA", "page_url": "https://www.youtube.com/watch?v=DXNqYSNvnjA", "title": "Demis Hassabis, DeepMind - Learning From First Principles - Artificial Intelligence NIPS2017", "show_or_channel": "The Artificial Intelligence Channel", "date": "2017-12-09", "duration_s": 2140, "type": "talk", "speakers": ["Demis Hassabis"], "confidence": "medium", "notes": "NIPS 2017 invited talk 'Learning from First Principles' (9 Dec 2017); third-party upload. Alt: FMBVBYnBYmQ. (YouTube upload 2018-01-24)", "people": ["demis-hassabis"], "id": "2017-12-09_the-artificial-intelligen-demis-hassabis-deepmind-learning-from", "audio_duration_s": 2139.648, "worker": "jc-w15", "transcribe_s": 27.8, "model": "large-v3", "segments": [{"start": 5.81, "end": 34.88, "text": "Thanks. I hope you can all hear me well. Sorry for my croaky voice, and I'm coming down in the cold, so I just have to go in my coffee. So I want to talk today about learning from first principles, and I'm going to cover quite a few different things, and talk a little bit about AlphaZero, our latest work at the end. So our mission is to firstly try and solve fundamentally solved intelligence."}, {"start": 34.88, "end": 64.88, "text": "And then it's our belief that if you're able to do that, you should be able to use it to pretty much solve everything else, or at least it should affect almost every aspect of our lives. And when we started DeepMind back in 2010 now, we had a sort of philosophy behind our approach. How are we going to approach this sort of big ambitious mission? And really we split intelligence among a number of dimensions. And I'm just going to mention four of the key axes."}, {"start": 64.88, "end": 93.95, "text": "that we use to sort of inform our philosophy about how we're going to approach building a general intelligence. So the first and most important is committing to this idea of learning versus hand crafting solutions. So we wanted to build systems that learn for themselves from first principles, directly from the raw data, rather than being spoon fed or hand crafted with solutions that the programmers themselves had come up with."}, {"start": 93.95, "end": 123.33, "text": "Secondly, there's this notion of generality. So we wanted a system that was able to work across a wide range of environments and tasks, potentially out of the box, so including potentially normal tasks that it's never seen before. And that's versus specifically kind of handcrafting a system to deal just with one problem or one task. Third was this notion of groundedness versus logic-based systems."}, {"start": 123.33, "end": 153.1, "text": "So we're firm believers that a true thinking system or true cognitive system needs to be fully grounded in the sense of remote reality. So you need to be able to trace the origins of the knowledge that it's created all the way back to the raw basic inputs. Rather than in contrast logic-based systems which are sort of these ethereal systems where the symbolic logic floating kind of unconnected to the actual data and really not."}, {"start": 153.14, "end": 180.91, "text": "Then the fourth thing is this notion of active versus passive learning. So a lot of systems, a lot of AI systems are what I would call passive. So for example, a classifier gets an input and then it outputs a decision about that input. But what we're after is agent-based systems that are active participants in their own learning, much like Alison was mentioning earlier about how children..."}, {"start": 180.91, "end": 210.46, "text": "And indeed, of course, as Liz was mentioning about how animals learn too. So these are sort of four axes we were thinking about and we committed to be, we thought the right approach was to be in the left-hand extreme here on all these four axes. And taken together then, that gives you the idea of a system that's a general purpose learning agent. And that's what we internally call at DeepMind artificial general intelligence."}, {"start": 210.46, "end": 239.5, "text": "And that's what we're trying to pursue. Now, in order to do this, we actually look to neuroscience, system neuroscience, and cognitive psychology ideas, including child development and animal intelligence, for inspirations for algorithmic ideas, representation ideas, and architectural ideas that we can borrow and build our systems around. Our first attempt at building this type of system was a system called DQN."}, {"start": 239.5, "end": 269.1, "text": "And we basically put together deep learning and reinforcement learning at scale to form deep reinforcement learning, which is the subject we now call it, and to try and do something impressive with it in terms of how far we could push the system. So we built an end-to-end learning system, an end-to-end learning agent, that just takes more inputs and a reward signal, for example the score from a game, and then has to figure out everything else for itself."}, {"start": 269.1, "end": 299.34, "text": "We used games and simulations as efficient platforms for testing and developing these AI algorithms because we could run millions of versions of these at once in the cloud versus something like using real robots. And our first big landmark result, as many of you know, was applying this to Atari games, classic Atari games from the 80s, and having only one system that could master all the dozens and dozens of different, very different Atari games and play them better than humans could."}, {"start": 299.89, "end": 330.74, "text": "So we trained up our neural network system, which we call DQN, DQ network, and basically it's a very large convolutional net that uses the reward signal to modify its weights. And that successfully played many of these dozens of Atari games. I don't want to spend very long on this work because I want to spend the rest of the time talking about our next piece of work, AlphaGo. So AlphaGo,"}, {"start": 330.74, "end": 360.59, "text": "is our system to play the game of Go. So for those of you who don't know, this is what a Go board looks like. It's played on a 19x19 board, and black and white take turns to place stones on the vertices of the board. Now the aim of the game in Go is to surround your opponent's pieces with more pieces, or to swallow off empty areas of territory. When you get to the end of the game, so this is their position at the end of the game, you basically count up the amount of territory that you surmounted,"}, {"start": 360.59, "end": 389.76, "text": "And compare it to your opponent's territory. Whoever has the most points wins the game. So in this case, White wins by just a single point. It's a very close game. Now Go is an extremely simple game to learn. There's only two rules really. But it's incredibly elegant because from that simplicity comes incredible complexity. And one easy way to illustrate that is the fact that there are 10 to the power 170 possible ball positions in Go."}, {"start": 389.76, "end": 416.74, "text": "Which is way more than there are atoms in the observable universe. So that's one of the reasons why Go has been so hard for computers to play. Obviously in the late 90s, IBM's Deep Blue famously beat Garry Kasparov at chess. And since then, people for the next 20 years have been trying to apply those same techniques to the game of Go. But really they were never able to get further than strong amateur level."}, {"start": 416.74, "end": 446.19, "text": "Let alone kind of world champions. So one of these problems is this enormous search space. So there's 10 to the power 170 possible positions. But actually an even more interesting and harder problem is the problem of trying to create an evaluation function. So it was thought impossible, in fact it may still be impossible, to directly write down an evaluation function for Go that will allow the computer system to determine whether the current position is good or not for it."}, {"start": 446.19, "end": 471.28, "text": "And that's obviously very critical to the sorts of traditional techniques like alpha beta search. The reason that's so difficult in Go, and I'll come back to this at the end when I talk about chess, is that there's no concept of materiality in Go. All the pieces are the same. So there's no easy heuristic to tell you who's currently winning. Another problem with Go is that Go is what I call a constructive game, whereas chess is a destructive game."}, {"start": 471.28, "end": 501.09, "text": "What I mean by that is that in chess you start off with all the pieces on the board and as the game goes on, pieces are removed. So the game gets simpler. In Go, the board starts empty and you fill it up. So if you're in a mid-game position in both chess and Go, in chess all the information is there right in front of you in the current board position. But in Go, although it's a perfect information game, you also have to project forwards in time and think about what might happen in order to figure out who's going to get what territory."}, {"start": 501.09, "end": 527.33, "text": "It's not obvious from the current position because there's still a lot of construction to go. So you might ask, well, how do humans deal with this level of complexity and professional Go players? How do they do that? Well, if you talk to the top Go players, they'll tell you that the game is primarily one about intuition. That's what they do. They rely on their instincts and intuition rather than calculation, which is what Chester and I do."}, {"start": 527.33, "end": 558.62, "text": "They would explicitly calculate out particular plans, whereas Go is much more about feel, that this move feels right in this particular situation. So we thought this was a great challenge for us and our learning systems. We know that the traditional brute force systems don't work with Go, and so that's why we chose Go as a good target for us for the types of learning systems we wanted to build. So we dealt with these two problems by using two neural networks."}, {"start": 558.62, "end": 577.33, "text": "So the first neural network, which we call the policy network here on the left in green, we trained on 100,000, just over 100,000 human games, human amateur games that we downloaded from the internet. And what we did with the policy network is we tried to train it to predict what the move was that the human was going to play."}, {"start": 577.9, "end": 605.81, "text": "And I got reasonably good at doing this to the extent that if you give it any random ball position, here in blue, then it will output a probability distribution of how likely it thinks humans are going to play certain moves in the current position. So that means you can basically look at the top three or five most likely moves rather than looking at all the 200 possible moves there are on average in a particular going position."}, {"start": 605.81, "end": 634.26, "text": "So this narrows down the width of the search. And the second year, most importantly, intriguingly, I would say, was we trained a second-year network on the data produced by this first network playing against itself many millions of times. And we took positions from those games, and we also knew which side ended up winning and losing, ultimately. And we trained the second-year network, the value network, here in pink, to predict from the current position who was likely to be the winner."}, {"start": 634.26, "end": 661.23, "text": "And what was its confidence level over that? So the value net returns a real number between 0 and 1. 0 meaning 100% confident whites can win, 1, 100% confident blacks can win, and 0.5 that currently predicts it's an even game. We took this first version of AlphaGo and we put it together with Monte Carlo Tree Search, which is a very important component of the Oracle system, which called these two neural networks."}, {"start": 661.23, "end": 687.42, "text": "And then we challenged one of the legends of the game of Go, a South Korean genius called Lee Seung-ho. He's a legend of the game. He has 18 world titles. You can think of them as like grand slams. And I call him the Roger Federer of Go because, effectively, that's the same sort of echelon he occupies. So he's acknowledged as the greatest player in the past decade. We had this $1 million challenge match nearly two years ago now in South Korea."}, {"start": 687.42, "end": 715.26, "text": "And, you know, it was a huge event and really kind of an amazing experience for us, a once-in-a-lifetime experience. And over 200 million people watched the matches online and on TV. And to everyone's surprise, including Lisa Gold, AlphaGo won 4-1. And this is sort of proclaimed to be about a decade before its time. So when I spoke to several of the world experts in ComputerGo a couple of years before we built AlphaGo,"}, {"start": 715.26, "end": 738.54, "text": "They were all predicting, and in their interviews, that they were at least 10 years away on using the current methods that they were using, traditional methods, to be able to get to anywhere close to world champion level. So this was pretty surprising for everyone. But also what was more interesting was not just if that Africa won, but how it won. What kinds of moves it made and what sorts of ideas it had."}, {"start": 738.54, "end": 767.73, "text": "So, because it was mostly trained through this notion of self-play, it actually came up with new ideas and new motifs that it discovered through that self-play. And here's one famous example from game two. This is the second game, AlphaGo is Black. And on move 37, AlphaGo played this stone here on the right-hand side. You can see it indicated at the end of the board with the white triangle. And this was totally shocking."}, {"start": 767.73, "end": 795.81, "text": "For the commentators viewing this match, and there's several funny YouTube clips you can look at where they literally fall off their checks. And the reason is because this move is being played on the fifth line. So there's 19 by 19, this board, and this move 37 was on the fifth line. Now in the opening part of the game, you generally only play on the third or fourth lines. And playing on the fifth line gives away territory on the side of the board."}, {"start": 795.81, "end": 825.54, "text": "From the fourth line, which is quite a lot of territory. So it's quite a lot of points at the end of the game. So it's basically inconceivable for a professional to play a move like this. And if you're an amateur and you play the move like this, your teacher will quickly correct you. So this is an original kind of unthinkable move. But yet in Go, Go is in China considered to be a sort of art form. And in Asia it's considered to be kind of an art form. But it's interesting because it's like objective art."}, {"start": 825.54, "end": 845.04, "text": "So, you know, any one of us in this auditorium could come up with a novel move by playing a random move in this position. But the key thing is whether it's a sort of beautiful move and considered to be a great move, is whether it actually affected the outcome and has really worked. And that's why this move, 37, is considered so beautiful, because about 100 moves later..."}, {"start": 845.04, "end": 866.99, "text": "These two stones here at the bottom left-hand corner, that I've marked with white triangles, which are in a bit of trouble because they're surrounded by white stones, that fighting in the bottom left-hand corner, 100 moves later, spills out into the middle of the board and perfectly connects up with move 37, that stone. And that stone is positioned perfectly to decide that fight decisively. And that's what ends up allowing AlphaGo to play the game."}, {"start": 867.31, "end": 883.9, "text": "So it's almost as if it sort of thought presciently ahead of time that a few dozen moves later, that stone would be in the critical position, even though it looks like a bad move locally. Of course, I should mention that Lisa Dole herself"}, {"start": 883.9, "end": 910.29, "text": "He was inspired to do his own genius move on move 78 in game 4. I haven't got time to cover where that is, but there's actually a documentary now about AfriGo, which was directed by a brilliant guy called Greg Coase, and now it's available on all sorts of streaming devices. I recommend you check that out if you want to know more."}, {"start": 910.8, "end": 940.51, "text": "So I've talked a little bit about intuition and a little bit about creativity, but what do I mean by those terms? Well, I'm just going to caveat this first of all by saying I don't think this encompasses all the complexity of what these terms mean, but I think these are at least reasonable operational definitions that we can think about. So I think of intuition as simply implicit knowledge that we have and that's been acquired through experience, of course, but that's not consciously expressible or accessible."}, {"start": 940.51, "end": 957.26, "text": "So we can't access it consciously ourselves, and we certainly can't express it to anyone else. The second thing is, you might think, well, if it's that kind of knowledge, how do we know that it's there? Well, of course, we can test it behaviorally."}, {"start": 957.26, "end": 987.49, "text": "So we can verify the existence of this knowledge and the quality of it by testing it behaviorally. So for Go, this is very easy. You just give the entity or the person a Go position and you see that you can evaluate the quality of the decision that they make. Secondly, what about creativity? So again, there's many dimensions of creativity, but I think one reasonable operational definition is the ability to synthesize knowledge that you already have in a new way to produce a novel or original idea in the service of some"}, {"start": 987.49, "end": 1015.57, "text": "And I think within the domain, albeit constrained domain of Go, I think AlphaGo clearly demonstrated these abilities and most professional Go players would agree. So we continued with the development of AlphaGo because our mission was not to just create a program that was good at playing Go, but we wanted to make progress towards this idea of a general learning algorithm."}, {"start": 1015.76, "end": 1041.86, "text": "And so the way our approach at DeepMind is to try and make something work in a general way as possible, and then look at it and analyze it after it's reached a certain performance, and to try and extract things from it, simplify it, and actually make it more and more general. So that's what we did with AlphaGo Zero, which is the latest and final version of AlphaGo. We wanted to remove this part of the bootstrapping from human data."}, {"start": 1041.86, "end": 1065.68, "text": "Because often you don't have any human data and it's also specific to that particular domain. So we started in this case completely from random play with zero knowledge and had AlphaGo zero play against itself millions of times, learning incrementally from its mistakes. And what we ended up with was an even stronger, more efficient and more general version of AlphaGo."}, {"start": 1065.68, "end": 1095.3, "text": "So we changed the architecture a little bit. We created a dual-headed architecture. So AfrikoZero only has one neural network that tries to predict two things, the policy net and the value net. But it's only one neural network. And we think this is one of the reasons why AfrikoZero is stronger, because this results in better regularization. And then it's actually quite a simple algorithm now. So Afriko plays about 25,000 games in one batch against itself at the current full strength that it has."}, {"start": 1095.3, "end": 1122.99, "text": "Then we take all that new data and we train a new policy network to predict that version of AlphaGo's moves, what that version would do. And we also train a new value network that predicts the winners from those games from mid-game positions. Then we put that together into a new iteration of AlphaGo's train automatically and then we pitch that new version"}, {"start": 1122.99, "end": 1153.34, "text": "Played a few hundred games against the old version and if it wins 55% or more, significantly better, then it replaces the old version as the new now generator of the next level of data. If it's not significantly better, we continue with the old version and collect more data and then try and retrain the game after another 25,000 games. So this is the curve of improvement. Performance is on the y-axis. Days and time is on the x-axis."}, {"start": 1153.34, "end": 1179.5, "text": "And you can see that just after three days, AlphaGo 0 surpasses the version of AlphaGo that we played against LisaDot. So it's already world champion level, here in world champion level after three days. After 21 days, it beats the best version of AlphaGo that we ever created called AlphaGo Master, which beats a lot of the other top professional players."}, {"start": 1179.5, "end": 1208.22, "text": "And then you'll notice after 40 days, we stopped running the program, but it still has an asymptote. So we never actually found the asymptote of this self-improving process. It's got to something kind of pretty crazy, like 5,000 ELO, but it's still improving. So who knows actually what the final optimal playing asymptote would be. And then we went back and looked through the different..."}, {"start": 1208.22, "end": 1233.92, "text": "And we found that at the beginning, as you would expect, it's playing like a total rank amateur, over-concentrating its tones, playing in stupid places like the corner of the board. And then over time, it starts rediscovering the best human strategies. So after 36 hours, 48 hours, it's now playing quite common, what's called Joe's Accusers, which are the best plays that humans have found."}, {"start": 1233.92, "end": 1256.37, "text": "And then the frequency of those plays drops off again as it gets towards 72 hours. And we think that's because AlphaGo Zero has decided that there are actually some better types of approaches and then these just actually quietly starts inventing its own ones. And these ones now are being analysed and utilised by human players and professionals around the world."}, {"start": 1256.37, "end": 1274.56, "text": "So in a way, you can think of AlphaGo Zero as we discover 3,000 years of human knowledge. So Go has been around for 3,000 years in 72 hours. So you can read more details about this in our two papers that we published last year and this year."}, {"start": 1274.56, "end": 1300.19, "text": "And then finally, I want to talk a little bit about our latest creation, AlphaZero, which is now even more general. So now we've removed the word Go from this, and it's just AlphaZero, a generic games-playing algorithm. And, you know, several people were skeptical, I think, about even AlphaGo Zero, that could it play even other games, like chess and shogi."}, {"start": 1300.19, "end": 1325.87, "text": "So we were pretty sure it could, but we decided to actually test this thesis out. And we used AlphaZero, and we trained it on three major perfect information games, Chess, Shogi, and Go. Shogi is a Japanese chess, which is a very fun game in itself for those of you who don't know, but I haven't got time to talk about that in this talk. So I'm just going to focus on AlphaZero's play in chess here, which is the game I know best."}, {"start": 1325.87, "end": 1354.22, "text": "Now, before we start this off, I talked to a few of my friends from the chess world, including some ex-world champions, and I basically asked the question that how far off did they think from optimal play were the current chess engines? So as you all know, the current chess engines are far beyond human world champion level, far beyond where Geekly was, and they've got really good now. They're sort of 3,400 each, and they're kind of unbeatable by human players."}, {"start": 1354.22, "end": 1378.0, "text": "And as actually the chess player myself, I figured that they were probably pretty close to optimal, maybe one or two, 200 ELO points off optimal. And so it wasn't clear that a learning system could be as good as these handcrafted systems, given the amount of effort that got into building these systems over the last 20, 30 years."}, {"start": 1378.0, "end": 1403.94, "text": "So computer chess is probably one of the most, if not the most, stunning domain in the history of AI. So its birth of computer chess was really at the birth and the dawn of computer science. So people like Turing, Shan, and even Babbage were thinking about computer chess in the early days of computer science. And current chess engines are highly specialized systems that have been obviously very successful in playing chess."}, {"start": 1403.94, "end": 1429.49, "text": "And they used Alpha Beta Search and a whole bag of handcrafted heuristics that have been distilled from human grandmasters. So if we look at Stockfish, which was the program that won the 2016 World Championship, this is the list of special case heuristics and extensions that Stockfish has. It's a gigantic list of..."}, {"start": 1429.49, "end": 1457.12, "text": "You know, specific types of techniques to help it play chess and rules to help it play chess. So instead of that, we just replaced all of that in hand engineering with self-playing reinforcement learning and self-playing anti-card research. So there's no opening book in AlphaZero. There's no in-game database. There's no heuristics of any kind. It just starts completely from random play."}, {"start": 1457.12, "end": 1480.3, "text": "And I wasn't actually sure, as a chess player, that this would work, given how important openings and endgames are. And, you know, Snogfish plays, the programmer Snogfish, plays the opening and endings nearly perfectly. But, oh, and the other thing was we wanted, obviously, again, just a notion of generality, the same algorithm, the same type of parameters for all three games."}, {"start": 1481.07, "end": 1507.31, "text": "And much to our pleasant surprise that, you know, almost the first time we tried to run this, that we found that AlphaZero beat the state-of-the-art chess programs in all three games. So in chess, we beat Stockfish 28-0, so we didn't lose one game out of 100 with 72 draws, including some wins with black. And in shogi, it beat the world champion Elmo 90-8."}, {"start": 1507.31, "end": 1534.67, "text": "And in Go, we beat AlphaZero, beat AlphaGo Zero, 60-40 after three days, comparing them both at three days of training. So AlphaZero was able to surpass Dotfish's ELO strength in four hours, ELO in two hours, and AlphaGo, what we call AlphaGo Lead, the LisaDoll version, in eight hours. So this whole thing, all three games in less than 24 hours."}, {"start": 1534.96, "end": 1557.81, "text": "The other thing is you might be thinking, well, is it a question of compute power? We honestly don't think it is. I mean, we use Google's Tensor processing units, four of them, in the cloud, and that's quite a reasonable amount of power. But actually, one way to measure this, an easy way to measure this, is how many moves are searched per decision made for AlphaZero."}, {"start": 1557.81, "end": 1585.22, "text": "And for human grandmasters, they search about tens of moves per decision they make. State-of-the-art chess engines like Stockfish search tens of millions of moves for each decision they make. So Stockfish has about 70 million moves per decision. And AlphaGo Zero is right in the middle of the spectrum in that it's using about tens of thousands of moves per decision."}, {"start": 1585.22, "end": 1603.73, "text": "So it's right in the middle, so it's three orders of magnitude less than the state-of-the-art chess engines. Obviously it's still three orders of magnitude more than humans do, but it's moving towards the right direction in terms of being more human-like. We did a quick analysis, we're going to do more analysis in that full paper."}, {"start": 1603.73, "end": 1633.86, "text": "As opposed to the sort of archived paper is we looked at the openings that were discovered by AlphaZero. And I'm pleased to say that my favorite opening, the English opening, here in the top left, survived this eight hours of training. Whereas surprisingly, some really popular openings, like in the bottom left here, the King's Indian Defense, was not really favored by AlphaZero at all, at any stage. And if you look at AlphaZero's style, it's not like, I would say it doesn't play like a human."}, {"start": 1633.86, "end": 1650.51, "text": "But it also doesn't play like computer engines. It sort of plays in a third, almost alien way. What AlphaZero loves, and we've got to analyze this a little bit more, but we've had a few chess grandmasters look at this, including one on our team, and AlphaZero loves long-term positional sacrifices."}, {"start": 1650.51, "end": 1679.74, "text": "And that isn't really even a thing in chess, because sacrifices are mostly about tactics. So the idea of sacrifice is, you know, you give your queen away because you think you've calculated it out, then you can checkmate your opponent, you know, in a few moves later. So these are tactical sacrifices. They're quite unusual, but they're playing, and there's famous, you know, world champions like Lee Kang Tao, who's famous for doing those kinds of sacrifices. But what AlphaZero does is it makes sacrifices for some very long-term additional gain."}, {"start": 1679.74, "end": 1704.1, "text": "Which, you know, I was surprised that chess had the capacity to allow for things like long-term positional sacrifices. So it appears that this is an actual thing in chess. And I think the reason my kind of initial take on that is that because it doesn't have a concept of materiality, so it doesn't have a rule telling it that rooks are five points, knights are three points, pawns are one point, it doesn't know anything about that."}, {"start": 1704.1, "end": 1729.3, "text": "So instead of that, everything's kind of contextual. It's sort of things like, in this kind of position, these assets I have, this rook that can move like this, isn't that useful? Whereas if my opponent's pawn or knight is really powerful in this position, so I'm going to swap them. And it doesn't have to overcome some vicious rule it has in its system that tells it that it's going to lose two points by swapping the rook for the knight."}, {"start": 1729.33, "end": 1757.15, "text": "Now obviously, a stock fish and those kind of chess engines can do sacrifices, but they have to calculate enough to see some endpoint that they will get that material back or there will be some other compensation for it. Whereas AlphaZero doesn't need to do that. It can be patient about recovering that material at some point down the line. So I think the key thing here is that rigid rules, like the traditional chess engines have, can't be dynamically rebalanced in a situation."}, {"start": 1757.15, "end": 1783.12, "text": "You can't just say, these systems can't have another if-then rule saying, in this specific kind of position, this rook is not worth five points, it's worth three points. It's always worth five points in its hard-coded databases. So the initial conception from my chess friends would be telling me is that maybe our conception of chess has been too limited, so this could be a very exciting moment for chess. And I think humans can learn from this because these ideas are strategic rather than tactical."}, {"start": 1783.12, "end": 1805.09, "text": "So we can actually graph that into our own play. And I just want to, near the end now, just talk quickly about, show you a couple of, for those of you who know chess, and as you probably gather, I love games, so there's quite a lot of game theory in this talk. But here's a position I want to show, a couple of positions showing these kinds of long-term sacrifices."}, {"start": 1805.09, "end": 1833.82, "text": "So AlphaZero is white here, and basically what it's done is sacrificing a bishop. You can see it's just taken a pawn, and it's going to give away its bishop. So now it's a bishop down in this position. You can see the black bishop there in the top left corner on b7. But it's completely calm about this. And I think what's happened is it's decided that that bishop there that black has, that it's hemmed in by the pawns, isn't worth very much. So it's definitely not worth three points."}, {"start": 1833.82, "end": 1855.68, "text": "So he's even allowed his queen to be swapped off. This is kind of like if you're a bishop down. I mean, this looks ridiculous in some sense. But it just can't. I mean, it's only got one pawn for it. So it just can't be carried on today. And eventually, Stockfish is sort of thrashing around with his bishop, but he can't do anything. And so eventually, he just marches his king in."}, {"start": 1855.68, "end": 1885.2, "text": "And then, you know, Stockfish just resigned. So it's kind of amazing. And then there's another one, because I like this one. It's a mid-game knight sacrifice. So this is sort of the middle of the game. And AlphaZero has just played this knight move to the corner of the board to check the king. And it basically gives up this knight incredibly. So it just allows Stockfish to take the knight. And it just can't be good. It develops its pieces."}, {"start": 1885.2, "end": 1907.97, "text": "And then it does this brilliant maneuver, which I love, and I showed one of the grandmasters this, and he just couldn't believe it. So it takes his pawn here, and then it moves his queen into the corner. I mean, I haven't got time to explain to you how just kind of alien and incredible this maneuver is. Because now it's lined up with the bishop to attack that critical pawn on c6, right? But what a maneuver. So, you know, I was just thinking..."}, {"start": 1907.97, "end": 1929.01, "text": "You know, there's a lot more here to explore, but I just think this is kind of chess from another dimension, I would call it. And so, Derek has brought up for himself, actually, even before he saw the AlphaZero results, he was talking about Deep Blue was an end of some kind of era, and he felt AlphaGo, because it's a learning system, equals a Type B system versus a Type A system, is some kind of beginning."}, {"start": 1929.3, "end": 1954.66, "text": "So I just want to end by talking, and I know that Gary will obviously be talking later, and I wanted to pre-empt some of his comments. So there's some limitations, obviously, with this system. Where I think of it is intelligence search to a huge commentorial space. So what you have to have is a clear objective function, and you have to have either large amounts of data or an efficient simulator, in this case, to generate your own data."}, {"start": 1954.66, "end": 1978.35, "text": "And this system still lacks a lot of things after zero. It can't generate hypotheses or design its own experiments, and it hasn't got any kind of abstract knowledge or causal reasoning abilities. So we're interested in all those things. But I do want to say, a caveat to all of that, a few years ago, I think pretty much everyone in this room would have thought that these results were kind of unthinkable. And, you know, this system is very general."}, {"start": 1978.35, "end": 2005.18, "text": "It is the general games-playing machine I always dreamed about 20 years ago and some of my colleagues did. And I think we have it now, at least for perfect information games. Obviously, we're not claiming this is solely AI in any shape or form. There's a lot of huge challenges to remain. Learning the model, learning the rules, dealing with partial information, uncertainty. There's questions of efficiency, being able to do one-shot learning and abstract knowledge. I think Josh Turnaround is going to talk a lot about that."}, {"start": 2005.18, "end": 2034.5, "text": "But one point I would make, and all these things that we're working on, all those projects, and we think that they're very important, clearly for general intelligence. But there's one thing about human efficiency I just want to mention, which is that, obviously, humans, it's not quite right to say that, you know, let's take out the Go Zero and have 30 million games, and we worked out that someone like Lisa Dole probably played about 30,000 games in their lifetime. And that seems like a huge discrepancy. But don't forget that humans are coming to this with some evolutionary priors already."}, {"start": 2034.5, "end": 2051.42, "text": "And also a lot of transfer learning, and most importantly, apprenticeship learning. So obviously Lisa Bowles read Go books, which are themselves distillations of many other humans' knowledge on the game. And they've also been taught by their Go masters. So I think in the future it would be quite interesting to actually test"}, {"start": 2051.42, "end": 2074.8, "text": "But you could actually maybe think about an information test on a book in the sense of you could have one version of AlphaGo Zero that learns purely through self-play and another version that wants to be able to read books that reads a book and then does self-play. And then you could kind of compare how many games of self-play was that book worth. So I kind of look forward to the time when we're able to do that."}, {"start": 2074.9, "end": 2100.62, "text": "And what we're after really, I think, and I discuss this a lot with my colleagues here, is that the minimum set of maximally general products is what we're after. And obviously, just to end, we're not looking to just play games with our systems. That's just our proving ground, our testing ground, and albeit it's a very fun one. And really what we're trying to do is acquire these ideas from the real world, and we've already started doing that in areas of healthcare to energy optimization and data centers and all sorts of other things."}, {"start": 2100.62, "end": 2115.06, "text": "And for me, the most exciting thing would be to try and apply this in areas of science. And although it's still quite limited in the ways I've just mentioned, I think there are already areas of science that may be quite amenable to this type of approach, like drug design, material design, and some aspects of biotech."}, {"start": 2115.06, "end": 2136.08, "text": "And I think ultimately this is the area I'm looking forward to as I work on AI, is the power of humans and machines together. You know, humans and us and our smart tools figuring out some complex scientific questions. And I think human ingenuity augmented by AI will unlock our true potential. Thanks for the opportunity. Thank you."}]}