{"url": "https://media.podcasts.ox.ac.uk/comlab/comsci/2016-02-24-strachey-lecture-no-q+a.mp3", "page_url": "https://podcasts.ox.ac.uk/strachey-lecture-artificial-intelligence-and-future", "title": "Strachey Lecture: Artificial Intelligence and the Future", "show_or_channel": "Oxford Strachey Lectures", "date": "2016-02-26", "duration_s": 3308, "type": "talk", "speakers": ["Demis Hassabis"], "confidence": "high", "notes": "Oxford Strachey Lecture, Feb 2016 (just before AlphaGo vs Lee Sedol).", "people": ["demis-hassabis"], "id": "2016-02-26_oxford-strachey-lectures-strachey-lecture-artificial-intelligenc", "audio_duration_s": 3308.136, "worker": "jc-w18", "transcribe_s": 137.5, "model": "large-v3", "segments": [{"start": 0.14, "end": 31.6, "text": "My name's Mike Waldridge. I'm a professor of computer science and currently head of the Department of Computer Science at the University of Oxford. And I would like to welcome you all to this term's Strachey Lecture. The Strachey Lectures are the distinguished lectures in computer science that the Department of Computer Science offers."}, {"start": 31.6, "end": 58.16, "text": "We do not usually host our straight G lectures in the Sheldonian Theatre. The fact that we are able to do this today is because of the generous support of Oxford Asset Management. We are very, very grateful to Oxford Asset Management who literally have made this event possible on a scale and type that would not have been possible otherwise. So we thank them very much for that and for their continuing ongoing support."}, {"start": 58.58, "end": 87.36, "text": "Let me introduce, let me say a few words about today's speaker. It's an enormous pleasure to be able to welcome Demis Hassabis of Google DeepMind to be this term's straight G lecturer. Demis gained his undergraduate degree from Cambridge in computer science. He then went on and was for some time a successful computer games programmer, designed a number of games which went on to achieve a degree of success in the marketplace."}, {"start": 87.36, "end": 112.99, "text": "He did a PhD at UCL in cognitive neuroscience. And then in 2009, he was a co-founder of a company called DeepMind. Well, all through that time, I think it's probably fair to say that he didn't hit the front pages of any newspapers. All of that changed in 2014 when DeepMind were acquired by Google for the very un-British sum."}, {"start": 112.99, "end": 136.16, "text": "I believe 400 million, which is not a figure that happens that much in any part of British industry and got him and DeepMind on the front pages of the international press. And computer science professionals like myself were all agog to see what this company, which I think is probably fair to say had been operating in stealth mode or something like it for a number of years, had to offer."}, {"start": 136.16, "end": 154.46, "text": "Well, we didn't have to wait very long. Very soon after that, the first results became public from DeepMind, and it became clear what Google were interested in. I'm not going to spoil them as a show by telling you about those results now, but some very impressive results to do with learning to play video games."}, {"start": 154.46, "end": 167.63, "text": "Being on the front pages of the national press just once or twice was not enough for DEMIS. They got on the front pages of the international press last year with some incredible achievements in the area of computer programs playing the game of Go."}, {"start": 167.63, "end": 197.31, "text": "And we now have the opportunity to see Demis speak at a really special time for DeepMind, because as Demis is about to tell us, they are heading up towards a competition to play Go with some of the world's leading players. So we're going to get an insight into a company that's doing remarkable things at one of the most remarkable points in their trajectory. So Demis, it's with very great pleasure that I introduce you and welcome you to give this term straight to your lecture. Over to you."}, {"start": 197.31, "end": 231.06, "text": "Thank you. Good evening, ladies and gentlemen. Good evening, ladies and gentlemen. Welcome to the Sheldonian Theatre. Before the lecture begins, would you ensure your mobile phones are switched off? I would also remind you that unauthorised photography and recording are prohibited. In the interest of safety, would you ensure emergency exits, walkways and window lights are kept clear of personal belongings? Guests in the upper gallery are asked to leave using the stairs to the sides and not to use the tiered seating as steps."}, {"start": 231.06, "end": 236.94, "text": "Thank you. Okay, well thanks from the voice from the sky."}, {"start": 236.94, "end": 263.81, "text": "Thanks Mike for that very generous introduction. It's a huge honor and real pleasure to be giving this lecture, be invited to give this lecture in these auspicious surroundings. So what I'm going to try and do today in my talk is to give you a whirlwind tour of what's happening at the cutting edge of artificial intelligence and then end with some of the latest breakthroughs that we've been doing at DeepMind. And then I'll probably talk a little bit about the bigger picture of"}, {"start": 263.81, "end": 292.4, "text": "Artificial Intelligence and where I think it's heading to in the future and then we can go into the Q&A. So Artificial Intelligence, AI, is basically the science of making machines smart. Now what DeepMind is, we founded it in 2010 and as Mike mentioned we joined Google in 2014 to accelerate our mission. And the way we think about DeepMind is a sort of Apollo program or Apollo program effort for AI."}, {"start": 292.85, "end": 323.54, "text": "We have about 200 research scientists and engineers now. So I think it's probably one of the biggest collections anywhere in the world of talent focusing around this topic. And not only is this a very ambitious sort of research program, but we also try and think about a new, more efficient, and productive way of organizing science and scientific research. And in terms of the environment we've created, we've tried to sort of build a unique environment that's a blend between"}, {"start": 323.54, "end": 352.43, "text": "The best of academia and how academia should function in an ideal world. And the best from the top sort of Silicon Valley startups. So kind of blue sky thinking from academia and collaborative interdisciplinary research. And then the focus and the energy and buzz and resources that really successful startups have. We tried to fuse this together into a unique environment that's uniquely suited to research."}, {"start": 354.13, "end": 382.98, "text": "So our mission at DeepMind, we basically articulate it, or at least I do, in this way. So step one, we try and fundamentally solve intelligence. And then step two, use that to solve everything else. So this may seem quite fantastical to you, this step two, but actually I hope that by the end of this talk you'll be convinced that it actually naturally follows on from solving step one. So more prosaically, how are we going to attempt to do this?"}, {"start": 382.98, "end": 413.54, "text": "Well, at DeepMind, what we're interested in doing is building what we call general purpose learning algorithms. So the key things of everything that we do is that our algorithms learn how to master certain tasks. They learn automatically from raw inputs or raw data. They're not pre-programmed or handcrafted in any way. The second important notion that we have is this idea of generality. So this is the sort of the idea that"}, {"start": 413.54, "end": 442.78, "text": "The same system or same set of algorithms can operate out of the box across a wide range of environments and tasks. So we call this kind of AI internally at DeepMind Artificial General Intelligence, AGI. And the hallmark of AGI is that from the ground up, it's built to be flexible, adaptive, and inventive. It can deal gracefully with the unexpected. Now, if we compare that with most AI that's out there today,"}, {"start": 442.78, "end": 472.34, "text": "Which we term narrow AI to distinguish it from AGI. Most of the AI that you interact with every day is handcrafted and special cased to a particular single task. And what that often means is that these systems are quite brittle. If you do something unexpected or something unexpected happens that the programmers of that system didn't cater for before time, it will catastrophically fail. And you can see that with things like Siri on your phone."}, {"start": 472.34, "end": 501.63, "text": "You know, it works fine if you stick to the templates that have been pre-programmed, but as soon as you start going off-piste with your conversation, the holes in the algorithms quickly become apparent. So, still today, probably the greatest achievement, one of the greatest achievements in AI was Deep Blue beating Garry Kasparov at chess in the late 90s. Of course, this was a huge technical achievement and an absolute watershed moment for AI research. But having said that,"}, {"start": 501.63, "end": 529.74, "text": "The question is, was Deep Blue truly intelligent? And I think even the designers of Deep Blue, and certainly we would argue, that it isn't really. And one easy way to see that, intuitively, is the fact that Deep Blue couldn't even play a strictly much simpler game like Noughts and Crosses without being totally reprogrammed from scratch. There was no knowledge in the algorithms that Deep Blue was running that would help it play any other game, let alone do anything else."}, {"start": 529.87, "end": 558.13, "text": "So I actually came away, I remember this match very distinctly, I was studying at Cambridge, and I actually came away more impressed by Garry Kasparov's mind than the computer. Because here was Garry Kasparov able to compete on more or less level terms with this brute of a machine, and yet, of course, Garry can do many other things, speak several languages, drive cars, tie shoelaces. So, you know, in a way it's quite amazing that the human mind, what the human mind is capable of."}, {"start": 558.96, "end": 588.98, "text": "So instead of this kind of regime, how do we think about artificial intelligence? Well, I would say the core of what we're doing at DeepMind focuses around what's called reinforcement learning. And that's how we think about intelligence at DeepMind. So I'm just quickly going to explain that with the help of a little simple diagram here, what reinforcement learning is. So if we start off with the agent system, the AI. And the agent system finds itself in some kind of environment trying to achieve a goal."}, {"start": 588.98, "end": 619.71, "text": "Now, that environment could be a real-world environment, in which case the agent would be a robot, or it can be a virtual environment, in which case the agent would be an avatar. And in fact, for most of our researchers, you'll see we use virtual environments. Now, the agent interacts with the environment in just two ways. Firstly, it gets observations through its sensory apparatus. We mostly use vision currently, but we're starting to think about other modalities. And one of the jobs of the agent system is to build the best possible model"}, {"start": 619.71, "end": 647.68, "text": "The second job of the agent is, once it's built this model of the world, is to use that model to make predictions about what's going to happen next. And if you can make predictions about the world, then you can start planning about what to do. So if you're trying to achieve a goal,"}, {"start": 647.68, "end": 677.3, "text": "The agent will have a set of actions available to it at that moment. And the decision-making problem is to pick which action will be the best action to take right now to get you towards your goal. And once the agent has decided that based on its model and its planning trajectories, it executes that action. And that action may or may not make some change the environment. And then that drives a new observation. And that's really it. That's the heart of reinforcement learning."}, {"start": 677.3, "end": 704.82, "text": "But although this diagram is very simple, those of you who know about reinforcement learning will understand there's huge complexity hidden behind this simple diagram. But we do know that if we could solve all the issues behind reinforcement learning and make this work perfectly, then that would be sufficient for general human level intelligence. And the reason we know that is because biological systems learn using reinforcement learning, including the human brain."}, {"start": 704.82, "end": 735.49, "text": "In fact, there are some seminal studies done in the late 90s on monkeys that showed that the dopamine neurons in the brain implement a form of reinforcement learning called TD learning. So reinforcement learning is then at the core of what we do at DeepMind. The second big philosophical thing we committed to at the founding of DeepMind was this idea of grounded cognition. So this is the idea that a true thinking machine"}, {"start": 735.49, "end": 762.67, "text": "has to be grounded in a rich sensory mode to reality or data stream. Now, when people commit to this sort of sentiment, often they then start working on real robots. Because after all, real robots are actually situated in the real world. And of course, through their sensory apparatus, they're getting data, real world data. But we actually made a different decision on this. We decided to use virtual environments in games."}, {"start": 762.67, "end": 785.33, "text": "And we think they're the perfect platform, if used correctly, for developing and testing AI algorithms. One of the important things you have to do and avoid is that when you use virtual environments, of course, if you want to, you could allow your agent to have access to all kinds of the internal states of the game that it couldn't actually directly sense through its normal sensory apparatus."}, {"start": 785.33, "end": 810.1, "text": "And of course, that's something you have to avoid. Otherwise, you'll think that you're making progress with our AI algorithms, but actually, it will be cheating in some way. So we have to be very disciplined about how you allow the interface between the virtual environment and the agent, and really treat the agent as if it was a virtual robot, only getting the information that it could be available to it through its sensors."}, {"start": 811.41, "end": 840.03, "text": "Now, if you use games like that, then there are many advantages. Of course, you can create as much training data as you like. This is very important when we were a small independent company and we didn't have access to a lot of data, but it's still vital now, even though we're at Google. There's no testing bias. One of the biggest things I think that held back the AI research field was that often you'll find that AI researchers are also the ones that are creating the tests."}, {"start": 840.03, "end": 863.31, "text": "And that can lead to unconscious sort of biases about the sorts of tests that you design. We end up designing tests that subconsciously, at least, our algorithms are well suited to. Of course, if we're talking about virtual agents in virtual environments, we can test thousands, perhaps even millions of these agent systems in parallel."}, {"start": 863.34, "end": 877.95, "text": "And games are very convenient in that a lot of them have scores or quite easily identifiable goals. So it's very easy to measure incremental progress and how your algorithms are doing when you incrementally improve them."}, {"start": 877.95, "end": 908.46, "text": "And that's very key for us. Actually, benchmarking is a hugely important thing that we have. We have a whole team who works on that. Because when you've got a very ambitious long-term goal, it's even more important to have short-term directional sort of waypoints that tell you if you're heading in the right direction towards this big, ambitious long-term goal. So putting all this together then, this brings to sort of the notion of what we call end-to-end learning agents."}, {"start": 908.46, "end": 936.24, "text": "So this idea of going all the way from the pixels or the raw data input and then ending up with making a decision about what action to take. And at DeepMind, we're interested in that entire stack of problems. So everything from perceptual processing to decision making and all the things in between. So our first attempt at doing this, which really scaled to something challenging, we call deep reinforcement learning."}, {"start": 937.04, "end": 967.34, "text": "And the essence here was combining deep neural networks, which is called deep learning these days, with reinforcement learning. And what this allows reinforcement learning to do is to actually scale up to work on very challenging problems. Until we sort of came up with this paradigm, reinforcement learning has been around for many decades, but has usually only been used for relatively toy grid world problems. It's been hard to scale it up to something challenging with high dimensional sensory inputs."}, {"start": 968.62, "end": 994.74, "text": "So I'm going to show you a few videos of this agent working. But before I do, I just want to clearly explain what it is you're going to see. So we started off with really the first iconic console, the Atari 2600 from the 80s. This has the benefit of there are hundreds of different classic games, many of which are iconic and everyone will recognize. But it's still quite a challenging sensory data stream."}, {"start": 995.86, "end": 1026.26, "text": "The agents here, they only get the raw pixels from the screen as inputs. So it's around 30,000 numbers per frame because the screen is about 200 by 150 pixels in size. And the goal here is to simply maximize the score. The agent system has to learn everything else from scratch, from first principles. It doesn't know what it's controlling. It doesn't know what the object of the game is. It doesn't know what it gets hit points. It doesn't even know that pixels next to each other"}, {"start": 1026.26, "end": 1057.01, "text": "And then there's an additional constraint or requirement we put on the system, which is this idea of generality again, that a single system has to play all the different games without any changes, with the same hyperparameter settings and other settings. So I'm going to show you a couple of videos. So the first one I'm going to show you is Space Invaders, with two parts to it, one where the agent"}, {"start": 1057.01, "end": 1087.17, "text": "has had no training, so literally the first time it's seen the data stream, and then after a day or two worth of training. So initially, you can see it's controlling the green rocket at the bottom of the screen. It's losing its three lives immediately, because obviously it has no idea at the moment what it's supposed to be doing, or even the fact that it's controlling that collection of pixels at the bottom of the screen. Now, after training by playing the game overnight for 24 hours,"}, {"start": 1087.17, "end": 1116.56, "text": "And now the system is superhuman level. So it can play Space Invaders better than any human can. So you can see here, every single bullet hits something. It's learned that the pink mothership at the top of the screen, coming across there now, which it hits with this amazing shot, is worth the most number of points. And as you'll see later, those of you who remember Space Invaders, if you're old enough to remember, the less of them there are, the faster they go. So just watch the last sort of predictive shot that it makes to get the last one."}, {"start": 1116.56, "end": 1141.3, "text": "So, you know, so it's built up these very accurate models, implicit models of what's happening in this game, in this data stream. So I'm going to show you another video now, this breakout, my favorite video. So here you control the bat and ball, and you're trying to break through this rainbow-colored wall. Now, at the beginning, after 100 games, you can see the agent is not very good. It's missing the ball most of the time, but it's starting to get the hang of the idea that the bat should go towards the ball."}, {"start": 1141.65, "end": 1169.52, "text": "Now, after 300 games, it's about as good as any human can play this, and it pretty much gets the ball back every time, even when it's coming back at very fast vertical angles. But then we thought, well, that's pretty cool, but we left the system playing for another 200 games, and it did this amazing thing. It found the optimal strategy was to dig a tunnel around the side and put the ball around the back of the wall. And you see how incredibly accurate, of course, it can send the ball around the back."}, {"start": 1169.55, "end": 1190.26, "text": "So the funny thing about that is that obviously the researchers working on this are amazing AI developers and programmers, but they're not so good at breakouts. And actually, they didn't know about that strategy. So they learned something from their own system, which is pretty funny and quite instructive, I think, about the potential for general AI."}, {"start": 1191.25, "end": 1222.06, "text": "I'm going to show a final video here of Atari, which is really showing a medley now of many different games, just to give you a feeling that this system, which we called DQN, really is a general AI within the constraint of Atari games. So here is the same system you just saw playing those other games, playing an early racing game called Enduro. Here it's playing a game called River Raid, which is a fighter pilot game. There's one of the very early 3D games called Battle Zone."}, {"start": 1222.06, "end": 1246.48, "text": "There's obviously the classic Pong. It's controlling the green bat here, and it wins 21-0 every time. Can't get a point off of it. Sequest, Submarine Game. So you can see the absolute diversity of the graphics and also the objectives. So here's a boxing. It's controlling the red boxer on the left. It does a bit of sparring. Then once it gets the inbuilt computer on the side, it just racks up an infinite number of points."}, {"start": 1246.48, "end": 1269.63, "text": "He's happy to just carry on doing that forever. So the very, very diverse range of games, same system out of the box mastering these things. Now, if you want to read more about that, that was featured in our Nature article at the beginning of last year. We also released the code. So if you want to play around with this system yourself, you can. It's freely available on the internet."}, {"start": 1269.63, "end": 1298.24, "text": "And so we then sort of took that, that was around a year ago, and now we've taken that further and we started looking at 3D games, at using simulators like robot simulators, and eventually we would like to get to start thinking about real robotics. Now I'm just going to quickly show you a couple of 3D videos now with effectively the same deep reinforcement learning system with a few tweaks now coping with this 3D data stream."}, {"start": 1298.24, "end": 1326.99, "text": "So here is a DQN-like algorithm driving a racing car very fast around the track, again, just from raw pixel inputs. So that's how it's learned. And it's driving around about 200 kilometers an hour, and it figures out overtaking maneuvers. It can also recover from spins, all sorts of things. So it has really amazing performance now in these kinds of driving games, just from the vision."}, {"start": 1328.11, "end": 1356.24, "text": "We started looking at harder problems in 3D mazes, collecting objects, finding your way out, remembering where you've got to go, the sort of things that maybe a rodent like a rat would be able to do. And so you can sort of think about where we're going next is trying to build a kind of rat-level AI intelligence. And rats are actually pretty smart. They can do quite a lot of things. And so here it is again just from..."}, {"start": 1356.24, "end": 1381.76, "text": "The visuals on the screen, just from the raw pixels, finding these green apples, which are rewarding, and then trying to find the exit, which is this little red floating object, and efficiently navigating around. So that's where we are on sort of 3D, and there'll be bigger announcements about that later this year. So I've talked about sort of reinforcement learning. I've talked about grounded cognition."}, {"start": 1381.76, "end": 1408.35, "text": "Another thing that I think is sort of pretty unique to DeepMind's approach to AI is taking systems neuroscience seriously as a source of inspiration for new algorithmic ideas, but also as a kind of validation testing, if you like. So if you have your own pet favorite algorithm or algorithmic technique, and the question is that you're not sure if this can scale up to become a component of general AI."}, {"start": 1408.35, "end": 1437.38, "text": "How much effort should you put into that? Should you spend five years doing that, ten years? How many people should be working on that? These are very difficult decisions if you're running a lab or a department or a company working on this kind of thing. Now, if you can point to something in the brain and show that, like with reinforcement learning, as I said earlier, we know that the brain implements TD learning through the dopamine system. That gives you confidence then that in the limit, this has to be sufficient."}, {"start": 1437.38, "end": 1450.86, "text": "For example, reinforcement learning. It's not crazy to think about that as a component, a vital component for the general AI solution. And that can be very important directionally when you're thinking about four or five year research programs."}, {"start": 1451.31, "end": 1475.09, "text": "So systems, but when I say neuroscience, I should be very clear we're thinking about systems neuroscience. So we mean the algorithms, the representations, and the architectures the brain uses, rather than something like the Human Brain Project, which is more interested in the low-level synaptic implementation details of how the brain achieves with things, with spiking neural networks. That's too low-level for us. We're more interested in the computational level of the brain."}, {"start": 1476.02, "end": 1501.9, "text": "So we're using many ideas for assistive neurons. I haven't got time to go into it today, but here are some of the things we're looking at. Memory, attention, concepts, planning, navigation, imagination. And all of these areas we're actively researching on right now and have very interesting prototypes on. And in fact, I'll just mention one thing. For my PhD, I studied an area of the brain called the hippocampus, and I studied memory and imagination in the human brain with fMRI."}, {"start": 1501.94, "end": 1531.82, "text": "And it turns out the hippocampus, which is shown here in pink, which is in the center of your brain, is actually critical for many of these capabilities, especially things like memory, navigation, and imagination. And the hippocampus has a very different structure to cortex. So it's quite interesting when people talk about intelligence in the brain, they usually talk about the cerebral cortex. But actually, there are other structures in the brain that are equally critical to this whole question of intelligence."}, {"start": 1534.32, "end": 1555.39, "text": "So now I'm going to talk a little bit about our newest work, AlphaGo. And the reason we took on this project, and I'll explain a lot more about what this is in a second, is that AlphaGo really combines pattern recognition with planning. So what you've seen so far with the Atari games is really a kind of stimulus response system."}, {"start": 1555.39, "end": 1583.14, "text": "So it's very smart, but it's stimulus response. So it learns about how to process Atari screens and, generally speaking, what to do in that moment in terms of an action that will maximize its score. But there isn't a lot of long-term planning. Now, contrast that with a game like Go. So Go, for those who don't know how to play it or don't know what it is, this is a picture of a Go board."}, {"start": 1583.14, "end": 1612.27, "text": "So Go is the sort of pinnacle of board games. It's the most complex game pretty much ever devised by man that's played professionally. And the way it's played is that you play on a 19 by 19 grid, and there's two sides, black and white, and you put down these pieces called stones. And the stones are placed on the vertices of the board. And black goes first, and they take turns placing one stone at a time."}, {"start": 1613.84, "end": 1643.15, "text": "Once the stones are placed, they don't move. Now, the rules of Go are incredibly simple. I'm going to teach you how to play Go in two slides. But it leads to incredible, profound complexity. That's why it's considered to be one of the most elegant games ever invented. Now, a quick history of Go, for those of you who don't know about it. It originated in China over 3,000 years ago, and it has an incredibly rich tradition."}, {"start": 1643.15, "end": 1672.37, "text": "In Asia, so in China, Japan, and Korea and other Asian countries, this is what they play instead of chess. But in Asian countries, this is regarded as more than just a game. Go is sort of elevated to the status of poetry or art. In fact, Confucius wrote about Go, and he considered it to be one of the four essential arts to be mastered by any true scholar. Japan also has a rich history around Go."}, {"start": 1672.37, "end": 1702.08, "text": "And in fact, during the Edo period, sort of 250 years, 1600s, 1800s, annual games were played. They were called castle games in front of the Shogun. And what would happen is each tribe or clan would send their top Go player to play in the castle game for the honor of the whole clan. And some real legendary players came out of this. There's one guy called Shusaku who won 19 years in a row."}, {"start": 1702.08, "end": 1732.53, "text": "And it's gone down in legend and has the nickname The Invincible. So they were really absolute heroes in that period. So it has this incredibly rich history intertwined with the culture of Japan. But it's not just an ancient game. Today, there are over 40 million active players. In many of these countries, like in Korea, for example, it's taught as part of the school curriculum. And there are specialist Go schools."}, {"start": 1732.53, "end": 1761.42, "text": "At Go at a young age, then you'll go to these Go schools from about the age of 10 instead of going to normal school. So it's taken very, very seriously. Now, as I was going to say, and I'm going to show you in a second, there are just two rules actually for Go. But the complexity is huge that arises out of these very basic rules. One quick, easy way of measuring, illustrating to you the complexity, is the fact that there are"}, {"start": 1761.42, "end": 1787.15, "text": "10 to the power 170 possible board configurations. And in fact, 10 to the 700 different possible games. And that's more than the number of atoms in the universe. So there's no way that you can solve Go through exhaustive search or even play Go well through exhaustive brute search, brute force search. It's just too large. So how do you play Go? Well, rule one is called the capture rule."}, {"start": 1787.6, "end": 1817.58, "text": "So here is a position from a game of Go. And we're just going to zoom into the bottom right of this board so I can show you how the capture rule works. So let's look at this little part of the board. Now, if you see that white stone there that's surrounded by the three black stones, the empty vertices coming out from each stone adjacent to where your stone is are called liberties. Now, when you run out of liberties, then those stones are removed from the board."}, {"start": 1817.74, "end": 1843.82, "text": "So here, that white stone that's surrounded by the three black stones only has one liberty left, that empty vertex above it. So if it's black's move and black were to play into that final empty vertex, taking away the last liberty, then the white stone would be captured. So now that white stone has no liberties and would be taken as a prisoner off the board. So that's the capture rule. And you can capture multiple large groups of stones like this, not just one at a time."}, {"start": 1844.56, "end": 1872.35, "text": "The second rule is that a repeated board position is not allowed. So this is called the co-rule. So let's imagine here's another little zoomed in part of the board. Now let's imagine it's White's turn. So White could here, I'll just replicate that board to the right so you can see how this is going to become a repeated board position. Let's imagine it's White's turn. They play here and capture that black stone like I just showed you. Now it's Black's turn."}, {"start": 1872.35, "end": 1901.36, "text": "Now Black, you might think, well, they could just play back where that stone was captured and recapture the white stone that was just put down. So you might think, why can't Black go there? And this is actually not allowed. Because if Black was to go back, that would recapture that white stone, and you'd see now this new position we're in is the same as that original position. So in fact, that recapture would not be allowed. Black would have to play somewhere else before recapturing. And that's it. That's how you play Go."}, {"start": 1901.36, "end": 1928.14, "text": "So the objective of the game is to not only capture your opponent's stones, but also to wall off and surround empty territory, empty vertices. And you can see here, this is a picture of a go board at the end of the game, and you just total up the number of the spaces that you've captured, and you add that to the number of stones that you've taken off the board. And the player with the highest total is the winner."}, {"start": 1928.34, "end": 1959.7, "text": "So here's the white territory and the black territory, and in fact, this is a very close game, and white wins this game by one point. So that's how you play Go. So why is it hard for computers to play? Well, I've just explained the complexity makes brute force exhaustive search intractable, and there are two main challenges. The branching factor is huge, and writing an evaluation function to determine who is winning is thought to be impossible. So an evaluation function is a function to tell you whether the black or the white side is winning."}, {"start": 1959.82, "end": 1987.41, "text": "And for Go, this is very difficult. Let me just unpack that for you by comparing it to the next most complex game, chess. So in chess, on an average position, there are 20 possible moves. And that's referred to as the branching factor. In Go, by contrast, in an average position, there's around 200 moves. So the branching factor in Go is one order of magnitude bigger than it is for chess."}, {"start": 1987.92, "end": 2013.78, "text": "The second issue, which is related to the evaluation function, is that Go is really primarily a game about intuition rather than brute calculation. If you ask a great Go player why they played a certain move, often they'll just tell you it felt right, and they'll use those words. Whereas if you ask a great chess player that, they'll never say that. They'll tell you exactly the reasons, how they calculated that that move was the right move to do."}, {"start": 2014.38, "end": 2044.77, "text": "And of course, what we know about computers is that when we start using words like intuition, computers are generally, traditionally not good at what we think of as intuition. But of course, they're very good at things that we think of as calculation. So that's one of the challenges of Go, making computers good at Go, is to replicate this kind of intuition that humans use to play. So this is the issue with writing an evaluation function and why it was thought to be impossible. For Deep Blue or any chess program,"}, {"start": 2044.77, "end": 2070.83, "text": "What you can do is write a set of handcrafted pre-programmed heuristics or rules. In fact, a first approximation for chess, if you just count up the value of the pieces on each side, that gives you a very rough and ready but reasonable estimate of which side, black or white, is winning. That's, of course, impossible for Go because each of the pieces are worth the same. They're just stones. So there isn't any idea of sort of materiality."}, {"start": 2072.62, "end": 2098.34, "text": "So Go then, and this is why we've taken this on as a challenge, combines intuitive pattern recognition with logical planning and search. So I'm just going to take you through the technicalities of how we did this. So what we did is we trained two deep neural networks to deal with some of this intuitive part of Go. So the first thing we did is we downloaded 100,000 games played by..."}, {"start": 2098.34, "end": 2125.97, "text": "They were relatively expert humans. They were still amateurs playing on Internet Go servers, but they were pretty strong club players. So we took those 100,000 games and we trained our first neural network, which we called a policy network. And this was done through supervised learning. And what we did is we got this network to try and mimic the moves, copy and predict what move in a particular position that human amateur expert would play."}, {"start": 2126.1, "end": 2155.18, "text": "So this network was trying to copy those expert players. So that was the first step. Once we had the first version of that, we then allowed it to play against itself many millions of times and improve its prediction capability through the use of reinforcement learning. So it learned through trial and error, and it's from its own mistakes, and then that would then modify the neural network to make it better, incrementally better over time."}, {"start": 2155.18, "end": 2186.0, "text": "And once we finish this self-play process, the new policy network could beat the original policy network 80% of the time. Now we freeze this sort of final reinforcement learning policy network, and we allow that to play a final 30 million times on our Google Cloud servers. And that generates our new data set. And we take one position from each of those 30 million games. So we have 30 million positions. So now we finally have a data set"}, {"start": 2186.0, "end": 2212.48, "text": "That maybe is big enough to try and learn an evaluation function. So what we do is we have these 30 million positions and we have the end result of the game. So we can try and learn sort of correlation between that position and who ends up winning. So we then train this final network which we call the value network. And this value network learns to predict"}, {"start": 2212.48, "end": 2242.03, "text": "Who is winning the game from a particular position and by estimate by how much. So this is really the core of the breakthrough with AlphaGo was the value network is this fabled evaluation function. But instead of writing it out by hand, like something like Deep Blue, where we as expert Go players or chess players wrote out all the rules that could evaluate position by hand, a big database of rules. We instead have a neural network that learned for itself."}, {"start": 2242.03, "end": 2271.89, "text": "So we take these neural networks forward and we have two neural networks. Policy network, this network in green that I was showing you earlier. That takes the board position in blue here as the input. And the output is a probability distribution over the possible moves. And you can see here the height of the green bars is the probability mass assigned to that particular move by the policy network."}, {"start": 2272.3, "end": 2299.74, "text": "So what this means is that our AlphaGo system doesn't have to consider all these 200 possible moves every time it's looking at a decision point. It can maybe just look at the top three or four most sensible or most likely moves. The second network, the network in pink, which is the value network, that also takes the board position in as an input. But this time the output is a single number, a real number between zero and one."}, {"start": 2299.74, "end": 2329.49, "text": "Where 0 is meaning white is winning by a huge margin, 1 is blacks winning, and 0.5 is the game's even. So estimates who is winning the game and by how much. So we now take this forward and combine it with search. And I'm going to show you that in a second. But I just want to give you pictorially an idea, illustrate to you why using these two neural networks helps with making playing Go tractable."}, {"start": 2329.58, "end": 2359.1, "text": "So imagine here, we're searching through the game of Go, and each of these little nodes here, represented by these mini boards, are a position in a particular game we're playing. Now, the tree of possibilities branches out almost to infinity, these huge number of possibilities, which are completely not tractable to search. So what we do is, we firstly take the policy network, this network in green, and what that does is reduce the breadth of the search."}, {"start": 2359.1, "end": 2383.81, "text": "So we can hone in to only look at the moves that are plausible and sensible. So that reduces the breadth of the search. And then the value network you can think of is reducing the depth of the search. So instead of having to search through the entire game tree until the end of the game to tell you which side is winning, we can call the value network at any time."}, {"start": 2383.81, "end": 2413.1, "text": "And estimate which side is winning. So we can truncate that search at any depth level that we want. So as you can see, by using these two networks in tandem, we've cut down that enormous search space to something much more tractable. So I'm going to show you how we do our search now. So we use Monte Carlo tree search. And we also use another thing called rollout policies. And we combine that together with the two neural networks I've just shown you."}, {"start": 2413.46, "end": 2440.94, "text": "So let's imagine that we are making a decision. AlphaGo is in the middle of thinking about what move it should make next. And it's done a bit of searching from the current position, which is that node at the top of the tree. That's our current position. And it's found a couple of promising moves. Now, the value of each move is represented by the letter Q here, the action value of each move. And what we're trying to do is find the move, in essence, that has the maximum Q."}, {"start": 2441.2, "end": 2467.44, "text": "And what we might do is we might follow a trajectory that has quite high Q value. And you can see this in the bold black arrows. And we follow that trajectory until we hit a node that has not been explored yet in the game tree. So here on the left-hand side of this tree that we're unfolding. Now, once we hit that new node, what we do, the first thing we do is we call our policy network, the green network."}, {"start": 2467.6, "end": 2498.18, "text": "And we ask the policy network to expand the tree at that point, but only expand it with a few moves that it thinks are most probable. So with the highest P, the prior probability of that move. So once that's expanded, then we call the second neural network, the value net, to evaluate that position and give an estimate of who is winning. We also do a second thing. We call, if we have time, we do rollouts."}, {"start": 2498.18, "end": 2527.9, "text": "To the end of the game, maybe a few thousand of them, to collect statistics, also true statistics, about who ends up winning the game from that position. And then we combine both these two estimates, the estimate from the value network and the estimate from the rollout policy, and we combine them together to give a final evaluation of the promisingness of that branch of the tree. And once we get this new Q value, we then back that up the tree."}, {"start": 2527.9, "end": 2556.5, "text": "And update the connections and the choice, the decision points. And then finally, once we run out of time for searching and thinking and we have to make a decision, we then, in essence, pick the action, pick the move that has the most promising Q value associated with it. So once we built AlphaGo, how did we evaluate it? Well, the first thing we did back in April last year was play against"}, {"start": 2556.5, "end": 2586.02, "text": "The other strongest Go programs available at that time. So we tried it against two, Crazy Stone and Zen, which are the strongest programs out there other than AlphaGo. Now, I'll just explain about the scale that we're going to show here on these bar charts. On the right-hand side are Dan and Q levels, which are the ratings that you get when you play Go. And they go, when you're a beginner in Q, from like 25Q down to 1Q."}, {"start": 2586.02, "end": 2615.33, "text": "And then as an amateur, you go from one DAN to about six or seven DAN. And then you can become a professional if you pass certain accreditions, and you start again from one DAN to nine DAN, but professional level. So that's what the three bandings are, the yellow, beginner, orange, amateur, red, professional. On the left-hand side are numerical equivalents of those DAN ratings. So we call them ELO ratings. And this is our ELO rating scale from zero to about 3,500."}, {"start": 2615.33, "end": 2642.38, "text": "And you can think of the way of thinking about this is that if you have an ELO rating difference between two players of 200 to 250 points, that translates to about an 80% win rate for the higher rated player. So it's a kind of Bayesian sort of comparison between the strengths of these different players. And what we found is AlphaGo, when we played it against these other programs, could beat them more than 99% of the time."}, {"start": 2642.38, "end": 2669.34, "text": "In fact, nearly 100% of the time. And there was a huge margin between AlphaGo and the next best program, Crazy Stone, of around 1,200 ELO points. And some of you who've been following this may know that Facebook also have their own program that they're working on called Dark Forest, but that's not even as strong as Zen or Crazy Stone. In fact, it lost to Zen on an online tournament last month. So it's estimated to be around the same level as them."}, {"start": 2669.34, "end": 2693.5, "text": "So there's around 1200 ELO point difference between AlphaGo and these other programs. So we needed a greater challenge. So we thought, well, we're ready to play a top human professional. And so we contacted Fan Hui, who is the reigning three times European champion. He's a two-down professional. And he started playing Go at the age of seven back in China where he grew up."}, {"start": 2693.5, "end": 2723.78, "text": "And he turned professional in China at the age of 16. And in China, it's one of the most competitive places to try and become professional. So this was very exciting for us. This was back in October. And we didn't really know how well we were going to do. We knew that we were much stronger than other commercially available programs. But we didn't know. Obviously, it was a lot better than any of us on the team. So we didn't know how strong it would be against a human opponent. So this is what happened. I think after first, again, maybe he don't like fight."}, {"start": 2723.78, "end": 2757.04, "text": "He's a really great guy actually. He's a really good sport. So AlphaGo won 5-0."}, {"start": 2757.04, "end": 2777.07, "text": "Which was very surprising to us. We were hoping to win at least one game. But 5-0 was pretty amazing. And this story ends well, though. Don't worry. He looks anguished here. But we actually then hired him as a consultant for our team, ready for the next world match. So in the end, he's on the side of the computers now."}, {"start": 2777.07, "end": 2801.57, "text": "But one interesting thing actually is that he's since played a few more games against AlphaGo informally and he feels that it's actually improved his own play. And very recently he won the European Professional Championship again with a full score. He beat every single other professional in Europe. So he feels he's got stronger by training against AlphaGo in fact, which is quite interesting."}, {"start": 2801.57, "end": 2827.95, "text": "So Fan Hui, he's around here on this measure. He's around 2,900 ELO, and AlphaGo at that time was around 3,100. Again, this was covered in a Nature paper that came out a couple of weeks ago, front cover, and it's caused a huge stir in the AI community. And I encourage you to read that if you want to hear much more of the technical details which are outlined in that paper."}, {"start": 2829.07, "end": 2857.42, "text": "So I just want to just take a minute to explain the critical difference here between AlphaGo and Deep Blue. So although this is a big achievement, beating a professional player at Go is a longstanding grand challenge of AI research. And many people have been working, many smart people have been working on this for over a decade. And in fact, this happened about a decade earlier than many experts in the field."}, {"start": 2857.42, "end": 2886.94, "text": "The top programmers of the other Go programs, for example, thought it was going to happen, even from last year. But the key thing for us is not actually the achievement, but how we did it. So we've used general purpose algorithms, deep learning, reinforcement learning, tree search. These are general purpose algorithms, and we've put them together in a way that learns how to play Go. It's not a handcrafted set of rules and heuristics like Deep Blue or chess programs."}, {"start": 2886.94, "end": 2909.58, "text": "And it's also a modular system that combines pattern recognition with planning algorithms. So that's another thing is that, you know, deep learning is hugely popular right now, very fashionable. But we think and we think it's critical. And of course, we have a huge deep learning team, many amazing deep learners at DeepMind. But we don't think that's the whole story on its own."}, {"start": 2909.58, "end": 2937.22, "text": "We think that other things are going to be required, like reinforcement learning and memory and other advances combined with deep learning to reach full intelligence. And because of the way we trained AlphaGo, many people have commented, many professional players have commented about how human-like that it plays, its playing style is, and how it thinks. And if you think about it, AlphaGo has been trained in a way like a human expert player."}, {"start": 2937.22, "end": 2966.48, "text": "Starts off by studying professional games and learning from that. And then it improves through practice, by playing games of Go. So for us, what's the next step now? As Mike alluded to, actually we're only about a week and a half away from this, is the next step is to take on the world's best player, Lee Sedol. And he's from South Korea. He's a legend there. He's sort of like the David Beckham of South Korea."}, {"start": 2966.48, "end": 2992.22, "text": "Believe it or not. And I describe him as the Roger Federer of Go, because he's been at the top of the game for a decade, but he's still one of the top three players in the world. And he's won 18 international titles, kind of like Grand Slams over the last decade. And we're challenging him to a million dollar match, five game match in Seoul in March 8th to 15th. And you can follow that on YouTube live stream."}, {"start": 2992.22, "end": 3010.85, "text": "And he's taking this pretty seriously. Obviously, there's the money on the line, his reputation. But when he was asked by the South Korean press how he felt about the game, he said, I'm not sure if I represent the whole of humanity, but I think I am. So it's good that he's confident that he's going to win the match."}, {"start": 3010.85, "end": 3039.07, "text": "So he's actually a lovely guy as well. And I'm really looking forward to going out there. It's crazy out there. We did a press conference yesterday by video call. And there were over 300 journalists, including live TV cameras, for a video call. So it was pretty crazy. So we're very excited to see how it's going to be like when we go there. But Lisa Dole, on our ELO measures, he's significantly better than Fan Hui. He's a couple of notches better."}, {"start": 3039.07, "end": 3066.93, "text": "Fan Hui is kind of like a grandmaster level, but there's another level to reach sort of the world elite. So he's at least 600 ELO stronger. So we have to go some if we want to beat him from where AlphaGo was back in October. So my final slide and go is talking about how do we do this testing? Well, we have our own internal testing where we have running 24-7 different versions of our program playing against itself."}, {"start": 3066.93, "end": 3093.25, "text": "And we can make accurate estimates of how strong we think our program is from this continual live tournament that's going on in the cloud. But every now and again, we have to calibrate those internal tests with external testing. So we need to test against these external benchmarks. So in April, we tested against Zen and Crazy Stone. We won over 99%. Then in October,"}, {"start": 3093.25, "end": 3115.06, "text": "Our new version, our October version, could beat our April version 100% of the time. And obviously we were playing Fan Hui, who we also knew could beat these other top commercial programs 100% of the time if he was to play it. And so we knew we were at least roughly matched, but in the end, as you saw, we won 5-0. So now we're at March."}, {"start": 3115.06, "end": 3138.74, "text": "or coming up to March, and we're playing Lee Sedol. And Lee Sedol, on the ELO ratings, you would expect him to win around 97% of the time against Fan Hui. So it's a huge step up. So it's obviously confidential, our number, till the match that we've got on the left-hand side. And obviously, the million-dollar question is, what's going to happen when we play him? So it's going to be very exciting to see."}, {"start": 3139.09, "end": 3150.9, "text": "So I just want to give a big shout out to the amazing team that's worked on AlphaGo, led by David Silver and Aja Huang as the team leads. But some incredible work has gone on on this."}, {"start": 3152.05, "end": 3181.94, "text": "Now, of course, playing games is great fun, and it's very efficient for advancing our AI research, but we also want to apply these technologies to the real world. And we plan to make some announcements about this over the next year in healthcare, in robotics, and in smart assistance. All these different areas we feel that extensions of and components of what we're building for things like AlphaGo can be used very powerfully in these areas. So I just want to end the talk with a couple of"}, {"start": 3181.94, "end": 3204.9, "text": "High level thoughts and why I've been so obsessed with AI for my entire career and why I think it's so important. I see two big challenges facing society today. Information overload, we're just deluged as users and scientists with data everywhere. Big data from genomics, entertainment, every field, sphere of human life."}, {"start": 3204.9, "end": 3229.44, "text": "Now, personalization might be one technology to try and combat that, but unfortunately it doesn't work very well because it mostly is based today on the averaging of crowds rather than actually adapting to you as a personal, as an individual. Then, secondly, the systems that we would like to master are so complex today, from climate to disease to energy, macroeconomics, high energy physics. So, you know, you have to think..."}, {"start": 3229.44, "end": 3249.98, "text": "That maybe the complexity of these systems is so great, it's difficult to imagine how even an Einstein, someone at that level, can master these systems in their own lifetime and still leave enough time for innovation. So we think at DeepMind that solving AI in a fundamental way like we're trying to do is potentially a kind of meta solution to all these other problems."}, {"start": 3249.98, "end": 3279.17, "text": "If we can solve AI in this way, we can bring it to bear on all the other issues that we would like to solve. So the dream is really to make, for me anyway, to use this kind of AI to create AI scientists or AI-assisted science. And finally, I should mention a word about ethics. As with all powerful new technologies, they have to be used ethically and responsibly, and AI is no different. And even though human-level general AI is decades away, we should start the debate now. And as a neuroscientist,"}, {"start": 3279.17, "end": 3300.5, "text": "I think trying to distill intelligence into an algorithmic construct and then comparing it to the human mind, well, actually, this journey we're on will be one of the best ways to better understand the mysteries of our own minds and shed light on things like dreaming, creativity, and perhaps even the ultimate question of consciousness. Thanks for listening."}]}