[0:00:00] Great. Thank you very much. That was a very generous introduction. So, yeah, my name is Mustafa. I started a company called DeepMind Technologies six years ago, and our mission was to build general-purpose learning machines. It sounded...
[0:00:16] Pretty crazy at the time. It still sounds a bit crazy to most people. But I'm going to talk you through a practical example of how we've taken deep science, deep technology, right the way through from theoretical models, publishing papers, to a practical implementation at scale in a large industrial system. So our mission is to solve intelligence and use it to make the world a better place. When we approached
[0:00:41] Building general purpose learning machines six or seven years ago now, most people thought that we were...
[0:00:48] Pretty crazy, pretty ungrounded. Why would you want to start all the way from scratch and try to train machine learning algorithms to learn their own representation of what is effective and rewarding in a space when in fact we already have machine learning systems that do a pretty good job in very narrow environments? That was the typical feedback that we got from many other people in academia and in industry.
[0:01:12] Artificial intelligence was taking a very different trajectory to the one that we had decided to embark upon, and it seemed very odd at the time. But to us, it made complete sense. If somehow we could distill the essence of this incredible intelligence that we have evolved as humans and that we have used to...
[0:01:32] Create all of the civilization and the complexity and the culture and the progress that you see around us. If we could just distill some of that intelligence into an algorithmic construct, then potentially that gives us the opportunity to parallelize and scale up and work with perfect memory and take advantage of all of the things that we get from silicon and that we lack in our basic human physiology. And I think for us, it was always about
[0:02:02] Using these kinds of tools and methods to have a really beneficial impact in the world today. That was important to us personally and is a central part of the culture of our company. And I think it's important to us as business people, people focused on making money and building great companies as we are, to take a moment periodically to think about what else is at stake in the world today. What is the state of our global environment?
[0:02:30] So for a moment, let me take you to 40,000 feet and look at a couple of our most intractable social problems. 800 million people today don't have access to clean water, and this is actually predicted to more than double over the next decade. So despite the incredible progress that we're making with smartphones and transportation systems, some of what appear to be the most basic human rights seem to have eluded our complex civilization.
[0:02:55] 800 million people are malnourished every day and yet a third of the food that we produce every year goes wasted. This is kind of a remarkable statistic that says something about the incentive structure and the means of production and distribution that we have evolved as a species. How might that change in the future? How could we imagine a more equitable, more rational distribution of the resources that we have available on the planet? And it's been estimated that something like 3.1 planets
[0:03:23] of resources would be required if everybody were to live across the world at the level of consumption that we do per individual in the UK. This is kind of an incredible statistic, especially when you take into account that China has a rising middle class and Brazil and India and Russia and many, many other developing nations around the world are looking to consume just as we do, eating meat that we eat and traveling the world for our holidays as we do.
[0:03:53] The incomes of the top 1%, though, over the last three decades have actually increased by 300%, whilst the bottom two-thirds of people on our planet have seen their real inflation-adjusted incomes either stagnate or actually fall over the last three decades, which is pretty remarkable. Amidst all the progress, it's worth thinking about the question of inequality and redistribution.
[0:04:19] And so there's a lot at stake. There's an enormous opportunity for us, for companies like us and for other organizations to try and tackle some of the world's most challenging problems. And in many ways, I believe this is our kind of calling card. These are the things that really matter in the world.
[0:04:36] As a species, as a global community, what we really need to do is figure out new ways to increase our problem-solving capacity. Groups of the smartest experts still seem to be struggling to predict the trajectory of the stock market or the long-term impact of climate change. How can we enhance this problem-solving capacity? That's what we're motivated by at DeepMind.
[0:04:58] At our heart, we're fundamentally a research organization, but we've really sought to build a hybrid type of organization that takes the very best from academia, where we focus on really hard and ambitious and complex long-term problems, and ground that in the kind of pace and scale and agility and commercial motivation that you get in a company.
[0:05:19] All the while underpinning it with the kind of values that you often find in the public sector or in the charity sector and making ourselves fundamentally committed to a social mission as we are. So three or four years ago, before we joined forces with Google,
[0:05:35] Getting access to data was really, really difficult. And so we figured out that we could actually train our algorithms in the old school Atari testbed. And this is about 100 or so Atari games from the 70s and 80s. Many of you will be familiar with playing Breakout or Pong at the arcade. And basically what we have created is a kind of small, controlled, but relatively complex world where we can adjust...
[0:06:00] The complexity of the environment in accordance with how increasingly capable our algorithm is becoming. And all we pass to the algorithm is essentially the raw pixels that describe what is happening at each frame at any given moment. So the algorithm simply sees a red, green, blue pixel at position XY at any moment 24 frames per second for hundreds and hundreds of thousands of games. And so the key intuition for...
[0:06:28] Why we're training in these environments and what we're trying to do is that everything is learned completely from scratch. We haven't handcrafted a bunch of clever rules into the system that say things like, you know, if you see a bat and a ball, then move to the right because you're going to get a reward and, you know, so on and so forth. These rules are incredibly brittle and they don't scale to novel environments, which is exactly the opposite of what we're trying to do when we're solving many of our toughest social problems. We want to train algorithms that can learn their own representation of...
[0:06:57] What is rewarding in some environment? A representation that is often much more complex and much more non-linear and much more distributed and sparse over time than we can if we hand program a set of heuristics using human knowledge. So everything is learned from scratch. There's no pre-programming. And the goal is simply to maximize score.
[0:07:15] So get as much score and as minimum number of time steps as possible. And so really just imagine a robot standing at an arcade being fed the pixel images and being wired up to the control button so it can fire and move left and right and so on. And so we were very successful in training our algorithm to play all of the games. This is a medley of all the different types of games that we trained it to play. So this one's Enduro. This one's River Raid, which is a very different 2D god view to the previous version.
[0:07:45] This is another view. This is a three-dimensional game where you have to wander around shooting other tanks. This is more of an adversarial game where you have to play the inbuilt AI and beat them at Pong. This is a pretty complex game called SeaQuest where you have to collect divers and shoot the sharks. And this is another game, adversarial, where you're basically playing against the other opponent and it finds a pretty clever trick where it can get it in the corner and basically there's no way out for his opponent.
[0:08:14] So it's really interesting here that we've actually deliberately tried to train the algorithm to learn new knowledge. We don't want to take all of our knowledge, limited as it is, and program it into the system. We want the agent to learn things like this. So we were very excited when we saw this. And we were lucky enough to be able to get a Nature paper for this. And as any of you know from an academic environment, getting a Nature paper is the highest possible accolade. So we were very, very happy and we were lucky enough to be awarded with a front page too.
[0:08:42] So we extended this work earlier this year to play the ancient Chinese game of Go. This is by far the most complex and the most difficult board game man has ever invented, and essentially it has some of the most simple rules possible. There are black and white stones, and you and your opponent take turns of placing the stones on the board.
[0:09:03] And the objective is to surround territory, surround your opponent's territory. And at the end of the game, whoever has the most territory, simply by placing stones and surrounding your opponents, which means that you take them off the board, wins the game.
[0:09:19] The really interesting thing about the complexity of this game is that there's been something, it's been estimated there's something like 10 to the power of 170 possible board configurations or state spaces. That's a 10 with 170 zeros on the end of it. It's an enormously large, impossible to understand number. That's how many different board configurations there are on a 19 by 19 board, which is what we and the professionals play on. And just to kind of put that into context,
[0:09:45] In the game of chess, this is roughly what the branching factor might look like. You know, relatively easy to intuit. This rolls out for about 100 or so. And this is about 10 to the power of 45. So 10 with 45 zeros on is the number of possible configurations on an 8x8 chess board. But if you try to intuit the number of possible board configurations in Go...
[0:10:08] It's massively, massively, exponentially, double exponentially larger. And so this rolls out for about 200 or so branches forward, and then it has to go right back to the beginning and roll out another 200 or so. And so just for comparison, it's been estimated that there's something like 10 to the power of 65 atoms in the known universe. So in every liquid, solid, and gas, in me and in the air and in the floor and everywhere around us, there are...
[0:10:37] More potential state spaces in the game of Go than atoms in the known universe around us. So this is the size and complexity of the state space that we're dealing with. And so traditional methods where we write a rule using the best human players in the world that says when you're in this kind of position, you should probably behave like this. They don't really work. They don't scale. And so what we actually did was we trained two neural networks.
[0:11:02] was a neural network that learned a policy. So this was estimating the likelihood of all possible moves that were valid in a particular position and trying to pick out the ones that are most likely to be rewarding and then stack ranking those. And so all the little green ones here are potential moves that can be made that will be rewarding.
[0:11:20] And then once the algorithm has identified that one of those moves is really interesting, what it does is it rolls forward a value network that tries to calculate how useful this would be. And over time doing this hundreds of millions of times, it builds a representation of all the paths through the complex space that happen to be rewarding.
[0:11:39] Now, we were lucky enough to play against the human world champion, who's a legend of the game. He's won 18 world titles over the last 25 years, by far the greatest player of the last decade. And we played Lee Sedol in a live match in Korea earlier this year, and we were lucky enough to win four games to one. It was one of the most watched TV events of all time. Many of you probably didn't see it because it was over in Korea.
[0:12:07] But it actually had 250 million live viewers, which is way more than the Super Bowl. So it's like a national sport out there in the Far East. So kind of interesting what sort of filter bubble we live in sometimes that we didn't get to see it as much over here. So we were lucky enough to get a nature paper for this again. And again, the front page, which was very exciting to us. We've actually just had another nature paper. So the first lab ever to get three nature papers in 18 months. So we're very excited about that.
[0:12:37] What we're really interested in doing is figuring out how we can apply these sorts of methods in the real world to make progress with some of our...
[0:12:44] Most intractable problems. And none, I think you'll all agree, is more intractable than the challenge of managing our impending global climate change, racing towards two degrees as we are. And energy, of course, energy consumption is the largest contributor to carbon emissions. And most interestingly, I think it's the case that data centers actually account for about 3% of global energy consumption. So within the Google data center fleet, it was a non-trivial amount of energy consumption.
[0:13:13] And so it was very interesting for us to try and see what we could do here. And so as Jim mentioned, we were lucky enough to be able to succeed in training a model that reduced the overall cost of cooling Google data centers by around 40%. And so I'm going to explain to you a little bit about how this worked in practice. I mean, just to say at the outset, this was incredibly surprising to all of the engineers. Any of you know mechanical engineers who run data centers?
[0:13:38] They do an outstanding job. They're extremely capable. They've been doing it for 20 years. Incredible amount of human expertise. And, of course, an incredible amount of cynicism that we could come along and find such low-hanging fruit, especially at this kind of scale. So it's important to kind of keep that in mind as I talk through some of the intuitions that our algorithm was able to derive from looking through these large amounts of data.
[0:14:02] So a little bit on data center cooling. Basically, how does this work? Well, roughly speaking, it's sort of divided into two chunks. There's water that is passed around a bunch of pipes on the right-hand side. There's the server floor in the top right-hand corner of the diagram, of the cartoon diagram. It's passed through a heat exchange, which attempts to extract that through the cooling towers and periodically use the internal chillers, which are sort of much more expensive, of course, when the cooling towers aren't as effective as we would like them to be.
[0:14:32] And the overall objective is to maximize power usage efficiency. So essentially, a data center is a large-scale input-output system where you're passing a bunch of incoming compute load into the system that generates a bunch of heat, and you want to efficiently remove that heat without disrupting the experience that you get when you watch a video on YouTube.
[0:14:53] But all the while, you want to make sure that you're maximizing efficiency whilst always respecting known temperature and pressure safety constraints. And that's a big part of the breakthrough that we were able to deliver is to incorporate the best of human intuition and knowledge about what constitutes a set of safe parameters to deploy our system while simultaneously learning new optimizations in the data center. And of course...
[0:15:18] The key thing here is access to retrospective information. Data, data, data is the heart of us being able to extract new insight in how these systems operate. And there's two types of data to try and give you a better intuition for the kind of information that we trained on. The first is the state. So all of the sensor data that is collected every minute, every second, that describes the overall physical behavior of the utility. So of course there's incoming IT load.
[0:15:47] There's power meters, there's pressure sensors, temperature, water flow meters, pump and fan speeds, clock speeds, and so on and so forth. So all of this information describes the current state of the data center at any time t. The second chunk of data that we look at...
[0:16:06] is the action data. So how many cooling towers have been turned on? How many chillers are active at any given moment? How many pumps are there? And what are the set points of the various pressure valves, temperature valves, flow rates, and so on and so forth?
[0:16:20] And so all in all, there are about 1,200 or so different state variables, and for each one of those, there's 20 or so actions. And what we did is we aggregated those into about 120 state representations, combined them with a whole series of actions, which are both continuous and discrete, so where you can select from one of 10, for example, or where they're continuous values that you have much more fine-grained control over. We had a time step of about an hour, a target, obviously, of power usage efficiency.
[0:16:49] And essentially what we're trying to do is given some state, what are the suggested actions in order to optimize power usage efficiency at time state T plus one? Obviously always operating within safe constraints. So essentially this is a very general framework that we use to solve data center prediction. There's a bunch of state inputs, a bunch of actions, and just like we did with the Atari or just like we did with AlphaGo, we're learning to correlate state with rewarding behavior.
[0:17:19] And it turns out, if you look at how data centers have performed over many, many years across many, many different locations with all sorts of different hardware, there's loads of variation and fluctuation, both in the way that human intuition has developed over time, but also in the local temperature and the local hardware that has been used, the configuration and the setup. And that variation is kind of like an exploration of the state space of all possible configurations. It's like having a really big...
[0:17:47] that can see over all the different possible configurations and pick interesting insights from the various different setups and aggregate them as the temperature in that local data center and the incoming IT load is varying over time. And so we pass state and action representations into our neural networks and we essentially predict three outputs. What is the pressure likely to be at time T plus one? What is the temperature? And of course, ultimately, what is the power usage efficiency?
[0:18:17] And all of this is whilst operating within significant operational constraints. So of course there are defined heuristic driven boundary constraints. We obviously want to maximize future reward rather than achieve the best performance for the next 10 minutes or two hours. We want to estimate the confidence. So we want the model to tell us how confident are you in this prediction that you're giving us. And it turns out that we can fine tune the model in order to bias it towards more or less
[0:18:45] We also want to put in place automatic failover mechanisms. So if the algorithm does make a recommendation that is beyond the safe boundary constraints, then there are a bunch of backup systems in place. And of course, all of this is subject to human oversight, and that remains a central part of what we're able to do. Ultimately, we were able to, as I say, reduce...
[0:19:09] By 40%, the level of energy required for the cooling system, which overall resulted in a power usage effectiveness improvement of about 15%. Now, ultimately, what we're really interested in at DeepMind is discovering new knowledge.
[0:19:24] So reprogramming systems with human knowledge that already exists is useful and valuable and will continue, but isn't really going to help us to make the 10x contribution to our most complex social problems. And so just to give you a hint at some of the insights that we were able to discover, the first is that more cooling equipment, not less.
[0:19:43] That was brought to bear in order to run the system turns out to be much more valuable. So if you spread out load really thinly over lots of bits of kit and run them all at a lower level of capacity, it turns out that overall we can learn really interesting linear versus exponential power efficiency curves that were very surprising and unintuitive to the human operators that had been running the systems for a long time.
[0:20:04] The second thing is that higher flow isn't always better. So a lot of the engineers often believe that they should be concentrating flow through the chiller a great deal. But if we put less flow through the chiller, it turns out that they can actually run much more efficiency. And that overall increased the local energy consumed by that chiller, but the global energy consumed across the system was actually much, much more efficient.
[0:20:28] And then finally, the ability to shift loads to different parts of the system given the type of incoming compute demand that we saw and the temperature actually allowed a much more flexible and sort of fast adaptive response to the kind of conditions that the data center team were seeing at any given moment. And that turns out to be very, very valuable. So finally, just to say, you know, obviously these methods are inherently general. These are large-scale optimization systems that we believe will work.
[0:20:57] Reasonably, effectively, in a wide range of environments, subject to the data being particularly appropriate and having basically enough of it. And so we're really starting to think about what this might actually look like as a service that we bring to the market outside of Google. And I should say that we run at arm's length of Google. We're sort of an independent company, and we treat Google as a customer, and so we sort of help them with a bunch of different things, and we're starting to think, like, how might we be able to do that in the wider market?
[0:21:26] So the final thing I just want to leave you with is that we're really trying to build a new kind of organization that is motivated primarily and singularly to use our methods to make the world a better place. We've been making incredible progress with our AI and I think pretty remarkable and pretty exciting. But to us, partnerships are the key to delivering and scaling up these kinds of benefits. And so we're very excited to be invited here and to hear from as many of you as would like to get in touch.
[0:21:53] We're also hiring. My email address is Mustafa at DeepMind. Please email me. Thank you very much.