{"url": "https://www.youtube.com/watch?v=B_2ovkDycuk", "page_url": "https://www.youtube.com/watch?v=B_2ovkDycuk", "title": "Demis Hassabis, DeepMind CEO (UC Berkeley)", "show_or_channel": "AttentionX (UC Berkeley)", "date": null, "duration_s": 1424, "type": "talk", "speakers": ["Demis Hassabis"], "confidence": "medium", "notes": "UC Berkeley talk; uploaded 2023-10-20, event date unknown.", "people": ["demis-hassabis"], "id": "0000_attentionx-uc-berkeley-demis-hassabis-deepmind-ceo-uc-berkele", "audio_duration_s": 1423.1394375, "worker": "jc-w19", "transcribe_s": 68.4, "model": "large-v3", "segments": [{"start": 0.0, "end": 25.41, "text": "Over the last decade plus, we went from 2010 where we had to, myself and my co-founder Shane Legg, we were both postdocs at UCL, and we had to kind of whisper in the corridors what we were planning to do because people thought that working on AI, certainly AGI, artificial general intelligence, sort of human level artificial intelligence, the original goals of the AI field, was not even, almost not scientific."}, {"start": 25.41, "end": 49.09, "text": "The attitude from a lot of the professors was that we already know it doesn't work, we can't get there. But we felt that with the advent of some new technologies, things like deep learning, Jeff Hinton and colleagues have just formulated the first stages of that, and reinforcement learning specifically, which we're very expert in at DeepMind."}, {"start": 49.09, "end": 68.62, "text": "We felt that those things could be combined together along with the way compute was going. Things like GPUs, which I had experience on from my games career. So of course, that's what GPUs were originally invented for. So I used to joke to, I've joked to sort of Jensen, who is the CEO, amazing CEO of NVIDIA, you know, I've used..."}, {"start": 68.62, "end": 95.84, "text": "His hardware in many different ways, right? And it turns out everything in the universe is a matrix multiplication. So whether it's graphics, AI, or indeed crypto. So it's kind of amazing how useful these accelerators have been. So we sort of saw all of those different trends and then also some ideas from how the brain works and the neuroscience that I did. I focused on hippocampus and memory and imagination and planning."}, {"start": 95.84, "end": 121.33, "text": "And all these things, those terms, we haven't imbued our AI systems with those capabilities. So we were trying to sort of bring together from the beginning this multidisciplinary approach to try and make progress with AI. And then what we would imagine is we would create this kind of Apollo program effort, really ambitious, high-intensity effort to try and make as quick progress as we could in the field."}, {"start": 121.33, "end": 148.78, "text": "And we decided to do it as a company because I thought we could get more resources together, things like compute resources, much more quickly than we would be able to necessarily do as postdocs. So we kicked that off, and we quickly made quite a lot of progress. And now, as a bigger part of Google DeepMind, which is our new group, we have a new mission."}, {"start": 148.78, "end": 174.34, "text": "Where we articulate the mission which is to build AI responsibly to benefit humanity. So that's the purpose of and has always been the purpose of why we're trying to build AI. And for me personally my passion has always been to apply AI to science to help us understand the world around us and the universe around us much better. So as Jennifer mentioned our first big success was sort of back in"}, {"start": 174.34, "end": 204.22, "text": "2013, 2012, 2013. We started, we worked on games and I sort of tried to utilize all of my experiences and games has always been a great testing ground for AI. Right from the beginning of the field, you know, back in the 1950s, people like Turing and Claude Shannon and sort of the founding people of the field, they all kind of proposed or wrote chess programs and used them to propose gaming as a good way to have a hill climb towards AI."}, {"start": 204.22, "end": 233.01, "text": "So we used a lot of different games and we started with relatively simple ones, of course, like Atari games here from the 70s, the kind of most simplest game system that got popular. And the key thing about this system, VQM, was that it learned directly from the pixels, the raw pixels on the screen. We didn't give it any of the rules or anything about the controls. It had to learn everything from first principles and it was just told to maximize the score."}, {"start": 233.01, "end": 260.59, "text": "And it was the first time that any kind of reinforcement learning system was able to scale to something real world, I would say, beyond a toy. And it sort of was a proof point for us. And I remember when, for about a year, we were struggling to kind of get anything working at all, like even AV systems to even get a single point in a game like Pong. And we were wondering, well, maybe we're just like 10 years or 20 years."}, {"start": 260.59, "end": 288.88, "text": "To the head of our time, you know, perhaps this is just too difficult. And then I remember that we should have recorded it. But I remember the first day that our system actually won a point. And then we were wondering, is this random? You know, who most randomly won a point? And then after another month or so, then it finally won again. And then another couple of months after that, it was winning every time. And then we knew we were onto something. We generalized it to all the games. So that was a pretty amazing moment."}, {"start": 288.88, "end": 311.34, "text": "And then we pushed this forward probably to the pinnacle of the games work we were doing by applying it to Go. And for those of you who don't know, Go is an amazing game. I really recommend it to everyone if you haven't played it. It's more complex than chess and of course in Asia this is what they play. It occupies that intellectual echelon that chess does in the West."}, {"start": 311.34, "end": 334.24, "text": "But the complexity of Go is much larger. There's roughly 10 to the 170 possible positions in Go, which is far more than the atoms in the universe. So there's no way one could exhaust the research or the possibilities in order to find the right moves. And we basically ended up building AlphaGo system and then challenged..."}, {"start": 334.24, "end": 357.79, "text": "One of the legends of the game, Lisa Dole, to a million-dollar challenge match in Seoul back in 2016. And it was a really huge and kind of life-changing moment in a way, and sort of once-in-a-lifetime event for us. The whole of the country, it felt like, sort of came to a standstill to watch the match because Lisa Dole is obviously a national hero. And AlphaGo won the match 4-1."}, {"start": 357.79, "end": 386.91, "text": "But the most important thing wasn't that it just won. It actually came up with some creative strategies. Most famously, what's called Move 37, the 37th move in Game 2, which has now kind of gone down in Go legend, in a way, and has been massively studied. And it's this move here on the right-hand side, the black stone circled in red. And basically, this is played too near the center as far as experts are concerned."}, {"start": 386.91, "end": 414.9, "text": "The center of the board for this early on in the game. So if you were an expert in Go, this would be a totally crazy move to do. And you'd be told by your Go master teacher that this is a terrible move. And initially, that's what everyone thought as well. The commentators, if you watch the documentary, they literally fall off their chair. And they think the mistake's been made. They actually think that there's a misclick by the operator."}, {"start": 414.9, "end": 443.47, "text": "And if they know Adjo, there's no way he could have made a misclick. So we knew that was not possible. And then later on, obviously, a hundred moves later, this stone, this Myth 37 stone, turned out to be in the perfect place to decide the whole match. So the fighting for the bottom left, those groups of stones in the bottom left, spilled out into the middle of the board. And then that stone, almost presciently, was in the right place to decide the critical battle for the whole game."}, {"start": 443.47, "end": 470.4, "text": "So it's kind of amazing. And what this made me think in subsequent years is about creativity itself. So what does it mean that AlphaGo has come up with this creative move? And I think there are sort of three levels of creativity, one could say. And I think AI is currently at level two of this. So if you think about interpolation being the most basic level of creativity, which is coming up with something original."}, {"start": 470.4, "end": 489.14, "text": "Let's sort of call that the operational definition. And you can think of that as averaging all the examples it's seen. So maybe you say, here's a million pictures of a cat. Now draw me a picture or come up with a picture, a photo of a new cat. And it might be some kind of prototypical average of all the things it's seen."}, {"start": 489.14, "end": 513.52, "text": "So that's the kind of lowest level of creativity. Then there's extrapolation, which I think is what AlphaGo has done here, which is, you know, experience some of the data, some of the environment, some of the structure, and then extrapolate from that something new that's outside of the distribution it's seen, but extrapolate it for what it's seen. And I think something like Move37 in Game 2 is an example of that."}, {"start": 513.52, "end": 537.34, "text": "Obviously, it's never seen any human play that kind of move. It was unthinkable. But then this last level, and it's an interesting question, is are we going to get to, can we get to true invention or out-of-the-box thinking? And my example for that would be not just come up with move 37 in game two, but could you invent Go or invent chess from a kind of very verbal description of..."}, {"start": 537.34, "end": 567.76, "text": "You know, you want an elegant game that lasts X amount of time and so on, and it's beautiful aesthetically. You could give it some high-level abstract, say, description, and then it would come up with something that was as incredible and as deep as goes. Now, obviously, we can't do that today, but it's interesting for us to be able to think of what's missing in order to be able to do something like that. So how does AlphaGo work? And I think this is instructive because later on I'll explain how we're using this in science, these same methods."}, {"start": 567.76, "end": 591.63, "text": "So AlphaGo and the subsequent systems called AlphaZero, which ended up... AlphaGo needed human games first to learn from, but then we generalised that to AlphaZero, which could play any two-player game. So it can play chess, Go, Shodij, you know, anything that you want, two-player, perfect formation game, better than all the specialist systems. So AlphaZero could beat AlphaGo."}, {"start": 591.63, "end": 615.92, "text": "And AlphaZero, hence the zero in the name, did not need any human games to bootstrap it. It would start playing randomly and then learn for itself purely from scratch. So it's kind of the most pure essence of a general games playing system. So the way that worked is, if you imagine version one, it would play 100,000 games against itself."}, {"start": 615.92, "end": 642.48, "text": "So version 1 doesn't know anything about Go or just the rules of the game. And so it starts off pretty much playing randomly. And then what you do is you generate a pile of data from the positions from this 100,000 games. And then you try and train a version 2 of the system that can predict what the right move is and who's going to win from certain positions in the middle of those games."}, {"start": 642.48, "end": 665.68, "text": "And so version 2, in theory, should be slightly better than version 1. And in order to check that, you have a 100-game sort of tournament at the end of training, and then you have a statistical threshold. In our case, we picked 55% win rate. And if it reaches a 55% win rate, then you replace the version 1 network with version 2, and you generate a new part of data, which is incrementally slightly better."}, {"start": 665.68, "end": 694.78, "text": "And then you train at version 3, which now tries to predict the version 2 data and who's going to win and so on. And obviously if you're below the 55% win rate, then you continue on with the current version for another 100,000 steps. And now you've got 200,000 games to learn from. And eventually you will be able to train a better system. And it turns out after 17 versions of this, at least for Go, it goes from random play to superhuman play."}, {"start": 694.78, "end": 720.06, "text": "And so it's kind of an incredible process to watch because you can actually watch it through the course of a day and play it at different times. And I've tried that with chess, and it's kind of amazing. In the morning, it starts off random. You're still able to beat it by lunchtime, and then by afternoon tea, it's better than world champion. It's kind of amazing to sort of see that evolution in real time. So that's the essence of these systems."}, {"start": 720.06, "end": 749.31, "text": "So then what can you, you know, what's the sort of, if we step back, what is the system doing in terms of this massive search space, say, in Go? How is this helping kind of navigate that search space? So what happens is what we're building there with those neural networks are good models of Go, right? What are likely moves to be played in certain positions and also who's likely to win, present a chance of winning from the current position."}, {"start": 749.31, "end": 769.92, "text": "And those are two key things that you need to know in order to narrow down your search base. So if you imagine this is the tree of possible go positions, and it's 10 to the 170 down at the bottom, how do we pick something that Africa has to move within a minute? How does it do that? So it uses the model to guide the search."}, {"start": 769.92, "end": 796.91, "text": "So that we're only searching useful parts of the tree and obviously a tiny, tiny fraction of the tree. And then when it's time to execute an output move, it picks the best line in fact. And it turns out these systems explore a lot less, orders of magnitude less than brute force systems that you get in chess like Stockfish, which search millions of moves per decision."}, {"start": 796.91, "end": 815.33, "text": "These AlphaZero systems only search tens of thousands of moves for each decision made. Still not as efficient as humans, so human grandmasters only search a few hundred moves in order to make a decision. So clearly our models are still not as good as the human expert models are."}, {"start": 815.33, "end": 838.75, "text": "But we can compensate with a little bit more search, which obviously we can't do Monte Carlo tree search in our brains. So it has an advantage there. So it's a kind of combining a pretty decent model with a good amount of search is the best possible combination. So then we went on and took this a little bit further with things like AlphaStar, which"}, {"start": 838.75, "end": 863.89, "text": "Beatground Masters at the game Starcraft, which is probably the most complex real-time strategy game of the computer games, and it has the additional challenge of having partial information. So you have Fog of War, you can't see the whole game board at once, so you have to deal with that kind of uncertainty too. So that pushed us further with dealing with uncertain environments, which of course is more like the real world."}, {"start": 864.14, "end": 890.11, "text": "So then we got to an exciting point around, well actually pretty much the day I got back from Seoul in 2016, I had in mind that we wanted, at some point we'd be ready to move out from games, obviously games are super fun and I could stay working on games my whole life, but it was just a means to an end, which is to test these systems, and then obviously we were trying to build general purpose algorithms that would be useful for real world problems."}, {"start": 890.11, "end": 915.14, "text": "And specifically for me, for applying to these scientific challenges, grand challenges. So once we won the AlphaGo match, I felt we were ready. We had enough ingredients, enough know-how to tackle something really difficult. And I've always had top of my list, and I don't actually have it, it's not actually an explicit black book list of problems, but over the years, I love all types of science."}, {"start": 915.14, "end": 943.25, "text": "I'm sort of very curious, and I've collected over the years many interesting problems that I think would one day be suitable for these types of learning algorithms. And protein folding was right near the top. And I'm sure this audience all knows about protein folding, so I won't talk much about the problem, but obviously we're trying to predict the 3D structure of the protein from the amino acid sequence one-dimensional string."}, {"start": 943.25, "end": 966.86, "text": "What gave me hope here is, of course, that nature does do this spontaneously. So in some sense, physics does solve this problem instantaneously or near instantaneously. So it can't be intractable in some ways as long as we were able to model enough of the system. Now, obviously, the real issue was, and always the issue with these systems, is how wide do you go with your modeling?"}, {"start": 966.86, "end": 995.44, "text": "How much information do you need to know? Do we need to know about the chaperones? Do we need to know about the dynamics the proteins are in before you can actually predict the structure? So these were all unknown. It's not clear that you could do this in isolation and you can break off the right chunk of the problem. But it was a fascinating problem and I knew downstream it should have a lot of impact and it's been great to see that impact if we could actually solve this problem."}, {"start": 995.5, "end": 1018.91, "text": "So again, it had all the traits that I look for in a problem, massive combinatorial space to search through. So, you know, depending on who you sort of read, like Leventhal estimated sort of famously, maybe there's 10 to 300 possible confirmations an average protein can take. As I mentioned, obviously in nature, though, somehow this is done in milliseconds."}, {"start": 1018.91, "end": 1037.25, "text": "And then the other thing we look for actually is rigorous benchmarks. It's really, really important to have those. And we were very lucky and grateful to John Moult at the CASP organization. We've been running CASP as a kind of gold standard for protein prediction, structure prediction for nearly 30 years."}, {"start": 1037.25, "end": 1065.18, "text": "Painstakingly every two years. And it's a fantastic competition that's sort of double-blind, right? So you get these amino acid sequences. The structures have just been determined by experimentalists, but they've not been published yet. So it's sort of a perfect blind test. So there's no way you can contaminate your training data with that. So it's actually a beautiful and really super well-run competition. And that made it really attractive to us."}, {"start": 1065.18, "end": 1091.94, "text": "To enter and to test ourselves against. So we were doing that, and this is the... So the cast runs every two years, and this is the measure they normally use called GDT, which is roughly percentage accuracy of the amino acids in position, and within a certain tolerance of the correct position, 3D position."}, {"start": 1091.94, "end": 1114.3, "text": "And for pretty much a decade prior to us entering in CAS 13, there had basically been no progress on the hardest category, this Fremontian category, where there are not a lot of templates. And then for the first time sort of in 2018, so we were always told by the organizers and people in the field that we needed to get to within one extra error."}, {"start": 1114.3, "end": 1142.82, "text": "So we entered in 2018 for the first time with AlphaFold1 actually, and AlphaFold1 kind of made a lot of progress."}, {"start": 1142.82, "end": 1172.82, "text": "For the first time, I think, it used cutting-edge machine learning as the core part of the system. People had tried to experiment with machine learning, but mostly as peripheral to the main hand-crafted system. It was mostly hand-crafted, a bit like computer vision used to be done, with trying to explain edges and edge detectors and sort of painstakingly sort of building these specialized components. And it turns out it's better to just learn a bit, like modern computer vision systems do today. And so we took that kind of approach with Alpha Fold 1."}, {"start": 1172.82, "end": 1199.2, "text": "And then that had this amazing, you know, we won the competition by quite a large margin and we improved the kind of winning score by nearly 50% over the historical average. So that was a great result, but we were still very far off getting to this one extreme error. And we had a big debate on the team of like, and actually for the next half year after the competition, we tried to push this system harder and it reached the ceiling."}, {"start": 1199.2, "end": 1220.08, "text": "you know of around 65 gdt so that so this was about the competition the score was about 58 and we got to about 65 and you need to get to 90 for it to be roughly equivalent to one angstrom error so we're still miles off with alpha 1 and a lot of people on the core team didn't think it was possible actually so we had this sort of big debate about"}, {"start": 1220.08, "end": 1249.25, "text": "You know, it's great. We won the competition. We should sort of declare victory and move on to something a bit easier. And I always sort of remind everyone that the idea wasn't to win CASP. It was to actually get to the one answer there and be useful and actually solve the problem. So we changed up the team a little bit and actually brought in some new people that were more expert, had expertise in structural biology, not just machine learning."}, {"start": 1249.25, "end": 1270.34, "text": "And I think that was the key. So with the AlphaFold 1 team, amazing team, but they were mostly general machine learning people that we sort of converted to be interested in this problem. But they hadn't really done, you know, structural biology themselves. And that's where, you know, people like John Jumper, who ended up leading the technical lead on AlphaFold 2, was really important."}, {"start": 1270.34, "end": 1297.44, "text": "And he has a long history in molecular dynamics simulations and protein structures. And he tried many other computational methods. And then he realized that AI was maybe the way forward. And we agreed that we would sort of take the know-how from AlphaFold1, but re-architect it from the ground up to try and get past this ceiling that we reached. And it's always very scary when you do that, because for the first, the next year, we got worse."}, {"start": 1297.44, "end": 1325.7, "text": "Then AlphaFold1, much worse. And we had to build back to AlphaFold1 level performance. And then the AlphaFold2 system had a much higher ceiling. And there were critical things, you know, happy to ask some questions about that. They were different in AlphaFold2. Maybe you're familiar with both architectures. But AlphaFold2 was much more end-to-end. AlphaFold1, basically the output of AlphaFold1 was a distogram of pairwise distances of the residues."}, {"start": 1325.7, "end": 1348.0, "text": "So it didn't produce the 3D structure directly. It produced this histogram and then you had to do a Newton method to actually create the 3D structure. So the problem with that is if the 3D structure is wrong, you can't flow back the learning signal all the way back to the neural network because there's this other method in between that effectively blocks the learning. So it's always better, if you can, to make the system more end-to-end."}, {"start": 1348.0, "end": 1375.82, "text": "So that the learning signal can actually propagate through the whole process. So with AlphaFold2, obviously, we skipped that interim step of creating a histogram, although it creates an implicit one, and directly produces the 3D structure. So then, of course, as you're all familiar with, AlphaFold2 then did reach this threshold of atomic accuracy, and then the organizers and John Bolt sort of declared that the problem was satisfied."}, {"start": 1375.82, "end": 1403.28, "text": "So it was obviously an incredible moment for us to get those results back in the autumn of 2020. So one of the reasons that AlphaFold 2 is better is that it has this iterative recycling process. So where it starts off, in order to predict the 3D structure, it starts off with everything bundled with the zero point. And then over time, it kind of expands it into the final structure that you see on the left."}, {"start": 1403.28, "end": 1412.34, "text": "After many cycles of iteration and that was very key to the system working. We obviously released this all at all."}]}