[0:00:00] So let me start. I have the pleasure of introducing very briefly Demis Assabis. And this is continuing a tradition that I started with Christoph. He's much better known than I am, and he was a postdoc with me. Demis. And of course, he's ideal for today because he's a mixture of a neuroscientist,
[0:00:31] And a technologist and an entrepreneur. So it's great to be here. Can everyone hear me? Yeah, cool. So I'm going to talk a little bit about, because of the symposiums about neuroscience and brains and minds as well as machines, I'm going to focus sort of most of my talk around systems neuroscience and how it can be maybe helpful towards in helping us progress in our quest for building AI.
[0:01:01] So I'm the CEO of DeepMind. DeepMind was founded in 2010 and then we joined forces with Google in 2014 to sort of accelerate progress towards our mission of solving intelligence. One way you can think about DeepMind is a kind of like an Apollo program effort for AI. So we have over 150 research scientists now, so I think it's the largest collection anywhere of sort of machine learning researchers on this single topic. But also
[0:01:30] One of the things we've tried to do with DeepMind is come up with a new approach to organizing and doing science. So we've tried to combine and take the best from the best academic institutes and combine that with what's best from the greatest sort of Silicon Valley style startups. And we've tried to kind of create a hybrid environment that maximizes and is sort of optimized for research progress to try and make it as efficient and productive and collaborative as possible.
[0:02:03] So what we do at DeepMind is we're sort of interested in algorithms. We call them general purpose learning algorithms. And we're only really interested in algorithms that can learn automatically from raw inputs or directly from raw experience, so are not pre-programmed or handcrafted in any way. And we're also interested in this notion of generality. So the idea that a single system or single set of algorithms can operate out of the box across a wide range of tasks. And in fact, this relates to
[0:02:32] Our operational definition, if you like, of intelligence that we use at DeepMind, which is we define it as the ability to perform well across a wide range of environments. So while I agree with Tommy and I think some of the other speakers today about that intelligence is quite an amorphous term, we found that operationalizing it like this and putting generality at the center of it works very well for us. And you could argue, well, what about all these different properties that that doesn't cover, you know, creativity, et cetera.
[0:03:01] But actually, then it just depends on the tasks that you include in the set of tasks that you're going to try and attempt. So the wider and the more diverse, the better. So we call this type of AI artificial general intelligence, or AGI. And the hallmark of these kinds of algorithms is that they're flexible and adaptive and perhaps even one day inventive. And they're built to deal with the unexpected from the beginning. Now, contrast that with...
[0:03:33] A lot of what's called AI out there, which we mostly term narrow AI, which is often handcrafted or special cased for a specific problem and therefore can be very brittle. If something unexpected happens, it will just catastrophically fail. So one of the other sort of philosophies, I guess, that we committed to at the start of DeepMind was the idea of grounded cognition. So this is the idea that, or the notion, that a true thinking machine has to be grounded in a sensory motor data stream.
[0:04:05] This kind of gives rise to notions of end-to-end learning agents that perhaps start with raw vision, so maybe pixels on a screen, and go all the way to making a decision about what action to take. And we're interested in that entire stack of things that's involved, all the way from perception to decision-making. Now, usually when people talk about grounded cognition or embodied cognition, people start working on or thinking about robots.
[0:04:34] Of course, that's the ultimate grounding, robots in the real world. But actually, we decided to start off with simulations and specifically games. And we think that games are a pretty perfect platform for developing and testing AI algorithms, especially these kinds of grounded algorithms. So firstly, obviously, you can create unlimited training data by running the game as much as you like.
[0:04:58] There's no testing bias because the games were designed to be challenging for humans. Obviously they were designed by game designers for the purpose of entertainment, not for testing AIs. So often I think one of the things that has held back the AI field is that usually it's the algorithmic developers who also develop the tests. And that can lead to a kind of unconscious bias about what the tests really test for. Of course, compared to robots,
[0:05:27] You can parallel test these algorithms, run millions of agents at once, all playing the games. And it's also quite convenient in most games that have scores, so it's very convenient for us to measure incremental progress. If you make some small change to an algorithm, you can quickly see and quantify the difference it's made.
[0:05:46] So we really love games and of course we also like robots and we're very interested in robots as a sort of application of AI. But in terms of development, we mostly focus on simulation.
[0:06:02] So I guess our most sort of widely known work to date is our work on deep reinforcement learning, where I think we showed for the first time quite an impressive use of deep reinforcement learning when we used it on the classic Atari games from the 80s, Atari 2600 games.
[0:06:19] I'm just going to run a quick video of the agent playing these games. But just to be clear for those who haven't seen this video before, what the agent's getting here are just the raw pixels as inputs, so around 30,000 pixels per frame. And the goal here is to maximize the score. Everything else is learned from scratch. And here it's a single system. We insist on a single system that plays all the different games without any tweaking of the hyperparameters.
[0:06:45] So we call this system DQN, after Deep Q Network. And here's a sort of little medley of the same system playing half a dozen games. And you can see, for those of you who don't know about Atari games, how different in style they are visually, also in terms of their objectives. They're incredibly diverse. And this same system could effectively master all of these games.
[0:07:12] When we published our Nature paper on this, we were better than human at just over half the games, and now we're superhuman level at all but five of them. Here's the boxing example where it's the red boxer on the left-hand side playing the inbuilt AI. If you're interested in the details of this, it's all in the Nature article, including our code, which is available online, so you can play with DQN yourselves.
[0:07:44] So given we're here talking about brains and minds as well as machines, what about the brain? And in fact, DQN and why one of the reasons it works so well was partially inspired by neuroscience and specifically hippocampal replay. The idea of replaying your experiences so that you can maximize the use of that information. And that was critical. The use of replay was critical in making DQN tractable in a reasonable amount of time when it was training on these games.
[0:08:13] So I'm going to spend the rest of the talk now just discussing my own view on how systems neuroscience should be integrated with AI. So first thing to talk about is often I get asked, well, why bother with the brain or neuroscience at all? There are probably many other ways of building intelligence. And that might well be true. But I think there's a couple of arguments to take neuroscience seriously. First of all,
[0:08:40] I think it's likely that the number of possible solutions is actually very small compared to the size of the search space. So if that's actually true, then it's probably worth honing in on trying to reverse engineer, at least to some extent, the solution that we know exists. And after all, the brain is the only existence proof we have that general intelligence is possible at all.
[0:09:00] Now, that might all be very well, but it would be no point talking about it if we didn't have amazing now new techniques and data streams to actually analyze and to give us information about what's going on inside the brain. And pretty much yearly now, amazing new techniques get developed.
[0:09:18] So all the way from optogenetics, connectomics, two-photon microscopy, the list goes on and on now. And there's a huge proliferation now of amazing techniques to get ever closer to what's going on in the brain. And this is all resulting in a kind of exponential increase in our understanding of the brain. Of course, we've got a long way to go.
[0:09:38] I'm not saying we have a good full or even close to full understanding of what's happening with the brain. But there are many clues now, many interesting nuggets of information that if used in the right way, I think can be helpful for AI development. So I see there's sort of two buckets, if you like, of the purpose of neuroscience in terms of how it can help AI development. So firstly, and maybe more obviously, it can provide direction.
[0:10:08] So research direction and inspiration for new types of algorithms, architectures, and even analysis techniques for analyzing these machine learning systems. And I think we should be looking to neuroscience where we have the least idea or the most uncertainty, if you like, about what to do in machine learning to solve a particular problem or to have a particular capability. And also, I think something that's...
[0:10:33] We're pushing very hard on at DeepMind is building and taking inspiration from the analysis and visualization tools that are now quite mature in neuroscience, say for analyzing fMRI images, and then applying that in some analogous way to analyzing machine learning systems. And another interesting thing is actually the experimental techniques and design techniques that are
[0:10:57] What I call validation testing.
[0:11:22] So if you have some idea or notion of your favorite type of algorithm, maybe it's reinforcement learning, and you're sort of arguing with another machine learner that actually this should constitute or could constitute a viable component of an overall AGI system, how do you decide if this is a reasonable conjecture? So maybe you go away and you start trying to build a system and probably it doesn't work straight away. So then you've got to decide how much more effort...
[0:11:50] Should we put into that? Is it just a question of another few years and then something viable will happen and something interesting will happen? So there's very difficult decisions to be made, especially if you're running a large group or you have a large team of where should you put your effort.
[0:12:06] And I think if you can point to a system in the brain that analogously does that, sort of mimics that algorithm, or indeed that algorithm mimics that part of the brain, then I think that can give us confidence that we should put more effort into that area of research and that ultimately it will yield some fruit. And in fact, that's what we thought about with reinforcement learning and why we committed to that so heavily, because the brain indeed and most biological systems use...
[0:12:33] Forms of reinforcement learning like TD learning in order to learn about their environment. The next question then is, if you're going to take neuroscience seriously, there's so much neuroscience, so what parts of neuroscience should we be paying attention to? Here I like quoting David Marr's three-level analysis, or it should be called Tommy's three-level analysis as well, because I think Tommy was involved heavily in this.
[0:13:00] And David Marr used to say in the 70s, probably one of the founding fathers of computational neuroscience, that to fully understand a complex biological system, you need to understand it on three levels. The computational level, which is the what, if you like, the goals of the system. An algorithmic level, so the how, so the representations and algorithms the system uses. And the implementation level, i.e. the sort of physical substrate that realizes the system. So those are the three levels you need if you want to fully understand, say, the brain.
[0:13:30] But actually, I think for machine learning and AI development, really the top two levels are the most important. So at DeepMind, we focus on the kind of computational and algorithmic levels when we come to analyze neuroscience in the brain. So collectively, I refer to that as systems neuroscience. And really what we're interested in then is the algorithms and the representations and the architectures that the brain uses.
[0:14:00] And we're working on all sorts of sort of cutting edge areas where we're using systems neuroscience ideas as well as, of course, machine learning ideas to try and make progress on. So here's just a sort of a kind of summary of some of the areas that we're looking on at the moment and currently focusing on representations, memory, attention, concepts, planning, navigation, imagination.
[0:14:22] And in fact, in terms of parts of the brain, Christoph was talking earlier about prefrontal cortex and high-level cortex. But actually, one of the old parts of the brain, the hippocampus, which is this bit of the brain in pink here in the middle, turns out to be quite critical for a lot of these capabilities. So let's talk about some of those in order. Let's talk about memory first. So we've been experimenting a lot with memory and adding memory to neural networks.
[0:14:53] So one way you can think about the work we're doing is you take a classical computer, we implement a sophisticated recurrent neural network, perhaps with some LSTMs, and then we give it a huge memory store that it can learn to control. And then that whole system is differentiable end-to-end.
[0:15:10] And then what we end up with is what we're dubbing this neural Turing machine, as it's called a neural Turing machine, in the sense that it has all the components now that a Turing machine would need, but it's neural in the sense that it's learned from input and output examples. So here's a sort of cartoon diagram of the architecture. So you've got in the center there in the CPU, the controller is the recurrent neural network.
[0:15:36] And then there's the input and output tape you can think of. And it learns by example, input, output examples. And it learns to control this, a very large memory store on the right. And it learns to read and write from that.
[0:15:49] Now I haven't got time to go into this in detail today, and in fact I think Alex Graves is giving a talk on this right at the moment, but also Greg Wayne is giving a talk about this at 5 o'clock tomorrow in another one of the workshops. So he'll be going into much more detail about this work if you're interested. I'm just going to show you a couple of videos of some of the latest things we've been able to do with neural tube machines over the last year.
[0:16:11] So we looked at one of these classical problems from old-fashioned AI, I guess, back from the 80s, Shrewdiloo, which is a blocks world problem. And the idea is that you're trying to manipulate these blocks, put the green block on the red block, or put the blue pyramid inside the white box. And you can also answer questions about the scene as well. Now, we're not ready to tackle full Shrewdiloo yet.
[0:16:37] But we created a kind of mini version of Shrewdulu, if you like. So a mini blocks world where now we're looking side on onto this block world. And the sort of task we created was imagine a configuration of
[0:16:50] On the left, you have the starting configuration, and you've got a goal configuration on the right-hand side. The system has to convert the start configuration into the goal configuration by moving one block at a time. The only moves that are allowed are moving the top block to any of the adjacent columns. It's a little bit like a complex Tower of Hanoi problem. It's actually quite difficult for humans to do in some sort of reasonably optimal way.
[0:17:19] So I'm just going to run this video and show you how the Neural Tune Machine does. So in the middle here, you see it trying to change the start configuration into the goal configuration. And note, obviously, it's never seen this type of configuration before. So obviously, the start configuration and the goal configuration are totally new. It's learned this from examples, seeing other examples of solutions from other positions.
[0:17:42] You can see there it's solved the problem. And then we have actually a language, a sort of mini language version of this, where we describe now the constraints in terms of little language instructions rather than an example end board. And so if you see down at the bottom here, hopefully you can see that, the little constraints. So like block three needs to be down from block five.
[0:18:06] Block four needs to be up from block two. Block one needs to be up from block four and so on. And so we give a bunch of these constraints. And then from the start position, the neural Turing machine has to reach an end position that satisfies all those constraints. So I'll show you that again here. So it reads in the constraints letter by letter. And then when it sees the question mark, it knows that it's supposed to execute a solution. So you can see there at the end there.
[0:18:37] That final board configuration actually satisfies all those constraints. Three down from five, four up from two, one up from four, and six down from three. And it does that in an optimal way. So that's memory. So what about concepts? So we would like to have abstract concepts because it's going to be key for so many things, including transfer learning, which...
[0:19:02] I think it's one of the keys to flexible general intelligence. So we define transfer learning as the ability to apply previously learned knowledge appropriately to a new situation to help your performance in that new situation. And that breaks down into at least three sort of sub-steps. One is you've got to identify the salient features in your current environment, and more importantly, ignore the irrelevant features. Then you need to re-represent those features as an abstract concept, so sort of divorce from the perceptual features that you learned them in.
[0:19:31] And then once you have these library of concepts, you need to be able to select and appropriately apply those to any new situation that you encounter. So probably step two of this is the probably, I mean, all these things are very challenging, but step two is probably the most challenging and the most critical. And really, I think from the beginning of DeepMind, we've identified learning abstract concepts as one of the key breakthroughs that are needed towards getting us to AGI.
[0:19:58] And you can see this diagram, this sort of missing gap. On the left-hand side, you can think of information in the brain crudely split into three levels, like perceptual information, this abstract conceptual information, and finally symbolic information like words. And there's been a lot of work, obviously amazing progress in the last decade with things like deep learning in terms of dealing with perceptual information. And obviously there was a lot of work in the 80s on logic networks. But there's this missing piece in the middle.
[0:20:25] Which would allow the symbols to be grounded and also to go from perceptual information up to truly abstract information. Now I think as Gabriel showed, when I was doing my PhD in neuroscience in London, this was one of the...
[0:20:43] Key results that I noticed that were quite amazing to me in terms of clues about how the brain might do this. So these are Jennifer Aniston neurons. I think Gabe will cover this earlier. This is a neuron that only fires to Jennifer Aniston pictures and only when she's on her own, not when she's with Brad Pitt.
[0:21:01] So this is very specific. And here's a Halle Berry neuron, where here it can be drawings of her, but not particularly good drawings, in fact. And one where she's dressed up as Catwoman, and another one where it's just the text of her name. And so these are really abstract neurons, in some sense, that are representing the concept of Halle Berry.
[0:21:27] We've done a lot of experiments. I did some in my PhD with my colleague Darshan Kumaran, where we looked at conceptual learning in the brain with fMRI studies. In this study here, we had people learning about fractal patterns. They saw two fractal patterns on a TV screen, and they had to predict the weather outcome of the next day, whether it would be sunny or rainy.
[0:21:50] And they learned this by trial and error, and eventually what you learn is that there's some underlying pattern. So certain fractals you can ignore where they're positioned, and other fractals you can ignore their identity, and it's just important where they're positioned on the screen. So these are kind of underlying rules that were independent of the fractals themselves. And we scanned people learning this task.
[0:22:14] While they were learning the first task, and then once they mastered that, there was a second task where now we changed the fractals to new fractals, but the underlying rules were the same, were kept the same. And we scanned them learning the second task, and they were much faster at learning the second task. And what was really interesting is that the part of the brain that correlated with this increased learning in the second
[0:22:36] When we were scanning in the first task, it was actually the hippocampus. So the amount of activity in the hippocampus in the first task predicted later transfer to the second task, being improved on the second task. So it seems quite strong evidence that the hippocampus here, it's quite difficult for me to point at it from the podium here, the hippocampus here is very critical in learning concepts.
[0:23:02] Here's another famous study actually of a sleep study also related to probably hippocampal replay and the learning of concepts. So in this one, the concept you're trying to learn is this linear hierarchy that A is better than B, that's better than C, better than D, E, and F. But you don't see that whole hierarchy at once. You only see the individual pairs. So you see that you get shown B and A together, B and C together, and so on. And you have to learn by trial and error which one of the two is better.
[0:23:31] But you never see the whole hierarchy and the joins between. So your brain kind of has to infer this from those individual premise pairs. And what's interesting about this study is that when they did this and they tested people 20 minutes later on the separated pairs, so maybe like was B better than D, then people are at chance level 20 minutes later. But after 12 hours or 24 hours and a night of sleep, they're now up to 80%.
[0:24:00] Or 90% successful on the sort of separated pairs, the inference pairs. But even after 20 minutes, there's no difference on remembering the premise pairs. So it's really this sort of extraction of conceptual information that's happening during the sleep. So what about representations? So here's another thing that I saw that made me think about this is a huge clue to how representations are structured in the brain.
[0:24:28] So there's this very famous effect in psychology called the DRM effect. And what happens here, just briefly, is that you get shown, as a subject, lists of words. So maybe let's take that top list, dark, cat, white, and coal. And you get shown all these study words, and you're told to memorize them. And then later, you get tested on these words. And you get tested on, like, did you see the word cat? And you say yes, no. But there's also, critically, some lure words. So like here, the word black.
[0:24:57] Or the word River or the word Cold, which are related to those lit word lists but were not shown during the study phase. And actually people get fooled very reliably to say that they've seen the word Black or they've seen the word River when they didn't actually see it during the study phase. This is an incredibly reliable effect. So it's been known since the 60s. It was rediscovered in the 90s. And it's been repeated in thousands of psychology experiments. It's one of the most cited studies.
[0:25:25] But interestingly, no one had thought about doing this with fMRI. So what we decided to do is look at this with fMRI where now you can think about what's going on in the brain and why is this happening. And one of the reasons we thought was that a partial priming effect. So the idea is that the neural representations underlying these words are maybe partially overlapping. Perhaps they were learned because of the way they were learned through similar experiences. And we thought maybe the degree of overlap would predict
[0:25:54] How much the brain of that individual would be fooled into thinking that they'd seen that world. Because perhaps all these partial primings of these partial overlaps would end up creating a full priming effect on the world, which is maybe the way the brain judges whether you've seen something recently or not.
[0:26:11] And that's what we found is that actually once we started scanning people looking at these lists, these word lists and being tested on a DRM effect, we found that afterwards there was one part of the brain that predicted whether reliably whether individuals were going to get confused about whether they saw a specific word.
[0:26:28] And the part of the brain that was predicting that was the anterior temporal lobe, here highlighted in the yellow. And that's actually known to be the area that's involved in semantic dementia. So that's the part of the brain that goes wrong when you have semantic dementia. So it's actually where we predicted these types of conceptual semantic representations would be. And I think this study now shows a little bit about how those representations might be organized. So with imagination now, then there's a final part.
[0:26:57] This is something I also did in my PhD, was looking at how people imagine and plan for the future. And one thing we decided to test on was hippocampal patients. So patients without hippocampuses, damage to the hippocampus, could they imagine? We know that they're amnesic, we know they don't have episodic memory, but could they imagine things about the future? And so what we tested on was things like cues, like imagine you're lying on a white sandy beach in a beautiful tropical bay, describe what you can see.
[0:27:22] And when we looked at the patient descriptions, they were hugely impoverished, specifically in their spatial coherence. So what that means is that they couldn't bind the disparate components of a scene into a coherent whole. And you can see their performance here. They're the dark bar. This is a measure of kind of richness of their description. And they're massively impoverished compared to age and education matched and verbal matched controls on the right-hand side.
[0:27:48] We then took this in the scanner and we found that, of course, the hippocampus is critical, but there's also a whole brain network, a very reliable one, that mediates imagination and is also involved in episodic memory.
[0:28:01] So then one interesting thing you can think of, and we haven't touched on this, is also about animal intelligence. You know, there's not only human intelligence. I think we can also learn a lot from animal intelligence. So I was fascinated to see whether rats can actually imagine. And so we designed this study that I think was quite elegant and quite simple to show, I think, categorically that rats can imagine. So first of all, we looked at play cells.
[0:28:23] So for those of you who don't know, place cells are neurons in the hippocampus of a rat that signal where a rat is in a location. So here we've got top-down looking at a box that a rat is in, and these are two cells, A and B, that fire only in that position in the box.
[0:28:39] What we also know from previous rat studies is that sequences of these cells play when a rat runs along a trajectory, like a linear track. So here, A, B, C, D, and it replays in the order that the rat moved in. We also know that when the rat goes to sleep, these trajectories are replayed at very speeded rates, probably to aid learning. So what we did is we designed a team maze.
[0:29:04] And so now again we're looking top down. There's a barrier on this T maze that stops the rat reaching the arms, but the barrier is see-through, so the rat can see the arms, it just can't move there. So initially in the first phase, the rat runs up and down the stem of the T maze. And we give it a reward. We show it a reward on the right-hand side. It can see the food pellet, but it can't reach it. Then the rat goes to sleep, and we're recording from the rat's brain at this point. And it's very important this, because this is the critical data that we come back to to analyze.
[0:29:35] Then in the second session, we remove the barrier. So now the rat can freely move up and down the T maze. And so now on the arms of the T maze, we can find its play cells, so here A and D. And what we find is that when we go back to analyze the sleep phase, that in fact, these play cells were being replayed or preplayed, if you like.
[0:29:57] During that sleep, even though the rat had never experienced it yet. So it had only seen it. Maybe it was thinking about moving towards the food pellet. So this is the first time that anyone's ever shown imagined play cells. And what was very cool was that there were significantly more pre-plays, if you like, to the right-hand arm where the food pellet was than to the left-hand arm in every one of the four rats. So I think that's pretty categorical that this was actually behaviorally relevant as well.
[0:30:23] So we're now looking at imagination-based or model-based planning for our Atari agents. And this is an early version of some of the models that we're trying to build and use in planning. So I'm just going to end with integrated agents. So if you now combine all the things I've been talking about into a single agent, then perhaps we have something that could be deemed rat-level AI. So we'd be able to do unsupervised vision, attention,
[0:30:51] Have memory, episodic memory, navigate around mazes, and perhaps even do imagination-based planning. And I think, you know, RAD is pretty smart. So if we can get to this level, that would be pretty spectacular. And maybe you can think of the Atari agent as a sort of lizard level. So here's a little sneak preview of the kinds of things we're working on. So this is now true feed 3D environments. It's called Labyrinthus. We've extended it from an open source 3D engine. And this is our agent.
[0:31:19] Navigating around finding rewarding apples and also exits out of the maze just from pixel inputs. So just from the raw pixels. So we're doubling down on systems neuroscience at DeepMind. And I think there's an incredible wealth of information and ideas and clues if you know where to look. And I think we're actually just at the beginning of the influence of neuroscience and AI on each other. And one thing we're especially excited about is developing new tools and techniques.
[0:31:50] Borrowed from neuroscience to help with the analysis of machine learning systems. So I think we're sort of entering a very exciting era now, and we're doubling down on these efforts. And part of that is actually, I'm very excited to announce that Professor Matt Botvinick, who many of you know from Princeton, is joining us full time from February to head up the neuroscience team and join our incredible sort of in-house team. And you'll know a lot of his work.
[0:32:13] I'm sure already from decision-making and control and working memory. And we also have a bunch of collaborations with Oxford and, of course, MIT with the CBMM program, Harvard, UCL, and so on. And obviously, we're expanding the neuroscience team. We're putting more into this. So if you're interested in some of the work I've been talking about, then come and have a chat with me or Matt.
[0:32:32] So I'm just going to end by saying that actually we've talked a lot about how neuroscience can help AI, but I also think equally interesting, maybe we can discuss in the panel, is how AI might help us better understand the human mind, especially questions like that Christoph is interested in, like what actually is consciousness, and is it necessary, for example, for intelligence? And as Richard Feynman, one of my all-time heroes, said, what I cannot build, I do not truly understand. And I think that's true of intelligence too. Thanks. Thanks a lot, Dennis. We have time for a few questions.
[0:33:10] Thanks for a great talk. I was intrigued by your... You guys are now looking at imagination-based planning. I guess one of the problems you have with that kind of system is how does an agent who acts, how does it know when it's thinking about something that it's not currently doing versus when it should be thinking about things that it is actually doing? And it seems like, in humans at least, we have some sort of cognitive level access to...
[0:33:39] Our own sort of attention mechanism so we know what we're thinking about and we know perhaps the context and why we're thinking about it you know in space and time so if I think about my my birthday a few days ago and I'm not thinking about you know I'm not confused in that sense. Yeah sure I mean we work on attention both the sort of internal and external attention I think that's going to be very important.
[0:34:00] I think actually the biggest issue with our model-based planning stuff is that our generative models are pretty good. So if you measure them on compression or other things measures, but the small errors compound very quickly. And so if you're trying to imagine hundreds of steps out, the errors get quite large. I think part of the issue is probably you don't want to actually be imagining on the pixel level, ideally.
[0:34:25] You want to be imagining on some high-level feature level, which again comes back to concepts. I would say that's the bigger problem than not knowing whether you're imagining or not. In an artificial system, we can make that distinction quite clear. Does that work? Yes, it does. A quick comment and a question.
[0:34:46] You mentioned in the beginning this Blockswald problem that you were able to solve. And you also mentioned that you think this is a more complex version of the Tauss-Vernoy problem.
[0:35:01] I think that's not the case, simply because in the task of annoying the problem, you have this additional constraint where you cannot place a larger disk on a smaller one, which means that if you have n disks, you really need two to the n moves, which means if you have 30 disks, you need a billion moves to resolve the problem optimally, which is not the case in that particular reduced blocks world setting.
[0:35:25] What is maybe more important for the topic of this symposium here is in the beginning you mentioned that you were inspired by neuroscience as you did the Atari game thing. And you mentioned the action replay as an example. But that, of course, goes back to Lin, 1991, who had a PhD thesis on reinforcement learning about action replay. Which we did cite.
[0:35:53] Yeah, of course. Yeah, you should. Of course. That's good. There you do. And the point is, however, is that this was driven by engineering considerations. So the question is, do we have something where you really can show neuroscience inspired us to do something better than standard engineering, which would maybe address exactly the question of this? I think that's a very difficult bar because I think what you'll find is that neuroscience ideas are sort of seeping around everywhere and help.
[0:36:21] Formulate certain ideas. And of course, you should be doing both at the same time. So I think engineering should go hand in hand with the neuroscience. And lots of ideas have been thought of in machine learning and lots in the neuroscience. And I think they help each other sort of confirm that you're going in the right direction. I mean, even ConvNets with Hubel and Wiesel, which ones influenced who? I mean, I'm sure you could say that the ConvNets sort of came up.
[0:36:47] You know, with little influence from that. But actually, I think these ideas are kind of all mixing together and help with this progress. So, I don't know if you're asking for a specific example of exactly a neuroscience thing that was done and then interpreted in machine learning. Is that what you're saying? Well, confidence is actually a great example because that was, in my point of view, more or less the last time when neuroscience really produced important inspiration for AI, which, because Fukushima's convolutional networks in the 70s were inspired by,
[0:37:17] I think this is actually a good question for the panel also, where other people might comment. So let's do this later, and maybe one, two quick questions. We're here. I'll try to be very quick. I'm surprised that you rule out innate structure more or less altogether, and I wonder whether that's a methodological commitment. You said at the beginning you want to do everything kind of from raw pixels. You don't want anything built in. There's a lot of evidence from biology that the genome is
[0:37:47] Spending half of its effort, so to speak, trying to build a very carefully structured brain. So why ignore that side? No, no, so sure. So in fact, I should clarify that. So we're interested in sort of learning as far as possible sort of end to end. But we're not against having modules or, I mean, clearly evolution has given some initial constraints. So we're looking at those kinds of things as well. But we would like the prior knowledge that we build in to be as minimal as possible and as generic as possible. That's maybe another longer way of saying it. It's an empirical question what the right amount should be. Sure.
[0:38:19] Yeah, very brief. Collaboration and competition have some neural bases in the frontal cortex. So the agents are very independent. What role do you think competition and collaboration play between learning systems? So we're arranged in things like meta-control, where there's more than one type of control system, and then you've got to decide
[0:38:44] Which system to trust or which system to switch to. So the brain definitely does that between model-free and model-based, perhaps between episodic controllers as well. So we're looking at that and implementing something similar to that, maybe deciding around the uncertainty of each of the controllers. So we're looking at that at the moment. I should say thanks to all my colleagues, of course, at DeepMind, all the amazing people who have done all the work on the screen here, and many more. Thanks. Thanks. Let's thank them once again.