{"url": "https://www.youtube.com/watch?v=UMq4BcRf-bY", "page_url": "https://www.youtube.com/watch?v=UMq4BcRf-bY", "title": "How to Keep Strong AI Safe and Beneficial (panel)", "show_or_channel": "Future of Life Institute", "date": "2017-01", "duration_s": 2033, "type": "panel", "speakers": ["Dario Amodei"], "confidence": "medium", "notes": "Beneficial AI 2017 (Asilomar) panel with Shane Legg, Nate Soares, Stuart Russell, Viktoriya Krakovna et al.; Dario's share likely < half.", "people": ["dario-amodei"], "id": "2017-01_future-of-life-institute-how-to-keep-strong-ai-safe-and-beneficia", "audio_duration_s": 2032.4775, "worker": "jc-w06", "transcribe_s": 26.4, "model": "large-v3", "segments": [{"start": 9.04, "end": 39.25, "text": "So in the last panel, being all optimistic, in some cases we got so excited it felt kind of sci-fi, even though I'm sure many of the panelists would argue that this was really founded on a solid science. And if you catch me, especially after a beer or two, you'll also hear me getting all excited about space exploration and doing all sorts of things which definitely sound sci-fi unless you're an astrophysicist."}, {"start": 40.62, "end": 64.14, "text": "Now, before we go to dinner, we're going to bring it back down again to the very concrete question of what should we do here and now that will be useful in helping realize all these awesome things and avoid any pitfalls along the way. Namely, technical research in AI that can help."}, {"start": 64.14, "end": 94.51, "text": "Accomplish these questions. We had, just before this conference, a very nerdy workshop. Raise your hand if you were there. Where we discussed in great length a bunch of progress that had been made in the past two years by the grantees that were funded by this grants program. Thanks to Elon Musk and the Open Philanthropy Project. A lot of exciting stuff happened. And I think Mo Chavardi summarized very..."}, {"start": 94.99, "end": 126.02, "text": "Succinctly, a fascinating thing about this, he said that you know that things are going in the right direction when AI safety research is going boring, is getting boring when there are talks with slides full of equations and matrices. So without further ado, I'm going to ask each one of you here, don't be afraid to geek out a little bit here, but what kind of nerdy, nuts and bolts research do you feel"}, {"start": 126.02, "end": 153.68, "text": "It's really useful to work on right now. Not in 20 years, but now. It can be helpful for these things. Shane, do you want to go first? Right. I think a really interesting and important topic to look at at the moment is understanding the representations inside neural networks and other machine learning systems. A lot of people have already expressed this thought, so it's hardly new. But I think it's something that..."}, {"start": 153.68, "end": 182.42, "text": "It's going to be relevant in the near term, the medium term, and the long term. And it'll be, I think, very, very important for many reasons. Another topic which is not so important in the near term, but I think is a very, very interesting one, is wireheading and the idea that a powerful reinforcement learning agent may have access to its..."}, {"start": 182.42, "end": 204.22, "text": "Reward mechanism or someone saying source code or other things like this and may act to modify this instead of sort of achieving the goal that you think you built into it. I think there are, I think this is a tractable theoretical problem."}, {"start": 204.22, "end": 237.49, "text": "And I know a number of people working on it, but I think that it's getting an understanding of this and the types of systems that are likely to wire heed or aren't likely to wire heed and how and when and so on. I think this will be a really interesting problem. And in the long run, it might turn out to be quite an important one. I think one thing I'd say is many people have a high-minded idea or there's lots of sort of fanciful ideas about what you might do for safety work."}, {"start": 237.49, "end": 266.35, "text": "And I would say that the current state of safety work is if, like, can you take a black box that's maybe a function optimizer or some black box that you're assuming has quite a bit of capability and could you utilize this black box to program a robot arm to put a strawberry on a plate and then do nothing else? And if you could take a box of arbitrary capability and cause it to"}, {"start": 266.35, "end": 295.76, "text": "Put a strawberry on a plate and then do nothing else. Then you've solved. That's sort of something that we don't know how to do in full generality today. Of course, we can program robot arms that put strawberries on plates, but taking a box of arbitrary capability, I mean, even just saying we want a system of arbitrary capability that will flip this switch and then turn off. This is rife with lots of technical problems, lots of technical challenges. There's sort of no..."}, {"start": 295.76, "end": 322.85, "text": "There's no obvious boundary of turn off. If, you know, there's no, you know, Steve Omohundro, who's here, will tell you that if you've accidentally written some objective, that incentivizes the system to create subagents to make really sure that the strawberry's on the plate, then, you know, you can go down these rabbit holes. So I would say the sort of technical safety work that's interesting to me at the moment is focused on these really, these sort of toy problems."}, {"start": 322.85, "end": 349.33, "text": "And really understanding these toy problems where you're treating capability as a black box. You're saying, let's assume we had a highly capable system. How would we solve, thus and such, a very toy problem? And I think there's just a whole slew of those, many of which you can find in papers that have authors here on the byline. All right. So before I turn it over to Richard, I was very struck at the workshop, again, by this feeling that there's this new subfield."}, {"start": 349.33, "end": 373.84, "text": "It's just being born and it's just full of fascinating problems, AI safety research. And it spans, of course, the range from things that are very useful right now for very near-term issues to things farther into the future. So focusing, Richard, on things related to AI at the human level or beyond, what do you feel are some really, what is it, if you call out one really exciting area, what would you pick?"}, {"start": 374.83, "end": 404.67, "text": "Yeah, so I'd say being able to find bridges between sort of the top-down value specification and bottom-up value learning is sort of a research direction that I haven't seen too much of yet, and it would be really great if we have more of that. So this is very related to some of what Demis was talking about earlier with the grounded learning of symbologies and semantics. Because then we can go from..."}, {"start": 404.67, "end": 429.49, "text": "From Jan Talen's preference aggregation potentially to things like advanced versions of inverse reinforcement learning to actually be able to bridge those. Great. And Dario, before you tell us about your favorite area, you said some very interesting words at the workshop also about how you see the long-term research goals related to the short-term research goals and safety. Do you want to tell us about that first?"}, {"start": 429.49, "end": 458.29, "text": "Sure. So actually a lot of what I would say was contained in Shane Lake's talk earlier today where I think for a couple reasons it's really important and one of the things I'm most excited about to look for ways to have an integrated theory that ties together things that could go wrong with an AI in the long run with things that could go wrong with an AI in the short run. So I first got interested in the safety topic by, for example, reading Nick Bostrom's book. And my reaction to that book was that, you know,"}, {"start": 458.29, "end": 471.82, "text": "The scenario that was sketched out was at the same time sort of plausible yet vague. I could really believe that these problems could happen at some point, that they could really be threats to humanity."}, {"start": 471.82, "end": 495.33, "text": "There are a few features to the problem that I found kind of undesirable. One is that because we've never built a superintelligent system, we don't know how to study empirically whether one would be safe or unsafe. So we're kind of stuck in long chains of theoretical reasoning. The other is that it's something that we only get one shot at. So again, it's very hard to get a feedback loop and study empirically. So it seems very valuable both for..."}, {"start": 495.33, "end": 513.81, "text": "So I'll mention a couple things that I'm particularly excited about, although there's many. One is, one of the concerns that's raised in Nick's book is"}, {"start": 513.81, "end": 527.41, "text": "The idea that an agent with a misaligned objective function that was very powerful could be very bad for humanity. But that's something that also happens in agents that we have."}, {"start": 528.14, "end": 557.81, "text": "That we have today. It's in fact a very common failure mode in reinforcement learning. So both understanding when this occurs better and looking for modified versions of reinforcement learning that are based on say learning reward functions or learning from imitation or using transfer learning to learn reward functions is something that I find very interesting. The second is something that hasn't been mentioned much in the AI safety community, which is safe exploration. So this is the idea that if you're a reinforcement learning agent and you're learning,"}, {"start": 557.81, "end": 558.77, "text": "Maybe."}, {"start": 558.77, "end": 588.24, "text": "Your objective function is just fine. And when you fully learn to do what you want to do, you'll do things that are good and beneficial for whatever purpose you were intended. But while you're learning, you might do something destructive. So if you're a robot helicopter learning, you might crash and destroy yourself before you have a chance to learn a good policy. I think actually the long-term or a long-term version of that is something like the kinds of things Nate talks about when you have an AI that modifies its own internal state and its code. And even if you could kind of trust what it was going to do when it kind of"}, {"start": 588.24, "end": 617.98, "text": "Great. Would you like to add something to that, Vika? So I think that the problem of specifying objective functions for artificial agents specifically"}, {"start": 617.98, "end": 647.57, "text": "Reward functions for reinforcement learners is a very challenging problem from a safety perspective because this involves building in common sense and this is notoriously difficult. There are various sub-problems of that that I think are very interesting to work on today. For example, Shane mentioned wireheading and how to build systems that avoid wireheading and avoid finding some sort of"}, {"start": 647.57, "end": 678.03, "text": "Loophole in your specification of the objective that just gives it infinite reward without actually achieving what you wanted. And I think that's going to be fairly tricky to actually figure out how to build agents that don't fall for that. Another big part of specifying objective functions is building agents that learn to avoid negative side effects."}, {"start": 678.1, "end": 707.87, "text": "And this is something that Dario talks a lot about in the Concrete Problems paper. And I think, when I think about how we would get agents to avoid negative side effects, then you have to be able to specify, well, what is the baseline there? So what is the state in which it has no side effects? And you have to figure out how to measure."}, {"start": 707.87, "end": 739.46, "text": "What about you? Given that we've already heard a lot of interesting ideas and thoughts about how to make sure that the AI systems we build are not going to be harmful, they are not going to be malicious, I thought I'd share a few"}, {"start": 739.46, "end": 769.62, "text": "Let's say a few words about some ways in which I think we can maximize the benefits of AI to society. And of course, we've already heard about some very exciting applications of AI, like in health care or autonomous driving that could potentially save thousands or even millions of lives. But my point is that there is a broader range of problems, very important problems, that perhaps are not getting enough attention because"}, {"start": 769.62, "end": 799.3, "text": "The economic incentives to work on these problems are not as strong. I'm thinking about environmental issues or development issues like poverty or food security or climate change. And I strongly believe that a lot of these problems can be addressed in a better way using AI techniques. And there are some very concrete problems that we can tackle today using the narrow AIs that we have."}, {"start": 799.3, "end": 829.1, "text": "And just to be concrete and a few examples, I'm thinking about the kind of work that Tom is doing on sort of using reinforcement learning for coming up with better ways to manage natural resources, or find better ways to do environmental conservation, or using machine learning to come up with better climate models, or some of the work we've been doing on using AI and ML techniques to address issues related to poverty and food security."}, {"start": 829.1, "end": 859.01, "text": "There is growing interest in these kind of issues, but there are a lot of technical challenges and problems where I think it would be great to see more work in these kind of areas. Great. Stuart, we already heard some exciting future directions from you on the opening morning. Would you like to add something more that you're excited about? How much time do we have? So actually, I think probably the most beneficial thing would be to stop thinking of what we're doing as,"}, {"start": 859.01, "end": 888.26, "text": "Safety or ethics or something that's going to be layered on top of the real stuff, which is the AI. But actually to think of this as intrinsic to the design of good AI systems, just like not falling down is intrinsic to the design of a good bridge. And it's not two separate disciplines in civil engineering. There is only one discipline and it's bridges that don't fall down."}, {"start": 888.26, "end": 908.27, "text": "That would be a step. I was thinking about the people for the ethical treatment of reinforcement learners, actually, and I was feeling sorry for these reinforcement learners who are sometimes getting the negative rewards, which presumably is putting them in great pain and suffering. And then I was thinking, okay, well, we have a shaping theorem which allows us to add an arbitrary"}, {"start": 908.27, "end": 935.25, "text": "Potential function on the state space and then we can just push all those rewards into the positive area and they would behave exactly the same way, exactly the same things would happen, but there'll be no pain and suffering in the reinforcement learning agents. So I think we solved the petrol problem already. But that of course points out that these numbers that we're giving to our reinforcement agents are not actual rewards in the same sense that they are"}, {"start": 935.25, "end": 965.07, "text": "Rewards for a human or for an animal. They're just information. And one of the things I mentioned in my talk the other day was that if you just view the reinforcement signal as information, not as reward, then there is no wire heading problem because there's absolutely no incentive to destroy your information source, which is your only way of achieving actual reward is to act in such a way that the true hidden reward function"}, {"start": 965.07, "end": 994.24, "text": "So I'm hoping that I'll be able to publish a result on that and people can stop worrying about wire heading, at least for AI systems. It's still a problem for humans. And I just wanted to mention one more thing. So for those of you who weren't at the workshop, Dario, you were the one showing the speedboat thing, right? So they have a speedboat racing game and the reinforcement learning speedboat."}, {"start": 994.24, "end": 1023.82, "text": "Devises this incredible strategy which is far better than actually finishing the race according to the scoring function. It just zooms round and round in a very tight circle crashing into all kinds of things but collecting points for driving through certain sort of places where you get extra points for driving through. And so I think this suggests that one of the things we would really benefit from is simulation environments in which bad things can happen when the AI is misspecified."}, {"start": 1024.18, "end": 1054.06, "text": "Because we really need more examples of the kinds of things that can go wrong to help us get a sense of where we should be going and what problems we should be working on. We can work on a lot of imaginary stuff, particularly imaginary stuff that comes from verbal descriptions of problems and verbal descriptions of arbitrarily intelligent AGI's that actually evaporate when you code things."}, {"start": 1054.45, "end": 1074.91, "text": "Do you want to add something about your speedboat? Yeah, so I didn't mention this in kind of what I said before because I wanted to avoid plugging the software tool developed by my organization. But one of the reasons that we developed OpenAI Universe, it was kind of made for many purposes, was actually to create the kind of thing that Stuart is discussing."}, {"start": 1074.91, "end": 1104.54, "text": "Where if we have a wide range of environments where we can train an AI in one environment and then test it in a similar environment and see what might go wrong or give it kind of access to a broad set of things it can do, then I think richer environments actually make it much easier to exhibit these sort of problems and then after exhibiting them to try to solve them. Yeah, I find that very encouraging also what you're doing there, Dario, even though you know that you don't want to mess things up."}, {"start": 1104.54, "end": 1132.4, "text": "In our universe, if you only have one shot, fine. If you have a simulated universe, you can really go about it the traditional scientific way, experiment over and over again. Now, since we have the pleasure of having such an interdisciplinary audience here, my next question is going to be not a computer science question, but a sociology question. So, Shane, you mentioned in your talk the sociological issue of this taboo. I'm talking about this kind of research."}, {"start": 1132.4, "end": 1161.12, "text": "I definitely see that it's reduced quite a lot. I think it's a, I don't know, I think it's a combination of"}, {"start": 1161.12, "end": 1190.74, "text": "Surprising progress in the field. A lot of people, you know, a lot more has happened in the last five years than a lot of people expected. And so there is a much stronger sense of optimism. You see that in the, you know, exploding numbers of people going to conferences like ICML and NIPS. And I think the other thing is that, you know, just in terms of deep mind, when we were getting started and we were just, you know, five of us or something and we were saying we're going to start a company and..."}, {"start": 1190.86, "end": 1217.33, "text": "We thought we'd had a good chance of trying to build an AI in a few decades. We didn't have much of a reputation. Now we've proven that we can do good AI research. I guess that people tend to take us a bit more seriously. So it's some combination of those two things, but it's just sort of anecdotal evidence of talking to people. And I guess it's also listening to other people discuss these topics as well. I remember years ago, people would..."}, {"start": 1217.33, "end": 1248.24, "text": "Really not talk about these topics, but at conferences I hear lots of other people talking about this sort of thing now. Does anyone want to disagree with Shane on this or share any similar experiences related? Actually, I still think there's institutional resistance. I think there's an old god with an AI that believes that any mention of risk is an anti-AI. It's a treasonous act and needs to be resisted. We can only talk about the good things that AI can do."}, {"start": 1248.24, "end": 1270.83, "text": "As I mentioned in my talk earlier, I think it's a very misguided viewpoint that's likely to have a worse outcome. I really would encourage open, vigorous, healthy debate on these questions. I'm just waiting for the other side to say something actually sensible so we can have a debate."}, {"start": 1271.02, "end": 1299.15, "text": "Dario? So one thing I'd like to add is, I mean, I agree with what Shane and Stuart said, that I think there's been a lot of progress on this over the last year or so, and still some progress to go. I think that we may be at a stage, you know, before we were at a stage where, you know, just explaining the problem to people and maybe discussing a couple things that could be worked on was a limiting factor, but now I actually see a lot of people doing that. And I think Shane might have said this in his talk, but I think..."}, {"start": 1299.15, "end": 1333.04, "text": "Getting papers on these topics that get into prestigious conferences like ICML and NIPS, I think both validates in our own minds the value of what we're doing and also legitimizes it for the rest of the field. And that may actually be the limiting factor at this point. Wonderful. I think this is a perfect time to open up the questions from the audience. So let's see. Jan, first, if you want to stand up. Janneke."}, {"start": 1333.04, "end": 1361.22, "text": "Stuart just asked for some contrarian opinion, so I'm going to give it a try. Okay, so there's a number of things that have been said during the previous session with the talks, which I think need to be thought about a little bit in more rational terms. First thing is that there is a lot of catastrophe scenarios that rely on the idea that somehow one entity is going to come up with the solution to AI."}, {"start": 1361.22, "end": 1388.7, "text": "And then there'd be game over for everybody else. That's just completely ridiculous. That's just not the way science works. It's just not the way this community is organized. If anything, this community has become much more open in the last three years than it ever was. And so if we are seeing a phased transition, it's a phased transition from being closed to being open, not the other way around. And whenever a piece of technology appears, just the fact that it exists"}, {"start": 1388.7, "end": 1416.08, "text": "It's gonna happen simultaneously in lots of different places. The other thing is that"}, {"start": 1416.08, "end": 1444.58, "text": "AI, super intelligent or not, is going to be very vulnerable. To run it, it's going to have to run on a big computer, which needs to be fed, which needs to be maintained, and which is going to fail after a while. And as super intelligent as it is, if it doesn't have all the means to fix itself, it's not going to continue for very long. Second of all, if there is another AI system,"}, {"start": 1444.58, "end": 1472.48, "text": "With similarly sophisticated technology whose only purpose is to destroy the first one is going to win. The same way we can be killed easily by crocodile who are not nearly as smart as we are. We can be killed by blue bottle jellyfish who don't even have a brain, right? Or by viruses, right? So you don't have to be smart to be able to destroy a super smart entity."}, {"start": 1472.48, "end": 1501.39, "text": "And if your only purpose is to do it, you will win. So I think the scenario that Nick Bostrom was describing that is AI, or the question he was asking, is AI like nukes where it's much easier to be on the offense than on the defense? I think it's just not the case for AI. I think it's much easier to defend against AI than it is to attack with AI."}, {"start": 1501.39, "end": 1530.8, "text": "We already have dangerous autonomous weapons. And the most dangerous autonomous weapons that we have are actually banned legally by international treaties that were actually just recently signed by the US. And it's landmines. Landmines are dangerous as autonomous weapons, not because they're smart, but because they're stupid, which is why they are legally banned. They did a lot of damage, but you know."}, {"start": 1530.8, "end": 1559.49, "text": "People kind of came to the senses after a few decades. There was a question as to whether... Okay, we should probably give Stuart a chance to give a rejoinder here while he still remembers all the points that you made there. Stuart, do you want a quick response? Gosh. Well, so I guess the first thing was that there would be mutually assured destruction because as soon as there was one AI system, there would be another one. I mean..."}, {"start": 1559.49, "end": 1582.48, "text": "Arguably, the US developed nuclear weapons before anybody else. It actually took quite a while, and it wasn't that the Russians developed them themselves. They had to be really given them. And there was... So you think if Facebook develops, let's say, super intelligent AI, there's not going to be someone who's going to be recruited by Google to tell them how to do it?"}, {"start": 1582.51, "end": 1609.74, "text": "It's exactly what happened with the nuclear bomb. There were people working... It took four years. During that period, in fact, there was significant institutional pressure within the US to bomb the Russians before they could develop it, to establish complete domination. So I don't feel at all reassured by our history, and I don't feel at all reassured by the idea that..."}, {"start": 1609.74, "end": 1635.52, "text": "You know, these super intelligent AI systems that are far beyond anything we understand are all going to kill each other and not cause any problems for us. So let me just briefly mention the autonomous weapons thing. I mean, yeah, landmines up to now have been the most dangerous autonomous weapons. That's because for most of our history, we haven't had AI that's good enough to do anything more than just blow up."}, {"start": 1635.52, "end": 1653.12, "text": "But now we do. So imagine 10 million flying landmines, each of which has somebody's name on, arriving in New York. That's a pretty scary proposition. It's entirely feasible with today's technology. We're almost out of time for this session. I do want to..."}, {"start": 1653.12, "end": 1679.71, "text": "Give a chance to the other people who have hands up here. But I would very strongly encourage both you, Jan and Stuart, to sit next to each other at dinner tonight because I can guarantee you both a really exciting and fun discussion. So let's see who is next here. Stand up. I'm Ian Goodfellow from OpenAI. I'd like to comment on something from Shane's talk and just offer a partial solution to one of the problems he identified."}, {"start": 1679.71, "end": 1708.29, "text": "He quoted Andrew saying that a lot of researchers don't know how to work on anything specific and actionable right now, that dealing with the superintelligence control problems of the future seems just not sufficiently concrete and operationalizable right now. One way that I deal with that myself is by studying problems revolving around controlling models that exist now. In particular, I work a lot on adversarial examples."}, {"start": 1708.29, "end": 1731.97, "text": "Where we want to make things like object recognition algorithms understand that an image of a bus is a bus. And it's easy to find ways that you can take an image of a bus and have the system say that it's an ostrich. So I think these techniques that we study for how to prevent these kinds of problems now might actually scale up in the future. And there's something that people can work on right now. Great. Do you want to pick the person you want the answer from? Shane?"}, {"start": 1731.97, "end": 1759.79, "text": "Yeah, so that's sort of part of the point I was making about trying to connect near-term, medium-term, and long-term questions. And I think there are many things in the near term, such as understanding representations and being able to inspect representations that will continue to be important in the future and may inform us in very, very useful ways in the future. So by all means, if you can take what some people may see as longer-term questions and actually come up with related..."}, {"start": 1759.79, "end": 1786.35, "text": "Questions in the near term, this could be a very fruitful avenue of research, and it also may open people's minds to some of the long-term questions as well, because they will seem more tractable, as you point out. Dario? Yeah, I mean, I'll actually second Ian's comment on adversarial examples. I think one thing about adversarial examples is it relates to thinking about how an adversary could cause..."}, {"start": 1786.35, "end": 1816.37, "text": "Did you have a quick remark to Vika?"}, {"start": 1816.43, "end": 1846.7, "text": "Did Daria already say it? It looked like you were waiting. Well, I had a quick remark for Jan LeCun's question. Actually, I think I still have a bit of pessimism about the proposed solution of having the AI whose only purpose is to shut down the original AI. I think if you look at the example of the dangerous animals that could, let's say you have a crocodile or a tiger that could kill a human,"}, {"start": 1846.7, "end": 1877.1, "text": "I think we might also notice that, yes, those dangerous animals exist, but we don't have any of them in this room, and we mostly keep them in the zoo because we know they're dangerous. So I think if an AGI was aware of a different system that was trying to shut it down, it would have incentives to keep it in some kind of zoo as well. So I think we might need to come up with potentially more of our solutions."}, {"start": 1877.23, "end": 1908.9, "text": "Great. The staff here has prepared a really delicious dinner for us, and we don't want to let it get too cold. So we're going to do one last question. And you, Sean, have been holding your hand up the longest. Hi, I'm Sean Hegarty, Center for the Study of External Risk and Center of the Future of Intelligence, both in Cambridge. My question is motivated by having the opportunity to be in an exciting hub where I can set up workshops and discussion groups."}, {"start": 1908.9, "end": 1934.0, "text": "For a whole lot of people from a wide range of disciplines. So my question is, who would you like to be here in two years who isn't here right now in terms of source of expertise, disciplines, things that are underrepresented at the moment in the community which we could use a lot more of or that might crack open some of these problems in ways we haven't anticipated? Thank you. Mate? Yeah, so I'd say this conference is, I mean, it covers a very wide range of topics."}, {"start": 1934.0, "end": 1963.5, "text": "Yesterday was focused on economic impact. Today was focused on more long-term impact. Tomorrow is focused more on law, also ethics, if I recall correctly. I'm not sure I got the schedule right. I would say that what I'd be really excited about is maybe in two years, maybe at some other venue, having a workshop or a conference more focused on actually doing the technical work. We had a workshop earlier which was focused on reporting technical work, and that was great. But I'd really like to see a workshop focused on"}, {"start": 1963.5, "end": 1991.57, "text": "What can we get done over the span of time with lots of interest and talented people in the room? And at those sorts of things, the difference between the people here and the people there, I would say, are more top folks in the field. I suspect there's fewer deep learning people here than are representative of the composition of the field. So yeah, just getting a lot of skilled people in a room to work on specific technical stuff. Great."}, {"start": 1991.82, "end": 2022.64, "text": "So you're going to get the last word from Vika. I'd just like to briefly say that I think it would be good to have more people who work on security and could provide a security perspective to getting the adversarial examples and other sort of adversarial challenges that we face with AI safety more resolved. Great."}, {"start": 2022.96, "end": 2026.93, "text": "Let's give our panel a warm round of applause."}]}