{"url": "https://www.youtube.com/watch?v=6sCKa5and1I", "page_url": "https://www.youtube.com/watch?v=6sCKa5and1I", "title": "AI Safety and Cybersecurity Panel - Artificial Intelligence and Global Security Summit", "show_or_channel": "Center for a New American Security (CNAS)", "date": "2017-11-01", "duration_s": 3311, "type": "panel", "speakers": ["Dario Amodei"], "confidence": "high", "notes": "Dario (then OpenAI) also gave opening remarks: https://www.youtube.com/watch?v=VrPc4_Mylr4 (641s).", "people": ["dario-amodei", "dario-amodei"], "id": "2017-11-01_center-for-a-new-american-ai-safety-and-cybersecurity-panel-arti", "audio_duration_s": 3309.0525, "worker": "jc-w06", "transcribe_s": 45.9, "model": "large-v3", "segments": [{"start": 0.37, "end": 28.08, "text": "Okay. Thank you, Kathleen, and before that, Dario. My name is Tom Simonite. I'm a senior writer with Wired, and I'm happy to be hosting this panel where we'll delve more into some of these issues. Dario and Kathleen are coming back, and we are joined also by Joanna Bryson. She is a researcher at Princeton and the University of Bath, and Amir Hussain on the end there. He is CEO and co-founder of the company Spark Cognition and the author of the forthcoming book, The Sentient Machine."}, {"start": 28.5, "end": 56.08, "text": "Now, I thought we could start off by picking up on a point that both Kathleen and Dario made, I think. So when you talk about safety and security in relation to AI systems, there's a tendency to think about intentional evils, maybe planned or plotted by a human or maybe even a machine. But in the two talks we just heard, you both made the point that accidents are an important aspect of safety as well, right? We need to think about..."}, {"start": 56.08, "end": 86.22, "text": "Kind of unexpected surprises. And Joanna, that put me in mind of some of the work you've done thinking about ethical frameworks that people who build these systems might need to work within and maybe the idea that we need to audit systems or have ways of doing that. Okay, well, thanks. I've slightly forgotten the format. I think I have a few minutes to answer that question, right? We're just going to have a general panel discussion. Okay, all right. There's no technical time limit, but don't take too long. Right, okay."}, {"start": 86.22, "end": 114.5, "text": "I'd really like to respond to the two talks we saw before and to reinforce some of the things that the two speakers before us made too in this framework. It's very important when you're thinking about the safety element to realize that in some ways artificial intelligence and natural intelligence aren't different at all. So if you do have two million people sitting, hacking, trying to get through defenses,"}, {"start": 114.5, "end": 144.13, "text": "Or if you have one quantum supercomputer trying to fight and get through the defense. Either way, the right thing to do is to try to solve the cybersecurity issue. And I'm really glad that we had this talk we just heard about cybersecurity. I want to emphasize, you cannot trust AI until you solve cybersecurity. And yes, AI is an issue for cybersecurity as well, because as I just said, you can use it to hack."}, {"start": 144.13, "end": 172.21, "text": "You should be having conversations in separate rooms. You cannot have a system in charge of something when you don't know for sure who currently owns that system. And that's why I think it is very important not only to have humans in the loop, but really being core through the chain of command all the way through, not just for military, but for corporations and things too, that this is a big problem."}, {"start": 172.53, "end": 201.09, "text": "On the other hand, I know this stuff with AlphaGo is a wonderful piece of engineering, but I want to emphasize one of the things Eric Schmidt said. It works within a very simple set of rules. So in some ways, it's amazing that a machine is able to beat us at Go. In other ways, it was incredibly amazing when books could remember huge amounts of information. And people used to think that was super freaky. And it did change the world. I really liked the metaphor about the air."}, {"start": 201.09, "end": 227.46, "text": "How AI changed things too. But having these open-ended problems, we still will have ways to go. And I don't agree that, I have been working on AI ethics for a long time, and I also build emotional systems for real-time AI systems. But I think the right place for the ethics to occur is at the social level as we design these systems."}, {"start": 227.46, "end": 256.35, "text": "The biggest difference between artificial intelligence and natural intelligence is that every artificial intelligence is an artifact. That means someone has deliberately built it, and we need to maintain the idea of human responsibility, and that's where the auditing comes in. We shouldn't allow people to release systems that they aren't willing to have audited. Some people are saying, this is going to slow down the rate of innovation. Actually, having a system that is clear makes it easier to maintain and extend."}, {"start": 256.35, "end": 285.76, "text": "So I'm not even sure it will slow down. It will change the way we innovate a little bit. But it's absolutely critical that people remain in a position of responsibility. We cannot make the system itself be the responsible entity because it isn't coherent to talk about putting suffering into a system. It's not like an evolved system where that's part of our entity. We are deeply social. We are evolved to be social. It's the same for fish. If a fish is socially excluded,"}, {"start": 285.76, "end": 313.44, "text": "It gets less healthy. Same thing for us. We are not going to build AI like that. So it's important not to reason about AI that way and to recognize that we cannot make the system itself responsible. Okay. I think, Dara, you wanted to come in. Yeah. A couple points. So the first one on the idea that security is a safety issue, I basically strongly agree with that. And the way I've often described it is that machine learning and AI systems operate on a stack."}, {"start": 313.44, "end": 330.11, "text": "Where they're at a very high level above software systems. And so any vulnerability to a software system is also a vulnerability to the AI system. And then there are these additional kind of new and exotic kinds of vulnerabilities that exist as well. But that's not to say that the old vulnerabilities don't already exist."}, {"start": 330.11, "end": 358.05, "text": "On the issue of AlphaGo, it's been said a couple times that it's just a game with kind of deterministic rules. I mean, I think that is to some extent a limitation, but our work on Dota and work on some real-world robotics shows that we're actually also beginning to make progress in environments where the whole rules of the game aren't known, where you have to figure out the rules of the game as you go."}, {"start": 358.05, "end": 383.36, "text": "These techniques are proving to be very general. So while I agree that we have a ways to go, I think we should be cautious about saying that certain things can't be done or that we're very far from certain capabilities because history has surprised us in both directions actually. Amir, you seem a long way away."}, {"start": 383.36, "end": 413.36, "text": "At Spark Edition, you work with customers, you're actually building systems and working in the commercial realm. Do any of the points that we're talking about here, do they come up yet in that area? Like customers asking you about safety and security at this point? There's a lot of deep issues that have been raised here. First of all, there's this idea that you can't really implement ethics or create a proxy for ethics in an automated system."}, {"start": 413.36, "end": 443.54, "text": "Then there's this other underlying assumption that you can always have a man in the loop. You can have a man, a woman imbue an artificially intelligent system with some initial purpose. And therefore, by imbuing that system with that purpose, the responsibility of the consequences as that entity conducts a journey towards obtaining its purpose rests with the individual. This is a fallacy."}, {"start": 443.54, "end": 471.44, "text": "Because the system that has initially been imbued with purpose is now learning. If it's a system of any level of complexity, it's a system that is learning. What you can do is you can neuter the system and you can say you can learn in a gym, but when you act in the real world, you cannot adapt your knowledge. There are solutions like this because"}, {"start": 471.44, "end": 499.97, "text": "We work, for example, in environments where we have to go get stuff approved by the FAA. There are near-term solutions to problems like these. I hinted at one. But in the long term, I think there are really groundbreaking technological innovations. One of the areas that we're working on, for example, fuses this idea of a distributed view of the world implemented as a blockchain-like storage system."}, {"start": 499.97, "end": 528.67, "text": "That is shared by all agents. And all agents are given one additional quality. In addition to being able to do their work, they can also judge the efficacy of another agent. And this becomes a social system where individual agents are commenting on the efficacy and the safety and the accuracy of all the other participants. So there are other ways. Ultimately, if we stick to this position that"}, {"start": 528.67, "end": 557.62, "text": "Full autonomy will not happen. We will basically have human beings take responsibility for every semi-autonomous system. One day in the battlefield, you'll have an entirely autonomous set of drones flying at you, and you won't know what to do. Joanna, you? Yeah, I'd very much like to answer that. I didn't actually say it was impossible to build an ethical system into an AI. I want to make this very clear. I am making strong normative recommendations, all right?"}, {"start": 557.62, "end": 587.5, "text": "In fact, the European Parliament has suggested earlier that we should consider making AI systems legal persons like companies are. But my argument is that because of the nature of artifacts, because of the nature of control we have over them, following the kind of model that we just heard leads us to enormous moral hazards"}, {"start": 587.5, "end": 615.73, "text": "That could create the kinds of social disasters that we were hearing discussed earlier. So it's important, not that we can't possibly conceive. Now actually, I do have a PhD in artificial intelligence from MIT. I do actually build these systems too. And I actually am not convinced that we would suddenly be able to create this wonderful world of sentient machines that would be stable and live longer than humans."}, {"start": 615.73, "end": 645.58, "text": "I think technologically they'll just fall over. But secondly, I think, well, that doesn't make sense to strive for because all of our goals are about our own species. So our aesthetic and morality is about trying to keep this species of apes alive on this planet. So I worry about things that displace that. So I was making recommendations about the responsibility. And that's a point of view that I've heard often, that all our goals are about,"}, {"start": 645.58, "end": 670.37, "text": "Maintaining the status quo and keeping the species of apes on this planet. The only question I would ask you is if you'd rather keep the species of apes on this planet as it is. Because from my short view of human history, the human condition is pretty miserable and I would like to do everything possible to change it. I think the apes need to probably improve themselves a little bit."}, {"start": 670.37, "end": 681.36, "text": "We are. I think it's both the case that we can get great benefits from AI, including AI that's fully autonomous."}, {"start": 681.36, "end": 710.66, "text": "But also with some of the kind of economic and even military races that we've talked about, it's fine to set down ethical guidelines that says a human has to be in the loop, a human has to take responsibility. But there are lots of domains, including cyber hacking, probably including many military applications where autonomous, completely autonomous systems and systems that integrate all the information at a high level have a huge advantage."}, {"start": 710.66, "end": 740.45, "text": "In both speed and in their ability to understand what's going on. So, I mean, I kind of agree it's a dilemma that we should be very nervous about turning things over fully to machines, but we also have to be realistic about the fact that there are advantages. Now, my personal view of this and informed somewhat by my research is that we may be able to create a kind of alignment with human values."}, {"start": 740.45, "end": 743.22, "text": "By in the process of training these systems."}, {"start": 743.22, "end": 771.26, "text": "We make sure that humans are in the loop when the systems are being trained. That while we're teaching the systems what to do, we make sure that the strategies they pursue are strategies that humans would approve of and that that's where the responsibility lies. But that when we actually deploy the systems, they're going to need to operate very fast. And the particular actions they're going to take are not actions that we're going to be able to supervise, give the go-ahead to, or maybe even understand in detail."}, {"start": 771.26, "end": 799.25, "text": "Rephrase that and tell me if you agree. In many situations, you would support full autonomy with human oversight over training, not on actual action execution in the field. That is for cases where full autonomy has such big advantages that we can't really do without it. Sure, and it just so happens that just take one, war, which is a major human endeavor. We all strive to avoid it, but it happens."}, {"start": 799.25, "end": 823.18, "text": "It's a major human endeavor. There, it's a fait accompli. So the point is that somebody has to invest in the AI research to make those things happen. I think basically saying, look, never. This is just on a stone tablet. I've carved it and I've brought it down from a mountain. And no system will be autonomous. Don't think it'll work. Because not everybody is going to pay attention to your stone tablet."}, {"start": 823.18, "end": 843.5, "text": "I have two very specific examples to give, but you haven't had time. I wanted to bring Cathy in here. So you gave the example of the Cyber Grand Challenge, and I was in Vegas to witness that last year. It was maybe the weirdest show ever to grace a Vegas stage, like five liquid-cooled server racks. Anyway, seven, there you go."}, {"start": 844.11, "end": 867.46, "text": "That was interesting because, as you mentioned, those systems were entirely isolated. They made a big show of you could walk around and see that they were not attached to any external network. And so they operated entirely autonomously. But when that technology moves into practical application, they won't work like that, presumably. They'll be more of a kind of human-in-the-loop guidance. Do we know how that technology will be deployed?"}, {"start": 867.46, "end": 891.36, "text": "I don't think we know exactly. It hasn't happened yet. But I suspect your intuition that it's going to be a partnership between computers and people is accurate. The computer has the ability to be very comprehensive, to be able to look at lots more than a person can and apply the rules exactly correct and not make mistakes in certain ways. But the person has the intuition and the..."}, {"start": 891.36, "end": 917.09, "text": "That we don't yet know how to put into the computer. So it's also a domain where the interface between the human and the computer is likely to be relatively clear. We could build a system where the computer can come back and say, I think this is an interesting thing because of this and this. Can you look more? Where in other domains, the partnership between humans and computers is more difficult because how the computer sort of shows you what it's thinking and vice versa is much more challenging."}, {"start": 917.09, "end": 944.35, "text": "Can I give two very specific examples of the systems that we're talking about? People here probably that have better security clearance than I do can tell me whether this is true, but I've been told that the Iron Dome system still has a human in the loop that can intervene, and they are talking about very short missile strikes, and that they say that the reason they've kept that going is because there's been a bunch of cases where that has proved to be very important."}, {"start": 944.35, "end": 973.47, "text": "So there's an extent case there that I encourage you to go find out more about than I would be allowed to know. And the second case, which again is slightly secret, but secret in a different way, and there was a paper published about it recently, is how Facebook keeps its code base going. So Facebook switched to giving up on having weekly releases and instead just decided that every time a single programmer saw a problem in their giant code base, they could just go in and fix it."}, {"start": 973.47, "end": 1001.98, "text": "And when they go in and fix it, now you can think of this like, why are you talking about Facebook? It's a toy, right? But it's a massive, incredibly complicated software system that a billion people use, right? It's huge. And what they do is they run a test, they run a small test suite that the user has to use, and they roll it out to 1% of the possible users. And then they have constantly, not just the humans, who of course will complain,"}, {"start": 1001.98, "end": 1026.32, "text": "But also software systems that are watching for any kind of weird thing that might happen. And I would imagine that this is the kind of way that we're going to keep humans in control, is that we're going to design our systems around, first of all, making sure the core humans are empowered, know where to go if there's a problem, report it, that kind of thing. And secondly, that we'll have a whole ton of AI processes."}, {"start": 1026.32, "end": 1054.19, "text": "Basically, they're gatekeepers. They're watching the fences, the AI technological fences, and noticing when you exceed some kind of parameter. But that would be the way I would expect this kind of system to be working. Kathy, you previously ran, I think, a DARPA program that was thinking about how we build AI systems and trying to make it a bit easier and also more understandable. Is that right? Oh, right. So that's about probabilistic programming languages. So right now, a lot of machine learning techniques..."}, {"start": 1054.19, "end": 1084.21, "text": "Work kind of only from the data. They don't have a model that they're starting from. And there are some AI researchers who argue that if you have a model, that's bad because you're prejudicing the data. You're like, your model's going to be wrong, so why start with a system with a wrong model? The premise of the PAMEL system is that purely data-driven approaches are not going to be the only answer in the long run, that we have a lot of information that we already know, that we can codify in models, and we can"}, {"start": 1084.21, "end": 1108.74, "text": "Have a sort of a hybrid approach where you have both the model and the data and that that will then get you a better solution in the long run. Right now, like machine learning is having this huge heyday where there's tons of like once we develop this technique and once we had enough computer cycles and enough disk to have enough training data, there's lots of these kind of perception problems that machine learning and neural nets of various flavors turn out to be fantastic at. And that's kind of we're like picking off all those low hanging fruit areas right now."}, {"start": 1108.74, "end": 1119.79, "text": "But we're going to reach the end of that path. And I think that that's what was mentioned in the earlier presentation, how that kind of layer of the AI stack is at a mature level. And the program at DARPA was focused on"}, {"start": 1119.79, "end": 1149.66, "text": "Kind of helping to go to the next level where we're integrating a model that we know about with the data to be able to get much better answers faster. So an example of this is handwriting recognition. You can use the low level machine learning techniques with no models to do handwriting recognition. But you need tons and tons of data of different digits that people have drawn to be able to do that. And it has to be classified. Somebody has to give you labels to tell you what the right answers are. But instead, you can write down a very short model of how people draw numbers."}, {"start": 1149.66, "end": 1175.49, "text": "Two examples, or one example of each number, more or less, that fits the parameters of the model, and then you're done. So no training, essentially no training data. So that's an example of how merging a model in can make a big difference. And that's how AlphaGo Zero works, too, that they basically came up with a much better representation so that learning was faster. OK. So I mean, I definitely kind of think that this"}, {"start": 1175.49, "end": 1203.1, "text": "This approach of kind of take the perception system and kind of add our knowledge to it. I mean, I think it's a really reasonable thing to try. It could be the way that we end up going forward. But I think we should also be careful because there's kind of another direction that I think things could go. And we don't kind of know which one it's going to be yet, which is often when you're training these AI systems, often you can just take"}, {"start": 1203.1, "end": 1206.91, "text": "A whole bunch of different kind of sources of data and views on the world."}, {"start": 1206.91, "end": 1233.98, "text": "And just throw a massive amount of data and hardware at it and just kind of train it across several abilities. And it kind of learns to sort out everything on its own. So there have been a lot of good results on captioning images and soon captioning videos. So you'll show an AI system an image and it'll say, this is some children playing with a Frisbee or something like this. And that combines language skills."}, {"start": 1233.98, "end": 1260.13, "text": "Vision skills, very rudimentary reasoning skills, although we're not really that good at it yet. And it's an open question whether to really get the reasoning components in, we need new ideas, ideas from more traditional areas of AI. But we should also keep open and at least be ready for, in case it happens, the possibility that this seamless integration of everything is possible. Because if that's possible, I think we'll have systems that"}, {"start": 1260.13, "end": 1289.9, "text": "It's harder to understand what's going on inside them. Transparency will be harder. And it will be harder to get humans in a loop because when you have something like a vision captioning system, there's no discrete boundary between what are the objects in the image and how do I parse that into language. It's all just this kind of formless set of numbers that flow throughout it. And so if that ends up happening, we're going to have to think a lot about how to make these systems do what we want."}, {"start": 1289.9, "end": 1317.97, "text": "And the criticism that neural networks, connectionist systems are black boxes. There's a lot of work happening there. DARPA's funding explainable AI. There's a lot of companies doing work in that area. The issue is really on autonomy. There seems to be a very sort of broad view, and it's a far more nuanced conversation than I think a broad view can support. Let me give you an example. There's two specific engagements that I've been involved with."}, {"start": 1317.97, "end": 1340.22, "text": "One of them has been the development of an autonomous takeoff capability for a large aircraft. And one of the challenges as you're taking an aircraft off is you can experience a condition that's called rejected takeoff. And what that is is that at a high speed, say if you're in a large commercial airliner somewhere between 80 and 120 knots on the runway,"}, {"start": 1340.22, "end": 1366.24, "text": "You can see some kind of a system failure. And very quickly, looking at a large number of individual sensors in the cockpit, you have to make the decision whether you want to accelerate and try and use the remaining runway to actually take off or whether you want to stop. If you make that decision incorrectly, you can wreck the plane. You can either run out of runway or you can take the plane off and the plane isn't sustainable."}, {"start": 1366.24, "end": 1396.43, "text": "So it's a very, very life and death sort of a decision. And the other problem with this is that it doesn't happen, well thankfully, it doesn't happen too many times. So it's not the kind of thing that you have tons and tons of data available to train a run-of-the-mill type application on. So in order to build a system around that, we built a reinforcement learning system, which I'll keep all the details out, but the net-net I'll share with you. It cuts down two seconds."}, {"start": 1396.43, "end": 1421.65, "text": "Off of the RTO decision when compared with a large average collected across many expert pilots. Two seconds as a plane is speeding down the runway is the difference between a comfortable stop and death. And the quality of the decision in each case where the systems made that decision, with statistical measurements you can show that they are better than..."}, {"start": 1421.65, "end": 1453.26, "text": "Very qualified human pilots. I don't want to say best human pilot because I don't know what that is, what ranking system, but very, very good human pilots with dozens of years of experience, military experience, civilian experience. If you allow a human being to interfere with that, you lose all of the benefit of that system. The only thing that a human being will do in interfering with that is just get people killed. That's one example. And it's a practical example that is in place today."}, {"start": 1453.52, "end": 1483.3, "text": "I totally agree with that. I just don't agree that it makes the argument you want to make with it. I don't know that I want to make a very specific... There's no agenda. My point is I'm just making the observation that removing people from some loops is okay. At some granularity. So necessarily it's going to happen that, as we've just said, necessarily..."}, {"start": 1483.3, "end": 1511.89, "text": "And we aren't conscious of things that even our own bodies do faster than we can be aware of, right? So necessarily anything that the machines are going to be doing in that speed, we'll have to be thinking about in terms of primitives, and then we're going to divine our system. That's what we call, you know, just that you treat it atomically. You treat it as one thing. So we'll have to treat those kinds of skills atomically, and then we'll have to decide how do we coordinate, when do we instantiate and use those sorts of things. But that doesn't change."}, {"start": 1511.89, "end": 1527.89, "text": "Whether or not someone is actually sitting there pushing buttons or using a joystick, that's not the only form of human responsibility. It's about making sure that there's somebody who cares about their lives, that cares about their careers, that will lose their careers or their lives or something if they..."}, {"start": 1527.89, "end": 1553.17, "text": "Deploy a system that is not adequately tested. Sure. And that's what maintaining... It's a very longer discussion because one of the, obviously the inherent points that you're making is that the way to save lives is to put somebody in charge that cares about saving lives. And modern science would suggest that there are many other ways of saving people's lives. Also, it's not just that unless there is a person that cares about saving lives, in no situation can lives be saved."}, {"start": 1553.17, "end": 1582.1, "text": "In fact, that's at the heart of a lot of the debate around AI and a lot of the outcomes, the inventions that AI makes. Eurisco was able to produce a three-dimensional transistor that no human being had ever even dreamt of, and it became a huge contribution to the electronics industry, to humanity. Eurisco, I don't think if you asked it, would tell you that it cared about humanity, but it gave one of the most amazing gifts."}, {"start": 1582.1, "end": 1597.18, "text": "I am in no way challenged by the kinds of systems I build. I am perfectly happy that they do well and I take credit for the things that happen in my lab."}, {"start": 1597.18, "end": 1626.34, "text": "And if I was working in a company, in fact, when I did work in companies, I took credit and responsibility for the things that I built. The fact that it's AI does not change the credit and responsibility. And if we do change that, then we need to figure out a different system of justice. Okay. So I think that discussion illustrates how deep these questions are. They're really hard. But we've heard multiple times on the stage today that everyone's in a hurry, right? That there's pressure, there's competitive pressure between countries or companies. Everyone wants to get there."}, {"start": 1626.34, "end": 1637.25, "text": "First, I mean, Kathleen, you mentioned that you think an arms race in sort of intelligent in some way is inevitable and already happening, you think? I mean, is there evidence that that's the case?"}, {"start": 1637.25, "end": 1664.67, "text": "Well, I mean, there's certainly evidence. The whole cybersecurity space is an arms race, right, where you have hackers working to find exploits and you have companies kind of scrambling, trying to patch holes. I mean, the challenge there is that we have essentially, what, 50, 60 years of technical debt in terms of deployed software that was built with terrible security standards. I mean, built during a time where there wasn't a cyber threat, so there was no need to spend extra time or resources."}, {"start": 1664.67, "end": 1678.54, "text": "Bulletproofing it against insidiously clever hackers. Hackers are some of the most amazing people. The things that they can make a computer do, kind of against all odds, that's just shocking. It's hard to write a program that does the right thing."}, {"start": 1678.54, "end": 1695.31, "text": "When nobody's making it hard, like they just give you a blank screen and you build the program, it's really hard to get it right. Hackers are writing programs that are working under incredibly adversarial circumstances. They're working in an environment where somebody's trying desperately to not have them be able to be successful, and yet..."}, {"start": 1695.31, "end": 1724.59, "text": "Time and time again, they show that it's actually really pretty easy to break into software systems. I think we might be reaching a turning point in that as, well, for one, companies and governments are getting much more motivated in building software that is more secure because we're seeing huge financial losses and costs as a result of those vulnerabilities. And the science is reaching to the point where we actually now know how to build software to much higher levels of correctness standards. Like the work that I did in the HACMS program at DARPA, for example, engineers..."}, {"start": 1724.59, "end": 1750.98, "text": "We're able to build programs that were like 100,000 lines of code, so small potatoes for Microsoft Windows software, but actually like a reasonable size for a UAV or useful things that came along with a proof of memory safety. There are no buffer overflows. There are no buffer underflows. There are no integer overflows. There are no integer underflows. All of those things that hackers managed to miraculously convert into taking over control can't possibly be there."}, {"start": 1750.98, "end": 1778.05, "text": "And then what they're showing next is how you can take kind of exquisite pieces of software that have been proven to be fully functionally correct, not only do they not have memory problems, but they do exactly what they're supposed to according to a specification, and they don't do anything else, and leverage that. It's like the SCL4 microkernel, which was built out of NICTA or Data61 in Australia, is a microkernel that allows you to partition the programs running on a computer so that they can't interfere with each other. They provably can't interfere with each other."}, {"start": 1778.05, "end": 1788.94, "text": "Up to assumptions about the hardware being correct and things like that. There are always assumptions. And we're able to leverage that to build much more complex systems that hackers, red teams, haven't been able to break into."}, {"start": 1788.94, "end": 1817.62, "text": "The technical solutions are starting to be there, but this is kind of showing this arms race between people sort of trying to build software that is much more secure, harder to break into, with hackers who are insidiously clever and very, very motivated, either for profit or for nation-state kinds of reasons, so that the whole formulation is built as an arms race. It's already an arms race. And then you have AI technology coming along, and it's like, wow, this could basically give us our vitamin pills or super power boost if you're a superhero."}, {"start": 1817.62, "end": 1839.26, "text": "And make our system much more effective than the adversary system. Of course, that's something that the relevant players in that domain are going to jump all over. It may be that it doesn't work. It may be that like what we saw in Las Vegas at the Cyber Grand Challenge was a, you know, a carefully crafted artificial environment where we could sort of barely scrape by but still didn't do well against the humans a few days later, you know, the day after."}, {"start": 1839.26, "end": 1862.86, "text": "The question is, is it going to follow the curve of chess and go, or is it going to follow a completely different thing? Program synthesis, which is what this boils down to, is a super hard technical problem. And the search space is extremely complicated. You can't order programs in some way that you can kind of smoothly search through the space. It's hugely discontinuous. So it's a very hard problem. So it may totally not pan out. But absolutely, people are looking at it."}, {"start": 1862.86, "end": 1892.27, "text": "If China figures it out or if Russia or North Korea figures it out, we better figure it out too or we're gonna lose because everything is connected. If you have cyber dominance, cyber warfare dominance, then you can shut down everybody else's systems. If you have a nation state that controls the Northeast power grid, you can make it so Manhattan doesn't have power for three years. If you take out power for Manhattan in three years, you've changed the world in a really bad way for a really long time. So I'm not saying that that will happen, but I'm saying that we can't not"}, {"start": 1892.27, "end": 1920.27, "text": "We do our best to make sure that we're at that forefront as well so that our adversaries don't get to the point. I'm not saying, I don't want to scare people too much. I think it's too late, actually. I'm basically saying this is a really important battlefield. We have to compete and we have to win or at least tie. And I don't know if AI will really be the magic bullet here or not, but it looks like it might. And if it might, we have to really invest and find out and be at that forefront. Okay. Dario, so you're working on..."}, {"start": 1920.27, "end": 1948.1, "text": "Safety, you know, you want fewer unexpected surprises in the research lab. Now, so Kathleen describes what sounds like a pretty promising development in cybersecurity, which is that we can sort of formally verify certain pieces of software and know there are no bugs there or know how they'll behave. Like, can't we do that for the systems you're working on? I mean, that sounds like, you know, it would be great if you can somehow prove that they can't surprise you. Yeah, so, you know, I think whenever that's possible, it's a good thing."}, {"start": 1948.1, "end": 1974.74, "text": "And I think particularly when you have systems deployed in the real world, if there are very well-defined kind of entry and exit points, so this was true of the aircraft collision avoidance system, where you can say that subject to certain assumptions, we can verify that a certain thing is happening. I think that's possible, but I am concerned that that will be the minority of cases for situations where AI is used in the real world."}, {"start": 1974.74, "end": 1987.2, "text": "So in particular, if we look at the go-playing agents or the agents that play Dota or many video games or systems that do robotics in the real world."}, {"start": 1987.2, "end": 2014.88, "text": "To the extent that those are based on deep learning and reinforcement learning, it's very difficult to formally verify all but the very smallest amount of the smallest versions of those. And I suspect though I don't know that for fundamental computer science reasons related to algorithmic complexity that it may be the case that no such verification is possible. And so we're going to have to rely on a mix of empirical testing, making sure that"}, {"start": 2014.88, "end": 2044.13, "text": "The design and the training process of such systems points them in the right direction to various kinds of testing and simulation, to just making techniques that are more robust to things going wrong statistically. It would be great if we could formally verify everything, and we'll certainly want to do a lot of interrogation of what's going on inside the system, again, to the extent that we can do it. The question is whether that's possible, and if it's not, then we have to kind of rely on an arsenal of"}, {"start": 2044.13, "end": 2072.02, "text": "You know, tools that don't give us strong guarantees. I don't mean to say that I think it's impossible to verify everything. So absolutely, that's never going to happen. But I think what we can do is verify pieces and then use those sort of exquisite artifacts to build larger systems that have higher security guarantees kind of built on the exquisite artifacts. Even if parts of the boxes, the things that you fill in, are not verified at all or there's a much lower safety standard."}, {"start": 2072.02, "end": 2100.88, "text": "And a simple example is some processes, like you don't really care how it comes up with a decision. You just care about the quality of the decision. So you could ignore the entire process by which the decision was produced and only look at the question of evaluating how good a decision is this. And that could be a much smaller part of a system. And indeed, verification is part of a whole suite of things that we do to increase the confidence in our software systems. So yeah, absolutely. In no model will we ever verify all software. The cost is just prohibitively high."}, {"start": 2100.88, "end": 2126.77, "text": "Okay. And in other areas like architecture, for example, we guarantee safety or try and ensure it using legal frameworks and things like that. So you've been involved in the IEEE process? Yeah, the IEEE process. Can you summarize what that is? I really want to say very quickly the answer to the question you asked Kathleen before about everybody's pace."}, {"start": 2126.77, "end": 2151.97, "text": "I was just at an OECD meeting, the OECD meeting on AI policy in Paris, Thursday and Friday. And we're only talking right now about China and America. If you're looking at the rest of the world, there's a lot of people that are very concerned about the long-term stability. And actually, coming back to something you said earlier about that there's lots of forms of safety, it is actually incredibly natural to want to be in a world that is going to persist."}, {"start": 2151.97, "end": 2181.2, "text": "So I do think there are a lot of people thinking about how can we cooperate, how can we do things like Eric Schmidt mentioned about releasing the software and making sure all the nations are on a level playing field. So don't feel like there's only the arms race. I do think that this encryption stuff is incredibly important to maintain that kind of security. But there is a lot of people working on how are we going to stabilize societies and those other kinds of big questions about how AI will be affecting human life."}, {"start": 2181.2, "end": 2209.66, "text": "But the IEEE is, one of the problems with all this is that legislators can't keep up with the technology fast enough. And so one of the models that's been used in other domains is to have professional organizations come up with, for example, ethical standards, and then have the legislatures and basically the governments work to enforce what the professionals, and have the professional organizations maintain the bleeding edge and try to keep up that way."}, {"start": 2209.66, "end": 2235.22, "text": "And so IEEE is putting forward, they have about, I think, 12 parallel standards, including the one I'm involved in is 7001, which is transparency for AI systems. And transparency in this case doesn't just mean open access and open code. That's neither necessary nor sufficient. It means being able to be understandable, so understandable by lawyers, understandable by developers, understandable by users."}, {"start": 2235.22, "end": 2251.98, "text": "Understandable by professionals. So that's the kinds of things that we're worrying about. Okay. And I think now we have some time for questions from the floor. So the microphones will be roving around. Do we have any questions? One right here."}, {"start": 2255.25, "end": 2284.34, "text": "Hi, I'm Matt Chuston from the Department of State. So I want to ask a question about cognitive security, which I see as sort of a subset of cybersecurity. So you have all these AI systems now, chatbots, affective computing, dynamic content creation, psychometric profiling, that have a lot of potential to influence human beings or actually control their behavior. When you look at what AIs are going to start doing and how they're going to start generating content that actually shapes our culture and shapes the way that we think,"}, {"start": 2284.34, "end": 2308.02, "text": "How do we actually introduce safety and security into that so that those AI systems, when they generate that content, are actually promoting principles like equality and liberty that sort of backstop our civilization as opposed to other models of political systems that we may not favor? That's a very interesting question. It's been interesting to me for a long time. In fact, my book that's coming out in..."}, {"start": 2308.02, "end": 2329.97, "text": "A couple of weeks now, it's called The Sentient Machine. There's an entire chapter in it that's called Mind Hacking, which starts off with the question that you've asked, walks through the entire scenario, and gives sort of a proposed answer. Not to suggest that at this stage there's a perfect answer, but first of all, what's happening now is that let's just take a scenario."}, {"start": 2330.03, "end": 2339.01, "text": "So you are a nation state and you're wanting to undermine, say, the science and technology capabilities in a very particular area in a different nation state."}, {"start": 2339.01, "end": 2365.28, "text": "You can start off just with a few names of known researchers. And now there's enough data streams and enough spidering frameworks and so on for those names to then lead to hundreds of other names that are connected to these individuals. And using sort of these graph relationships between those names and applying known graph theory algorithms, you can also figure out what the rough relationship between these names must be."}, {"start": 2365.28, "end": 2384.16, "text": "Closeness and therefore even machine-learned patterns on graphs can imply who this person is and so on. Once you do that, you can start to collect profiles from social media posts that they might have done, even writing. And there's this psychographic model that's called Ocean."}, {"start": 2384.16, "end": 2413.17, "text": "And it can take a small number of words, 66 Facebook likes and a few dozen words, to be able to build a complete psychographic profile on an individual. Once you do that, and this is all automatic, you just gave the names, the ANI is doing everything else, this is all doable now. Taking those psychographic profiles now, you can start to generate, through natural language generation technology, messages that are designed to"}, {"start": 2413.49, "end": 2438.27, "text": "Transform, Opinion, Shift, Shape, and so on and so forth. Part of this we saw, if the whole Facebook story is true, they did it in a less sophisticated way because they just took the demographics available through the Facebook interface and just advertised certain pre-made messages. The messages weren't being generated dynamically as you would have through NLG. And all of that runs through and then..."}, {"start": 2438.27, "end": 2464.94, "text": "You can actually also combine that with automated cyber capabilities. Because once you get to somebody's Facebook, maybe you create a fake account. Maybe you now, looking at all of their past history, are able to shape messages that get them to accept you. Now you have their email address. When you have their email address, you can start to use tricks and phishing and so on. Let's say you get into their email address. So far, a human hasn't touched this pipeline."}, {"start": 2464.94, "end": 2490.98, "text": "You can start to extract and build models from that email and very soon you will have a sequence of automated threats that you will be able to use against that individual or their family or people that are close to them. So this scenario is totally not unrealistic and some versions of the scenario are in place now. They're being used for different things, but if you were to use this at a nation-state level,"}, {"start": 2490.98, "end": 2519.66, "text": "To really shape opinion in very targeted communities, you could do some pretty insane things. How do we stop this? We'll stop you there because I want to make sure there's time for other questions from the floor. Some of the other panelists want to jump in. All of them, in fact. All right. Yeah, let me agree. It's a really important thing. It is something people are talking about. It is a major problem with the idea of humans on the loop or whatever because what if they're being manipulated?"}, {"start": 2519.66, "end": 2548.83, "text": "And I want to say that I think while we in China are worrying about the things we're worrying about, this is what the EU is worrying about a lot. And maybe still not enough. And actually going back to something you said very earlier, much earlier about that cyber warfare is going to be like airplanes, like fundamentally changing the nature of warfare. I hesitate to..."}, {"start": 2548.83, "end": 2574.78, "text": "OK, I'll just do it. One of the definitions of warfare is having your infrastructures destroyed. And so that might happen without bombs. And you might actually convince people to destroy the infrastructure of their own nation states. So for example, you might convince Hungary to get rid of the CEU, even though it's one of their best higher educational system."}, {"start": 2574.78, "end": 2602.93, "text": "Or you might convince a nation in the EU to leave the EU, and it might be one of the two military powers within the EU. So it's possible that you could say we already are, this kind of instability that we're seeing now tends to correlate with inequality. It doesn't correlate with the amount of AI, but we had exactly this kind of situation in World War I, and we might be in the equivalent of World War I already."}, {"start": 2602.99, "end": 2630.02, "text": "Yeah, so I was at a workshop actually on Friday that was talking about exactly this kind of question. And one of the people who was there, Dana Boyd, was talking about how, for a variety of reasons, society is in a place where many individuals' sense of identity is threatened. And one of the consequences of having a threatened sense of identity is that you're susceptible to basically brainwashing, right? That's when a group brainwashes you, they take you, they deconstruct your identity, and then they build you a new one. So we're in a situation now where people are kind of..."}, {"start": 2630.02, "end": 2643.39, "text": "Many people have a weakened sense of self-identity and then they can go online and kind of on their own find the brainwashing material to become brainwashed in a particular way. So the shooter in North Carolina kind of had this..."}, {"start": 2643.39, "end": 2672.5, "text": "His sense of being a white male was threatened. He went online about reading about the Trayvon Martin case and didn't find the sort of mainstream narrative fit his understanding of the way world works. He found the sort of alternative media, liked that definition instead, kind of dove down and really deeply became a believer in that philosophy. So that's like one individual case, you know, tragic for the people involved. But if an adversary can cause that to happen at scale, you have a potential of like decentralized"}, {"start": 2672.5, "end": 2693.74, "text": "Widespread brainwashing of creating an army of people. And with the social media connections, the adversaries can find those people and can target the material. They can create synthetic materials. Now we have technology to be able to take a person's writing style and generate material, kind of combine a semantic model with their writing material to produce tweets, small..."}, {"start": 2693.74, "end": 2721.52, "text": "Small pieces of text that sound like that person. We have the technology to take written text and make it sound like that person if you have training data for what that person normally sounds like. So you can make audio that sounds like this person said something even though they didn't. You can take headshot videos and make that look like the face actually says what you want it to say instead of what the face... Those are still the place where we can detect the face, the audio. The examples here are Obama in press conferences talking."}, {"start": 2721.52, "end": 2742.03, "text": "And you can still detect it if you're paying attention. But the AI technology is likely, in the future, will be able to do this in a case that would be pretty hard to detect. So you can create tons of synthetic materials that pass the smell test to susceptible populations. So I think this cognitive hacking is actually a huge problem. And in a society that values"}, {"start": 2742.03, "end": 2771.71, "text": "Not censorship, open press. It's really hard to figure out what do you do about this in a way that doesn't sort of compromise our society to the point where we lost anyway. Okay, Daria. Google doesn't spread that in real time, in real time at the OECD. We can now do that video manipulation and speech stream interference with a live broadcast. So the technology on this is moving super fast. So the technical name for the area of research is generative models, and we've done it for speech."}, {"start": 2771.71, "end": 2798.99, "text": "We've done it for video and I think it's even a little bit already out of date that it's just short snippets of text. We're very recently starting to make the snippets of text longer that you can make whole paragraphs that make sense that look roughly like they were written by a human. And I think the equilibrium of this is that we're going to be able to make things that there's really no way just by looking at them without having some context or authentication to distinguish."}, {"start": 2798.99, "end": 2820.8, "text": "Between whether these were made by a human or not by a human. There was a paper from NVIDIA just two weeks ago basically looking at a data set of faces and making synthetic faces. Basically they looked at a huge data set of celebrity faces and they made a bunch of new celebrity faces of people who don't exist that were indistinguishable to a lot of humans from..."}, {"start": 2820.8, "end": 2843.66, "text": "Some of the existing celebrity faces. And I think in a couple years, we're going to be able to do that for video as well. We can already kind of do it for speech, where we can synthesize speech that really sounds like a person's speech. On a lot of these, we're kind of just below human parity, but I think we're going to zoom past it. Can you use the same technology, you know, effectively in reverse to spot the fakes?"}, {"start": 2843.66, "end": 2868.24, "text": "So you can, but I think the equilibrium is that you're not going to be able to tell the difference, right? It's like if you have a money counterfeiter and the mint, and they're fighting against each other, in the end the counterfeiter wins. Okay, right. Let's move on and have some more questions from the floor. There's one in the back there, I think. Hi. I'd like to follow up on that question just from a..."}, {"start": 2868.24, "end": 2894.38, "text": "The other side of it. So we've been talking about it from the threat side. But what about the risks of sort of benign manipulation? So we say that we want to use AI to improve decision making. What are the risks that we end up just creating artificial sycophants that indulge our own cognitive biases and tell us the things that we want to hear, a la sort of like the Facebook news feed bubble? Facebook isn't designed to give you the best news. It's designed to give you the news that you like."}, {"start": 2894.38, "end": 2925.09, "text": "How does that figure into design? What are the risks in that area of inadvertent psychological manipulation? Kathy? One thing that comes to mind is the work being done at CMU on bias in the data that is being trained. There's a lot of, oh, an AI system said this, so it must be right. It's like, well, no, an AI system said that, so its model based on the training data said that that was the most likely outcome."}, {"start": 2925.09, "end": 2930.61, "text": "There's a lot of research that shows that if your data is biased, your outcomes are going to be biased."}, {"start": 2930.61, "end": 2960.29, "text": "People are working on trying to figure out, like, measuring, is it biased? And if it is, like, what do you do about the fact that an algorithm might be biased? This is used for things like, you know, job screening for candidates for job positions or loan applications and things like this where the data, you know, skews along racial lines. Actually, I was one of the three authors of the paper that was in Science that showed that if you just use the English language web, you will get the same implicit biases that Americans show."}, {"start": 2960.29, "end": 2992.35, "text": "For racism and sexism. And also that your sexist attitudes about who's a programmer and who's a nurse or whatever have a 90% correlation with the actual proportion of women in those jobs. Oh, interesting. I think we've got time for one more question. If there's one out there. One in the front here. Thanks. So the panelists have talked a lot about potential..."}, {"start": 2992.35, "end": 3017.89, "text": "We talked about risks of unintended outcomes and safety and talked a little bit about formal verification as a way to address that. But I know that there's also other work that you're all doing to make progress on that. So I was wondering if Dario, who works in OpenAI and also other panelists could talk a little bit about the kind of work that they are doing and maybe that they plan to do in the future. Sure. So a lot of the work that I and the people on my team are currently"}, {"start": 3017.89, "end": 3046.11, "text": "One way you could describe it is a thing I said earlier, which is that while there are some contexts where a system has to act so fast that a human can't supervise it or even be helpful in what it does, that can we still design the process by which the system is trained from scratch, such that the things that it would do are things that humans would approve of? And so in particular, we take reinforcement learning, which is this standard method that's been talked about."}, {"start": 3046.11, "end": 3057.49, "text": "Several times here that is used to train AlphaGo and the Dota bot and a lot of things in robotics. Usually that's given kind of a fixed reward function, a fixed criterion for success."}, {"start": 3057.49, "end": 3083.36, "text": "And we've developed a method where that reward function can be developed over time by the machine using feedback from humans. So the machine acts. It presents some examples of its behavior to the human. The human tells it which behaviors are better. And then it incorporates that into its training loop. And so during the training process, the human and the machine act in concert. And the machine begins to understand what the human wants and then to act on what the human wants."}, {"start": 3083.36, "end": 3112.67, "text": "Then at actual test time, at deployment time, when you would actually use the system, the system can act completely autonomously, but it contains within itself the knowledge of what it is that the human wants it to do. That may not be enough. That may not be the only thing that we want to do. But it's one solution for cases where a system has to act so quickly or just knows things so much better than humans that we can't supervise it while it's acting. Maybe we can supervise its process of creation."}, {"start": 3112.67, "end": 3130.88, "text": "Okay, we're sitting directly between the audience and their lunch, so maybe we could, the remaining panelists, just give us a quick sound bite on what you're working on next in this area. Yeah, well, actually the reason I got into artificial intelligence in the first place is I'm really interested in the laws of nature as they pertain to cognition."}, {"start": 3130.88, "end": 3149.02, "text": "And so that we can understand how even in nature you see different solutions and strategies with respect to cognition. And so I'm actually trying to work on, and that led me into working on public goods because exchanging information like this is a public good. And now I'm working on political polarization and why it's correlated with economic inequality."}, {"start": 3149.02, "end": 3178.4, "text": "However, in order to do this stuff, I did a PhD in systems AI. So how do you put together an AI system out of the box is exactly what Andrew said is now the big thing. I'm glad to know my PhD is now coming in finally. But my PhD students and I have been with that trying to work on the transparency question. So how we can get people over this hurdle of over-identifying with artificial intelligence. And we've shown that just exposing even the basic real-time development tools and debugging tools"}, {"start": 3178.4, "end": 3207.06, "text": "The same things that help developers do actually help ordinary users. Okay. I mean, so you're in a different environment to these researchers. Absolutely. So our work, I mean, we're doing several areas of work in safety, but primarily we're looking at swarm systems, and we're looking to produce basically these shared stores of the world, a shared worldview, if you will, amongst all the members of the swarm. And the active members of the swarm, in some sense, are a democracy."}, {"start": 3207.06, "end": 3235.76, "text": "They are able to judge themselves, their performance, and the performance of others. And in a very simplistic level, that metric globally, and it can't be messed with. It's sort of like a blockchain-like phenomenon. So you would have to take out more than half the swarm to affect the global view. And in doing this, then, you have self-enforcement. So if any one of those elements finds that something's going awry, it can stop that. And Kathleen, what do you think?"}, {"start": 3235.76, "end": 3259.57, "text": "I would say something I'm really interested in is the computational ethics systems. So when you build a system, I think you're not going to be able to anticipate in advance all of the difficult decisions that an automated system will have to make or an AI-enabled system will have to make. And I don't think you'll be able to, in training, expose it to all of the circumstances that it will have to make. So I don't think the pure reinforcement learning approach will be sufficient."}, {"start": 3259.57, "end": 3287.3, "text": "People don't have a, you know, we have ethical systems for a reason. I think we'll need to have a similar kind of ethical system for our AI system. So I think we do need to use the reinforcement learning to produce kind of human-like values to the extent we can, but I think we'll need to also imbue it with an ethical reasoning framework that allows it, when it gets into a sticky situation, to like, okay, I'm going to do this, here's why I'm going to do this, or produce kind of a log that can be audited downstream so that later on, you know, humans can say, oh yeah, I made totally the right decision, or, you know,"}, {"start": 3287.3, "end": 3303.57, "text": "I can see why it made that one. I probably would have made the other, but they're both reasonable. Or no, we need to go and change either how it was trained or the data or the ethical framework so in the future that system or ones like it would do something very different. Okay. Well, I hope we can all just continue the discussion during lunch. Please thank our panel."}]}