{"url": "https://www.youtube.com/watch?v=_5wEF6oyT9E", "title": "AI Policy Panel - Artificial Intelligence and Global Security Summit", "show_or_channel": "Center for a New American Security (CNAS)", "date": "2017-11-16", "duration_s": 3311, "type": "panel", "confidence": "high", "notes": "CNAS AI Policy Panel with Crootof, Parker, Vignard; same summit as yGmvaOast_k", "src": "y3", "id": "2017-11-16_center-for-a-new-american-ai-policy-panel-artificial-intelligenc", "audio_duration_s": 3309.056, "worker": "jc-w10", "transcribe_s": 135.7, "model": "large-v3", "segments": [{"start": 0.05, "end": 21.78, "text": "My name is Phil Stewart. I'm the military affairs and intelligence correspondent at Reuters. In addition to Jack and Rebecca, I'd like to introduce Kirsten Vinyard, who is the deputy director at the United Nations Institute for Disarmament Research and has been deeply involved in international discussions on AI and on cyber. And also, I'd like to introduce Dr. Lynn Parker."}, {"start": 21.78, "end": 43.01, "text": "who is an AI researcher at the University of Tennessee, and helped lead the development of the Obama administration's national AI R&D strategic plan. So, actually, I'd like to start off by referencing the first panel we had this morning with Eric Schmidt, where he spoke kind of alarmingly about how the United States could lose"}, {"start": 43.01, "end": 72.27, "text": "It's competitive advantage when it comes to AI that China was on track to challenge the U.S. dominance. And I wanted to ask, Lin, if you could just, from your position, having seen the U.S. government's approach to this, is Congress, is the White House, is this administration or the last prioritizing AI the way it needs to be prioritized? So first, I'm speaking for myself. The answer is no. Would you like me to elaborate? Yes."}, {"start": 72.27, "end": 88.83, "text": "No, I don't think we have a sense of urgency. I think there are reasons for that lack of urgency. Certainly, if you look historically, say the last 50 years or so, we have had the great pleasure of living in a country where"}, {"start": 88.83, "end": 113.78, "text": "Many of the technological advances are coming from this country. We have invested in research and development for 50 years or more. And we have benefited from the investments in research and development. Now we have very strong economic high-tech companies that are benefiting from decades of investment in basic research. I think we've forgotten that. I think we've forgotten our history."}, {"start": 113.78, "end": 127.79, "text": "We've forgotten that where we are today as the world leaders in economic development and other areas is because we have been investing in R&D for years and years. And since we're at, you know, we're talking about America."}, {"start": 127.79, "end": 155.78, "text": "If we agree that we want America to be the best in the world in these areas, I think what it takes is a recognition that we got here on the shoulders of many people who worked very hard and who recognize that it's the research and the advancements in science and technology and engineering that give us all these great benefits. And so the question is, why is that today that we've lost our sense of urgency? So I'm a computer scientist and an engineer."}, {"start": 155.78, "end": 182.72, "text": "I would argue that I think as a culture we've lost our value in recognizing the importance of science and technology and engineering. I think about my dad's generation. My dad was an engineer and most of his friends were engineers and I don't think it was just because they were engineers that they were friends. I think it used to be that many people studied engineering and science as a matter of their course of living."}, {"start": 182.72, "end": 212.02, "text": "I think I'm correct to say that most of the Politburo in China is composed of scientists and engineers. So it's not surprising that they recognize the importance of science and technology, including AI, because they are scientists and engineers by and large, plus economists and so forth. And so I think we as a nation, in some sense, don't understand the technology. And I think that has led to some of these fears that you see in the media and in movies and so forth is really a fear that"}, {"start": 212.02, "end": 237.79, "text": "These technologies are going to do something that they aren't going to do. And maybe if we understood them better, we wouldn't be quite so afraid. So I think there are a lot of different reasons as to why we're at the situation where we don't really have a sense of urgency. But I do think in looking historically at how we got to where we are today, to me, it's investments in the R&D that have led to..."}, {"start": 237.79, "end": 264.29, "text": "The position that we're in and I think because the technologies today particularly for AI and the charts that the examples that Jack just showed and others earlier today showed about how quickly this technology is advancing that I think we're going to be surprised and recognizing that today's technology is not the same as what the engineering advances are were in the past that perhaps took a little bit longer."}, {"start": 264.29, "end": 293.86, "text": "In summary, I think we don't have that sense of urgency, largely because I think as a nation we don't really understand collectively what this means. Jack, you spoke about how in your gut you thought that China understood this. What do you think the implications are? If the U.S. isn't prioritizing this, China in its gut does kind of have a sense of its importance. What are the implications of that? And then also, what role should the U.S. government have"}, {"start": 293.86, "end": 322.22, "text": "And maybe shouldn't have in prioritizing investment in this area. So two things. One, in the case of what happens to China, I think a number of good things are going to happen to China. You're going to see cheaper medical technology. You're going to see better healthcare mediated via AI systems. And you're obviously going to see a number of applications in the kind of security domain. It's going to change the character"}, {"start": 322.22, "end": 349.14, "text": "What should the government do?"}, {"start": 349.14, "end": 376.67, "text": "The reason why I talk about measurement is why don't we care about it? I think it's because we're in a plane, and most of the dials are completely missing. We have no idea how fast the plane is going, the trajectory it's on, and our legislators will see occasional examples like AlphaGo and like other things, and they'll find that exciting for a moment, but they're not going to be hit with a steady drumbeat of, this happened this week, this happened this week."}, {"start": 376.67, "end": 406.93, "text": "The progress of drone automation technology just over the public papers that have come out in the last two years, you would find it exciting and hair-raising in equal measure. And if you don't do that, you're not going to get the sense on the government side that this is a matter of urgency. I don't think government should yet regulate on things like bans of technologies or theorize about far-future technologies. Some people have spoken about superintelligence. I think that would be premature."}, {"start": 406.93, "end": 435.14, "text": "But I think government should at least be tracking what the technology can and can't do, because otherwise also a more pernicious effect is the people that track it will be outside of government, and as AI becomes more and more important, those people will have motivations about how they present the data. And it's very dangerous if the government is solely informed by people outside of itself, because then it's not going to get things which are also germane to its own interests. Well, I mean, Kirsten, on that point, you know, right now,"}, {"start": 435.14, "end": 463.62, "text": "Governments around the world with varying degrees of information are engaged in this discussion about what to do about AI. And you're going to be in a meeting in a couple weeks, I think, that's going to look at this issue head on. So if you could talk to us a little bit about that meeting. And before you do, I wanted to read, there was an open letter signed by various people. One of them was Elon Musk. I'm just going to read one clause in this letter. It said, lethal autonomous weapons threaten to become the third revolution in warfare."}, {"start": 463.62, "end": 492.77, "text": "Once developed, they will permit armed conflict to be fought at a scale greater than ever and at time scales faster than humans can comprehend. These are weapons of terror, weapons that despots and terrorists can use against innocent populations, and weapons hacked to behave in undesirable ways. We do not have long to act, etc. So could you talk a little bit about what this meeting will be? This was an open letter to inform that meeting. There was also a food for thought paper. Give us a sense of what's going to happen there and what should..."}, {"start": 492.77, "end": 521.87, "text": "All right. Well, the meeting, of course, you're referring to is the meeting of the group of governmental experts that was founded or established by the Convention on Certain Conventional Weapons who have been discussing lethal autonomous weapon systems in informal meetings of experts since 2014. And so they've been meeting one week a year since 2014 to discuss developments in autonomy and weapon systems. So, spoiler alert."}, {"start": 521.87, "end": 549.71, "text": "Autonomous weapon systems will not be banned two weeks from now in Geneva. So I hate to break it to you, but that's the case. There's a lot of expectations, though, for this meeting. And a lot of states expected and have been preparing for, based on the mandate for this group of governmental experts, that they would be discussing definitions, because that's one of the issues right now. We have a lot of policy responses on the table. There should be a ban. There should be a moratorium. We should do nothing. We should wait and see."}, {"start": 549.71, "end": 579.65, "text": "There's a lot of what are we talking about conversation still needs to happen. And some states believe that the best way to do that would be first to come up with a definition, right? So states have been preparing for that. There's also a sense of urgency as evidenced by the open letter. And there's some frustration because there were supposed to be two weeks of meetings this year. And for financial reasons, due to the state of the treaty and the contributions to the treaty, have reduced that to a five-day meeting."}, {"start": 579.65, "end": 604.14, "text": "of the meeting, which is Ambassador Amandeep Gill of India, put out his Food for Thought paper, which is something that chairmen and chairwomen often do in preparation for the meeting. And that paper has raised a lot of questions. In fact, it's three pages of questions, and it's three pages of questions on AI. And that has shaken some expectations about the meeting in the sense that"}, {"start": 604.14, "end": 633.79, "text": "He wanted to focus on a thorough review of the current state of developments in the technology domain and their instantiation, I can't even say that word, in military systems, right? This Food for Thought paper has generated a lot of talk. It's generated some criticism publicly by groups like the campaign to ban killer robots. Obviously, they have a strong policy position."}, {"start": 633.79, "end": 660.96, "text": "Questions from states as well about if this is useful. Essentially, there's two criticisms. The paper is too narrow. It's left out many of the issues that have been central to the discussions up until now, which include human rights issues, issues of proliferation, issues of human control, which has been a big theme for the last three years about meaningful human control, appropriate levels of human judgment, depending on what community and practice you come from."}, {"start": 660.96, "end": 689.36, "text": "And those things are absent from the Food for Thought paper. Other people say it's too broad, that we've gone way meta-level on AGI-related issues, and rather than sticking to perhaps the more narrow, nearer-term applications of AI in weapons systems. So it's not real sure what's going to happen in two weeks' time, so watch this space. Well, Rebecca, I mean, just going to you, could you..."}, {"start": 689.36, "end": 716.5, "text": "As someone who looks at the legal aspects, international legal aspects of autonomous weapons and the discussions that are happening in Geneva and elsewhere, is there any kind of legal consensus on any aspect of this? Is there anything that could help guide these discussions as they stand right now? Because it seems to me, based on discussions we've heard here today, there's a lot of debate of whether there's even going to be or there is an arms race that's going to..."}, {"start": 716.5, "end": 748.06, "text": "We heard this morning there wouldn't be or there isn't any time soon. Later in the day we heard there might be. Where are we right now? I would say there is a huge risk that the discussion of regulation of autonomous weapon systems is going to focus on autonomous weapon systems with, as the UK defines it, a human cognitive level of judgment."}, {"start": 748.06, "end": 769.36, "text": "of some futuristic weapon system that may actually be banned because it's defined out of existence. Or there might be a ban of a weapon system that nobody actually wants to develop. And I'm really concerned that a lot of energy in this space is getting poured into"}, {"start": 769.36, "end": 798.11, "text": "Narrowing in on exactly what should be banned at the expense of talking about what should be regulated. And what are the issues we're concerned about right now? What humanitarian protections are being undermined by weapon systems that exist right now, that are being deployed right now with increasing levels of autonomy and autonomous capabilities? What military systems could be made better by different weapon systems with increasing autonomous capabilities? And so I'm concerned."}, {"start": 798.11, "end": 826.78, "text": "As a legal matter, the focus on the ban is the solution to one specific problem that's not sucking all the oxygen out of the room for talking about other problems raised by autonomous weapon systems, or depending on how you define them, weapons systems with varying levels of autonomy. And just on the legal aspects, I mean, is there, when it comes to accountability, you know, how does current law, you look at U.S. law,"}, {"start": 826.78, "end": 849.2, "text": "On weapons manufacturers, how might that inform how policymakers look at the issue of autonomous weapons? So domestically, there's a lot of tort law and also tort exceptions for weapons manufacturers. At the international level, there's not a lot of accountability for accidents."}, {"start": 849.2, "end": 869.52, "text": "The more you have automated systems and systems with high levels of autonomy on the battlefield or engaged in decision-making trees, the more you're going to have accidents that arise."}, {"start": 869.78, "end": 887.44, "text": "As a result of problematic human machine interactions, you're going to have skill fade, right? You're going to have situations where humans are not able to judge when best to jump in. And this is going to result in accidents in wartime, which tend to be fatal."}, {"start": 887.44, "end": 909.78, "text": "And when that happens, the existing system of accountability in international law is primarily international criminal law, holding individuals accountable for willfully committing war crimes or recklessly committing war crimes. There's not any real individual accountability for negligence, at least in criminal law. I think that's a good thing. I think negligence..."}, {"start": 909.78, "end": 939.44, "text": "It tends to be a balancing standard that's not something you want to introduce into criminal law. And so I've made the argument that we should have something called war torts, where states are held strictly liable for accidents that occur in war, particularly in result of the use of autonomous weapon systems, as a way of limiting the use of them, recognizing their utility. Just like in domestic law, we recognize that many ultra-hazardous activities have benefits. Dynamite has beneficial uses in construction."}, {"start": 939.44, "end": 965.04, "text": "But there are certain accidents that no matter what you do, no matter what precautions you take, you can't protect against. And in those cases, the victims deserve compensation, and there needs to be some limiting factor on how often that kind of technology is used. A couple more questions before we open it up to the audience. Jack, you were talking before in your presentation, you used one word that kind of struck me, which was governments. You're looking at governments. And so a lot of people here..."}, {"start": 965.04, "end": 982.37, "text": "You know, with an eye shot at Congress or thinking about this government. But you're looking at it more globally and regulations that could be imposed on industry. And what I'm wondering is, given the very different take that countries like China impose on, the different kind of regulations that countries like China impose on industry,"}, {"start": 982.37, "end": 998.67, "text": "How do you think industry is going to navigate this? I mean, the U.S. is obviously very concerned about losing its dominance while China is able to kind of extract quite a lot of information from companies that operate there. How do you think industry is going to navigate this space in the coming years?"}, {"start": 998.67, "end": 1026.75, "text": "And by navigate, what do you mean precisely? Well, navigate the different patchwork of regulations that arise. Is it going to be a battle in each corner? I mean, I think some of it really is a definitional thing. And we heard about the meeting which is happening at the UN and how some of that letter is a series of questions about the technology because there is a lack of information about the technology and a lack of shared understanding in governments, plural, around how it works. So I think..."}, {"start": 1026.75, "end": 1054.77, "text": "On one level, you want to have governments who are sort of communally measuring different aspects and communicating that to each other so you have a shared discourse. As to how you navigate regulations, I think that it will depend very much on the culture that implements the AI system. You can imagine that in the US there will be regulations around things like bias. These regulations may not exist in other countries. I don't have a solid answer for you because I think at the moment we're..."}, {"start": 1054.77, "end": 1081.78, "text": "Going to deal with a series of definitional problems so large that it will be difficult to even sell in countries which are that different sort of in their economic structure to your own. Have you followed all these conversations that are going on in Washington about CFIUS, whether CFIUS might need to be reformed in order to keep AI in the United States? What are your thoughts on that? It risks mistaking the sort of..."}, {"start": 1081.78, "end": 1109.33, "text": "There's a risk here which is inherent to the generality of the technology. And I'll give you a kind of motivating example. If I have a video analytics company that does personalized analytics for footage that people take on their consumer drones so that I can automatically identify my family members from overhead and so that I can kind of page through my memories and be like, there's me and my mom and spot the dog. That's nice. This technology is a general..."}, {"start": 1109.33, "end": 1138.58, "text": "Thank you very much."}, {"start": 1138.58, "end": 1166.96, "text": "Foreign actors, possible sort of interference. And that feels so broad to me that it will cause problems in terms of economic competitiveness that will outweigh the sort of benefits you might get from such a broad piece of legislation. So again, this is where I think it comes down to definitional questions. I think the lack of an understanding of exactly what this technology does and what its capabilities are is leading us into a path of very general regulations motivated out of"}, {"start": 1166.96, "end": 1197.06, "text": "To some extent, a sense of fear and also a sense that we haven't got a handle on the progress in our own country. Well, fascinating discussion. I'd like to open it up to the audience for questions. We've got a great panel, so I'll get as much time in as I can. Yep, over here. Hi, this question is for all the panelists, but maybe especially for Jack as someone who's close to the AI research itself, is how should governments, regulators..."}, {"start": 1197.06, "end": 1226.53, "text": "Multilateral bodies go about interacting with AI researchers themselves as they're developing these policies. I think I've heard some hesitation from AI researchers in, for example, some of the European regulations that are coming through and how sort of applicable and actionable those will be. So I'd be curious for your thoughts. I mean, I'd like to get lists of what people are worried about earlier. I have had discussions with people where they say, here are issues we are concerned about. It is like very, very..."}, {"start": 1226.53, "end": 1253.84, "text": "Many periods of time away from doing any actions on it, but we're currently trying to figure it out. And then that seems like an easy conversation to actually bring the technology community into because it doesn't have this implication that you're then going to do something, which is where you can have these issues of ideological differences among the engineers and researchers that work on this technology. So I'd say sort of talk early and talk often and also try and talk a bit more openly so that you don't treat these..."}, {"start": 1253.84, "end": 1272.74, "text": "These participants are like a black box, which is how people sort of view government sometimes. So I would just throw in on that, on the Musk letter that you mentioned. It was really commonly portrayed in the media as Elon Musk calls for a ban on killer robots. And that's not what it said at all."}, {"start": 1272.74, "end": 1302.32, "text": "The word ban is nowhere in that letter, and Elon Musk, of course, is just one of the signatories. What it was calling for was it was saying, we are technology companies who have something to contribute to this discussion. We need regulations, right? We need to address the problems these systems can cause. And it was sort of a call for saying, let us be involved in helping to craft those regulations. And so I was very disappointed with much of the media coverage for not acknowledging that."}, {"start": 1302.32, "end": 1312.21, "text": "The sort of takeaway seems to have been like this letter is over and done with, right, as opposed to it's the saying, you know, let us be involved. They should be part of the process."}, {"start": 1312.37, "end": 1341.97, "text": "I think there's another level of governance in some sense about the connection with the research and development and its societal impact and some of these negative consequences. And certainly that's the researchers themselves governing themselves. So professional societies such as IEEE have started activities to try to help the R&D community itself decide, for instance, what's ethical as it relates to automation, robotics, AI, and that sort of thing. And I think that plays a really important role."}, {"start": 1341.97, "end": 1368.42, "text": "Because R&D scientists are really the ones that probably know the technology strengths and limitations the most, the best. And so certainly having that community self-regulate, understand what the consequences are, is really an important piece that can go on independently of the regulation. And I think it's very interesting some of the comments that Jack made about dual use and the publication."}, {"start": 1368.42, "end": 1391.28, "text": "You could even imagine editorial boards, certainly they do in other areas like with CRISPR, they will not publish certain topics or certain areas of research because of the implications of it. And perhaps this community as well needs to start considering more deeply the dual use impacts of some of the research that's published. Can I just follow up on that though? How would you, who would decide that?"}, {"start": 1391.28, "end": 1420.62, "text": "Who would be the body that decides whether or not to publish? Well, professional societies such as IEEE, I'm not trying to put them on the spot, but I am a fellow of IEEE, so I know a little bit about that society. And certainly, it's a self-governance thing. It's where the community itself recognizes that this is a challenge and we need to be responsible engineers and scientists in order to address this. Many of the signatories probably on this letter are members of IEEE. So I think it's really the research community getting..."}, {"start": 1420.62, "end": 1449.81, "text": "Getting together itself, recognizing that this is an important challenge, and then trying to come up with solutions within the research community. I think that this points to a larger level issue that we have in our multilateral framework, at least in the United Nations, where the international security implications of developments in science and technology don't have a home there, right? And member states"}, {"start": 1449.81, "end": 1462.93, "text": "They don't have a home in the secretariat or the existing structures. And member states, i.e. our governments, don't want creation of new mandates, new organizations, new bodies. All of that's very costly. And so we're shoehorning."}, {"start": 1462.93, "end": 1484.93, "text": "New science and technology issues into existing frameworks, most of which are arms control frameworks, where they're not very comfortable fits. And we're seeing that in cyber, and we've seen that since the cyber negotiations started in 2004 in the United Nations, and then now on increasing autonomy in weapons systems, which has been put into a very flexible..."}, {"start": 1484.93, "end": 1502.0, "text": "Convention. The CCW is very flexible. It has a chapeau and then you negotiate protocols for specific needs as you go along. But it's the convention on certain conventional weapons. And we really need to think about this. Is this the fora? Member states have said this is the right fora to discuss."}, {"start": 1502.0, "end": 1529.54, "text": "AI and increasing autonomy in weapon systems. But there was a steep learning curve and it's still going on for CCW delegations, many, not all, but many CCW delegations, for them to understand that this is not a conventional weapon. We're not talking about delivery platforms like drones, right? So as a global community, we're concerned about how to develop appropriate regulations at the international level."}, {"start": 1529.54, "end": 1555.94, "text": "for science and technology applications. We have a lot of thinking to do about what the architecture of that would be, and the role of the private sector in that, because states really want to believe that international security is the domain reserve of the state. And that is just, in some ways, at least in science and technology, no longer the case. Between the research community and the private sector, we must..."}, {"start": 1555.94, "end": 1580.96, "text": "Involve them in not just a token way, invite them to come speak at the meeting and then ask them to leave the room when the real negotiations happen. We really have to find new ways of working on these issues together. That's fascinating. Jack, what do you think about that? I just want to sort of, I agree, but I also want to kind of illustrate this. Like AI, it isn't necessarily tied to a specific capital item. It's not like you're going to buy..."}, {"start": 1580.96, "end": 1604.06, "text": "Thank you very much."}, {"start": 1604.06, "end": 1633.02, "text": "Andrew Moore spoke about had in 13 weeks been able to create this kind of smart rock climbing sort of tutor by integrating a bunch of open source technologies which are lying around. And I think that that should really challenge our notions of what, how, if this could at all fit into traditional arms control frameworks because the properties are absolutely different. And I think it's so counterintuitive and I'm basically agreeing vigorously with this point. Excellent. Next question."}, {"start": 1633.02, "end": 1661.1, "text": "Ian McLeod from the Applied Physics Lab. And maybe for Jack and Rebecca in particular, but I suppose anyone, are there useful analogies or parallel technology domains that, like this, were very difficult to measure and pose maybe novel challenges to the legal community that you think the government actually got right? And are there lessons from that? Thank you. I'm going to spin off of that."}, {"start": 1661.1, "end": 1687.54, "text": "So one thing I think is really fascinating about autonomous weapon systems is that the better you get an analogy, the less law there is in existence. So right now, probably the most apt and common analogy is just another weapon. It's a weapon that you direct, that you use. It might have a certain degree of unpredictability in it, but we can analogize it to a weapon. This works right now."}, {"start": 1687.54, "end": 1714.77, "text": "But it's not going to work going forward because it has an X factor, right? It has an element of unpredictability to it, particularly systems that might have emergent capabilities. You're not going to put them through a normal weapons review process. And so as these systems get more and more autonomous, the weapons analogy isn't going to work, and the whole body of law that we've constructed about regulating the acquisition and use of weapons isn't going to apply."}, {"start": 1714.77, "end": 1738.51, "text": "The other really common analogy in conversation are that they're combatants. We're going to have robotic soldiers. They're said to be, you know, they're going to be less bloodthirsty or less vengeful or less racist or less bored or tired. And all these different comparators with human soldiers. But that's not a good analogy either, right? Human soldiers can't get turned off and turned back on again."}, {"start": 1738.51, "end": 1765.22, "text": "They can't be switched from an autonomous mode to a semi-autonomous mode. They can't be hacked. And they might be spoofed in certain ways, but they can't be hacked in the same way. And we control human autonomy through a combination of training on the front end and punishment on the back end. Punishment doesn't work for autonomous systems in the same way. So that whole element of using law"}, {"start": 1765.22, "end": 1785.68, "text": "To constrain action and constrain autonomy doesn't work. So that analogy goes out the window. Other analogies people have used or talked about, one is child soldiers. Children on the battlefield are autonomous actors that can't be held accountable under existing liability regimes for their actions. And we have a ban on them."}, {"start": 1785.68, "end": 1799.84, "text": "Problem is we have a ban for a completely different reason than we're talking about a ban on autonomous weapon systems. We have a ban to protect the children. No one's interested in a ban on autonomous weapon systems to protect the robotic system. So the law there goes out the window. And then finally,"}, {"start": 1799.84, "end": 1827.06, "text": "Maybe the best analogy is the law of animal combatants. Again, autonomous entities on the battlefield that can take action and yet can't really be held accountable, but they can be trained and they can be used in teams with human trainers. We have no law of animal combatants. The only thing that's out there is one protocol in the CCCW that says you can't hide booby traps in animal carcasses."}, {"start": 1827.54, "end": 1851.63, "text": "We commonly use law by analogy to try and figure out what the best solution is. The problem is, at least for autonomous weapon systems, there's not, we don't have the legal regime. And just to, I think that the work, I mean, Lynn mentioned this with regards to CRISPR, but work done by the scientific communities themselves, not only on CRISPR, but prior to that on things like gain-of-function research when you look at, you know, dangerous."}, {"start": 1851.63, "end": 1881.52, "text": "I'm lapsing on the word, but I think you know what I mean. The creation of powerful viruses or making a cold worse. I think those are actually good models that we should look to as a community because there haven't yet been as many really dangerous things that have emerged, partly because the community sort of took hold and said, let's create a social cost to doing things that go against this norm that we're trying to create. And that seemed to be somewhat effective."}, {"start": 1881.52, "end": 1913.42, "text": "Kirsten, I'm just curious, are there any lessons from cyber? Yes, no. I mean, I see a lot of parallels, obviously, in the cyber discussions. There is no regulation yet, but we're discussing very similar sets of issues. We're discussing how international law applies. We're discussing what are norms of responsible state behavior. How would we recognize norms that are being developed? What norms should we impose, right? What's the role of the private sector in normative development? Another thing that we see in AI."}, {"start": 1913.42, "end": 1940.86, "text": "The private sector and the tech community are leading normative development and states in both cyber and in AI and autonomy are lagging. And what does that mean? Are we going to delegate? Are we going to kind of delegate normative development okay to those people out there while we meet one week a year or two weeks a year to discuss these issues as a community of governments? So there are similarities."}, {"start": 1940.86, "end": 1951.46, "text": "For the moment, they're not happy endings. If I can just add real quickly on this issue of do we have any good examples of measurement? I think one school of thought says..."}, {"start": 1951.46, "end": 1981.07, "text": "Maybe the way to go is not trying to measure directly, but to measure behavior more broadly. So driver's licenses, testing for driver's licenses is a good, the physical driving test is an example. We don't actually have a really clear quantitative set of measures that says this is what you have to do physically in your car in order to get a license. But you do a set of tasks, and some human decides you have done that task to some sufficient quality. It's subjective in the mind of the tester."}, {"start": 1981.07, "end": 2015.33, "text": "Then you're awarded now a license to drive. And so we don't seem to argue about whether or not people can get driver's licenses based on how they're being evaluated in the car. So that's kind of a different way of thinking about evaluating behavior, measuring systems. Question for Rebecca. So I have some legal law-related questions. The first one is, if you look at AI, what's the current law status of you about"}, {"start": 2015.33, "end": 2044.0, "text": "The liability, accountability. I mean, autonomous, you know, self-driving cars, one example, but it can be applied to a lot of other things. Who's accountable? And if there's a cyber attack involved, you know, compromised system, it's just getting more complex. And second is from IP perspective. I'm not talking about the military use, just for the commercial use. So AI right now is really driven by software. Is it copyright or it's something else that..."}, {"start": 2044.0, "end": 2071.44, "text": "What's the status of the law now? As I was saying, absent specific laws regulating new technology, you're going to have a fallback to what existing laws are already there."}, {"start": 2071.44, "end": 2091.38, "text": "Ryan Calo has a great piece that he just put out where he traced the history of robotics in the American legal system and what happens when robot cases go to court and found that judges almost inevitably look for a human to blame, right? And not just any human, not just a programmer back."}, {"start": 2091.38, "end": 2120.48, "text": "Or the manufacturer, they tend to focus in on who is the human involved in this specific accident. And this comports well with Madeline Elish's idea that humans run the risk of serving as the moral crumple zone. They might stay in the loop or be a supervisor on the loop, and they might exist in that space just to have someone to blame when something goes wrong, just to have someone to hold accountable. That doesn't mean that that individual is necessarily the most appropriate."}, {"start": 2120.48, "end": 2145.86, "text": "There is a, at least in American judges, there's been a bias towards holding that person who maybe could have grabbed the steering wheel accountable. Focusing on the person who was, say, the guy in the Tesla who was watching a movie and ignoring warnings instead of saying what was it about the design of the Tesla system that encouraged him."}, {"start": 2145.86, "end": 2176.62, "text": "To think that he could do so. What was it about the marketing of the autopilot and the language and the analogies used around it to encourage him to maybe think of it as a driverless car as opposed to a highly automated vehicle. And so I think these accountability issues are huge and are going to be, need to be worked out hopefully with an awareness that the person in the loop might not be the person actually at fault. That's great. And on IP protections?"}, {"start": 2176.66, "end": 2212.93, "text": "I don't know if you're interested in tackling that one. I, no. Because I think, so we're a non-profit, and so it's not something that we directly have to interface with in our day-to-day. Okay, our next question. Thank you, Patrick Tucker with Defense One. For Director Vigneron, I was wondering if you could talk a little bit about what you were talking about earlier. You said that the current framework"}, {"start": 2212.93, "end": 2237.52, "text": "That we have for discussing these things around conventional weapons is like a shoehorn. It's inadequate. And I wonder if you could touch a little bit on what a better framework to your mind would look like. And I also am curious, I know when we talk about conventional weapons and an international dialogue about what they should be about, I've got a sense of all the players, right? And a country that's made a decision to get rid of clustering munitions is in favor of limiting clustering munitions for everybody else."}, {"start": 2237.52, "end": 2264.21, "text": "When it comes to these sorts of weapons and their future, I want to just give us a sense of who are the players on a big sense, China, Russia, the United States, what do they seem to want? Is there an agenda emerging that's Chinese, that's Russian, that's American, in terms of what a framework should look like? And I'm not sure if it is yet, but I'd love to know your thoughts. Thanks."}, {"start": 2268.37, "end": 2294.4, "text": "What would a better framework look like? That's an excellent question because it goes to the heart of really many challenging questions that go to the UN Charter. Territory, sovereignty, the role of the state versus what the state controls and doesn't control. Again, we see that in cyber, we see that in autonomy. So I don't have a clear answer of what it would look at. What I do know is that our traditional..."}, {"start": 2294.4, "end": 2321.94, "text": "Arms control toolbox, our go-to things of bans, tight regulations. If we're not going to go for an all-out ban, which many people have rightly stated is very impractical for many reasons, at least on the nearer term scales of autonomy, we open up that toolbox. And what do we have in there? We have technology control regimes, export controls. That's our go-to when it comes to these dual-use sensitive technologies."}, {"start": 2321.94, "end": 2348.53, "text": "Those aren't going to work this time around. I mean, these are not even dual use. They're omni-use. They're everywhere. And governments don't control them. They don't develop them for the most part. They don't own the infrastructure. They don't own the brains that create it. We are in really a brave new world. We open that toolbox and we say, oh, there's nothing there. And that's why I'm saying we need a different community."}, {"start": 2348.53, "end": 2378.05, "text": "Practitioners around the table to develop appropriate solutions to put tools back in that toolbox. Because I think we need tools in the toolbox. But they're not going to be control regimes. And I think there's a bit of blinders in the traditional arms control community that will somehow, will be able to fit it in there somehow if we just keep on kind of shoveling it in. And I doubt that that's going to be practical. Or the sorts of regulations that"}, {"start": 2378.05, "end": 2407.65, "text": "We have to be very careful. And all the people sitting in CCW, they certainly have their areas of expertise. Don't get me wrong. They're intelligent, thoughtful people. But they don't know everything about everything. So I think we need to..."}, {"start": 2407.65, "end": 2443.74, "text": "Bring other smart and thoughtful people to the table to develop appropriate solutions which allow us to get the benefits. Next question. Anybody? Over here. There's been a lot of discussion today about international competition and limitations and restrictions. Do you think that's dominating at the expense of discussions about cooperation? And are we missing an opportunity here? Can I take that? I mean, yes. But to extend it, there are a couple of areas where"}, {"start": 2443.74, "end": 2468.14, "text": "International collaboration is not only kind of needed, but we believe necessary. One is safety. You want a shared toolbox of safety techniques for these AI systems where the incentives inherent to the technology are such that governments of different ideologies will choose to adopt them. So if we get safety right, it means that it will be seen as a way to"}, {"start": 2468.14, "end": 2497.17, "text": "Augment your system so your system is in some direction more performative and that by doing so it doesn't hinder you in your international competition with the other players. And I think if we don't get that right, you open yourself to accidents which will be of a scale that are broadly unparalleled and of a potential for harm that is kind of nightmare inducing. So yes, I feel quite strongly that collaboration is important."}, {"start": 2497.17, "end": 2523.46, "text": "Can you give an example before you move on? Can you give an example of what you mean by safety? Because it's such a broad term and there's so many different possibilities there. So what I mean is the way that I think of it is these systems will take a series of actions that are autonomous over a period of time. That could be a drone or it could be a large-scale cyber defense system or it could be something else. And what you want is for those actions to be..."}, {"start": 2523.46, "end": 2552.3, "text": "So when we talk about safety, that's really what we mean. And that is somewhat in reference to these."}, {"start": 2552.3, "end": 2582.35, "text": "Reinforcement learning type systems which take actions. It's not quite so relevant for existing systems which classify and their safety may mean stuff like bias or interpretability, which is also being worked on. So I think there's a, we all know, there's a path dependence to narratives, right? And so I really appreciated the picture of the elephant. The thing is, you could have a lot of people without blinders on looking at that elephant and seeing very different things, right?"}, {"start": 2582.35, "end": 2601.62, "text": "One weapons procurement person could see a force that would be of great use to ground troops in knocking down a wall. A construction worker could see something that could help him or her move blocks. And what has happened a little bit with the international discussion on autonomy is it's..."}, {"start": 2601.84, "end": 2623.2, "text": "And so that's necessarily going to make people think about competition, make people think about relative military advantages. And there hasn't been as much discussion about health standards or about safety or about different ways of addressing poverty issues."}, {"start": 2623.2, "end": 2651.41, "text": "And so what I would like to see is an opening up of the narrative. I'm glad the conversation has been started. I'm so glad it's happening. I would like to see it open up a lot more. And then I think one thing that international law is actually great at and doesn't get enough credit for is creating standards for cooperation so all these different states and countries can have airlines flying and not hitting each other. I would like to see more of that kind of cooperative law emerge."}, {"start": 2652.02, "end": 2681.76, "text": "Certainly, I think for AI broadly, there are several point examples of cooperative activities. For instance, the UN hosted this AI for Good Global Summit a few months ago, and that brought together governments internationally, academics internationally, and certainly the user community and industry to really have a conversation about how AI can be used for good. It wasn't in the context of security or defense or that sort of thing, but I think it really helped"}, {"start": 2681.76, "end": 2706.37, "text": "Bring these communities together to really understand what the real challenges are. So for the development sustainability goals of the UN, things like hunger and education and so forth, these things can lead to improvements in our overall security as well. So I think there are some point examples of collaborative activities. For instance, I know NSF had some"}, {"start": 2706.37, "end": 2731.86, "text": "Conversations with Japan as it relates to smart cities, for instance. So I think there are a lot of these examples of the willingness for folks to collaborate internationally. I don't know that it's led to any sort of broad-scale activity, but I think there's a willingness that's been demonstrated. I'm really glad that you mentioned the AI for Good summit that was in June, hosted by the ITU in Geneva. So that was a system-wide UN..."}, {"start": 2731.86, "end": 2760.06, "text": "And as well as private sectors and others. That's great. And also just to mention that the new Secretary General has a very keen personal interest in what he calls frontier issues. And he's identified, of course, AI as one of the frontier issues. So there's been kind of a mapping exercise going on right now in the UN by the Chief Executive Board on who's doing what in the UN system on these issues. And then that will help position or make recommendations to the Secretary General of how"}, {"start": 2760.06, "end": 2795.18, "text": "As a normative body, the United Nations should be showing leadership and more leadership on frontier issues than it's shown in the past. So again, let's kind of watch this space. I think the CEB meeting is coming up in the coming weeks, and so November, and I think that we'll see some strong statements coming out from the Secretary General and leadership going forward. So on non-weapons issues. Interesting. All right. A question here in the back. To return to the question of dual use, I agree it's a"}, {"start": 2795.18, "end": 2820.48, "text": "An important concept for the community to develop going forward, given the potential for misuse of some of these technologies. But at the same time, I think if the analogy AI as electricity does hold up, I think certainly the notion of omni-use is also quite appropriate in this case, given that you could find military applications for just about any form of AI technology going forward. And certainly the potential for dual use and tech transfers..."}, {"start": 2820.48, "end": 2849.09, "text": "I've been provoking a lot of the debates we've seen on CFIUS. I agree that sort of limiting investment based on national origin might be an extreme approach, but at the same time when China's pursuing a national strategy of military civil fusion that is trying to take advantage of this dual or omni-use capability and deliberately as its new AI plan states trying to take advantage of international innovation resources and a strategy of tech transfer, I think these issues will continue to be problematic going forward. So I'd be curious to hear more thoughts on"}, {"start": 2849.09, "end": 2878.26, "text": "Are there new metrics or frameworks or ways to think about the risks of technology transfer in the context of AI, perhaps beyond the notion of dual use given the multiplicity of uses and options to think about what sort of ways to think about the risk in a particular case, be it capability or likelihood of misuse or the particular actors involved? I have one contribution. So I think that..."}, {"start": 2878.26, "end": 2904.11, "text": "The notion of stuff like red teaming would be helpful here, as in take a given technology and have a group of people, ideally partially composed of people from industry who may have even helped develop it, work out all of the ways in which it can be repurposed for malicious or military uses. And I can imagine developing impact scales and understandings on the basis of that which can help condition."}, {"start": 2904.11, "end": 2930.99, "text": "Some of what you do. Because as you point out, it is an omni-use thing. And you're going to be playing whack-a-mole if you try and specifically lock down individual parts of it. So it may make sense to think of it as more of this sort of holistic thing where certain things will have a very, very broad scale that goes up to a very, very large scale of impact. And those are where you want to concentrate your efforts. And others may seem like they could be repurposed for..."}, {"start": 2930.99, "end": 2953.87, "text": "There's some dangerous uses, but they'll be small within some kind of acceptable boundary by the people who come up with this. But it's an open area. Lynn, do you have any thoughts on that? I think broadly speaking, we're now in a situation where we've democratized AI to the point that everybody can use it for anything they want to use it. Creativity is the only bound."}, {"start": 2953.87, "end": 2968.59, "text": "So, in some sense, omni-use is, I think, a really good term because it's not that you develop a technology and we have to think about it for a little while to figure out some nefarious use for it. I think it's pretty easy."}, {"start": 2968.59, "end": 2987.23, "text": "These days to think of these ways of doing it for almost any kind of AI technology that can be used for good can also be used not for good. Bringing up this UN summit, Global Summit for Good again, I found it extremely valuable because there were people there say from the Red Cross that said,"}, {"start": 2987.23, "end": 3015.47, "text": "You know, on the one hand, as an industry, you might say, well, I just need data, and if I have data, then I can help your refugee crisis. But then the Red Cross will say, the problem is that the data is politicized, and the same despot that can get access to this data can actually harm the refugees with the exact same data. So you can do good with it, you can do bad with it. And it's very difficult to even imagine for any of these technologies that we could say, okay,"}, {"start": 3015.47, "end": 3031.04, "text": "This one, it's okay because I can't imagine how to use it in a bad way. So I think we're in a whole new era now with this technology. In some sense, we've really opened up Pandora's box that we've created technologies that can go either way."}, {"start": 3031.04, "end": 3061.95, "text": "As a researcher, you might say, well, maybe we can invest in more R&D for inherently safe AI systems. And certainly it's not an area that's been deeply explored, and I think there's certainly, it's important to study those sorts of things. There are no clear and obvious answers to me. I think we have time for at least one more question. Right here in the front. Hi, thanks. Carl Robichaud, Carnegie Corporation of New York. Countries are..."}, {"start": 3061.95, "end": 3090.1, "text": "Developing their own policies on each of these questions. But they're also looking to their neighbors and their potential adversaries. So in some ways we have a real collective action problem. And I was wondering if you've seen any cases where there's a potential race to the bottom and how we might prevent that. And whether there are any steps countries can take to be more transparent about their intentions. Race to the bottom. Anyone? Rebecca?"}, {"start": 3091.28, "end": 3120.4, "text": "Whoever goes to me for the best. I'll start with the good stuff. So the U.S. DOD published its policy on autonomous weapon systems, right? And this publication coincided with Human Rights Watch's report, which really presented two different views on this new technology in a way that sparked a conversation."}, {"start": 3120.43, "end": 3149.5, "text": "Because the report was out there, it could be celebrated, it could be criticized, but it's an example of a government publishing its thoughts and its policy and its approach. And perhaps for governments that hadn't thought to even do that, it provided something against which other governments could say, what do we want to do? And if the U.S. is doing this, maybe we should be thinking about it too. It gave the issue a legitimacy that it hadn't had before."}, {"start": 3149.5, "end": 3179.22, "text": "Too often, if I talked about being interested in autonomous weapon systems, people scoffed and were like, oh, so you really like Terminator, as opposed to something legitimate that governments were concerned about. Simply the act of making public policies can go a really long way. I think that we're yet to see the emergence of international data marketplaces, but I think that it's fairly likely these will happen. Most governments have access to"}, {"start": 3179.22, "end": 3208.7, "text": "Some forms of data which are inherently useful or useful for AI. And you can imagine them setting up systems where you can exchange data with other governments if you fulfill some kind of norm set collaboratively. And in this way, you can create more of a kind of carrot as well as presumably having a big stick somewhere as well to help with that. I mean, just on the race to the bottom issue, we heard during the discussions today that there was concerns that different governments don't know."}, {"start": 3208.7, "end": 3239.78, "text": "What their AI systems can do yet, right? They don't really know where we're at. And there's obviously competitions at DARPA and others fund that helps get us a sense of the private sector and researchers. Is that a potential race to the bottom issue that we'll see different governments in the dark about each other's capabilities? And then if there were ever a kind of a conflict, you'd have an unknown scenario. Is there any concern about the lack of sharing or the lack of transparency?"}, {"start": 3239.78, "end": 3250.18, "text": "Yes, I don't have a good simple answer for it though, so it's just another thing which makes me slightly nervous."}, {"start": 3250.18, "end": 3278.91, "text": "Certainly in the CCW framework, there's been discussions about the strategic stability aspects of this, and the fact that there are so few policies that are out there in public. We don't know what the intentions are, and that's destabilizing. And it's only going to get more destabilizing as the technologies, we all see the technology progressing, but we don't see the openness about the policies developing. They always develop behind, but if that lag gets too great,"}, {"start": 3278.91, "end": 3308.88, "text": "It's going to be very destabilizing. So, again, that should motivate us. And the U.S. comes under a lot of criticism for its policy, but good on them for putting it out, right? I mean, I look to that, and I don't agree with a lot of it, but it's there, and it's something that I can react to, and I encourage all governments to do the same thing. So, yeah, it's good to have that. That's going to help us. It is. Great. We'll finish on a positive note. Thanks very much."}]}