{"url": "https://feeds.soundcloud.com/stream/387450086-cnasdc-tech-podcast-paul-helen-jack.mp3", "page_url": "https://soundcloud.com/cnasdc/tech-podcast-paul-helen-jack", "title": "Artificial Intelligence\u2019s Transformative Effects on Society", "show_or_channel": "CNAS Podcasts", "date": "2018-02-02", "duration_s": 1094, "type": "podcast", "confidence": "high", "notes": "Panel with Paul Scharre, Helen Toner; Clark as OpenAI strategy & comms director", "src": "p2", "id": "2018-02-02_cnas-podcasts-artificial-intelligence-s-transformative", "audio_duration_s": 1094.6986875, "worker": "jc-w03", "transcribe_s": 25.5, "model": "large-v3", "segments": [{"start": 0.0, "end": 29.78, "text": "Hi, I'm Paul Shari, Director of the Technology and National Security Program here at CNAS, and we're here to welcome you to the first of our 2018 podcast series on artificial intelligence and national security. I'm joined today by Helen Toner, Senior Research Analyst at Open Philanthropy Project, and Jack Clark, Strategy and Communications Director at OpenAI. Helen, Jack, thanks for joining us. Thanks for having us. Good to be here. You both work at some interesting spaces, thinking about artificial intelligence and some of the transformative effects that it's bringing to society."}, {"start": 29.78, "end": 58.0, "text": "I want to start with a question, how people should think about AI as a whole and the types of changes that we're seeing in AI. Let's start with you first, Jack. How do you think about the types of changes that AI is bringing to society? I think one of the initial effects is going to be stuff that we use every day will get cheaper in the background. So to the point of view of people, not much is going to change. But from the point of view of the companies operating many services or utilities, things are going to get dramatically cheaper."}, {"start": 58.03, "end": 81.1, "text": "When people will become aware of this is when you have the first major products in their hands. This will initially be stuff like self-driving cars followed by robots, at which point I expect there to be a much larger conversation nationally and internationally around first things like automation, and then as they see how this technology develops, issues of safety and progression."}, {"start": 82.03, "end": 111.73, "text": "Helen, what about you from where you sit at Open Flythrough Project? How big is AI? How significant is this to you? Yeah, I mean, I think the indications that we're seeing is that this is going to be a really big deal for the rest of this century. We're already seeing a lot of progress in kind of the basic science of AI, so machine learning research, particularly deep learning research, deep reinforcement learning. They're getting results that seem really likely to transform a lot of different sectors,"}, {"start": 111.73, "end": 140.86, "text": "Things that humans interact with every day. And I think it's obviously, how does the saying go? Predictions. Making predictions is hard, especially about the future. But I do think we're seeing a lot of evidence that this is going to be maybe the most significant theme of this century, if that's not too grandiose a statement. Jack, OpenAI is really one of the groups at the cutting edge of this research and putting out some very surprising results regularly. How much are you surprised by the pace of progress?"}, {"start": 140.86, "end": 167.17, "text": "Internally, we regularly surprise ourselves. We attempt to be conservative in some of our forecasts and in some of our projects we've overshot those, as in stuff has happened earlier than we expected. However, that's traditionally where you're dealing only with software. So if you're dealing with software, progression has been faster than the intuitions of experts. But the moment you go to hardware, like real robots..."}, {"start": 167.17, "end": 190.35, "text": "Yeah, I think this is a great theme, and I think it's something that is..."}, {"start": 190.35, "end": 202.94, "text": "Not reflected very well in media coverage of AI, which often focuses a lot on robots. The dystopian sci-fi stories are often about robots going wrong in some way. But I think of the most influential..."}, {"start": 202.94, "end": 227.57, "text": "AI systems that are currently being used every day as things like Google's search algorithm or Facebook's newsfeed algorithm. And I think those kinds of things potentially, you know, increasingly Siri, Alexa, Google Home, as they get more capable, as Jack says, the possibility to develop and roll things out very quickly in software, I think is underestimated compared to the sort of glamour of robotics."}, {"start": 227.57, "end": 256.83, "text": "Tag on to what Helen said. You know, robots, they're obviously very, very glamorous, but a smartphone is kind of a robot. And so in the last decade, we produced, you know, tens, hundreds of millions of tiny little cameras, which are billions, actually, which are embedded in the world now. And so in the background, kind of invisibly, all of this information is now open for analysis and open for insight. And I think that we, as kind of a community, aren't ready for the impacts this will have, and especially when it comes to issues like..."}, {"start": 256.83, "end": 280.78, "text": "So can you give some examples of those kinds of changes that we might anticipate in the national security space or things that would be relevant to national security?"}, {"start": 280.78, "end": 306.64, "text": "Systems for collecting visual information about the world, whether in the form of satellites, drones, high altitude and low altitude, and embedded cameras on the ground, wherever they are. Many nations still use, you know, analog analysts. They use humans who are powered by hamburgers and sleep to go and look at these photos and draw bounding boxes around them. I think in the future you will see a transition to partial automation there, or to..."}, {"start": 306.64, "end": 334.7, "text": "Yeah, and I think going back to the theme of"}, {"start": 334.7, "end": 364.93, "text": "Software being able to make progress in software more quickly and then roll it out more quickly as well. I think cyber is obviously the national security domain that involves the least kind of hardware or physical machinery. And I think there's good reason to expect that current approaches in machine learning and reinforcement learning, experts that I know and I'm far from an expert in cyber operations, but experts that I've spoken to seem to expect that we will see"}, {"start": 364.93, "end": 392.72, "text": "How should we think about proliferation of some of these technologies? This is something that comes up all the time in the national security context. Now, on the one hand, something that software can proliferate fairly easily."}, {"start": 392.72, "end": 417.95, "text": "On the other hand, some of the most advanced systems require a lot of computing power to run them and to train some of these neural networks. So how do you guys think about the requirements to use some of this technology and who might have access to that? In my head, I break it into two distinct categories. If you're doing what I just call a classification task, then not only are the technologies widely proliferated, but the ability of people to..."}, {"start": 417.95, "end": 444.67, "text": "Perform that action is also sort of distributed. Those technologies tend to be well understood and require the sort of computational resources that most people can have access to on their own. Now, if you want to do larger things, like tasks that Helen referred to involving things like reinforcement learning, where you need to model the world, simulate it, and run games within it, that's where the large hardware requirements come in, and that may be a potential."}, {"start": 444.67, "end": 456.98, "text": "Point of control for policymakers because we'll know that you'll need certain huge amounts of hardware to accomplish things and these will likely be the things that are thought to be potentially dangerous as well."}, {"start": 456.98, "end": 484.37, "text": "I agree with that. I just had a couple of points. I'm not totally sure how these will play out, but I think they're relevant to be thinking about. One being the distinction between the computing power required to, because I agree with Jack that computing power is likely to be a big determinant in who can use what. So the distinction between the computing power required to train an algorithm and to run the algorithm once you have a trained model, and the computing power for running these models is often many orders of magnitude less than that required."}, {"start": 484.37, "end": 514.94, "text": "So that's kind of one interesting factor. And then another thing worth keeping an eye on is the progress in hardware and the beginnings of deep learning specific computer chips, which will potentially be deployed in smartphones and things like this, which I think on both of those elements, I'm not completely sure how that played out, but I think they're going to be really relevant sort of inputs. I think that this speaks to a really important point in security that we're already familiar with,"}, {"start": 514.94, "end": 543.81, "text": "When it comes to widely deployed encryption services, we build our encryption far ahead of the computational envelope. So you know that at some point you're going to be able to crack a given encryption algorithm, given enough computation. So you scale the difficulty of that encryption far, far ahead of what you model as potential likely progress in computation. I think that we're going to have similar issues in AI where we need to figure out ahead of time."}, {"start": 543.81, "end": 569.06, "text": "What sort of dangerous things look like and what their computation might require so that we're ready to think about it. And as Helen said, hardware is getting better over time. So once you've got a thing out there which is expensive, potentially dangerous, and currently limited in the number of actors that can deploy it, check back in five years and everyone will be able to deploy it on a mobile phone or something. And that effect we haven't really dealt with as a..."}, {"start": 569.06, "end": 594.48, "text": "As a community of researchers and scientists, yes, and I think we need to somewhat urgently. So what about some of the vulnerabilities in these systems, whether they're vulnerable to spoofing attacks or different control problems that can come from them? How should we think about risk of these systems, particularly when they're used in high consequence settings, when it might be a cyber tool being deployed by a nation state, and then they have these kind of vulnerabilities?"}, {"start": 594.58, "end": 621.3, "text": "So I think the way that I would hope we will be able to think about these risks is as something that I think will be very relevant to their use and something that we should be proactively tackling. I think there is the potential for systems to be deployed without vulnerabilities or issues being noticed and then that having really negative consequences on the ground. And I think, I know there's a lot of concern in national security communities about"}, {"start": 621.3, "end": 649.49, "text": "Relative advantage and competitiveness, and I think that's all very valid. But I think I'm interested to see and hopeful that we can see work to help these systems be robust and work well and reduce the chance of unintended negative outcomes that can be shared even between competitors, whether that be states or private companies who are also in very tight competition."}, {"start": 649.49, "end": 675.66, "text": "So I think I'm about to start an argument with Helen, so you can position yourself between us if that seems appropriate. Good, I think it's good, yeah. You know, my view is that it's actually good that these technologies are risky today because I think that it puts a brake on deployment and it puts a brake on rollout. Now, yes, it does increase the chances that a less careful actor will roll something out that's non-predictable."}, {"start": 675.66, "end": 704.75, "text": "Very damaging consequences. But I think broadly the technology today is sort of at such a point of immaturity that it's good we have this essential risk innate to it which is going to force us to slow down and think about it. And just to add to that point, Ian Goodfellow and his team at Google published, I believe this week, a paper on his work called Adversarial Examples. Adversarial Examples is the best evidence we have that you can spoof these neural networks very, very easily."}, {"start": 704.75, "end": 731.36, "text": "And his group has done some work to show that it seems like as you get to more and more complex domains or domains that we call higher dimensional domains, adversarial examples persist. So this lack of surety seems likely to persist in the future with at least some of these components we have today. And again, I think that's a good thing because it forces us to sort of maturely study it and hopefully it allows policymakers to..."}, {"start": 731.36, "end": 759.81, "text": "to plan a thoughtful rollout program that takes that into account. I mean, I would agree and disagree. I think it's a good thing as long as, to the extent that we have common knowledge about the immaturity of these systems and about their vulnerabilities. I think one thing that I worry about somewhat is a version of the security dilemma playing out in which maybe two parties both are aware of issues with these systems and reasons to be cautious in deployment, but they're concerned that their counterpart"}, {"start": 759.81, "end": 786.26, "text": "So I'm going to just present a quick counterpoint to that. No, yeah, go ahead. Continue to give counterpoints. I love it. I think..."}, {"start": 788.05, "end": 815.09, "text": "I still believe that this ultimately nets out to most people not deploying this stuff, and I'll give you a tangible example as to why. So there was recent work showing that you can take these adversarial examples and you can make them work in the real world. And the way this worked is an MIT student got a small 3D-printed turtle. I love the turtle. The turtle's great. He painted a pattern on its back, and now our neural network classifies it as a rifle. So that's obviously somewhat disturbing."}, {"start": 815.09, "end": 839.73, "text": "I spoke to this student and I did a write-up with Tim Wang for our review of NIPS. And we were saying, if I am a party and I have a bunch of machines, like let's say, you know, tanks or Humvees or buildings, and I don't want the automated classification systems running over satellite imagery from another government to accurately know what these things are."}, {"start": 839.73, "end": 862.48, "text": "Thank you very much."}, {"start": 862.48, "end": 887.63, "text": "You will assume that if you've rolled something out, people are going to try to compromise you constantly. And if some of these flaws seem like they are, for the moment, unfixable, that could, at least among the larger nations, lead to a break. Though I agree with Helen that this doesn't deal with the problem of small, malicious and or ignorant actors who could pose a threat. Yeah, I think to the extent that these vulnerabilities and issues are known about, I agree with you."}, {"start": 887.63, "end": 902.05, "text": "Yeah, I think one observation I'll make, and I'm interested in each of your reactions to this, is when I think about digital systems today, we know that they have lots of vulnerabilities. We know that software can have bugs in zero days and people get inside them."}, {"start": 902.05, "end": 924.29, "text": "And so that hasn't stopped the proliferation of digital systems throughout our lives. They're in our pockets and they're in our cars. And even though they have these vulnerabilities, the technology is so valuable and useful that it's being deployed everywhere. We seem to not be able to do a good enough job of securing large databases."}, {"start": 924.29, "end": 950.99, "text": "The U.S. government's OPM database, that is information on people's security clearances or Equifax data. So is it fair to say that these vulnerabilities would hold people back from deployment? Or is there a world possible where that might not be the case? I think it's fair to say it will hold you back from deploying it in critical circumstances where you're dealing with lives that are at stake. I also think it's fair to say that if you're making..."}, {"start": 951.12, "end": 975.95, "text": "Things for the consumer that don't risk the consumer, then you're going to be much more cavalier about this because it's a kind of tragedy of the commons where we don't have systems to enforce costs on people for not paying attention to this. Well, good. Any last parting thoughts from each of you? Let me ask one final question. Each of your organizations is doing really interesting work thinking about..."}, {"start": 975.95, "end": 1006.13, "text": "AI in different respects. What are some of the top things on your agenda for the next year, things that you're excited about in the field of artificial intelligence? So for OpenAI, there are two things that we're really excited about. One is safety, and this relates to what Helen's been talking about. We think that making people aware of both the flaws in this technology and also the ways it can be made safe is important. So you want all players in this sector to be aware of that."}, {"start": 1006.13, "end": 1036.06, "text": "And the second is meta-learning. I think many AI research organizations are betting right now that now that we can learn how to act in a simulator, the trick will be to learn how to learn to act effectively in a new simulator you haven't seen before. So learning to acquire new skills. And the jury is somewhat out as to how effective this will be, but we have many examples in small domains where it seems to be effective. And I think if we can crack that,"}, {"start": 1036.06, "end": 1058.75, "text": "Then we will further inflect the sort of AI progress curve in a steeper direction, which will probably lead to more podcasts like this. Good. And one final, final question for you, Jack. OpenAI recently beat Dota 2. AI companies are constantly sort of tackling some of these games. What's next? Can you share with us what the next big sort of..."}, {"start": 1058.75, "end": 1086.96, "text": "Dean, grand challenges. We beat a somewhat limited version of Dota 2 involving one player versus another player, and we have a goal of having a team of five players operated by an AI go up against a team of five human players. We think that's quite a big enough challenge to have taken a bite off, but that's what we're focused on for now. After that, I expect you'll go to much higher resolution games with more moving parts, but I can't predict for you what they are."}, {"start": 1086.96, "end": 1094.7, "text": "I play games myself, so it's in the back of my mind at Christmas when I download a new game. Excellent. I will look forward to that. Thank you both for being here."}]}