[0:00:00] Twas many nights before Christmas, and one woman knew she needed a Christmas tree. So to Wayfair she flew, finding just the right one, and for years it will last. Wayfair shipping was free and remarkably fast. What's this? Garland? A wreath? Essentials to host? Good thing she celebrates way longer than most. As her cart filled with magic and decor from the heart, Merry Christmas to all. Get your Wayfair head start. For holiday decor you can count on year after year, Wayfair delivers.
[0:00:30] Still waiting in line? Again? That's time you'll never get back. Save time and money with Stamps.com. Over 4 million businesses have skipped the line with Stamps.com. Join them to save up to 90% off carrier rates from your computer or phone right now. Print postage for certified mail, registered mail, and packages in seconds. Then schedule a pickup right from your home or office. For a limited time, go to Stamps.com and use code podcast for a free welcome gift. Taxes and fees apply.
[0:01:06] Now, with the spread of capabilities, the ability for people to actually do things that is like use APIs, trigger actions in third party environments, actually initiate physical actions in the real world with software. That's a different quality of impact to simply the spread of misinformation where we have the traditional challenge of freedom of speech. So that's going to be the challenge that we face in.
[0:01:34] in democratic, liberal Western countries. But as I write in the book, the flip side of that is that for regimes that don't have any desire to pursue freedom and openness, then they're actually going to have, quite surprisingly, the opposite sort of turbocharge of their power. I'm Jack Goldsmith, and this is the Lawfare Podcast, September 5, 2023.
[0:02:02] There is no more consequential technological development in recent years than widely accessible artificial intelligence. And there are few more consequential contemporary figures in the artificial intelligence field than Mustafa Suleiman, who is the co-founder of DeepMind Technologies, an early leading artificial intelligence firm later bought by Google, and more recently co-founder of Inflection AI, a firm devoted to personalizing artificial intelligence.
[0:02:29] I sat down with Suleiman to talk about his new and somewhat frightening book, The Coming Wave, Technology, Power, and the 21st Century's Greatest Dilemma, which is his take on the novel threats posed by artificial intelligence and synthetic biology. We focused on the artificial intelligence components of the book, discussing AI's promises and especially its dangers to both individuals and the state, and what governments and firms can realistically do to redress the dangers.
[0:02:56] It's the Lawfare Podcast, September 5, The Coming Wave. Mustafa, you spent your whole adult life in the field of artificial intelligence. Can you just kind of give us a sketch of that before we talk about your book? Yeah, for sure. I co-founded DeepMind when I was just turned 25 years old. DeepMind's the artificial intelligence research and applications company that we started in London. Our mission was to build general purpose learning systems.
[0:03:27] And we were acquired by Google in 2014. And I spent six years at DeepMind working for Google, essentially. I ran a division called DeepMind for Google, which was tasked with applying all of our breakthroughs from AlphaGo, AlphaGo Zero, AlphaFold, all our healthcare work, energy work to various products and services across Google.
[0:03:57] During that time, I probably did about 70 or 80 launches at Google in many, many different product areas. And then I went to work full-time at Google as VP of AI products and AI policy. And then bringing right up to the present, last year in February, I left Google and started a new company called Inflection AI with Reid Hoffman and one of my friends from DeepMind, Karen Simonian. And we're now building an AI called Pi.
[0:04:26] Which stands for personal intelligence. Which I've been using. It's really fun. Oh, nice. So your book is about the profound dangers of artificial intelligence and what to do about it. It's also about the benefits, but you're more worried about the downsides. And before we get to what those downsides are, at what point in being involved in the development of these models, and you were centrally involved, did you...
[0:04:52] I think the long-term impact of being able to distill intelligence, the thing that has made us so productive and capable as a species, into an algorithm or a piece of software that could be paralyzed and scaled up and reproduced and shared so easily.
[0:05:20] I mean, that prospect was, you know, the prospect that that would create, you know, incredible instability and chaos was very clear to me right from when we started the company in 2010. So our mission was to build AGI, Artificial General Intelligence, ethically and safely. That was the strap line on our business plan that we took to Silicon Valley back in 2010, in the summer of 2010, when we first met.
[0:05:48] Peter Thiel, of all people, because he was the person who was most, to be fair to him, forward thinking and most interested in the singularity. I think he was just very kind of into sci-fi. And so it was sort of easy for him to kind of imagine a world where you had these sort of AIs producing things and helping us organize businesses and run entire economies eventually. So yeah, it was very foundational to our thinking and my thinking personally.
[0:06:18] From the moment I started the company, when we were acquired by Google, we made it a condition of the acquisition that we had an ethics and safety board process, as well as some red lines about how our technology would and wouldn't be used. And then I've been involved in a succession of variously successful, more often than not, unsuccessful efforts to work on the ethics and safety of AI throughout my entire career. And you talk early in the book about how when you were
[0:06:47] Early on, pitching your ideas, pitching the ideas on the company, you would make very clear in the presentation, in your slides, what these downsides were. And you describe a kind of blank stare, a kind of we don't really care, a kind of disbelief. Is that a fair way to describe it? Yeah, there was always a kind of awkwardness. It was always a bit of a taboo. You were kind of looked upon as, you know, I don't know whether...
[0:07:14] I don't know quite what the judgment was, but there was a sense that it was taboo to be critical of big tech. And I don't want to be seen as, and I am not an angry critic, I'm an accelerationist of anything. I believe that we have to build this technology, but I think that...
[0:07:35] Particularly, I think in the US, I think people get caught up between whether we're optimistic about things or we're fundamentally pessimistic about things. And I think both are biases, you know, lenses which color our thinking, when in fact, the conversation should be around.
[0:07:51] Observable facts, you know, what can we measure? What gives us an indication that this is likely to cause harm? What is going to make the security paradigm more challenging? What is likely to be fundamentally labor replacing rather than just labor augmenting? All of these discussions, I think, that are now part of the discourse post-ChatGPT and increasingly in the last few years with the various kinds of governance efforts, you know, now they're sort of everyday part of the conversation.
[0:08:19] 2010, 2012, 2014, 2016, it was just a very awkward thing to raise. And in the book, that's what I refer to as the pessimism aversion trap. The danger of being so sensitive to having potentially dark, gloomy, difficult conversations that we end up ignoring what might be some fundamental trajectories, which...
[0:08:46] Where there is actually potential for us to intervene in the course of those trajectories and steer them towards the kinds of ends that might be more in the public interest. Okay, before we get to those fundamental trajectories, can you just tell us more what you mean when you just said that you think we have to build this technology? I think what's at stake in the world over the next 30 years is going to be fundamentally driven by a warming climate.
[0:09:15] And the demand of the next 5 billion people who are going to want to eat and be educated and be cared for and have access to healthcare that is akin to what we've luckily experienced in the top 1 or 2 billion people in the world. And so there's this massive onswell of demand for more, cheaper, and better.
[0:09:44] I believe that they're entitled to get access to it. And indeed, of course, they are. And so there's this question around how do we produce more high quality goods and services with less? And I think that is fundamentally the quest for artificial intelligence. We want to take what has made us so productive and capable, our intelligence, and make that cheap and abundant so that we can solve all the big challenges that we've got ahead of us in the 21st century.
[0:10:13] And talk just a bit about Pi, if you could, and this idea of everyone having a supercharged personal assistant, researcher, et cetera, in their pocket. I mean, isn't that kind of the philosophy of the company? Yeah, that's exactly right. I think in the next few years, hundreds of millions of people will have access to a personal AI, right? There are going to be many different AIs in this new wave of technologies.
[0:10:41] Some of those AIs are going to represent businesses in the normal way and try and sell you things. Other AIs will be marketing and promotional things. They'll be persuasive AIs. They'll represent digital influencers, entertainers. They'll be part of music and film. They'll be kind of productivity AIs. Governments will have AIs. NGOs will have AIs. And of course, you as an individual consumer will also want your own AI, your own AI that is going to
[0:11:11] Help you find and access credible information. It's going to prioritize your busy schedule. It's going to help you plan your life and make trade-offs just like a great chief of staff would. It's also going to do, like you said, research for you, help you synthesize tons of information that's coming your way, and fundamentally communicate with you in plain natural language, fluently and smoothly, just as you and I are communicating now.
[0:11:38] And I think that what you're going to want is a personal intelligence that is really aligned to your interests. There's a fundamental fiduciary connection between the pair of you because you're going to give your AI, you're going to give your Pi the ability to enter into contracts, make agreements on your behalf. It's going to be a kind of legal proxy in a way. It'll certainly make purchases for you. And it's going to interact with other AIs and try to plan and coordinate and book things and learn new things.
[0:12:07] But it'll also call other humans on your behalf, just as a chief of staff or a personal assistant would or something like that. And so this is, in many ways, your digital representative. And what we're trying to establish is that close, trusting relationship for an AI that you can use to make you way, way more productive and allow you to focus on the things that are really important to you.
[0:12:32] So there are many potential upsides, but the book is mostly about the potential downsides. So could you just kind of summarize the thesis? I mean, it's got many moving parts, but what are you worried about? And over what timeframe are you worried about it? I think the timeframe is actually the critical piece to begin with, because one of the things that I found most frustrating over the last five years is this complete sort of hyperbolic obsession with superintelligence.
[0:13:01] And this is the idea that once you build an AI, it'll very quickly become an artificial general intelligence, and that will become a super intelligence, one that has recursively self-improved. It is updated and improved on its own code in order to make itself smarter and add more capabilities to itself. And naturally, a system like that would somehow get out of the box.
[0:13:28] breach its containment, it would acquire more resources, it would persuade people and influence people to prevent any kill switch being activated. And this would then lead to what's known as an intelligence explosion, this recursively self improving system. And I think that framing, you know, the Terminator framing, if you like, has just come to dominate.
[0:13:50] And I think that's completely wrong. And it's meant that we've missed, we've totally missed
[0:14:19] The much more prosaic and incremental nature of what's happening. And so start with something that you are already very familiar with and many of your listeners will be familiar with, the spread of misinformation. I think this has been quite a predictable trajectory over the last 10 years. Slowly, our models got good at identifying pixels in images and understanding the content of those images. Slowly, they got better at being able to understand.
[0:14:48] Audio and do perfect transcription for dictation. And once you can classify perceptual input, right? So you can sort of understand the contents of images, audio, and text. The next obvious step there is that you should be able to generate new examples of that. So if the model understands the idea of cat well enough to distinguish it from dog and rabbit and donkey,
[0:15:18] It then, over time, is clearly then able to develop the capability to generate a new example of that particular classification of a cat. And so the generative AI revolution, to many, I think feels like it's just come out of nowhere, like a bolt from the blue.
[0:15:38] But actually, there's been this very steady and I think actually quite predictable rise in capabilities as the models have got bigger and bigger and as we've gone from classification to generation. And so if you just take that kind of clear-headed observation of the trajectory, you can start to think, okay, well, what are the next things that come?
[0:15:57] You know, in line, what are the capabilities that are likely to come next? And what are the implications of those capabilities for the workplace, for politics, you know, for culture? And that's really the approach that I've taken in the book is to try to lay out what this means for each one of those different areas. Okay, well, I want to march through some of those areas, but I want to pause first on the first thing you talked about, this super intelligence, this intelligence explosion.
[0:16:25] You're skeptical of that? Is that a skepticism? Is this the same skepticism as a skepticism about existential risk that everybody talks about? Or is that an autonomous AI? Is that what you're skeptical about? And are you skeptical about it in the short term, medium term? Are you just saying that's not where we should be focused right now because that's not where the main problem is on the horizon? Could you just clarify that? Yeah, that's exactly right. It's a question of timelines. I'm skeptical of the obsession with existential risk because
[0:16:54] I don't think that these systems represent an existential threat to the future of our species anytime soon. But it is a question of timeline. People often say to me, well, what would be your probability of arrival of an existential threat by 2050? And I'm like, well, okay, you're forcing me into a kind of like...
[0:17:14] So as a rationalist, I have to assign some probability mass to that outcome, which is non-trivial, which in itself could be seen to be very scary. I'd probably give it single-digit percentage, less than 10% or something. People say, well, one in 10 chance of existential threat. So I guess what I'm trying to say is I think over the next few decades, we've got many more practical near-term...
[0:17:39] High probability events, which are quite easy to wrap our heads around, the consequences of misinformation, for example, which we should spend our time talking about rather than, you know, the sort of Skynet Terminator thesis. Okay, so the kind of more medium term or more immediate problems are that you talk about in the book are, and I'll just mention a few and you can mention others if you like, misinformation on steroids, cybersecurity problems on steroids.
[0:18:07] The possibility of individuals being empowered to do terribly bad things by building weapons or other offensive vectors that they'll have the power to develop. Can you talk about, just say a bit more about the misinformation problem and then talk about cybersecurity and why you think that's a big problem in the short to medium term? And anything else important that I missed out on in the kind of problems we should be focusing on now?
[0:18:36] Right. So, I mean, I think for the misinformation problem, I mean, we're basically reducing the cost of production to near zero marginal cost. And, you know, that has been the trajectory of technologies throughout our entire history as a species, right? Like everything that is useful over time tends to get cheaper and easier to use, and therefore it spreads far and wide.
[0:19:05] Almost law of technology is even more acute when the substrate is actually software and it's bits in information rather than things that are manifested in atoms. Atoms have physical constraints which provide a flaw on the cost of moving things around and recombining them into new types of products. Whereas information has...
[0:19:31] It still has a fundamental cost because it's hosted somewhere in the cloud, but that is so infinitesimally small that it means that we're really able to effectively have zero marginal cost of production on a lot of this new synthetic material. And so the simple idea is that that will drive a proliferation and empower everybody to produce new content. Now that in itself isn't necessarily bad. I do actually believe that...
[0:19:59] Some of those kind of harms have been slightly overstated. I do think that we're going to adapt and be able to mitigate some of the downsides of that. We're already seeing pretty significant progress, which I think was expected on things like digital watermarking. So there will be an accredited account that represents Jack in this specific setting on these platforms and material that you produce will be cryptographically signed to your public key.
[0:20:28] And so I think that if someone then comes out and makes a claim about something that you've said, and it hasn't been signed or verified by your account, then I think that that's going to at least cover that portion of the risk. Now, clearly, that only represents a small portion in the grand scheme of things of the ways in which misinformation is causing instability. So I think the broader point is...
[0:20:55] Anybody is going to be able to broadcast, right? And that's actually been the trajectory of the last couple of decades with social media anyway, right? Is that we no longer require people to have, you know, formal affiliation with magazines and newspapers and TV stations. It is an entirely new medium now, you know, the internet and of course podcasts. And that's been amazing. That has been a hugely, you know.
[0:21:19] Liberating thing that we should praise. And in many respects, we want a lot more of it. I think the challenge is going to come when, as has classically been the case, bad actors exploit this new lower cost of production, lower barrier to entry to carry on doing what they've already always been motivated to do. And so it's that old game of security and freedom coming up once again. But let me ask you this on that.
[0:21:49] And you hinted at, I think, part of an answer. I mean, why do you not think that the technologies that you're talking about won't also be available to ameliorate these harms? I mean, I don't understand. The book seems to be confident that, to use a phrase, the offensive concern is going to defeat any defense.
[0:22:17] In other words, you talked about digital watermarking, but how do we know that artificial intelligence models won't be able to detect and defeat misinformation in ways that leads to a sustainable equilibrium? Why do you think that won't happen? Yeah, that's a really good question.
[0:22:42] Just to say, this is held lightly, so I'm not 100% sure either way. But my instincts are that this time is different because the pace of innovation is happening at an unprecedented rate. And typically, that, to my mind, tends to favor those who want to develop offensive capabilities. Because it basically means there are more...
[0:23:06] Creative ways that they can get into systems, given that this is a porous system. We're not talking about a walled garden here. We're talking about culture itself. And so this is really just adding new, vast amounts of new information, which many people who don't have access to capabilities for defense will be able to verify that that information is correct or not. And so we'll increasingly rely on...
[0:23:33] Actually, the walled gardens, Facebook and Google and other platforms to have to figure out how to police that content. And obviously, they haven't always done a great job of that. But I think it's different this time if you also look at the kind of cyber weapons case. As power gets concentrated in smaller and smaller transferable units, it can be used by a much wider range of people.
[0:23:58] And therefore, those in centralized bodies who are trying to defend against those proliferated bad actor uses are going to have to move faster and more efficiently to create defenses. And I think in the limit, defense will catch up and we will find equilibrium. The question is, given that things are evolving so quickly, is it going to end up causing instability in the period before defense is able to catch up?
[0:24:28] But that strikes me as, that's exactly the right way of putting it, I think, but it strikes me as an open question. I mean, the cybersecurity example is a good one because, you know, the argument you're making was an argument made 25, 30 years ago about cybersecurity, that offense would beat, defeat defense. It only takes, defense can't defeat everything. Offense only needs to find one vulnerability. And there were all sorts of sort of confident statements about.
[0:24:56] Government's being unable to control and government's going to be brought down because they wouldn't be able to defend their networks and the like. And the truth is that it's been a kind of ugly arms race and the government has proven to be vulnerable and weak in some respects, but remarkably resilient in others. And when you take all that into account over and against all the enormous benefits, it seems like it hasn't been a terrible trade-off. And I'm just wondering,
[0:25:25] I'm wondering why that, I just don't understand why you're confident that won't continue. Your main answer was that the pace of innovation is increasing, but the pace of innovation on defense can increase as well. And if I could just say one more thing and then I'll let you comment. I agree with you that when new technologies akin to this come on the scene, the state is thrown back on its heels. We saw this a bit with the internet at first.
[0:25:51] And there is this messy period where the state and centralized power has to readjust itself, but it does tend to readjust itself and able to defend and control the new technology. And we're speaking in very abstract terms now, but I wonder why that won't happen here.
[0:26:21] Many motivated bad actor groups that run across international borders that require international cooperation to repress them, from Wagner to various kinds of militias and drug cartels in South America, to various kinds of terrorist groups in the Middle East. Each of these non-state actors now have hundreds of millions of dollars at their disposal.
[0:26:49] In some cases, billions. And their traditional weaponry was, like I said earlier, sort of grounded in atoms. Physically, it can be observed and it was hard to move around. And if increasingly their weaponry is the spread of misinformation or can be exercised through hacking and putting pressure on our cyber infrastructure, I think that is an observable tilt.
[0:27:18] Right. I mean, again, this is the last time I'll make this point because I think you've answered it. But all of those claims were made about cybersecurity 25, 30 years ago. And it's not really true.
[0:27:47] At least for the US government that we require international cooperation to defeat these threats. I mean, all you need to do is to read what the NSA and Cyber Command says about its defend forward capabilities, about living in adversary networks, about stopping the threat before it starts. So look, I'm agreeing with you. I have no doubt that there's going to be a disruptive period. I just don't know. I just am not confident that it's going to be catastrophic is all I'm saying.
[0:28:15] I don't know if you have any reaction to that. Yeah. I mean, I would say that it depends on the snapshot in time in terms of the capability. So I kind of thought you were more asking me about the spread of misinformation and whether that was going to be a catastrophic tool, right? So, you know, and I think that, as I was saying, I think that's probably overplayed. And I think that that's a traditional threat that we're going to be capable of dealing with because it just means more people have the ability to broadcast information.
[0:28:45] But I think if you roll forward five to 10 years, let's just take a moment to lay out what kinds of capabilities are likely to be unleashed over a 10-year period. So we've moved from this era of classification and understanding, so we can now understand and read text, to being able to generate and produce. And we're now about to approach a third moment in the AI evolutionary era.
[0:29:15] Which is of interactive models, right? So they will be generating or predicting a recommendation or a conversation or an action to either another AI or another human, right? And those three pieces, you can think of them as sort of understanding and responding and acting, taking action. They are very sort of foundational to what makes up.
[0:29:41] Many roles and skills and qualities of being a human professional in the workplace. And so in the book, I frame this as an artificial capable intelligence, right? So rather than focus on an artificial general intelligence, the super intelligence we talked about earlier, I'm sort of trying to get us to fixate on a nearer term, sort of generally capable system that can talk to you in plain English, that can...
[0:30:09] carry out actions that can book things, buy things, do things, and that those agents don't require vast amounts of computation. They will be available to anybody in open source, and they will spread far and wide. And I think that one of the ways that I've proposed that we try to measure that is by a new modern Turing test. So the old Turing test tried to measure intelligence on the basis of what the machine could say.
[0:30:37] Could it speak to a human for a few turns of conversation and trick the human into thinking that it was actually human rather than a machine? And the problem with the imitation sort of test is that what an AI can say isn't necessarily a great indication of whether or not it's intelligent. Whereas what an AI can do is...
[0:31:06] Far more compelling and useful as an indication of A, its intelligence, and B, what its consequences are for what it can do in the world, right? And so that's, I think, where I start to get more concerned about catastrophic outcomes. If the ability to take action in the world at almost zero marginal cost, at huge scale in open source, and then I think that's really the model that we have to think about in terms of political consequences.
[0:31:35] And by the way, on open source, is the open source of these technologies, is open sourcing these technologies, is that good or bad for the things you're concerned about? Does that exacerbate the concern or diminish it? Yeah, I think it's quite straightforward. I think it is amazing for innovation, is essential for innovation. And I also think it amplifies the fragility from these kinds of potential catastrophic harms that I'm describing.
[0:32:04] I think it has the potential to create and empower a lot of bad actors to take very material action in our societies. Support comes from Wise, the smart way to manage the currencies you need around the globe. Fed up with losing out to hidden fees when you send money abroad with your everyday bank? Choose the smart way. Wise. You can count on the exchange rate you'd usually find on Google. No unwelcome surprises. Plus...
[0:32:35] Ditch that where's my money feeling. Most transfers arrive in under 20 seconds. Join millions saving billions on hidden fees. Be smart. Get wise. Download the Wise app today. T's and C's apply. You save. You invest. You keep an eye on spending. So why do your finances still feel so complicated? Quicken helps you put the whole picture together and make sense of what to do next.
[0:32:59] Connect accounts from more than 14,000 financial institutions and see how all the different pieces of your financial life fit together. With Quicken, you can make sense of your options and decide what to do next. Quicken. Know what to do next. See how at Quicken.com. There are two types of business owners. Those who are busy and those who want to be busy. Toast is designed for both with tools to keep you humming and help turn grind into growth. That's how you turn busy into business. Toast.
[0:33:28] So I detected something of a tension in the book, but maybe you can tell me I got it wrong. Between your concerns about the decentralization of power and the kind of democratization of power down to small actors who can do bad things as a threat to the state, I couldn't tell if it was a threat to the state or a threat to the West and Western states and democracies, because at times a lot of your concerns are about how
[0:34:00] You know, misinformation undermines confidence in government and the like, and that's going to be exacerbated. But you also talked a lot about these technologies sharpening the tools of repression by authoritarian governments and how authoritarian governments could use these tools, and this is clearly China's aim and they're not hiding the fact, to enhance surveillance, enhance control, social control and the like.
[0:34:27] Does the threat play out equally for authoritarian and democratic states? Is it mainly a threat to democratic states? Will China be able to control this technology and use it to maintain control within its borders? Yeah, I think there is a bit of a tension there, although I think the contradiction is a real and observed one.
[0:34:52] That is the path of this wave, is that it does both simultaneously. And on the face of it, it looks like they're contradictory statements because the open source proliferation of these models is obviously going to be the overriding characteristic of open liberal democratic societies. And if you buy my framing that the proliferation of capabilities is really a proliferation of power, anyone who wants to yield influence and persuasion.
[0:35:19] Both for good and bad is now going to have an easier and cheaper time of it. I mean, that for sure is the trajectory that we're on over the next decade, I think. So the consequences for open societies are that our belief in openness is going to be tested and strained. So far, we've had a mild version of that, I think, with the moderation of the platforms, because we've seen what...
[0:35:43] The spread of both untrue or deliberately misleading content has done for the fabric of society. And we want to hold true to our belief in the freedom of speech, which is obviously a good thing. But it's tested it. Now, with the spread of capabilities, the ability for people to actually do things, that is like use APIs, trigger actions in third-party environments, actually initiate physical...
[0:36:12] Actions in the real world with software. That's a different quality of impact to simply the spread of misinformation where we have the traditional challenge of freedom of speech. So that's going to be the challenge that we face in democratic, liberal Western countries. But as I write in the book, the flip side of that is that for regimes that don't have any desire to pursue freedom and openness.
[0:36:41] Then they're actually going to have, quite surprisingly, the opposite sort of turbocharge of their power. Because with the centralization of identity cards or passports, with the tracking of mobile phones and IP addresses, and we're already seeing that in many countries around the world, these tools...
[0:37:02] Enable you to find the needle in the haystack that surveillance agencies in the West struggled to find, you know, debatably maybe over the last 20 years. Whereas now we actually do have the power to search through that big data. And so, you know, warrants are going to become even more important than ever in the Western world, but they're not going to be an issue in China or in Russia or in various parts of the Middle East. And as we've already seen with many of the, you know, tools, you know, Pegasus being
[0:37:30] But actually, there are lots more mundane tools for quote-unquote legitimately collecting information about citizens and tracking their sentiment, tracking their purchasing behavior, tracking their savings. I mean, many of your listeners will be familiar with the China's social credit score system. Various versions of that are going to spread and make their way around authoritarian regimes. And I think that are more likely to entrench that
[0:38:00] That power and give them an easier time of repressing various kind of sparks of uprising. So to summarize, tell me if this is correct, that you're saying that AI asymmetrically empowers the Chinese government as compared to the US government. Is that correct? Yes, I think that's true. Okay, I want to come back to that. So I want to talk about Mark Andreessen wrote an essay, which you probably read, Why AI Will Save the World.
[0:38:29] And one way to read his essay is as a counterpoint to your book. It came out before your book. You wrote your book before his essay. Basically, he is much more optimistic about the upsides and much more skeptical about the downsides. I'm just wondering, do you have any general reaction to that essay? And then I have a couple of particular questions. Yeah, I mean, I think I read the essay. I mean, I think the question for him is who's the we, right?
[0:38:57] I think if you focus just on the US and Europe, then I am as bullish as he is about the upsides. We haven't got onto this, but I actually truly believe that we're headed towards radical abundance over the next 30 or 40 years. We are massively reducing the costs, exponentially reducing the costs of producing knowledge.
[0:39:24] Everything that you can see in your line of sight at this very moment has been touched by our species' intelligence. What makes us unique is our ability to create and use tools to create comfort and happiness, health and well-being in our environment. Every piece of steel in your line of sight, every cup, every computer, every part of the landscape, we have produced that with our ability to...
[0:39:51] Process information, make predictions, do reasoning, imagine, simulate, and then create new things. And those tools, those new things, then go and improve the world around us, right? That whole construct is a function of intelligence. And we're now making rapid progress towards...
[0:40:10] Removing the biological substrate that drives that intelligence. It's not going to be dependent on our biological brain. It's going to be dependent on silicon, which can be massively paralyzed, sped up, scaled up, which can highly reliably draw on masses of data to learn from the entire corpus of human experience many, many millions of times more than any one individual can consume in their lifetime.
[0:40:39] Absolutely, I'm optimistic. I'm wildly optimistic. We are going to produce unprecedented value. This is going to be a Cambrian explosion of productivity. I predict that it's going to be the most productive few decades in the history of our species. But that doesn't mean I'm not also simultaneously able to point to some of the potential massive instabilities that might arise.
[0:41:05] And to proactively want to talk about what we might do around those things. So I don't see those as intention. Whereas I think Mark, and he said it to me before a bunch of times, is I think he just doesn't have the stomach for having the conversation about the potential downsides. And he thinks that it immediately leads to regulation. And he's got an emotional trigger, I think, about any prospect of any regulation. Okay, let's talk about regulation. Because you have... You've got big plans for your money. You invest. You track.
[0:41:36] You plan ahead. But does your financial app actually help you decide what to do next? Quicken helps you go beyond balances and basic reports. Project your cash flow, dig into your investments, and use Quicken AI to understand what matters across your finances. With Quicken, you can make sense of your options and decide what to do next. Quicken. Know what to do next. See how at Quicken.com. Twas many nights before Christmas, and one woman knew she needed a Christmas tree, so to Wayfair she flew.
[0:42:05] Finding just the right one, and for years it will last. Wayfair shipping was free and remarkably fast. What's this? Garland? A wreath? Essentials to host? Good thing she celebrates way longer than most. As her cart filled with magic and decor from the heart, Merry Christmas to all. Get your Wayfair head start. For holiday decor you can count on year after year, Wayfair delivers. The holidays are busy enough without worrying about missed calls or customers.
[0:42:36] But just because you're swamped doesn't mean people stop calling. And if nobody picks up, that call goes to your competitor. That's why over 90,000 companies choose Quo, the business phone built for busy teams. Smart call routing sends calls to the right person. Shared numbers let your team cover for each other. And an AI agent picks up when nobody can, capturing every lead so nothing falls through the cracks. Money is on the line. Always say hello with Quo.
[0:43:06] Try Quo for free. Plus get 20% off your first six months when you go to quo.com slash tech. That's Q-U-O dot com slash tech. A set of solutions. I want to frame this by talking about the titles of your first and 13th chapter. Chapter one is called Containment is Not Possible. And chapter 13 is called Containment Must Be Possible. So tell us what containment is and explain why those.
[0:43:39] Titles aren't contradictory. They are. They are. And I think that, frankly, I think wisdom in the 21st century is the ability to hold multiple contradictory and conflicting claims in working memory at the same time and use them both for rational reasoning, but also to accept the emotional consequences of those at the same time.
[0:44:03] It's absolutely true that on the face of it, it looks like containment is not possible because everything proliferates. That's been the history of science and technology in our species. If I could just interrupt you, explain what you mean by containment, please. Sure. So containment is basically the ability to slow down or stop any technology either in development or in deployment at any stage.
[0:44:30] It's a broad spectrum of the ability to slow down or introduce friction up to and including stopping something entirely. And so in the final chapter, when I sort of try to reflect on strategies for containment, my goal is not to say, this is a hard and fast, perfect solution. It's just to say, here are some sort of levers. Here are tools that we have in the system.
[0:45:00] That might allow us to introduce friction at any stage in development. And I think, just to be clear, right now, I don't believe we're anywhere close to needing a pause in AI or indeed any kind of slowdown. We're in an era where we should be racing towards the benefits as fast as possible. But I don't think it's contradictory to race towards those benefits whilst also thinking about moments when we might be open to slowing down.
[0:45:30] And that includes things like, you know, cooperating with the governments proactively in order to introduce audits and oversight and to make declarations about where we're at with respect to capabilities development. So your first proposal is what you call an Apollo program for technical safety.
[0:45:53] Just tell us what it means for these models to be safe and how confident are you that they can be made safe? It seems there's a debate about this, about whether the general models can be made safe. Every week, it seems I'm reading a story about some researchers who showed that the safety measures can be circumvented. So talk about, please, what safety is and how confident you are it's achievable and under what circumstances.
[0:46:18] So I think it is achievable. I do feel confident. I'm building one of the largest large language models in the world today. I mean, we have 22,000 H100s, which is the most powerful NVIDIA GPU chip on the market. And we have designed Pi to be incredibly safe. So if you try to use Pi today, you'll see that it doesn't suffer any of the jailbreaks or prompt hacks that a lot of the other...
[0:46:46] OpenAPI models suffer. It's also pretty guarded in its language, so it won't produce long racist diatribes and it isn't subject to a lot of the sexist bias of last year that many of the models were vulnerable to. And that's because we set out with really trying to make one of the safest.
[0:47:08] AI is in the world and put a lot of guardrails around it. And I believe that it's better to start with guardrails and then slowly release them and introduce more flexibility on the way that you design it. But I think that if you look forward, the kinds of capabilities that I think we should have an open mind to wanting to contain are actually not the ones that we see in production today. I think that we've seen that the bigger the models get, the more controllable they are.
[0:47:36] And if you pursue a safety agenda, it is possible.
[0:48:05] There are capabilities which we should be concerned about in the coming years. So for example, things like recursive self-improvement that I mentioned earlier, or agents that have the ability to define their own goals, or models that are completely general. So they're explicitly designed to do many, many, many different tasks at the same time. If you design models to do that,
[0:48:28] Then naturally, they're going to bump into false positives and they're going to misclassify or mispredict things. And that's challenging because that's also where much of the value comes from. I mean, these are inherently general purpose systems. They have many, many different uses and that's one of their strengths. But at the same time, it's one of the ways in which we can get tripped up on the safety front. So I think that maybe just to give you one other capability is autonomy.
[0:48:58] If we're explicitly designing these systems to have their own agency and act independently of human oversight, then we're taking a few steps closer to potential harms and risks that are more difficult to contain and manage. So I think that the task over the next few years is to try to honestly lay out all of these capabilities that are being developed.
[0:49:22] And start to demarcate ones which are potentially, you know, more harmful or carry greater risk, you know, and those which are acceptable and we want to storm ahead with them as fast as possible because there's huge benefit to gain from them. And then you can start to lay a containment framework on top of that. So you call for, you have a bunch of containment suggestions, audits you've already talked about, the need to verify the safety integrity.
[0:49:51] And the uncompromised nature of the system. You talk about choke points, which you describe as sensible rate limiting factors, checks on the speed of development to ensure that good sense is implemented as fast as the science involves. And there are a whole bunch of obvious questions here about how do we know the government is going to get this right? And because they off very often don't. And how do we know that this isn't going to and won't stifle innovation and put a.
[0:50:19] Put a check on innovation. And then I have some follow-ups after you answer those questions. Well, just on the subject of choke points. So with the export controls, we've already implemented a containment strategy. We've basically said that we want to deny China access to the most powerful AI models, the frontier models. And it now means that China won't be able to train a GPT-5 scale model.
[0:50:49] So just to sort of explain that and put that in context, it means that a model which is 10 times larger than GPT-4, I think a lot of people don't quite appreciate that each increment or generation of these models is in fact an order of magnitude. So it's 10 times more computation used to train the models, which obviously includes much more data at each order of magnitude as well.
[0:51:17] They'll be able to get access to a GPT-4. In fact, many of their teams, many of the private company teams in China are already doing it. But denying them access to chips is an obvious choke point, which has already been sort of triggered. You can imagine that given that these models will always depend on running on these chips, if not just training on them, right? So we still serve on the same trips that we chain on. That is going to remain a significant choke point for a long time.
[0:51:44] You know, independent of the trajectory with respect to open source. So that will be one very obvious, you know, sort of path. In terms of your point about innovation and, you know, the risks there, I think that, you know, we have to be wide-eyed about the harms, but not act, you know, too soon, right? So I don't, like I said earlier, I don't see any reason for us to take an intervention, you know, today.
[0:52:12] I do think that there's probably reason for us to take an intervention sometime in the next five years or so, right? That would be my best bet. And I think that given the scale of the capabilities involved, I think it's pretty reasonable to discuss a precautionary principle. And I appreciate that hasn't been done before and that has potential consequences for slowing down innovation, et cetera, et cetera.
[0:52:37] But I'm pretty confident that on the trajectory we're on, many, many millions of people, developers, are going to continue to have access in open source, on APIs, being able to train their own models. And I don't really see any sign that that's kind of slowing down. I think that's a theoretical risk that we should cross when we have to decide, are the potential harms greater than the potential benefits? And we'll have to wait and see when we arrive at that moment.
[0:53:05] But how confident are you that in the U.S. context, that the U.S. government is going to get that right? I mean, that they're going to implement the precautionary principle intelligently. Not super confident. Yeah. Yeah. Exactly. I mean, obviously, that does not fill me with joy. And, you know, I have deep skepticism about that. You know, one thing I would say is.
[0:53:30] I did go to the White House last month to visit President Biden to sign on to the voluntary commitments with six other frontier model development companies, Google and Microsoft, OpenAI, DeepMind, and so on. And I think that's a step in the right direction.
[0:53:48] They aren't binding commitments, but we have said publicly that we will expose ourselves to model audits, that we will share safety best practices with one another, that we will red team our models and publish our findings. They're non-trivial steps in the right direction. And I think that for the time being, that's the right flavor of what we need. So that event that you were at gave rise to...
[0:54:12] A kind of classic old traditional question. Whenever you see the leaders in an industry begging for regulation, it always raises suspicions that they're trying to lock in advantages and that new players won't be able to afford or be able to satisfy the new regulatory requirements that the big companies can. What do you say to that concern?
[0:54:35] Well, I think there's two separate issues. The first is you sort of can't have it both ways. You can't on the one hand say that the government doesn't get it and is moving too slowly and is going to over-regulate it and there's a risk that they're going to screw it all up. And on the other hand, be super skeptical when the technical experts who are actually building it sort of try to help and be supportive. So that's the conundrum that we face. That's the reality of the environment that we find ourselves in. And I think that we have been...
[0:55:05] Doing our best to genuinely and sincerely participate in trying to help governments get up to speed and get wise on this stuff.
[0:55:34] I definitely think there's going to be more and more concern about proliferation of dangerous capabilities in open source models. And we just got to make sure that we don't throw the baby out with the bathwater. I mean, it's pretty clear that these models have carried the risk of making it easier for somebody to manufacture a biological and chemical weapon. There's just no question about that. I've seen it myself.
[0:56:00] We've had independent experts come in and use all of the models at the big companies who have training in biological weapons development. And these models are great at providing coaching and feedback and support to learn about any discipline in the world. And because of the way they've been trained on the open internet, they have inadvertently hoovered up all kinds of potentially dangerous information on bomb making, weapons, cyber attacks, and so on.
[0:56:29] So one of the steps that we're all taking is to try to, when we say red team the models, we mean adversarially attack them, pressure test them, try to identify their weaknesses. And then once we identify various exploits, share them with one another. Now, there's no reason why the hosts of open source models, like Hugging Face and others, or the providers of APIs, like OpenAI and Azure and so on, can't subject their...
[0:56:57] You know, sort of hosted or API models to this kind of critical screening for, you know, potentially catastrophic risks in this case. And the proposal is not to do anything more than just make sure they're compliant with the law. I mean, this is not about inventing new law or inventing new regulation. It's just saying, like, here's a new threat vector that, you know, where existing illegal activities may now be, you know, available via another channel.
[0:57:26] with a lower cost of production or a lower barrier to entry. So that's one of the examples where I think there's collective cooperation between centralized providers and open source providers to try and remove those capabilities because no one wants that kind of thing to proliferate. And I think that's a good example of the kind of area where we still get the upside of innovation whilst being responsible about clipping the potential harms that arise in these models.
[0:57:53] So one of the things I fear, and you talk about this in the book, indeed, I think it's a certainty, is that the rise of these enhanced threats, misinformation, cybersecurity, individuals being able to learn about how to develop biological and chemical weapons, and the other threats you mentioned, is that the government is, and indeed, the precautionary principle.
[0:58:15] Is that the government, the national security establishment, is going to be increasingly concerned about these things. And it's going to want to know about them and stop them before they happen, which means significantly enhanced surveillance. And the history of growing threats in the country has been invariably met with a history of growing surveillance in the country. And is that an inevitable outcome here? How do you think about that?
[0:58:44] Yeah, I mean, this is probably one of the hardest parts of the book that I had to write because I've long been, you know, sort of privacy maximalist, frankly, and much more on the other side of that debate, skeptical of the post 9-11, post Snowden, or Snowden revealed surveillance state and the potential threat to civil liberties. You know, I think I was, you know, using encrypted communications.
[0:59:11] Before it was popular. And I think that I've definitely softened my position on that over the last sort of three to five years as I've really wrapped my intuition around the potential trajectory here over a 10 to 20 year period. And so in the book, I have this framing around the sort of potential catastrophic harms of a completely open environment versus the potentially dystopian.
[0:59:39] You know, at the same time, you know, I do think that if we haven't really touched on synthetic biology much here, but like we are now going to be making available the tools to
[1:00:08] Engineer new synthetic compounds, potentially pathogens, using desktop synthesizers that cost less than $20,000. So that means that you would still need a sort of undergraduate, maybe a postgraduate education in biology, but all the tools that you would need, plus the knowledge and know-how, is now going to be more readily available.
[1:00:31] Available over the next few years to manufacture something like a smallpox or an anthrax or anything else, or even a kind of modified version of those things which are more lethal or more transmissible. So in that kind of environment, we're going to want anti-proliferation tools. We're going to want licensing. We're going to not want it to be widely available.
[1:00:54] Just as with the arrival of drones, we don't see drones flying around all over our neighborhoods and parks everywhere, even though the technology has been invented. There's been an anti-proliferation regime, which has, for various reasons, stopped their casual use. And I would expect that kind of thing to continue as we see new forms of robotics come out. Like I said, the spread of desktop synthesizers for DNA synthesis.
[1:01:21] And various other kinds of tools and technologies. So we don't have to sort of panic here, like we need to invent some new regime. We've done anti-proliferation before. We've done containment before many, many times. And in many cases, we've got almost all the upsides with very few of the downsides. I mean, just look at airline regulation, for example, aircraft regulation. It's an incredible feat of engineering and technical governance that we have.
[1:01:46] A black box recorder on board recording all the telemetry of everything on the aircraft, plus everything that pilots say in the cockpit.
[1:01:56] And then anytime there is even a mild incident, let alone a crash, that is broadcast back to a centralized authority, the FAA, who then scrutinize it in complete transparent detail and then identify the lessons learned and then share all those lessons in new codes of conduct and training and whatever to all the airlines simultaneously, right? Overriding any kind of competitive instinct to make safety a competitive advantage. So I think there's just like many examples where we should...
[1:02:22] Take confidence from the fact that we've solved problems like this before, that we don't have to reinvent the wheel here or be overbearing and lunge towards large-scale surveillance. It can be targeted, it can be deliberate, and there's a lot to learn from past regimes. So my last question builds on that. The book is a scary book. It's a book that is heavy on things that can go wrong and downsides and possible catastrophes.
[1:02:52] But it points you, though you're more optimistic, you just painted a somewhat optimistic possibility in one context. How optimistic are you that we can have a happy outcome in terms of continuing to develop these technologies for the good and containing the downsides of the technology such that in some sign of cost-benefit analysis, the benefits outweigh the costs?
[1:03:18] Are you pessimistic about that outcome? Optimistic? Do you not know? No, I'm optimistic about that outcome. I think that the quest here is to give billions of people access to intelligence, not just access to knowledge, but the ability to invent and create. And I think that is going to put us on a path to producing radical abundance and be more...
[1:03:47] Productive and more creative than we've ever been in history. So I think the prize is enormous. We need it more than ever before. And there are a few bumps along the way that we have to mitigate. And I think that, you know, in some ways we've got to be more optimistic and confident and positive about policymaking. I think there's a general kind of malaise, like, you know, people accuse the, you know, the quote unquote masses of being apathetic. I actually think it's the
[1:04:16] The sort of elite class that have lost confidence and faith in the system more than anyone else. We have to be focused on being positive and optimistic and making things better and trying to improve the policymaking process because the upside is enormous. And I think it's less about predicting from a kind of optimistic or biased perspective and more about focusing on the practical things that we've got to try and do together.
[1:04:45] Mustafa, congratulations on your great book and thank you very much. All right. Thanks very much, Jack. I really appreciate it.
[1:05:18] Chatter, Allies, and the Aftermath, our latest Lawfare Presents podcast series on the government's response to January 6th. And check out our written work at lawfaremedia.org. The podcast is edited by Jen Patia Howell, and your audio engineer this episode was Kara Schillen of Goat Rodeo. Our music is performed by Sophia Yan. As always, thank you for listening. The quarter pounder with cheese is made with 100% fresh beef, cooked when you order it.
[1:05:51] Then comes the melted cheese, the slivered onions, the pickles, the ketchup, and the mustard. Add hot crispy fries and an iced cold Coke, all for $8. And if things go well, one loose fry at the bottom of the bag. The Quarter Pounder with Cheese meal, now $8. Ba-da-ba-ba-ba.