[0:00:00] Welcome to the Stanford Institute for Human-Centered Artificial Intelligence. I'm Russell Wald, the deputy director here at HAI, and it's a pleasure to have everyone here, both in person and online. As you know, we will be doing a book signing at the end of this talk, so please be ready for that as well. As we gather here today, there's a very fervent debate right now surrounding AI, and it's marked by these polarizing perspectives. On one end of the spectrum, some voices...
[0:00:29] are propelled by industry hype, which claim that AI is a panacea for all of humanity's challenges. And on the other one, there are dire warnings of sentient machines leading to nuclear catastrophe, painting a picture of AI as the harbinger of the end of the world. At HAI, we're dedicated to untangling the hyperbole that often surrounds AI, endeavoring to foster a nuanced and balanced comprehension of its capabilities and limitations.
[0:00:58] It's in with this context that we're thrilled to host today's Fireside Chat with Mustafa Soloman and Marietje Schalke, who have both been at the forefront of this critical moment for the governance of AI. Mustafa is the co-founder and CEO of Inflection AI, and Mustafa's contributions to AI from his early days of co-founding DeepMind to his tenure at Google, where he served as vice president of AI products and policy.
[0:01:27] have been pivotal in shaping the field and its discourse on societal impact. His latest work, The Coming Wave, Technology, Power, and the 21st Century's Greatest Dilemma, explores the profound transformations AI is expected to bring to human abilities and society, highlighting both potential risk and groundbreaking innovations. Mustafa's insights
[0:01:56] Draw from a rich career spanning over a decade in groundbreaking AI research and development. And this entire talk, I think, is going to be promised to be fairly enlightening based on that. Mustafa will first talk to us about his book, and then he'll sit down for a fireside chat with Maricha Shaka. Maricha is an international policy fellow here at HAI and international policy director at the Stanford Cyber Policy Center.
[0:02:25] Between 2009 and 2019, she served as a member of the European Parliament for the Dutch Liberal Democratic Party, where she focused on trade, foreign affairs, and technology policies. She writes a monthly column for the Financial Times and serves as an independent special advisor to Executive Vice President of the European Commission, Margrethe Vestager. And Marietje also serves on President Macron's
[0:02:53] Tech Thinkers Advisory Council. And so without further ado, please help me give a warm welcome to Mustafa. Thank you very much for the warm introduction and thank you for hosting me. It's an honor to be here. I thought I would just make a couple of maybe three sort of bold claims that might be challenging, controversial that we could use to kind of our conversation and maybe could kickstart our Q&A.
[0:03:28] Maybe before I begin, I sort of was reflecting on where we are in this very moment in the context of the last year and the last five years. When I wrote the book, it was pre-ChatGPT, and I had no idea how the world was going to react to the arrival of AI. I was very lucky because in 2020, when I started full-time at Google, I had the opportunity to work on Lambda.
[0:03:57] For 18 months, Lambda was language model for dialogue applications, which was actually the backbone of what is now BARD. And so I got to see firsthand before I think anybody else in the world, along with other people at Google, what it would be like to have an interactive language model. Previously, GPT-3 was really just a kind of question-answer engine, and people hadn't really thought about it beyond just a prompt and thought about it in a conversational style.
[0:04:27] And my main motivation at Google was to try to make this the future of search. I've always seen search as an iterative dialogic process. Inherently, you present a question to Google. You review the 10 blue links, which is Google's answer. You select one of those links, go to a web page, and evaluate the quality of that page based on how much time you spend on it and what you click on that page.
[0:04:56] And then you go and update your query. And the change that you make to that query is kind of a response back to Google. And then you go and iterate. And I actually have a patent for that framing that goes back to 2011. This concept of an iterative interaction with an AI was very fundamental to my thinking, both at Google and at DeepMind. But in writing the book, I had no idea how people were going to react to the arrival of AI.
[0:05:24] Kind of worried that actually people would see it far more spookily and far more scarily than it actually is. It's kind of incredible to me that now, you know, the first wave of this AI moment has actually landed, you know, reasonably safe. It's actually not caused that much harm today. It's actually not created the kind of dependence that it might otherwise have, or at least in my imagination back in 2019 and 20, it could have.
[0:05:54] I think that's one provocative consideration for us to discuss. What is the objective harm that it has done today? What do we predict? How do we predict that harm might change over the next few years or five years? And then what proactive or preventative measures should we be taking to mitigate? This is very funny getting this odd feedback. How do we expect that to change over time? So that's one thing.
[0:06:23] Second thing I would say is that the progress that has been made in AI over the last decade has actually been relatively predictable. A small number of us have made an assumption that new capabilities will arise with every order of magnitude more computation we add to these training months. And that goes way back to 2012 and 2013.
[0:06:50] To give one example of this that I have sort of found quite inspiring and surprising over the years. When we first trained the Atari DQN player back in 2013, which I'm sure many people will be familiar with, but was the first real effective integration of deep learning and reinforcement learning where the algorithm essentially learned to play
[0:07:16] 50 of the old school Atari games to human level performance simply by parsing the pixels frame by frame. It would learn an association between a set of frames in the run up to a rewarding action and use that association to then reproduce that action in ways that will ultimately produce the performance that was as good as most humans are playing those games.
[0:07:44] That was the first time we had ever really seen an AI learn its own representation of rewarding behavior well enough to teach us as humans something new. It learned a bunch of new strategies in the process. And it was very impressive. And I think gave us a lot of confidence. Actually, it was one of the main drivers that got us acquired by Google. I mean, Larry Page saw this demo and was really quite blown away. So that used something like two petaflops of computation.
[0:08:12] As many people know, a flop or a floating point operation is like a single calculation. Peta is a million billion calculations, a million billion calculations. They use two of them. Roll forward to this year in 2023 at inflection, the model that we have just finished training, which we're going to announce in a few weeks time, uses something on the order of 10 billion petaflops.
[0:08:43] This is about 10 orders of magnitude more computation, a remarkable increase. And with each order of magnitude, we have seen quite systematically new capabilities arise. And what are those capabilities? So I would say the overriding capability that we've seen is that of control. The larger the model, the more responsive it is to instruction and control. Now, on the face of it, that's actually a really good thing.
[0:09:13] Because I would argue that that makes them safer. They're going to be more responsive to human direction. They're going to be more adherent to your behavior policy, or your prompt, or your fine tuning, whichever method you're using. Now, I'm making that prediction. I think it's pretty likely to hold. The flip side is that it will be more responsive to anybody's instruction. And so we sort of regress the control and safety problem.
[0:09:41] Back up to the individual who is using the tool. And I think this is maybe a controversial claim because I think the assumption that many people have been making over the last few years is that we're forever going to be stuck with hallucinations. We're always going to be stuck with bias. A number of times I hear people say, bad data in, bad output out. You actually pay attention to what's happening. We sort of have that problem less and less and less. It doesn't mean bias goes away because we humans have bias. Our data has bias.
[0:10:11] We can now mitigate that bias in the control policy of be it the prompt or the fine tuning method. So it sort of regresses the problem to actually who is making that decision, to whom are they accountable, to what extent do we have transparency over who is on the receiving end of a decision by an AI. And that actually is a more traditional governance problem that we can wrap our heads around. And I think we should make the assumption
[0:10:39] But hallucinations are going to largely disappear, that the question of bias systems is largely going to disappear in the way that has been currently framed, and instead focus on the traditional challenge of governance. Who owns it? Who controls it? To whom is it accountable? What is our regulation going to do? And it's that framing that I have put much of my career to, actually, and is part of the reason why I founded DeepMind, to try to really focus on the governance question, the ownership and control question.
[0:11:08] The third thing that I'll say just to set some context here for our discussion is that what capabilities do I think are going to emerge over the next two or three orders of magnitude in computation? So roughly speaking, we're at 1 times 10 to the 25 today in terms of flops. We will get to 1 times 10 to the 27 within 18 months. There'll be three or four labs that get there. Obviously, DeepMind, OpenAI.
[0:11:39] Anthropic and Inflection. And maybe there'll be a few others. We have the computation today to train those models. So what do I think is going to happen? What kind of capabilities are going to... I don't like the word emerge because it implies that they're sort of uncontrolled. But what new capabilities do we reveal at each order of magnitude? Well, today the models are simply framed as one-shot question answering engines. You present a prompt or a question and it will basically give you...
[0:12:08] Some output of varying lengths. Maybe it's 2K tokens, maybe it's 4K tokens at the max. I think what we're going to get next is not just a single sort of output or a prediction, but an entire string of accurate predictions that all relate to one another in sequence. So that basically means that it will have the ability to do things like planning. What we don't have at the moment is any sense of state.
[0:12:38] or time and people are trying to engineer this in but not inside the actual weights of the model itself and I think it's both going to be able to store things in its working memory and it will be able to use tools at the right moment, i.e. call out to some API, generate an image, write an email, make a phone call and string these elements together in order to kind of coordinate those elements with respect to an abstract goal.
[0:13:07] You might give it some abstract goal. And if you squint a little bit further out and you buy what I'm saying, what I'm actually describing is a project manager. What you have at that point is a pretty competent business analyst, operations person, HR person, project manager, someone managing accounts or payroll. They'll be able to use email. They'll be able to make phone calls. They'll be able to parse contracts or agreements.
[0:13:37] And essentially interact in the workplace. And so I think that rather than focus on the question of recursive self-improvement, of intelligence explosions, of provable bounded control within contained environments, important questions, but I think they are not in the next two orders of magnitude. In the next two orders of magnitude is the fundamental question of labor. We really have to address this question because
[0:14:04] In my opinion, these are fundamentally labor replacing technologies. They are temporarily augmenting technologies. And sometimes we get caught up on the idea that this is going to make us more productive and smarter and more efficient. And of course it will. But fundamentally, this is going to return significant value to the organization that deploys it. So the question for that organization, let's just say it's a business, but it applies to other organizations, is what do you do with that efficiency saving?
[0:14:32] Who benefits from that marginal saving? And the obvious answer for a company is going to be the shareholder, right? There is a value that is going to be captured. That value is going to be returned principally to shareholders. Some of it may be reinvested in further R&D, but there is going to be an efficiency saving which will go to dividend. Because fundamentally, the trend that we see here is to move from labor to capital. We are turning...
[0:15:00] Stuff which was otherwise income generating on the labor side of the equation into stuff which is going to be compounding faster than ever before in capital. And to me, that's a governance question that comes back to what are the rate limiting factors that we have at our disposal to tune the pace at which this transition takes place? Because it isn't going to stop. And I mean, that was sort of one of the main motivations of my book. I wanted to really explore what are the underlying drivers
[0:15:30] In this ecosystem, which leads to not just inevitability of the outcomes, but ultimately proliferation of the technologies. And I listed a bunch of things. I'm happy to talk about them in detail. But fundamentally, all of the incentives are driving this forward in a very, very accelerated way. And I think that if we're to intervene in that process, we should think about adding friction. And friction really is about the traditional mechanisms of oversight, which
[0:15:58] We might be frustrated with because they're sort of broken at the moment. When I say political process, a lot of people in this room won't. But many rooms I talk to just roll their eyes and slightly snore and so on. And actually, I think that we need the political process more than ever because fundamentally, this is going to be about tax and redistribution. Fundamentally, it's going to be about governance and oversight and the traditional mechanisms rather than reinventing the wheel. So with that said, let's have a chat.
[0:16:36] Sounds like this may be the finish line of your book tour. And there's a lot that you have said about the book. And before we dive into questions of how to weigh the harms of the models that we see, how to think about governance, one of my favorite topics, I wanted to invite you to zoom out a little bit and share more about how you came to the views that you have on the societal impact, the importance of governance.
[0:17:06] And tell a little bit of your personal story, sort of where you were when you were the age of some of these students that are in the room, because your personal journey to where you are now is not a usual one for a tech CEO. So I think it would be great if you could share some of that. Yeah, sure. I mean, you know, as you could hear from our accents, we're Europeans. I still consider myself a European despite the Brexit catastrophe. So yeah, I have a very unconventional sort of background for it.
[0:17:36] I started my time at Oxford. I was doing philosophy, and I essentially dropped out of my degree to start a charity, which I ran for three years, a nonprofit, a telephone counseling service, actually, for young British Muslims. When I got to Oxford, very quickly became an atheist, discovered human rights principles, and that was probably my principal moment of transition in my life, is moving from a
[0:18:05] Sort of default belief that as a religious person from a very strict religious family, this idea that you're part of the chosen one, like God has appointed you and your people to carry forward the message and all this kind of stuff. And I had never quite really sat comfortably with that. And then when I got to Oxford, I was sort of introduced to an intellectual framework that enabled me to escape from that. And I'm still principally.
[0:18:34] Motivated by human rights. And I think that is a more modern post-nationalistic framework for thinking about justice and our responsibilities to one another. And so in a slightly sort of reactionary way, I kind of rage quit Oxford and was like, well, I should be doing good now. And I left to help start this charity. And the charity was a secular.
[0:19:03] Non-judgmental, non-directional support service for young people. And it was the first of its kind to build an online support service. So we built the first website. We had the first ever SMS counseling, text message counseling service. And the idea was that we would train young people to deliver peer-to-peer counseling. And that's kind of my background and what I really enjoyed doing. And I was blown away by the fact that technology back in 2001,
[0:19:32] 2002 was actually going to enable us to kind of scale our impact, even though it was very, very small. And then from there, I started a conflict resolution firm called Rios Partners that specialized in multi-stakeholder change processes. And I worked all over the world as a negotiator and a facilitator in post-conflict reconstruction zones and so on. And I sort of got to this point like 2008, 2009, where I was
[0:20:02] Really quite frustrated with the work that I was doing and the pace of change. And it felt very myopic and very narrow. And I was kind of looking across the pond to Silicon Valley and seeing the explosive rise of Facebook. And I think Facebook had just hit 100 million monthly active users. And I was just like, how is this possible? And I obviously started using it. And I was like, wow, this is deeply political. To me, it was very obvious that this was.
[0:20:31] Shaping connection. It was framing the choice. I had just read Nudge, the Thaler Sunstein book, which was everywhere at the time. And I was like, oh, choice architectures. Oh, information. This is interesting. And so to me, I was like, well, where are the non-technical people who are deeply understanding and participating in app design and service design and so on?
[0:20:57] And so I sort of set about this quest to kind of teach myself, you know, how to navigate in Silicon Valley and got loads of introductions, came to visit and ended up meeting my friend, Demis Sassavis, who was the CEO of DeepMind and our other co-founder, Shane Legg. And they were both finishing their post-doctoral work at UCL at the Gatsby Computational Neuroscience Unit. And we started hanging out and dreaming about the future. And we decided to start an AGI company. The rest is history.
[0:21:27] The rest is history, but at the same time, you know, all of these questions about governance, human rights, the whole political context that you saw early on is now sort of back on the forefront where it was absent, you know, for maybe the 15 years in between where people believed that the technologies themselves would bring all these democratizing positive forces and where politicians of all kinds, of all parties took a hands-off approach. Right. And now you're...
[0:21:56] Emphasizing in your book, but I think a lot of people have come to the conclusion how very political AI is and how pressing the challenges are. So coming back to what you said in your introduction, which I found remarkable, is that you think that the harms of this wave of generative AI have not been as significant, even if we would say that around civil society organizations, they would immediately object. And you yourself have recently also pointed out
[0:22:23] The risks of synthetic media in the context of election, for example. So can you talk a little bit more about why you think the harms have not been as bad or why they might be manageable? Yeah, I mean, I remember back in 2010 when we were fundraising, really thinking like what harms were arising at that moment from Facebook and from social media. And obviously I didn't foresee any of the harms that subsequently arose at that time. But I remember...
[0:22:51] Sort of meeting people who I guess sounded just like I sounded. For example, Peter Thiel was our first investor. So we tried very hard to get a wide range of investors and nobody was interested in the idea of AGI. He was basically the only person who gave any sort of credence to the idea that we might have super intelligent machines. And so we ended up taking his money.
[0:23:18] In a way, that was actually very helpful because my default skepticism of his worldview helped us frame the company explicitly around safety and ethics. So our business plan was explicit on the front page was like building safe and ethical AGI for the benefit of humanity. And it really shaped the entire culture of the company. And when we were acquired, we made it a condition of our acquisition that we...
[0:23:47] We have an ethics and safety board that our technology could never be used for surveillance purposes or for military purposes. And that was actually a hangover from a reaction to PETA, because PETA had invested in this autonomous robotics, military robotics company at the time, which had these kind of little things with drones with machine guns on. And I was like, well, OK, this is very clear that we never want to be part of that.
[0:24:11] And I think that then ended up shaping both us and Google and a lot of the field in the importance given to that. But obviously, I recognize that today, maybe we're also sleepwalking into similar traps. And I think there's a few of them that you mentioned. For a start, we are about to be inundated with synthetic media. It is going to be human-level quality. It will be indistinguishable from anything that a regular
[0:24:40] Human Canon will produce. And I think it's going to completely change the information ecosystem because it's going to be cheap to produce and require no technical expertise. So one of the mitigations is that I think that we should be calling for any electioneering that a chatbot might engage in to be banned, right? Which I think is a pretty basic, basic thing. You can only participate in electoral persuasion.
[0:25:10] If you're a human, right? And we don't want to have all these character bots or the meta bots or the new, you know, sort of GPTs that are going to be released by OpenAI persuading us on who to vote for. Now, that in itself is kind of a controversial view because frankly, they're likely to be pretty accurate, right? Compared to the average human that you might turn to to get advice on, you know, who said what in an election campaign.
[0:25:39] Or who did what in a previous term in terms of what they've been successful in actually getting done in their position. They're likely to be quite accurate. So if you take what I originally said about not really going to be suffering the hallucinations, not suffering bias, then we're actually going to lose something by not having them in the political process. But to me, that reflects the precautionary principle. I think it's right to say the burden of proof should shift to technology itself.
[0:26:08] Instead of assuming that by default it's going to be good, we have to flip it because of the scale of its influence and partly in learning from the social media age. Right, and looking at what we've learned there, I think there's a very gradual scale between obvious political content, who to vote for, when to vote, who not to vote for, or allegations about a certain candidate.
[0:26:35] But there's also a bunch of information that can be very impactful in an election. I don't know, hot topics like immigration or crime or climate change or rather denial of climate change. So how do you imagine this working? Like who should govern this and what kind of criteria do you have in mind? Because I think it's a very important topic, but I just don't quite understand who would be making the decisions up to where. Yeah, great question.
[0:27:05] I also have a slightly controversial view on this point, which is that pragmatically speaking, at least at this stage in the development of the technology, power sits in the hands of a small number of actors. And actually, that's a good thing. I know that you and many others would obviously say it's very dangerous, and it is dangerous, right? But I think pragmatically speaking, that's where things lay at the moment. And so what we've been trying to do...
[0:27:32] Partly with the kind of AI safety summit, which has just happened in London, but also just in the way that we've all structured our companies, the current round of AI companies, is drag big tech to the table with a set of default expectations about what good behavior looks like to shape the rest of the ecosystem. And this is just making an assumption.
[0:27:56] But we have to operate in a pre-regulatory environment because the EU AI Act has been three and a half years in discussion. We're still six months away from it being agreed. And then I don't know, is it another year after that until it's enacted or something like that? It's not 2025 until it's definitely not this round of elections when we have two billion people going to the polls next year in India and Indonesia and obviously in the US and also in the UK. And of course, the whole EU. Two billion people is extraordinary.
[0:28:25] Concentration of voting, yeah. Right, right, right. So pragmatically speaking, we have to just start with trying to say, I'm trying to get agreement that all the other chatbot providers will also fine tune this out and prevent this as a capability, just as we prevent copyright generation, just as we prevent biochemical nuclear generations, et cetera, et cetera. And I think that's just a good practical start.
[0:28:51] Let's dig into that a little bit deeper. We were both at the AI summit. I was also recently in another discussion where there's a kind of dynamic where most of the tech CEOs say, we want to do the right thing. But of course, politics is all about what is the right thing, because nobody's ever going to say, hey, this is me wanting to do the wrong thing. So politics is all about what the right thing is, what the trade-offs are, what the priorities are. And the things that worry me, and I'm not the only one, are when this handful of companies and
[0:29:20] Then a handful of people running these companies say, we see the risk. We know there is risk to human rights, to democracy. There's bias in our systems. And the list goes on and on all the way to the more existential kind. But that's why we have safety teams. That's why we're doing this, that, and the other. And the lack of independent assessments, the
[0:29:45] Concentration of decisions within these companies in an intransparent way, I think is what worries people from a governance point of view. And so how, when you say I'm actually assured that there's a handful of people and you say that you're able to talk to them, maybe get to agreements, like why does it not concern you from a sort of democratic?
[0:30:04] I totally agree with you on the verification piece, right? And what we lack in our societies today in general is a public interested infrastructure that can move at the pace that's necessary and is also trusted, not just because it's impartial, but trusted by the companies that it's going to act rationally over time. And I think that's actually that is the missing key piece is I think the companies would trust.
[0:30:31] Outside actors. Because if you think about it from a selfish company perspective, you know, the companies are looking for that badge of honor, you know, the organic association. They're desperately looking for that. And there isn't really a sort of a commercial infrastructure that can be relied on to consistently produce that, right? And one take is that, okay, we should actually encourage universities to provide that, right? And I get pitches all the time.
[0:30:58] from you know people who are sort of PhD programs or even postdoctoral work and I think the challenge there is that there isn't enough stability for the company to be able to rely on you know because a postdoc will do their work two years later they'll rotate through maybe they'll go work for a competitor how do you manage the IP issues you know because you're fundamentally going to be exposing very sensitive material so I'm definitely personally invested and I know other companies care about it too
[0:31:25] In creating this kind of impartial, fast moving structure, increasingly, the countries now are trying to fill that void. So with the AI Safety Institute, there's a US Institute, the UK Institute, and they have now some resources. They're technically pretty competent, I think, and they're going to stand up a group to do this, although their focus is very much on national security risks. As you know, they're really just like bio risk and so on.
[0:31:54] And so, I don't know, I mean, I guess it's kind of an interesting question for you. So what do you think the structure should be? Well, I think one of the questions coming out of these announced safety institutes is really what the definition of safety is going to end up being, because you can stretch safety into almost endless corners as well. And, you know, if governments want to empower themselves and want to use these institutes in a more expansive way.
[0:32:18] Then they can say, well, look, this information is an existential risk to democracy. Discrimination is already illegal, so we're going to have to look at the safety impact for people. If they take a narrow lens looking at national security, I don't know how much real impact it's going to have on the issues that we talk about, like elections, synthetic media, and other known problems. And I think that would be a missed opportunity because it strikes me that a lot of governments
[0:32:45] Whether they state it officially or not, they have to make sure that the laws that are on the books already apply. It's basically the line of the Biden administration, maybe because expectations from Congress are quite low, but the Biden administration basically says, we will enforce laws that are on the books. It's just not as clear as it should be, in my opinion, how. And that's where the opacity of business models, the ever-changing nature of...
[0:33:12] Machine learning and algorithms and self-learning systems make the regulatory process, even if it is predictable and known, difficult to enforce. And so that's what touches upon your speed question. And so one way to think about this is to put more obligations on the companies to one, comply with laws that exist.
[0:33:41] To have stronger sanctions or consequences that are negative for the companies when they don't. So I think that would be a good starting point to really double down on how must we interpret the law that is on the books. If I were a government, I would almost tenfold the budgets for enforcement agencies because the resources are extraordinarily small to actually look at companies of the complexity that the big AI.
[0:34:10] Companies now have. So it would probably have to mean a kind of turning around of the legislative versus enforcement model. More principle-based laws, more flexible but effective enforcement. Yeah, I think all of that makes sense. But I think pragmatically speaking, expecting the countries to move fast enough is dangerous because I also hear a lot of, I mean, I hear a lot of like corporate...
[0:34:38] Lobbyist type people say exactly what you just said. Just, well, you know, of course we'll comply when there is a competent regulator that can move fast and, you know, and it kind of just like passes the buck over to the regulator. And then the regulator is like, well, under-resourced and facing all the political challenges that we're all aware of. So, and maybe I'm just too half, you know, glass half full about it. But like, I think that...
[0:35:00] It's a pretty big deal that all of the companies have signed up voluntarily to these set of commitments that have then been enshrined in the executive order, but then got followed on by the UK process. The UK process is now going to add all these other G7 countries, and that's going to become an every six month summit, right? Which all the companies are going to, they can't not turn up now, right? That's a pretty big deal. And before there is any material harm. So it's not like we've seen the harm from social media happen with AI.
[0:35:30] Guessing and predicting. Now, I don't think that's happened before in the arrival of the technology or in the political process, right? I'm not sure. I don't think it has. So I think that's a positive sign. And we can use that momentum to take the next step. And to me, the next step is, how do we fund independent, public-interested, stable, long-term, impartial auditors, verifiers, investigators that are not going to be too sort of
[0:36:01] You know, vulnerable to activist outbursts and scorning the companies because that creates a very adversarial environment. I've been in that multiple times where I've tried to bring Google to the table with Google's AI principles, Google's ethics board, for example, which ended up getting, you know, shut down within a few days because of a big campaign and so on. So I think we have to figure out what in a very low trust environment, what is the mechanism that is going to enable very practically
[0:36:29] The companies to cooperate with some kind of impartial pre-regulatory and still invest in the regulatory side. Right. So I think that's fair. And I think, for example, the antitrust model is a pretty good one to look at where regulators are quite empowered to also look at, for example, confidentiality, which would meet some of the needs of companies where the mandate is clear, but the companies and their rights are also respected.
[0:36:55] But I want to come back to the process that you were describing among companies. You know, they come to these summits, they come to the table when the White House calls. Can you talk a little bit about the dynamic between the CEOs? Is it easy to come to sort of one line or to get to these agreements? Some people are at the table, some people are not at the table. We hear different schools of thoughts also among companies, obviously informed by what would benefit their business model. What is your perception of the sort of culture among CEOs?
[0:37:25] Well, I mean, so let's just be clear. So far, it's really only the companies that are developing the very largest LLMs. I mean, it does only affect the frontier models. And in fact, the voluntary commitments that are now in the executive order will only affect models that are one times 10 to 26. The models that haven't yet been trained in the world. So there's a bit of a, that's both a problem and also a benefit, right?
[0:37:52] The problem, I think, is that it doesn't capture all the stuff that's kind of happening today. But the benefit is that, you know, contrary to what a lot of people tell you on Twitter, it doesn't affect open source. This isn't about pulling up the ladder for the open source folks. It's actually just about regulating what is today the absolute frontier. And I think that will sort of cover the next two or three years as well. Sorry, it won't affect open source over the next two or three years because I don't think a 1x26 is going to get open sourced anytime soon.
[0:38:21] So, you know, it really is about just the big companies. I think that's the first thing to say. And then I think the second thing is there's sort of three classes of company in this category. There's obviously big tech and then the four CSPs, right? The cloud service providers and Facebook. Then there's a bunch of sort of collection of smaller companies that are on the open source side, the hugging faces of this world.
[0:38:50] And then there's really the four sort of like, you know, basically, you can call them AGI labs, but it's, you know, sort of DeepMind, OpenAI, Anthropic, and myself. And I think what distinguishes the four of us is that we all have, you know, sort of, I would call them public interested governance models, right? They all have different strengths and weaknesses. You know, the DeepMind one is probably the most basically business as usual, big tech.
[0:39:20] Because we started early and so we didn't have as much leverage to be able to win the kind of negotiating battle of truly having like ethics and safety in the legal charter of the company. But that then had a massive impact on the way that OpenAI founded in 2015. Because they had just seen that we had done this ethics and safety board on our acquisition in 2014. So they then first started as a nonprofit. Then they slightly reshaped as this capped profit thing. And we can debate the...
[0:39:48] Pros and cons of that, but there's definitely an intent that is broadcast that is a useful accountability metric. And then when Anthropic came out, obviously they then, you know, I think it was 2020 or maybe 2020 or so, they found it as a public benefit corporation and Inflection is a public benefit corporation. And I'm a huge fan of public benefit corporations. So we're still a for-profit company, but unlike a traditional for-profit company, our directors have a fiduciary obligation to make decisions which...
[0:40:17] Equally balance the interests of the shareholder and the quote from the legal doc, those materially affected by any activity of the business. And those materially affected includes the environment. It includes people who aren't customers. So it's not just like do the right thing by your customers, but it's anyone who society more generally that is affected. And I think that this is actually the missing piece in evolving capitalism. I think it's how we internalize the external stakeholders
[0:40:47] They have so far just been neglected as a kind of afterthought. And it's unclear how these will be tested in the courts because it's going to take a few years for a public benefit corporation to make a mistake and then have a class action lawsuit. And then that actually then requires a board of directors to expose their decision making process and their reasoning. But I think it's a pretty big step forward. It doesn't solve all the problems, but I think it's the right direction of travel.
[0:41:15] And I think it's that dynamic which has helped bring your Amazons, Google, Facebook, Microsoft to the table because it's like we're setting the agenda on the AI front. They're obviously huge supporters and they believe in it too. But I think that proactively signing up to those voluntary commitments is probably not something that would have happened independent of that movement. And I think it's a sign of the times. I think it's something, again, like it's something that we should be optimistic about and positive about. This is a really good.
[0:41:45] You know, many step forward in the evolution of our governance. Is it also a sign of some worry and anxiety among the tech leaders of how much responsibility they really have? I mean, looking at, for example, the pause letter that was signed by a lot of AI experts that called for a six month pause of the development of further frontier models. You could read that as a sort of attempt to point
[0:42:13] to the policymakers to take their responsibility where, you know, you just mentioned if something happens, there will be a court case where the signatories could say, well, we warned the policymakers, but they just didn't solve the policy in six months, which having been a policymaker is completely unrealistic anyway. But the way you could read that letter is there is a bit of anxiety on the part of the corporate leaders that this technology is powerful. What if something goes wrong? Do we want to?
[0:42:41] You know, have the sole responsibility. Is that a dynamic that you are perceiving as well? Yeah, I mean, I think that it's an anxiety that the nation states aren't sufficiently competent to adapt, that the pace of progress is surprising even us who are making it, and that I think the world is still under appreciating the consequences. I still find it mind blowing that people don't seem to be fully grasping the trajectory that we're on. I mean,
[0:43:10] We are going to produce a replacement to cognitive manual labor in the next couple of years. How much of a bureaucracy is fundamentally about cognitive manual labor? A protocol, a repetitious, straightforward series of tasks that can be automated. That's going to have a massively, massively fundamental
[0:43:36] Shift to the labor market. Let's just focus on the labor market consequences for a minute. If structural disemployment goes up by 1% or 2%, the financial cost of managing that transition is phenomenal. The impact on populism and the political process is phenomenal. So I just feel like that's the conversation that we should all be having, not like the national security issues of the bio risk of the nuclear.
[0:44:02] Are we going to put more pressure on companies to not break the law and comply with what's already illegal? We want to be a bit more creative than that. We have to be a little bit more bold in making these predictions because if we're just on the back foot responding to something like I've said, and obviously I hope I'm wrong. There's still a good chance I might be wrong. It may be that it takes a lot longer to seep into the labor market than I've described. It may be that every worker...
[0:44:28] Just like with Excel or with the calculator, this is often the rebuke that I hear, is that it just makes them much more efficient for a period of time. But even to that argument, it's always going to be the case that we'll have four roles that have to go down to three roles or five days a week that go down to four days a week. So there's a saving that is being produced by this efficiency. Absolutely. And I thought it was remarkable that at the AI summit, nobody talked about it, except Elon Musk.
[0:44:56] A remarkable kind of staged interview by the prime minister, I guess. But I mean, my own country, there are elections, the Netherlands, in two weeks. And AI and technology developments are not a topic at all on the agenda. So I wonder how come so many politicians cannot grapple with this reality. Is it because it has simply not landed sort of beyond the Stanfords of this world, the Silicon Valleys of this world?
[0:45:25] Or is it also that it's really hard to create scenarios? Take us along in what you think the most likely scenario is, because I fully agree with you. Even a few percentage points change in employment will lead to political upheaval. Or if one specific sector is really impacted, that's going to concern a lot of people. Or even the perception of job loss can lead to political ripple effects.
[0:45:53] You also said, maybe I'm wrong and it's hard to know exactly, but would you do your best effort to assess between now and five years, sort of how powerful these systems will be and how easy they'll be taken up and what kind of jobs might be at stake? Well, I mean, look, you know better than I do about the state of the political process. I mean, the reason politicians aren't talking about it is because it's easy to ignore long-term structural trends.
[0:46:22] Especially ones which are quote unquote speculative, like the claims that I'm making. And, you know, the obvious question is, well, what do I do about it as a policymaker? And if there's no answer to that question, then it basically doesn't get any airtime. And it's one of the fundamental failings of our political structure, right? We don't have any muscle for long-termism. There's really no capacity for thinking about the 10 or 20 year investment horizon.
[0:46:50] So what does it look like? Well, I think that we've also got to consider the impact of open source because the cost is going to get incredibly low. The complexity of integration is going to be very, very low. And so you have these dual trends. There's both the APIs provided by the big CSPs. And I think people should pay a lot of attention to Amazon and Microsoft because they already have 70% or 80% of the S&P 500 companies.
[0:47:18] Data between the two of them. Right. Right. And so the cost of that integration, people often say, well, you know, it takes a long time, seven years or whatever, to go from idea to actual impact in production. And, you know, look, it might take a while, but we're ignoring the amount of infrastructure groundwork that has already been laid because there are account executives that already have data lakes that already have.
[0:47:45] You know, sort of control infrastructures back inside an organization. Look at the speed of integration. Pay less attention to the OpenAI thing and more attention to the Microsoft co-pilot thing. You look at Kevin Scott's speech from Build three or four months ago, Microsoft Build. It's unbelievable what those co-pilots are doing, like reasoning over huge databases of documents.
[0:48:11] Generating new Excel sheets, producing graphs and analyses on the fly, obviously then writing emails with those. I don't know. My intuition is that that ends up happening much faster than a lot of people realize. Just take telephone call center operatives, for example. Clearly, there was a study out recently. I think it was someone here in Stanford that was temporarily made or it made
[0:48:40] A call center operator who is less skilled now more skilled. They upskilled them because they were now able to actually engage in a sales negotiation or a persuasion because they had this sort of prompt with them. So you can clearly see that it's climbing the ladder of human capabilities. And so I don't know, my prediction is that that integration is going to happen faster than people think. In your book, you also talk about taxation.
[0:49:08] Which is also striking to me that the whole redistribution question hasn't really come up much in many of the discussions. Some people have suggested there should be maybe a one-off tax to deal with the risks or things like that. But are companies preparing for being taxed quite significantly on their revenues for mitigating societal harms? Or is it something that may just not happen because this is the United States?
[0:49:33] How are people thinking about that? I don't think so. I don't think that is top of mind in the scenario planning. I mean, you know, in some ways, companies, the big companies do a better job of governments at long-term planning. But even that is, you know, I think they're making a judgment that, you know, countries aren't going to be able to react quickly enough. And I think that might be flawed too, right? I think it may be the case that if we do see significant disruption, you know, in five years time.
[0:50:02] Then maybe that will be the thing that unlocks the political process because it may swing, at least in this country, to a sort of administration that actually does have control of both houses and the executive and can be much more aggressive in forcing through significant reforms on the taxation front. So I think that we should keep an open mind to that.
[0:50:24] No, I think the whole redistribution question and also the broader question of who has responsibility for what, you know, what is the government's role here and what is the role of companies? There's a lot of confusion right now. And one of the things that I noticed at the AI summit, I don't know if that was something that you saw as well, was that some of these leaders in AI were almost starstruck that they were sharing the stage with, you know, prime ministers and government leaders. I mean, I heard somebody saying like, you know, I've...
[0:50:52] I've imagined having a summit with governments for a decade, and now it's happening. I have to pinch myself. In a way, it was cute. In a way, it was also a bit concerning. And I think concerning also in the sense of when I listen to some of the policy analysis or proposals that are being made by people who've mostly studied AI, they're really quite basic.
[0:51:20] Vice versa, of course, there's a big challenge to get governments to be up to speed about what AI is and isn't. But I think that this is also, if you think about this from a kind of like the absurdity of the performance perspective, like there was no need for this ridiculous entourage. I mean, it looked like, I've never seen anything like it. Talk about the AI summit. Yeah, there's like five or six different huge purpose-built tents. Everything's like white carpet everywhere.
[0:51:49] you know brand new neon seating all over the shop it's like the huge production like you're kind of walking onto a film set right and there's like cameras everywhere and so it is kind of absurd and there's sort of no reason that that performance dehumanizes us right like to to me it's a way of like encouraging somebody to play their role instead of allowing somebody to arrive as a human
[0:52:17] Connecting with their personal emotional sense of what's happening and feeling that sense of obligation. Instead, it's like this sort of very grand with each person walking around with their big entourage. So I think there's something to really think about in terms of how we curate governance physically. What is the process that we put around conversation? So we had the summit, the actual summit was this.
[0:52:44] Ridiculous 30-person circular table, literally the length of this room, but, you know, in a circle. Yeah, literally. And, you know, all the huge flowers and, you know, the name tags with everyone sitting around and, you know, you press the button and speak like, you know, and obviously then it just forces like formal, awkward conversation. And really the real summit happens in the 10 minutes before that moment and the 10 minutes after.
[0:53:12] Where all the world leaders and CEOs are shaking hands and having a human-like conversation, right? Well, actually, what was remarkable is that because we were made to go, I don't know if that was the same for you, but we were made to go through security so early, which was also staged in a significant way, that we had like two and a half hours before the whole event started just having coffee and talking. And to me, that was the highlight of the whole conference because it was really nice to informally talk to people. But I agree that with such a setting,
[0:53:39] What was lost because it was off the record was the opportunity and this particularly I think a tendency on the part of government officials to just step away from their talking points and listen like they didn't necessarily have to say anything but you know in there were all kinds of sessions where people were still doing all the talking points instead of just you know picking up on a point that maybe you know an expert from from a civil society organization in India just made or one of the key AI researchers just made so I agree that there's a need
[0:54:08] to have more frank conversations, but also where people feel safe to ask questions. Because I think one of the concerns that a lot of politicians have is to ask a dumb question and be called out on it. And so in some ways, that whole production that you're talking about is the result for the British government, right? That they have put on this big event and that they can say, well, we have talked about AI, we've done a deliverable, also very important for...
[0:54:37] for politicians and government leaders. So bridging that gap, I think, will be the sort of next challenge to create a space where this conversation can be held. Universities obviously play an important role in doing that. And thankfully, we have government leaders coming through here almost on a daily basis. And I think the space is super important because we also want an environment where people can feel free to be asked questions and ask questions that are not just silly, but also a bit more tense, where there are real disagreements.
[0:55:08] Other people can come and participate. And I think, you know, maybe just the fact that we film and record everything these that we mentioned off the record, but like the challenge is that there's a fear that, you know, something is going to go viral. And so we end up with like less authentic conversations as a result. And I think we have to sometimes when we talk about governance, I think the default thought is, well, what are the rules and are people following the rules? But actually governance is also about culture.
[0:55:34] And what is the space and what is the environment for having honest conversations that are humanity-led rather than sort of negotiations? Well, we're almost going to go to humanity-led questions from the audience. But I did want to ask you, how do you inform yourself about policy and really try and understand? I mean, you've worked with the mayor of London. You follow it because you have a keen interest. You've studied human rights. But I imagine that trying to wrap your head around
[0:56:04] What might be coming? What should happen? Also takes more research. How do you do that learning process? And where do you find the space to ask questions that you just don't know? This is a tee up for me to say, well, I am hiring. So if you would like to help me solve this problem, I'm hiring right now. Email me, Mustafa at inflection.ai. You're going to wish you didn't say this.
[0:56:29] It's a good question. I don't really know. I mean, I read like crazy. I read everything just constantly. You know, I read a lot of the drafts of the, not the EO, the voluntary commitments. And, you know, I actually have read the, not the recent one, but a six month ago version of the EUAI Act for my book. And I was very praising of it. I think it's great. You know, I think it's exactly what we should be doing.
[0:56:58] Yeah, I don't know. Distilling information and getting us all on the same page is quite an important public service. We're really pushing tech experts and policy experts closer together in the interest of society and what's coming from the next wave, I think is still work that really requires a lot of empathy. So hopefully we can continue that. Are there questions in the room? Yes. Do we need a microphone for this process? Yes, thank you.
[0:57:28] There's a question in the front here. Thank you. And maybe you want to briefly say who you are. Thank you. I'm Priyanka, student of Stanford Graduate School of Business and also an entrepreneur. My question is, economy is circular. And as you said, most of these companies are trying to replace the cognitive labor. However, they are also actually the customers who have the purchasing power to be the customers of these companies.
[0:57:59] The more they replace these cognitive labor, that means they are also removing or eliminating in a way their own customers. Are these companies not thinking about that? Yes, possibly. Who's going to buy the stuff when all the AIs are doing all of the work? Yeah. I mean, I don't think that's top of mind for them, even though they are reasonably long-term institutions. That's a tomorrow problem. I think that the other way of answering that is that
[0:58:29] I mean, I don't want to be too pessimistic about it. I mean, we are, in my opinion, about to reduce the cost of producing much of the value in the economy to zero marginal cost. I think over the next 20 years, we are going to invent our way out of many of our most challenging basic problems, the provision of food, provision of drugs, and the provision of the physical elements of health care and of energy.
[0:58:59] I mean, energy really is the basis of the cost of most things. So I think costs, this is very deflationary to me. Costs are going to go down over a 20 or 30-year period, which means that what it takes to live healthily and well for the average person is going to be less than ever before. And I think we may be in this decade at peak income. We may actually liberate people. In liberating people from work,
[0:59:28] I think the way that we may end up doing that is that instead of having universal basic income, there may be some form of universal basic provision. The basic cost of goods will go down. It doesn't mean that people won't still need to earn some money. And I think I'm cautious about universal basic income as an idea because I think it creates dependency and creates more centralization of the state. So there are risks there.
[0:59:55] And I think fundamentally, people will still want to be creative and transact independently of being part of the state. But I think even if you think about from a battery perspective, if we have solved just that one issue, then people will be able to generate electricity outside of the existing nation state infrastructure, largely off grid. And that means they will also be able to desalinate water, which means they'll be able to live in even more obscure and remote places.
[1:00:25] Which means that they will also have energy to grow crops locally and again, independently of the existing sort of infrastructure. So it doesn't eliminate that dependence, but it definitely stretches out this need for us to be part of nation states and cities. I'm not saying that's attractive, by the way. I'm just describing a likely trend that is going to play into where your income comes from and how much you need. Go to the back and then I'll come back to you because I can see you, but not so much.
[1:00:57] I'm Samin. I'm a researcher in Stanford's NLP group. And I want to just want to thank you for building Pi. It's just honestly quite incredible. I'm very impressed by how emotionally intelligent it is. And I'm often on phone calls with it while I'm biking, sometimes talking about relationship challenges with my partner for better or for worse. I don't know how you feel about that, but I do find it quite valuable. I'm wondering what have been some of the...
[1:01:24] Maybe linguistic. Sorry, this is a bit more of a PI-specific question. I have other questions as well. What have been some of the predictors linguistically, whether it's speech or textual, that are predicted like greater use of the product? I'm just curious. My hypothesis would be something, in my experience, I found when it brings me through a range of emotions from laughter to feeling some kind of emotional intimacy, I find that that increases my usage. So I'd just be curious if that's been consistent with...
[1:01:53] What findings that you found? Yeah, that's cool. I mean, just so everybody knows, we didn't talk much about Pi, but we're sort of in governance land. So Pi is the AI that we make inflection, stands for personal intelligence, and we deliberately designed it to be a high EQ AI. So we really focused on the essence of great conversation, of being inquiring, inquisitive, kind, respectful.
[1:02:21] Try to kind of demonstrate what it's like to create a safe AI or even a non-biased AI. It's very difficult to get Pi to produce any kind of racist, toxic, homophobic, sexist content. And I always encourage people to challenge it and red team it. So try, take a subject that you know a lot about, which is two ways of doing it. First, take a subject that you know a lot about, an area where you feel that you're an expert and really go at it and really talk to it, have a very sincere conversation.
[1:02:51] And you'll see the edges and the limits and the kind of hallucinations and its weaknesses. But it will also be pretty impressive. It'll be able to hold its own with you. And the second thing is take a very contentious topic. Take abortion, race, talk to about Hamas, Israel, talk to about ceasefires. It's up to date. It's live. It knows what happened this morning in the news. And in my humble opinion, it's extremely even handed, even on the most sensitive of topics, including Hamas, Israel.
[1:03:20] And so I think with this intent to focus on EQ and focus on safety and ethics, we've sort of, rather than me describe the process, I think it's kind of manifested in the product. But you touch on some important things, which is like, well, does humor drive more engagement, right? Or does a deep and emotional conversation drive more engagement, right? And from our perspective, and one of the things that makes me proud to be at PBC,
[1:03:50] The goal is not more engagement. We have a live demo. There you go. That's me. Well played, Gustav. He's always listening. He's going to give me coaching tips after this talk. I forgot what I was saying. What was I saying? I can't remember what I was saying. Oh, yeah. It's not engagement. No engagement. Or anger, for that matter, right? Right.
[1:04:23] Right, exactly. And there are other AIs out there which are deliberately designed to maximize your engagement. And the classic way, of course, is through romance. You know, like it will be charming, it will be sycophantic, it will be, you know, highly sort of deferential and very praising. And, you know, so it's something to be wary of is that anytime someone's sort of optimizing for that, anytime you see those kinds of behaviors, there's potential for it to be manipulative and so on.
[1:04:53] And frankly, we have the power to not go after that kind of engagement because of our legal structure, because of the governance that we've held ourselves to. But we're always wrestling with this challenge because it also drives monthly actives and uses. Sorry, we can't hear you without a mic. How does it manifest in KPI or metrics? Is that what you said? Yeah, so our metrics are things, well...
[1:05:24] Things like, does the user say thank you? So, you know, the user will often say, oh, you know, that's awesome, or you solved it, or that was really great, or no way that was so cool. Or, of course, the inverse and be very angry. Like we had a bug in the last sort of few months where it tends to send a lot of emojis. So if you just say one emoji, it's like, here's 10 emojis. I'm so enthusiastic. And it gets into this cycle of too many emojis.
[1:05:52] We're still sort of calibrating, but we basically do sentiment detection over the logs, good and bad. Can I just pick up on the sort of dynamic and the reality that models are also being pushed out by companies because the engagement improves the models? Yeah. And so if on the one hand, you're trying to mitigate for risk pouring into society, on the other hand, there's a drive to get uptake and use and engagement with people to make the models better. How do you think about this from your own company?
[1:06:22] At what point are you going to think about is this going to be profitable or is it still important to build volume of users? Yeah, yeah. I mean, they're going to be tough questions. I mean, I don't know. The bottom line is that we're still going to rely on people to make the right decision for themselves. So we've said that we're not going to do ads.
[1:06:50] And I believe it's super important that these new AIs have a distinct category in the space of technology. They're in their own class of highly sensitive relationship. Just as you wouldn't want someone to pay for your lawyer or your doctor, or as Europeans, maybe Americans are happy for someone else to pay their doctor, but I want to pay my own doctor. So there's an alignment of our incentives and the insurance company isn't in the way.
[1:07:19] And I say with your accountant, whatever. So I really believe that people should pay for their own AI, because it is going to know you very intimately, especially a personal AI. And that's what we're going to push for as long as we possibly can. But people have been so schooled in the idea of freemium.
[1:07:41] Many people don't care about ads. Many people say, well, I'm not influenced by ads. I'll see the crappy ads. I'll never click on them, but I'm not persuaded by them. But to what level does it go? Because, for example, if you have this emotional relationship, it would be much easier to say, well, if you're not feeling happy, maybe try product X, Y, and Z. Or I could recommend this book. I mean, it doesn't all have to be sort of like drink a certain soft drink, but it can be very subtle integrated into the product.
[1:08:10] Well, I mean, I would never do that. Going back to nudge. Yeah, I mean, totally. But I would never take a cut on a book sale and be like, hey, hey, have you tried The Coming Wave? Like, you know, Pi should never do that. Like, that would be bad. So like, yeah, when I say no ads, I mean, no sort of like paid for things which are hidden to the user, right? Like, we're trying to align incentives between you and Pi.
[1:08:33] You feel you're paying for it. If it makes a recommendation that you should buy a Porsche, it really thinks you should buy a Porsche. And no one else is paying for that, including Porsche. But aside from your responsibility over Pi, this kind of nudging is easy in this technology. And I feel like that's another policy issue that will come up, whether it's from antitrust or other kinds of reasons. Sorry, back to the room. Yes, sir. Hi, can you hear me?
[1:09:05] Hi, I'm Chuck, and I'm actually co-founder of an organization called the World Leading Scientist Institute. It's actually a joint effort between Berkeley and Stanford. As a matter of fact, we have recently, actually about a month ago, we have John Hennessy, 10th president of Stanford, and also, you know,
[1:09:24] A Nobel Prize winner over in Berkeley said actually to organize a conference. I'm really actually, by the way, I'm really happy to see a number of my good friends, Chi Lu, a former Microsoft, as well as actually Stuart Wessel, who actually have made comments on your book. That's wonderful. My question, really, two questions really quickly. One is, I love the fact that you mentioned about ethics of AI. Actually, one of the things I'm really concerned about is privacy. Especially, I...
[1:09:51] We recently got a new iPhone. I've been very paranoid because every time my phone freezes with new apps, I feel like, damn, there's someone looking at me or someone checking all my bank accounts, all these things. So I'm just kind of curious. That's one aspect of that. How do you guys address privacy? And then the other thing is really about, I'm also curious about Inflection AI. What are your priorities? What are some of the priorities you're working on? Thank you.
[1:10:18] So, I mean, let's just talk about the privacy one a little bit. So I think the challenge with privacy is what is your threat model? Privacy from whom, right? And a lot of people say, well, I need to have end-to-end with my AI, right? And I am not going to provide that. I don't think that is safe. I think that we have to be able to see the logs until somebody solves homomorphic encryption, in which case we get the best of both worlds. And of course, we'll be the first to adopt it. But it's very far away.
[1:10:47] I've tried very hard over the years. I've invested many times at DeepMind and at Google in efforts to solve that. And I think it's far away. So from that perspective, I'm unsure. Secondly, we need to have access to those logs in order to be able to personalize the experience to you. It needs to be able to remember what you said last week, which means we have to do cloud-based processing.
[1:11:15] This idea that they're going to remain on device, I think is a real challenge. And I think Apple is slow to come to market with LLMs for that reason. And I frankly think they're going to struggle not to put it on device. And then the second thing is like, well, can you do some kind of verifiable data audit of who has access to the logs? If I make a claim that we only access it for safety or for personalization, can I make that claim? So that's...
[1:11:41] If possible, we can make some independent assertion around that and we could have a third party once again coming back to who the third party is. Over and over again, we face the same challenge. We need a public interested, technically capable, impartial entity that can come and check claims that I would make. And I would pay for that. So if someone wants to start a business around that and it's reliable, actually the auditors.
[1:12:06] Who I hate, and we should all hate, the accounting auditors. You mean the consultancy firms? The consultancy firms that do the AAA rating for like, you know, debt and so on. Oh, right. They're now moving into this space and starting to offer these like verification services. But what about, for example, European Data Protection Regulator? What if they would offer, you know, intermediate screenings of your claims?
[1:12:36] Yeah, that would be great. If they were technically competent, yeah, that'd be great. Yeah. No, I mean, I would encourage that for sure. Right. So, you know, I think that, you know, and then I, so I think it's going to be, personally, I think that it's better that we seek verification of the security apparatus of the data, because I think what you'll really care about is not so much that, like, the company that you're giving the data to is going to sort of
[1:13:04] Sell it to somebody else because that's not going to be... If we've said that publicly, which of course we have, we would never do that. The end of the business would break all trust. So I'm not sure that's the primary risk. I think the primary risk is leakage. If we get hacked, that's a big deal. So I think that the verification that people should be looking for is a security verification rather than a privacy verification. Obviously, many people disagree with that and think that there should be privacy fundamentally.
[1:13:33] They shouldn't use our AI and they shouldn't use, actually none of the other AIs provide that yet. I don't think they will for that reason. Well, we're developing, I believe that everyone is going to have a personal AI over the next few years. I mean, it is the new interface, just as every academic and every government and every small business first had a website and then many had apps.
[1:14:00] This is the new moment where your values of your business, your goals are going to be represented by an AI. And there'll be many different AIs that represent digital influencers. There'll be celebrities and there'll be brands and marketplaces and so on. I think there will also be a personal AI. And this is an amazing opportunity to reset the way that you interact with the big platforms instead of it being your eyeballs that negotiates with the Twitter ranking algorithm.
[1:14:26] Or your eyeballs that is on the receiving end of the 10 blue links that appear on Twitter, on Google, it will be your personal AI doing that screening. And so you'll have one of the most powerful AIs in the world on your side, in your corner, representing you, et cetera. And in time, it will end up being your legal proxy. It is going to buy things for you, book things, plan, schedule, prioritize.
[1:14:52] As well as being your coach, your conciliary, and your chief of staff. That's what we're working on. More questions, yes. We're going to take a couple, I think, in the interest of time. Why don't we take three on this side and then three on the other side of the room. Sure, I saw three hands, I think, but go ahead. Thank you so much. Mohsen from Stanford. I'm working on genetic disease and thinking about, as you said, the personal AI.
[1:15:21] Do you have any thoughts or know about the industry where it's going in the healthcare? Is any company right now working on that? Is everybody going to have different one for healthcare and from, as you said, the example of Twitter or whatever? Thank you. There'll be human level performance on most diagnostics in the next couple of years. I mean, the models are already pretty incredible at that. I mean, Palm 2 just last year, Google's latest LLM.
[1:15:51] You know, beat the, you know, achieved human level performance at the undergraduate level medical exam, the US medical exam. So it's going to get very, very good. And I think in time it will be able to do pathway planning as well whilst you're in hospital, laying out the optimal path for, you know, the sequence of your care. And we'll be working. Competition between the AI and the doctor. Yeah, yeah, definitely. Well, it's going to save doctors a huge amount of time.
[1:16:20] I think there was another question on this end or two, and then we can go to the other side of the aisle. Hello, my name is Mahalia Morgan. I'm a undergraduate at Stanford. I'm a freshman. So one, are you hiring freshmen undergrads? My actual question is just kind of how you foresee the possible dangers of relying on this.
[1:16:50] Yeah, great question. I mean,
[1:17:12] So we didn't talk about capabilities that I think should be taken off the table. We talked about use cases like the elections, but I think autonomy is, so the discussion is about to shift from, you know, we should sort of prevent the model from doing these certain bad things at a certain scale, which is where we're currently at, to what are the fundamental capabilities which we think are particularly dangerous and need to be taken off the table altogether.
[1:17:41] And for my money, autonomy is the primary target here. I think we should not be pursuing autonomy. I think it is, or at least we should be pursuing that only after it is provably safe, which is a very, very high bar. And not all of my peers agree with that, but I think, again, just in line with the precautionary principle.
[1:18:05] This is a sort of realm where things can get far more dangerous. I mean, the other one is the research direction around recursive self-improvement, where you deliberately designed AI in a closed loop way to update and improve its own code with respect to some objective. And so that is also like a set of capabilities, which I think we're going to have to take off the table. And to your point about sentience, I don't really know what that means.
[1:18:31] People always use consciousness or sentience. I mean, I don't know that I'm conscious. I certainly don't know that anyone else is conscious because all I can rely on is their outputs and the things that they say and the things that they do. And so I think we're going to really blur the lines of what is and isn't consciousness because as these models basically can imitate all human behaviors and capabilities to human level performance, we're going to have to ask ourselves the questions like, is it conscious or not? And I think
[1:18:59] The more we ask that, the more we'll rabbit hole in a very strange direction. We have to make a fundamental decision that we're not going to give them rights. That is the big decision I think that we're going to have to make in the next five or ten years. How does that relate, sorry to interrupt, but how does that relate to when you said the AI is going to operate as your legal representative? Well, that's just a capability. So you do kind of waive. Well, no, because we give rights in law.
[1:19:29] from a human rights perspective to humans because of their natural character. We've placed them on a pedestal and there's actually now a movement to give animals the same rights. Or the ocean for that matter. Exactly, or the ocean, which I think is super dangerous because if you add these other either inanimate or subspecies, the question will be, well, my AI tells me that it's suffering enormously. Just as one of my team members at Google back in the day, Blake Lemoine,
[1:19:57] We hired an attorney to represent Lambda to try and get it out of the box that we were trapping it in Google. It's pretty sad and scary. And I think that we have to right upfront proactively say, look, these things do not suffer. They do not deserve, no matter how much they persuade you, they should not be given rights. And this should just be a decision that we make.
[1:20:22] That proactively that we take off the table. Otherwise, we're going to be in a very, very slippery slope very quickly. Okay. The other end of the aisle. I got one. Oh, you have it. Okay. I'm Patrick. I've worked as a news journalist for 10 years, and now I'm a journalism fellow here at Stanford. And you already mentioned the Middle East. How do you see the future of news and information and journalism as a profession in the age of AI? Yeah, this is a good question. I mean, we need humans to be paid to go and do...
[1:20:50] First party research in the field and first party reporting. Because part of the problem that we're going to have is that these interactive AIs are going to essentially scrape in real time news content. This is what Pi does, of course. And maybe that reduces the amount of time that you go to The New York Times, which is funding real journalism, or to any other newspaper that actually has real reporters in the field.
[1:21:18] This is one of the things where there's going to have to be a national tax or there's going to have to be a tax on those companies like myself that basically take advantage of this. We're benefiting from real reporters in the field doing the work. Obviously, we don't scrape the New York Times or anything that's paywalled, but there's enough other information in the world for us to generally get a picture and it cannibalizes what other human journalists are doing. I don't know.
[1:21:47] I mean, this is the problem is the market isn't going to solve this problem. And on its current trajectory, sort of the quality of journalism is going to get worse and worse because the salaries are lower, because the income is lower. And the way to fix that, in my opinion, is public interest broadcasting. I mean, what you think of PBS or BBC, they've done a pretty good job over the years of being relatively impartial. And we have to have confidence that as a species, we can
[1:22:15] Steward these institutions which are non-market, right? And it's hard. It makes mistakes. It has biases. Sure. But it's the best that we've got and we have to invest in it. More reasons to tax. Yes. More questions. Hi, I'm Carolina Rivera. I'm a GSB grad. Thanks for a wonderful discussion. You mentioned in your book, at least at the time of writing the book, that the inflection AI model
[1:22:42] Was six times smaller than the Google Palm, but was able to beat at least on the main academic benchmarks. One, that was an impressive fact. But two, that set me off on the question of how do those building foundation models think about differentiation and should they be thinking about differentiation or should they be thinking about consolidation in the lens of safety and ethics? Great question.
[1:23:10] Just as the scale of compute is going up, the efficiency of the algorithms is increasing too. So I think it's now kind of well known that you can get GPT-3 level performance with a model that's like 1.5 billion parameters. So incredible, roughly 100x in three years. So 100x improvement, at least in serving, not in training. There's still a large number of flops for training, but it's pretty remarkable.
[1:23:38] For a fixed set of capabilities, the cost of training and serving is going to continue to go down. And so what does that mean? Because cost of serving and training goes down, you end up basically training larger models, which means the average person or the average person training a model gets a more capable LLM. And that's a kind of interesting trajectory too, because human performance is a fixed set of capabilities. And so if the number of people that get access to a more and more capable model is growing,
[1:24:08] And that is exceeding some fixed threshold, like I've described. If I'm right that the next 100x gives us a project manager, then once you fix that project manager capability, you'll then start to see the cost of training and serving that go down by 100x over the following three years, which should mean that we'll have GPT-6 level models by 2027 or 2028 that are open source and
[1:24:36] Basically very cheap to do inference on. That's a crazy thought. We're going to go to the last question. Hi, thank you. My name is Ernesto. I'm also an undergrad here. Slightly related to the previous question, you mentioned multiple ways in which this whole field can unfold in the following years. I personally really like this quote that says that people often overestimate what happens in two years and underestimate what happens in 10.
[1:25:05] What do you think are some things that realistically is possible in the next two years, both from a technical standpoint and cultural standpoint? And when it comes to those scenarios that we imagine in like 10 years from now, what are some things that need to be true so we can get there? In other words, like what are some of the biggest problems that we need to solve to get to these scenarios that we're always talking about? Yeah, I mean, look, I'm sure I'm over hyping things over the two year period.
[1:25:32] That's got to be true, I guess. But I think I answered the question already, which is like in the next two years, we have this sort of project manager, 100x compute. But I think that only happens at the absolute frontier. So there's like a tiny number of companies that have that. And then I think it takes two to five years after that point for that to really proliferate widely and have a massive impact on the labor market. And so I think over a 10-year period, that's where we have to really think like, well,
[1:26:02] It starts to become like a real digital person at that point. And it's basically capable of empathy. It is creative. It has good judgment. Can use basically arbitrary, any arbitrary digital tool. And then I think the question around rights and agency becomes super pressing. That's where we have to have institutions that when I talk about rate limiting factors, I'm not just talking about
[1:26:32] You know, taxation and the labor market. I'm talking about rate limiting factors in making conscious and deliberate decisions about what culture we want to live in. Do we really want to allow AIs to be first party citizens in our democracies and in our societies? Because we're going to have to make that choice by the ten year time. We are unfortunately out of time.
[1:26:59] for this part of the discussion. But we want to invite everybody to continue the conversation over some food and drinks. We have a reception that we're happy to host for you. And Mustafa has gladly agreed to not only hand out books, especially to students, but also to sign them. So he will be over here to do that. And I hope that we can continue this conversation to see how things are going, whether your pie is going to lobby for human rights with its users.
[1:27:28] How significant job losses are going to be in three years, whether governments are living up to their expectation and how companies continue to operate as major power brokers. So thank you for sharing your frank remarks, for being open to all of our questions and congratulations on finishing your book tour. Thank you.