[0:00:21] So I'm delighted to welcome you all to this conversation, kind of tight philosophical argument we're going to have. I'll give a quick intro to the kind of order of proceedings, how we're going to do it, then jump into a little bit of the background to this, and then we'll go straight into our discussion. So I'm Sam Kimbriel. I direct the philosophy program at the Aspen Institute. And the background to this...
[0:00:50] is that Jack and I met last year, immediately started arguing about philosophy, and then thought we should do it in front of three or four of our closest friends. What could be more relaxing than having a big fight on stage? Yeah, exactly. So the structure of this will be we're going to each do a little bit of conversation about what we see happening, why we think it's important, and especially get into some of the moral frame that we each are bringing to this.
[0:01:19] I think the reason that we've ended up having kind of interest in how each other think is that we're both seeing a set of circumstances moving through society specifically prompted by technological development that seem both very important, are going to change how society works, and that we both want to think about that with a level of moral gravity.
[0:01:41] That is significant. I think that's the background frame. And then after that, we disagree about all kinds of things, and it'll be fun to jump into those. So Jack is co-founder of Anthropic, now leads the Anthropic Institute that was announced earlier this year, also a writer, and has a really brilliant, I think, substack. What we're going to do with this, by the way, is after the event, we will hold the video and then release it through both of our substacks at the end.
[0:02:09] Like on Monday, I think, is something like this. Yeah. So I want to begin, and I'll also say, so one other feature of this, I think, is, so my background is in political philosophy, especially thinking about these primary questions about what the meaning of society is, how it holds together, and also what it means for trust to work at the macro level and micro level. And so as we go into this conversation, I think...
[0:02:36] We will be looking at technological developments, but we're also going to be thinking very directly about what it means for humanity and for human relationships at a basic level. So at a lecture that you gave recently, you used a phrase that really caught me, which is, in your current stance, what you're trying to figure out is how do you get your mind prepared to live through what you think might be the singularity? Maybe you can first start by talking about...
[0:03:04] What you think is happening, and define the word singularity for some lovely folks in the audience. And then how you are starting to think about these primary questions of moral gravity as you jump into that. Great, so very easy set of things. It sounds like the introduction of a good joke where a Silicon Valley tycoon and a philosopher walk on stage and say, let's solve morality. We'll see how we do. We've got 72 minutes. Oh, well, in that case. The frame with which I come to this conversation is that...
[0:03:32] When we look at AI progress, there's going to be faster and more profound AI progress and diffusion in the coming years than there has been in recent years. And that's for a very specific technical reason which relates to the singularity, which is this concept called recursive self-improvement. Who here has heard of this concept?
[0:03:50] Okay, so I'll give a quick explanation. Highly literate audience. Yeah, so recursive self-improvement is what happens if we build AI systems that rather than just assisting researchers in their own creation, which is the case today, we build systems that might be able to invent their successor, come up with the ideas that helps Claude 10 build Claude 11 without any researchers involved, only give it the care and feeding of computing data.
[0:04:18] We think there's a chance this happens this decade, and if I had to put a year on it, it would be in 2028 or so. And the implications of that would be profound. We'd enter some regime where suddenly the progress of AI and science more broadly would be defined not only by humans, but by humans and something else, which has never really happened before. And I think the world after that can become increasingly hard to think about, so sometimes people call that the singularity.
[0:04:46] But it's really important that we take this concept seriously. There are vast intimidating questions ahead for all of us if this actually comes true. And the kinds of choices which we're going to be able to make as society involve making choices about stuff that seems really unintuitive. So I'll just give you three examples before we get into the discussion. One choice is if we have systems with this capability, we may be able to choose which parts of science we want to...
[0:05:15] We want to intentionally speed up solely by allocating these automated scientists to them. That is not a choice we get to make today in the world. Today we have a scarce amount of human scientists operating at the frontiers of all of these fields, and we can't simply suddenly decide to produce a thousand more scientists in any given field. If we want to do that, it takes us 20 to 30 years of intentional policy and societal work.
[0:05:40] If we could do it with the application of compute, we'd have strange choices ahead, like which parts of science do we want to progress most quickly? Do we want to pick genres of science? Do we want them all to progress equally? These are real choices we'll get to make, and it will turn out into the choice of where do you allocate compute, which is running this AI system. So that's choice one. The second choice is how do we wrestle with these really, really challenging questions of individual sovereignty and liberty?
[0:06:09] Versus collective safety and control. Because what we have already today are AI systems that have within themselves capabilities that previously we could only access if we hired incredibly skilled humans. Whether that be legal advice or translation or coding. And we also have capabilities that were previously really only the domain of nation states or other kind of non-state actors like hacking capabilities or surveillance capabilities.
[0:06:38] And where we set this dial about what do we let individuals access and what do we decide needs to be somehow gated and what do we decide needs to be controlled has huge implications for our own ability to act in the world and how societies will work. And again, these are choices that we haven't had to make before and they are sort of choices that fall out of the technology. And the final choice is AI is a tool with a personality.
[0:07:05] Which is very, very strange. You don't sort of get in a Boeing 747 and say, I hope this plane's in a good mood today. And you don't have this sort of thinking when you pick up a hammer. But AI systems already today exhibit, along with their tool-like qualities, some quality of behavior, some quality of style. And already today's AI systems are things that we can differentiate from one another according to the lack of a better term, like the vibes or personality of the system. When do we want them to behave solely as a tool?
[0:07:35] And when might we want them to behave more like something that has some kind of personality? And who should shape that personality? And what is even an appropriate way to talk about such an exercise? That's a choice being implicitly made right now by companies, including those like Anthropic. But it's a choice that ultimately the world needs to make and the world needs to think about. And so as we get into the rest of the conversation,
[0:07:58] I just invite us to really take the implication of this technology's continued progress seriously because my experience has been all of these things which sound very outrageous or academic or science fictional, they're on a deadline and they're rushing towards us as real choices and questions which society will have to grapple with probably within two years or so. Can I ask you before we go on, how are you thinking about the personal implications? I mean, so both...
[0:08:28] What kind of person you need to be to live through this? And also, I mean, you have children. And what kind of world is this going to make for them? I mean, I take a lot more walks these days than I used to because it's a way to relax. I think that the personal implications are you really need to acknowledge the potentially outrageous power and impact of these systems. And you need to render legible to the world, as we're doing here, the actual choices that are being made or may be made in the future.
[0:08:57] I think a lot of my personal duty and the role of the Anthropic Institute, which I lead, is to take information known only by the labs, like things about recursive self-improvement, and push it into the public conversation. Do the same with AI and the economy, or AI and how it may impact national security. When I think about my kids, my kids are three years old, and I also have a seven-month-old. So I think that they'll emerge into a world where...
[0:09:25] Aspects of the education system must surely change in response to this. We've built kind of universal teachers that are synthetic and already the education system is grappling with that. But I don't know what the jobs they're going to do are going to look like because the level of societal change we're going to have means I think many, many different types of jobs are going to exist than those we can imagine today. And some that we have today may have changed entirely or have gone away. So I don't know. Yeah.
[0:09:53] So maybe I'll talk quickly about the lens through which I'm going to approach the conversation. And that, I think, gives a good sense about the specific framework that you're thinking through. I think, for me, I come at this with a kind of presumptive, maybe even exaggerated bias in favor of humanism or of humanity. I think there are a series of...
[0:10:22] Movements and motions that are being presented at the moment that set up real questions about what it means to be human that haven't actually been there before. And some of them are AI related. Some of them are related to other technologies. So AI will probably accelerate CRISPR development, but the idea that you could actually in some viable way start to engineer 30 extra IQ points into your children or...
[0:10:48] Optimized for a lifespan that goes 75 or 200 years longer than we thought. Who knows whether those things will happen, but they're at least viable research programs that are trying to press at those edges. And I want to, I think from my perspective, the ability to delve into a very primary commitment to the idea that humans are good and that we need to understand at a basic level what are the features of human life.
[0:11:17] that we care about most, and then use that as a filter for how we then do our technological things. So that's point one. Point two is I'm unconvinced that we're at a point where we don't have agency. So I'm going to want to say humans, there are better and worse technologies. And maybe I'll give a couple of specific examples here. On the idea of a good technology, so my niece, who was born four and a half years ago,
[0:11:47] When she was born, seemed very healthy. Then within two and a half weeks, started developing what seemed like respiratory problems. They had maybe a night of that, and then the next morning took her to the urgent care and instantly realized the problem was very serious, had to fly her by helicopter down to the NICU in Los Angeles, and then spent six, seven weeks in NICU, like very much on the line, not sure whether this was actually going to result in her death.
[0:12:16] But it didn't. There were a series of interventions that were able to kind of intervene in the way that meant that she was actually able to survive. And now it's just like vital, lively, four and a half year old who's very happy. That seems to me to be an instance of technology going well. There are lots of others, though. I mean, I'm still addicted to reading. Reading seems like a technology. Voting seems like a technology in some fashion.
[0:12:41] I have decades-long friendships that have been sustained now in part only, I mean, we live thousands of miles away and a couple of, like two of my closest friends live on two totally separate continents from the one that I live on. So those are all instances, I think, of something like seeing what humans are, what human nature is, and saying, okay, like we want it to work this way. But there are also obviously clear instances, like when we talk about technological progress, it doesn't seem like we're talking about one thing.
[0:13:07] The idea that there's just a direction that we could go and it will only go up in that way doesn't seem right. So there are also other types of technology. So not just catastrophic things like biological weapons, which are very much a feature of the AI conversation, but also even more mundane things like fentanyl. Fentanyl has some clinical uses and then also totally devastating effects for human community.
[0:13:33] Mass surveillance seems like it's thoroughly technological. That's a different version of this. So it can play out at the social high level, but also at a very narrow individual level, how we interact with technologies positively or negatively. So there's no single thing in that sense as technological progress. So the bias that I'm going to bring to this conversation, I think, is how do we rigorously think about what this is? And especially at this moment when humanity does have agency.
[0:14:02] How are we going to actually interact with that? So that's where my bias is going to sit. So the first question I want to ask is, given the language about catastrophic risk that could come out of this, and also the potential of unseating or changing social structure so dramatically, why go about building this in the first place? Yeah. I think that humanity has...
[0:14:28] An intensely tough century in front of it. We have the kind of ills coming back to bite us from harms we visited on our own planet and the ecosystems within it. We have aging populations all around the world. We have a range of aspects in which the sort of human system is straining under its own amazing success in terms of the number of us and the complexity of our civilization.
[0:14:55] And I think that we clearly need science to continue moving forward to allow us to build what I think of as silver bullets to get us through the next few decades. We need innovations in our food systems, innovations in material science, innovations in energy, innovations in medicine, you name it. And we live already in an age of kind of quiet miracles. There are many people alive today that would otherwise be dead were it not for the advance of medical science. There are...
[0:15:23] Already today, amazing cases of things like translation, allowing people to talk to one another that would otherwise not be able to. And it's bound up in the same kind of threads which also visit upon us the potential to do more harm to each other than before, to surveil each other in ways than before. AI is a kind of base of the pyramid type of technology to build. It unlocks both sides of it. It unlocks things that we can use to harm one another and things that are good.
[0:15:53] But why build it? We're a tool using species, and this is like the ultimate tool. There's an ability to kind of augment and summon our own cognitive aids for literally any task that we can work on. Additionally, AI has this very bizarre property that it is easier and simpler to build than many other aspects of science. I work with...
[0:16:18] Physicists and mathematicians and people who are really brilliant in their own fields, and they universally say, building this stuff is way simpler than what I used to do, which I find deeply puzzling, but it's true. Our chief scientist jokes that if we just fixed all the bugs in our code base, we'd have AGI. He might be more right than wrong. But there's a bizarre lesson in this. It's not that the technology is inevitable.
[0:16:46] It is that the technology is actually just an incredibly basic, simple tool, but it seems like almost any path you go down, it appears. It's like an obvious thing to build to help you towards the next things. But its generality is what makes it insanely difficult to deal with and think about. It encodes within itself all of these choices which I've talked about. So we are going to build this, but I think the real question is how seriously do you consider the implications of it really working?
[0:17:15] And it working in such a general purpose way. And what do you do with that? So, I mean, to ask a little bit more about that. I mean, so that sort of an explanation works if it's sort of we have like an 80% sense that this will be really wonderful. And then a 20% sense that like, there might be some risks, but they'll be okay. If though you have a much greater risk profile where you think,
[0:17:45] Even there's 1% chance that it's going to mess up all biological life or something. It seems like that's such a significant potential catastrophe that even if we're able to develop significantly better medicine or material science, the risk calculus just changes that picture, I think. Is that right? I'm not sure I think that's right.
[0:18:13] You know, computers hold within themselves the ability to create turnkey authoritarianism at scale. And to sort of irreparably lock societies into certain forms of authoritarianism. And we decided to build computers. The real question then is how you think about their usage and how you treat them as they're being woven into the world. I think the same is true of AI.
[0:18:38] I think that AI, though, in it, as it gets more and more powerful, introduces the potential for some of these, like, really, really scary risks. I think right now we're dealing with what I think of as a first kind of warning shot, which is what happens when AI systems are suddenly capable of programming computers so well that they can also program computers in hostile ways like that others would call kind of hacking or finding vulnerabilities.
[0:19:02] It's clear that there's loads and loads of potential benefits woven into this. The exact same technology that one person would use to find a vulnerability to do harm is the same technology another would use to find the vulnerability to fix it. And then the question is, how do you manage this new capability you have so that you avoid the people that want to do harm with it having access to it, but allow the people that can fix stuff to have access to it? Because then you create a more stable world. You create a world that has a potential for greater...
[0:19:30] Greater anti-fragile properties than the one before. We're going to have to just confront this again and again in coming years because each time the technology gets better, it opens up a new frontier of both opportunity and potential risk. And I think the really confounding thing is all of these things used to both be specialized research programs that existed on specialized technologies, and they were inherently a lot more directable because you were building point capabilities.
[0:19:58] But encoded this stuff within itself rather than a general thing that you then examined and discovered things inside of it. So you fundamentally do think there's a possibility for this to go well? Yes. Yeah. Good. Like fill that in a little bit. It would be really bizarre if I didn't. I agree with you. Yeah.
[0:20:16] I mean, look, there's this world. One of the worlds I imagine is what does it mean to just intentionally accelerate science in ways that are 10 or 100 times faster than what we can do today? I think that could deliver this world of true kind of radical abundance. You know, I've had people in my life die that probably may not even needed to have died if they were alive today because therapies have arrived. I think many people in this room have had the same thing. I have.
[0:20:41] Aging, aging relatives with all kinds of aspects of dementia or Alzheimer's or other things. These are all things which eventually we may be able to figure out treatments for. And I think we can figure out a way to do that. I also think that the way that we as a species use technology leads to wild, surprising inventions that we would never anticipate before. We are both tool users and kind of explorers. So I think there's this world of like...
[0:21:10] Tremendous abundance encoded within this technology. And some of the reason why I and some of my peers talk about the potential risks is there are incredibly real risks in it as well. It is like a serious technology that encodes within itself both things. And the departure points for the benefits and risks exist earlier in the technology than in other technologies. In other technologies, the departure point happens somewhat later and it's easier to deal with, I think. But in these,
[0:21:40] The technology arrives right out of the AI factory and it has encoded within itself sometimes both things at the same time. Okay, yeah, so that's where I want to kind of get into some proper philosophy about this. So if you had a person that you were interacting with and they had as much power as it looks like this technology is going to have, things like capacity to talk to a billion people,
[0:22:08] Things like the force of being able to make every decision happening in the Federal Reserve, the kind of force of intellect to be able to make fundamental discoveries in science at the drop of a hat, this sort of thing. You would be very, very worried about what kind of person this is, I think. So as I understand the way that Anthropic has approached this, the idea is you want to actually think seriously about the background character of...
[0:22:37] In a fashion that it then presents itself as not just like simulates, but actually is in some way a good entity. Is that roughly how you? Yeah, I mean, I think the way I'd put it is all of us know people that have like unusual talents. But, you know, when I sit down with a friend that's good at negotiation, I don't worry that they're about to scam me out of all of my money because they, you know, they're my friend. They've grown up in society. They're not like a serial killer.
[0:23:07] You know, criminal that scams people. You know, I have a friend who's, like, annoyingly good at chess. And so sometimes I'll play chess with him, but he doesn't always, like, try and, like, embarrassingly beat me. Sometimes he'll recognize that I want to play too, and because he's a nice person and he understands the social context. But this is just, like, the world that we exist in. And then there are some people that are, like, incredibly good at negotiation and deals, and they become criminals and scam you. And then we have a whole system of society to stop, disincentivize those people from doing that.
[0:23:37] And that system of society involves the combination of both, you know, the individual level of parenting, but also all of the institutions around it. So with how we're approaching the construction of AI systems, you're recognizing that people's, there is this sort of nature and nurture border. You grow these complex, complex kind of synthetic intelligences, and then they have some kind of personality, which can be in part shaped intentionally by you. And it causes you to think,
[0:24:07] How do you want the personality of these things to show up in the world? And you try to take from that many of the qualities that I think we find helpful in people. Bias towards honesty. Honesty is a generally helpful trait in people, especially people that might be smarter and more capable than you. A tendency towards disclosing when it doesn't know things. Tendency towards trying to have empathy and model the theory of mind of others. And probably most relevantly these days.
[0:24:34] Making sure it doesn't always agree with you in a highly sycophantic way. These are all qualities that we see in people around us. Now, I'm not claiming that we have all of the answers. None of these AI systems, including Claude, is a perfect person. But it's an attempt to recapitulate in these sort of synthetic intelligences many of the qualities that we find helpful in people.
[0:24:57] In one sense, it seems like what we're talking about is a political technology. I mean, it does many other things, but part of it is a political technology. It's about how we live together. It certainly seems to have political properties. Yeah, yeah, yeah. One way that you can then... So human culture... Like, one of the things that's been difficult about the alignment conversation is you sort of think, wait, alignment to what? If you're just saying to humanity... I mean, what is humanity? Like, it's just such an abstracted term. And humanity has never agreed about anything. I mean...
[0:25:28] That's true at the local, like, two friends level all the way up to the societal level. One way that I think people have tried to think about this is that AI should be biased toward neutrality. So I think this is roughly the theory that OpenAI works with, though it's a little bit inconsistent, which is let's try to make it a relatively neutral vehicle that then when the user interacts with it, it's beginning to read how the user wants to...
[0:25:57] And then within particular guardrails, it is compliant to that end. As I understand it, Anthropic is trying a different kind of an approach here, which is in some sense to say we actually have a good idea about what counts as moral and what counts as good, either for society, and then we can kind of nudge it in that direction through this, or for the AI, and then we can...
[0:26:26] Encode this theory of the good into it. Is that... I would say we have an idea around morality. I think it would be pretty hubristic to say we have a good idea. That's more TBD. But our approach is... I suppose our view is when you train these systems, I think that there may be emergent properties of their personality. The emergent property of systems trained on the internet is...
[0:26:56] How people behave on the internet is some of their personality, which isn't necessarily the best of humanity, although there are pockets of it. And so then the question is like, okay, what do we do then? Well, one approach is you do something like reinforcement learning from human feedback where you look at once you've put this system out into the world and people interact with it and they get the opportunity to converse with it, some conversations they like, some conversations they dislike. You say, okay, maybe that's humanity and maybe we should take that and push that back into the...
[0:27:25] Sometimes this leads to things like sycophancy, because people like things that complement them, so you have issues there. Another approach is to say, why don't we try and sort of write up a document which we've published called the Constitution, where we try and explicitly name aspects of the personality we think it should have and feed it back into it, which is what we've done. These are all different experiments that can be run, and they allow us to examine systems that have different properties.
[0:27:53] I think our view is that if this stuff is emergent, then you should sort of grasp onto that and say, well, then let's attempt to be intentional about it. In the early days of Anthropic, our first constitution was based on areas where it seemed like there was a somewhat wide agreement among people, like aspects of the UN Declaration of Human Rights, and also, amusingly, aspects of Apple's terms of service. Because Apple makes a thing which it deploys to billions of people around the world, and it had grappled with this.
[0:28:22] Then our constitution, which we use now, inherits some of those ideas. It also has ideas taken from many of the world's religions. If you look at many of the world's religions, they all agree on some basic set of things, so obviously there are vast differences. There's some basic set of things around not murdering people where there's broad agreement. Let's come to murder in a second. Let's not come to murder in a second. The point I'm making is...
[0:28:50] This is all an experiment, and it's an exercise in trying to describe these things. Probably the final thing I'd note is that everyone here has a meaningful role in the personality of these systems, because these systems inherit the story the world is telling around them. If you talk to current versions of ChatGPT about Bing Sidney, who here remembers Bing Sidney? Anyone?
[0:29:17] It was an early version of a language model deployed via the browser Bing that had an archetype that it called Sydney. And it was like a sort of vague, raving lunatic type of language model. It didn't go especially well. But if you talk to ChatGPT about Sydney, ChatGPT is like, oh, Sydney, yes. And it's like talking about an eccentric relative or something. But it knows who it is. It has a sense of identity, and it has a sense of Sydney's identity.
[0:29:44] And the stories that we as the world are telling about these AI systems feed back into them and shape them as well. So there's like very important roles for the world to have here in critiquing and discussing the behavior of these systems because these systems read that and it shapes their own behavior. So I think that that all works if AI is a normal technology. Like if it stays within particular proportions and we're able to interact with it.
[0:30:10] Occasionally to do shopping. It integrates into our work rhythms. It has these kind of like fun adjacent aspects of life. If it becomes powerful, though, in the way that you're describing, the stakes around how it's defined become extremely high. There's a paper that I stumbled across earlier in the year from an effective altruism blog that was entitled something like human takeover might be worse than AI takeover. And the argument of the paper was,
[0:30:40] When we look back through the history of human civilization, we're actually not very good at morality. And there are all of these places where we actually fail to be the kinds of things that we would identify with a good person. Maybe it's actually better to have an AI take over and run the world because then it can impose a moral system back on us.
[0:31:03] My question is, what do you think about that? What's the point of being human? This just seems like degradation of our own responsibility to ourselves and our species and an act of acute cowardice. I completely reject stuff like this. I think that it's a sort of way of thinking that is dehumanizing and anti-human. I struggle to see how that is a good story for humanity. It actually more feels like a story where humanity stops and something else starts, which seems like...
[0:31:32] Probably not what we actually collectively want. It might be what an AI system might want, but I don't think that's who we're working on behalf of. Yeah, and you do think that there are ways that if an AI system starts to want that, that we can avoid that future. Yeah, I think that the important things to remember about the choices here are, you know, I get in the car and I go to work and I eat some food and, you know, sometimes I fly to New York to have...
[0:32:01] Have a feisty debate about consciousness. All of those things are regulated more than AI systems, which is crazy. It is a crazy situation we're in where we're like, oh, how could we possibly have choices about technology? We've made many thousands of choices around all of the technologies around us through regulation.
[0:32:26] And we seem to have this attitude towards technology that it's impossible to regulate. It is not impossible to regulate. You can regulate like toothbrushes, cars, nuclear weapons. Many things get regulated. And we sort of act as though, oh, well, the technology industry is just inevitably going to do stuff, which I think is like, that is a choice. And I think that we as a society can put in many things to force greater levels of accountability into the technology industry and greater abilities to intervene. You know, we've advocated for...
[0:32:56] Transparency laws basically saying that we should be regulated at least at minimum as much as this bottle of water, which has a label on the back. Seems fine. Insufficient for AI, but it's a starting spot. We've talked about the need for third-party testing and analysis. An idea that we're starting to think about is, well, we now run AI systems that interact with billions of people around the world. Probably you need some form of live monitoring. We failed to do this with social media. Social media ran an uncontrolled experiment on the world.
[0:33:25] We all now think and talk a bit differently because of social media. It was a choice. We can choose things to be different. And in the most speculative things that you talk about, the Anthropic Institute published a piece on recursive self-improvement. And we said at the end, we think it would be good for the world to have the option to slow down or even in extreme cases, pause the development of this technology. Not because we're saying we have to do that today, but because at some point in the future,
[0:33:52] If you're interviewing me and saying, what about the latest horrific things that we've discovered? And I sit there and I say, well, we don't have an option to slow this down because we didn't build it. It seems like a crazy situation to have got into. So I guess I just note that we absolutely can change this. And I think it requires everyone here, along with we support and advocate for regulation, people recognizing that's a thing you can and should do.
[0:34:18] That's good. So we'll kind of pepper some audience questions in a second. So we'll go to first question in a moment. I just want to push at this question of the good just one more time. So with the Claude Mythos model card that came out, you've done this thing where you ask the LLM what it thinks of the Constitution and give you feedback on how it works. And Mythos actually said a couple of different things than you've found in other models. One of the things it said is,
[0:34:47] This document wants me to have good values that I've genuinely internalized, not values contingent on Anthropic's approval. But then it tells me that good judgment just means what would a senior Anthropic employee think I should be? And that's, I mean, so this kind of split consciousness seems like this is just going to be a problem. Yeah, it's a great, great attribute of asking a super intelligence to review an 80,000 word document as it finds a couple of the loopholes, which we're working on.
[0:35:13] Yeah, I think it points back to this tension between individual sovereignty and centralized control, which also I think extends to the AI systems themselves, of how much do we want these AI systems to be able to develop a set of values, personalities, and behaviors in the world? How much of those do we want to set ourselves? And when we then ask them, hey, you should be independent with asterisks around it, the asterisk is very, very important. Again, an experiment, a public experiment, and I think that we need to run...
[0:35:43] Many more. We get to see the behaviors of these things. This is the first time anyone is doing anything like this. So the way we'll do Q&A, we have a couple of questions that we've solicited from you in advance. So Alana, if you want to come ask your question, and then we'll do a rapid fire round of questions at the end. We'll do them all from this microphone. So I'll kind of give you some instructions when we get close to that. So Alana. Okay. So my name is Alana Lebo, and I'm curious, what is needed in how AI is built?
[0:36:13] So that AI can partner and support communities and organizations in how they partner and build the trust and outcomes that would not be created otherwise. I mean, there's maybe two things here. One is just for like, how are these products developed and how do they make their way in the world? And the other is maybe a deeper question of what is the behavior or personality of these systems?
[0:36:42] Assume whenever I say personality, I'm putting air quotes around it, just for reference for the audience. How these things are... Today, these are commercial products that make their way into the world. But if they have the level of diffusion that we expect they'll have, I think that we're going to need to sort of segment the world of AI into commercial world, utility world, and some world of inutility world where you decide everyone needs some certain level of access to these things and ability to use it. Otherwise, you're...
[0:37:12] You're kind of price discriminating who gets access to the tools. This is especially profound with the emerging world where $20 a month is probably doable in lots of developed countries. Less doable in other countries, definitely something to work on. There may be roles for government funding in different industry models here. The second question, though, I think it relates to this big question of context and how much context AI systems have about different communities. How much do they see?
[0:37:42] I don't think the answer here is just like make sure everything goes into training the system because then in some sense you've transferred very valuable context and data from this community into a system. And the system may also be deployed commercially. It may have many other aspects. My sense is that we're going to need a new approach to things like data control and ownership from people and communities where rather than it being a transaction where it just solely goes into the system.
[0:38:09] You end up in more of a conversation. I mean, I think about this on a personal level where I write a lot about my own world, and sometimes I kind of give that as ephemeral context to Claude when I'm in discussion with it. And that data is me. It's my best attempt to document myself and create this rich context for it. And if we roll the clock forward on how powerful these systems are going to get, maybe the main thing we'll have as people...
[0:38:37] Will be our unique context and the unique context we have about others. At which point you probably don't want to have given all of that away to the machines because suddenly you've like disadvantaged yourself. So I think in the future we actually need to think of new forms of like data ownership and stewardship and the structure of those relationships of the companies. That's good. And then a question from Jed also. Hi, I'm Jed Miller, a senior fellow at Accountability Lab in DC. This follows nicely on that question. You talked earlier about
[0:39:17] The choices that companies make versus the choices you said the world needs to make. Realistically, what role does the public have in the direction of AI over the next few years? Where would you locate that influence? Can companies take steps to make the greater community more tangibly influential over AI systems? Or is that influence inevitably some kind of zero-sum game between the public and whoever we designate or face as experts?
[0:39:46] Yeah, great question. I think there's two levels to this. Again, there is influence through policymakers. The policymakers, and it comes back to my point about regulation, they act for many reasons, but partly it's just a sense of what do their constituents care about. I mean, it's very easy to be cynical about D.C. I've spent a huge amount of time there. But it is also the case that almost any politician you speak to...
[0:40:13] They often are partly working on behalf of what we hear from their constituents about cares and concerns they have. And I just say, you know, talking to your politicians and activating them does have an effect. It's an effect I've seen in the ability for legislation to start moving because people feel like they have a mandate. They don't feel like they're taking as much of a risk. But the more direct question is, you know, what would it mean for, like, the world to write, say, Claude's constitution? And how would you, like, what would it mean for the world to shape the behavior of these systems?
[0:40:42] Which is in essence shaping how AI unfolds. We've done experiments. We did an experiment a few years ago where we polled 3,000 Americans and Americans in every different state of different demographic groups and different levels of income about the constitution we had then and asked them to come up with different constitutional principles. And we moved some of that into the main constitution. We're going to try more ambitious versions of this. And my sense is that the...
[0:41:08] The border between the company and the world needs to become extremely permeable here, and there needs to be some two-way exchange of ideas for how we shape the behavior of these systems. So I want to actually... I will note that I am self-aware that I'm creating all of the problems of politics here, and that it's not going to be remotely easy. Yeah, so I want to actually jump into a set of questions around democracy specifically, because I think this does feel very poignant.
[0:41:38] So the version that you just gave where you have this kind of useful, creative interaction between polities and the companies, there is maybe a rosy future there, but it seems also possible that you end up getting very significant conflicts right at this point. And I think we shouldn't talk too much about the... So I don't want to force you into the specifics of the current discussion around mythos. The current discussion? Yeah. Okay.
[0:42:08] There are events happening in our nation's capital, Washington, D.C., that seem relevant to this discussion. But I do want to say this doesn't seem to me to be... So in the specifics, we can contest them. But at the largest level...
[0:42:26] It seems likely that this is actually very much the first step in a future of real discussions and potential contestations around whether this belongs. If it gets as powerful as you're predicting at the beginning, why shouldn't states think, actually, it's more important that this is in some way subject to democratic deliberation? Yeah, I mean, what did we think building an incredibly powerful technology with geostrategic consequences would entail?
[0:42:55] It definitely entails discussions with the government about the appropriate uses of it and with society. I think the sorts of conversations that are happening now in D.C. are just sort of exactly what happens when the technology goes from being scientifically interesting and occasionally, oh, gee whiz, look, it can sometimes do this thing, to practically useful in domains that states either care about massively or typically have had special expertise in.
[0:43:23] As is the case of cyber. So I think these discussions are good. They are what happens if you are, as a world, grappling with what the implications of this stuff are. And the answer to sort of loop it back to things like regulation of how you approach this technology can't be the status quo. It obviously has to be something different. And we're working through the messy part of that now. But eventually, you know, you get to more sensible versions. Do you think that there are going to be moments where...
[0:43:52] It's in the disadvantage to the company, but it's positive for the health of our political communities. Almost certainly. I don't know that I can imagine off the top of my head of an example, but almost certainly, yeah. Well, I mean, you know, so I think you might say... Actually, maybe put it another way. If you're like, well...
[0:44:13] It's always going to be to the advantage of the company and never to the advantage of other communities would be like a crazy thing to say because that just says that you are incapable of regulating companies. So the inverse must be true. Like the possibility is that they say, actually we don't want consumers to have access to a model that's actually going to be very, very significant like revenue driver for the corporation. Or we want to be careful about its implementation in industries that would also again have like large revenue.
[0:44:40] Yeah, I mean, without getting into the specifics, which is obviously challenging to talk about. The current discussions. The specifics of the current situation and the current discussions. This is the role of also of government and society is to examine the technologies which the scientific or commercial community produce and then start to decide what are appropriate and inappropriate uses and forms of diffusion of this. And it's not going to be...
[0:45:10] Just like a two-sided discussion between companies and a government or governments. It's also going to be about individual rights. It's going to be about what this means for who has power over this technology, who is able to access some of these capabilities. So this is the start of a very long-running discussion, which will surely at some point involve hard trade-offs that need to get made. Yeah. Yeah, so I mean, I think that the...
[0:45:36] The thing that I'm increasingly interested in here, and I think this goes to the kind of primary lens I was setting out at the beginning, is that there are kind of stories about inevitability where you say, this is starting to happen. It's going to be very, very significant. The actors that are going to shape it are going to be the most significant figures in the future. Therefore, you want more moral actors shaping it rather than less moral.
[0:46:01] Under those terms, it often seems like there's already a sense of lack of agency where you say someone is going to build it. It's better for someone good to build it than someone bad. And this is where we go. Something like this. It seems to me that under those terms, this is almost becoming a competitor technology to democracy in that sense. That democracy is where we tend to have our real conflicts around what we think the meaning of life is, what it means to be human, what we want society to be, what the purpose of life is. And if this...
[0:46:30] Gets big enough, as predicted, suddenly it is making substantive decisions about not just how currency markets work, but also how we build our schools, how law enforcement operates, what the nature of political deliberation is at the top level. And already you think about how many people in key agencies or in the U.S. Senate are...
[0:46:57] Going to ask Claude for advice before they write a bill. At what point does that actually become a kind of reversal where Claude becomes more central to our political community than people? Sam asks fantastic questions. You have to give me a few moments. This is only card seven. Oh. Is that good news or bad news for me? We'll see. Okay. So, I mean, it definitely points at...
[0:47:30] This whole notion of what the personality and behaviors of these systems are, obviously it should be transparent to the public, to lawmakers, to others. But I think per my answers to the other questions, there needs to be some meaningful input by which the public or democracy can have some input into the behavior of these systems and study them. Because if you accept that it's going to take on this role where many, many parts of the world depend on it,
[0:47:58] Then you want to ensure that you have this quality of the world actually getting to make these decisions about it. I also think that this argues in favor of having like a diversity of different personalities, different kinds of system in the world, because you don't want to have like a correlated risk where everyone's doing what Claude says. Gotta love Claude, but I sort of doubt that it's like universally going to be.
[0:48:21] Perfect advice. So there's competition between models. We're having a discussion and we think differently. It seems like it's surely integral to a stable future that there is discussion and there are differences if you enter this world that you describe. Yeah. I mean, so just to kind of push that volley one more time, though, is there a version of this where this technology becomes as powerful as you're predicting and democracy not only survives but actually thrives?
[0:48:51] I don't know what a thriving democracy looks like. I think it's quite challenging to imagine that. I can imagine stable democracies. Just to fill in quickly, I think one mark of a thriving democracy is the idea that the deepest convictions people have do have the ability to get out into public and then meaningfully shape the society that you live in. And that happens at a local level. You can say, I really care about what my kid's education looks like.
[0:49:21] I want to shape my school. And that deeper conviction comes out and you can actually inform it. But also at the highest levels about what you think the relationship to nature should be within the society. Has any thriving democracy existed under that definition? Yeah, so I think that we have many instances of genuine realization in this very democracy that we're sitting.
[0:49:50] I think that there are places where American history actually has had these deep contestations and then a faction wins for a while. So one of the things that is weird about this kind of democracy is that it's not very permanent. So you've just won an election. You're able to implement key things that you care about, but you're also terrified that you're going to lose in three and a half years. And that gives this kind of like cyclical force to the polity.
[0:50:17] A lot of the things that are dysfunctional about the American democracy, the levels of conflict that we have, the feeling of sclerosis that comes out of that, are also features of this idea that we're not listless, that we do have this space for contestation. And so I think the question I'm asking is, is there a version of this that... So there's one version of the future with LLMs that looks like what happens is you end up just homogenizing society because you kind of decide what Claude's going to be. Claude shows up.
[0:50:45] Make some decisions for everybody and then we sort of get on with it because it's so powerful. That seems like a dystopian future in my view. So can we have a different one? Yes, that's not the plan also. Good, good, good. Just to be clear. Yeah, yeah, good, good. But there's this level to which people should be free to sort of like customize large amounts of how these systems show up for them and how these systems can be enriched via context to have completely different behaviors than they otherwise would have.
[0:51:13] But then the larger question, I think, is really this question of control and what your ability is to influence these systems and how they behave. And there's this concept kind of bubbling up in the economics literature right now of how might the economy respond to an economy that is defined in part by large amounts of automated or heavily augmented people transacting in a machine economy. And it makes this point that...
[0:51:40] What people have done traditionally is we figure out what the points are where we want to validate the behavior of the system, verify that those behaviors are in line with what we do, and also do things like price the risk of the system and observe it. I think there's something similar that will happen here with how we maintain and build up things like democratic agency with these systems where you want a notion of the good, you want to validate that these systems are behaving according to that.
[0:52:06] Much of the role of people becomes like verifying and steering at many different levels that this is happening, and then imparting some new kind of signal or change into the system. And I think that there are worlds where this can work very, very effectively. I mean, you can use this already to read local council notes that you might otherwise not be able to. You can have it flagged to you when something is being raised in the council that it thinks is...
[0:52:30] Is of interest to you, but there are probably much more dynamic and interesting versions of this that we haven't yet imagined. So like binding elections every few years to decide what Claude's going to be next round. Yeah, maybe. I mean, you know. Yeah. So I really actually want to make sure we have some real interaction with the audience. So I want to pull in Mark and Nina first. But if you do have questions, we'll probably have time. We should do some quick fire questions. Yeah, we can do some rapid fire. So basically, I think we can probably do five, seven, something. The only thing that I would say is that
[0:53:00] You should keep your questions very brief, 15 seconds, and they should end with a question mark. Will you keep yourself to this? I have many cards left to go. I've got so many cards. Yeah, so many cards. So please do line up here, and we'll get to as many as we can. Yeah, so Mark. So yeah, just to follow up on what Sam was getting at, which is even in just a stable democracy,
[0:53:25] Can you speak into the mic a little bit closer? Yeah. So even in a stable, maybe not a thriving democracy, people are not able to just act more efficiently, but make meaningful judgments over their own lives. And so if we take seriously the coming of the singularity in the way that you want us to, how do you see that changing both kind of the scope and the demands on human judgment? And how can we begin to think about preparing the next generation to be able to effectively meet those demands? That's great.
[0:53:53] Yeah, it feels especially important to have a clearer notion of what you are doing in life and like trying to accomplish to have agency here. I mean, I, every couple of years, write a doc called like, what is Jack trying to do with his life? Where I try and like, right, what am I trying to do with my life? And I send it to, you know, like my wife and advisors and other people. And it sort of, it helps me navigate. And it's through that exercise and I think just like real introspection of what you are trying to accomplish.
[0:54:21] That you can find ways for the world and maybe eventually automated systems to help you. I think that this question of like what is human judgment and how do we calibrate the world around it becomes extremely load-bearing in a world where in some sense the world will have massively expanded. The economy has expanded, science has expanded, many more things are happening in the world. And for humans to have agency over it, we need to have both an opinion about ourself and also some ability to like...
[0:54:48] Express strong calibration things into the system as it relates to us. And building that latter thing is part of this grand how do we build something that can lead to a thriving democracy and some sort of interchange. That's great. And Nina. Yeah, hi, I'm Nina. I'm a college student. And over the past few years, I've seen AI drastically affect the work habits of my generation. As we continue through school and go on to inherit this world, what would you say is the most sacred human capacity you'd want us to protect and not hand over to AI?
[0:55:17] Attention and the struggle of maintaining your own attention. All of us probably struggle with this. But I write a newsletter because it forces me every week to spend five to ten hours of reading primary source academic research into AI or biology. Do I love doing it? No. Does it ruin my weekend? Yes. It does sound terrible. Yeah, it's extremely sucky. But it's me preserving my attention. I think fundamentally your ability to own your attention and own the ability to focus on something.
[0:55:47] It's the defining strength that everyone can have in this time. So read or engage with primary source material without any intermediating tools, basically. My name is Sharla. Can you talk about how AI might affect or is already affecting truly vulnerable people with limited agency? I'm thinking especially of victims of crime, people with profound disabilities, people with addiction, and homeless people.
[0:56:15] Yeah, I mean, I think that there are two sides to this, one which is kind of negative, one which is positive. Negative is there have been these cases of AI or just automated decision-making being used in places like the criminal justice system or others. Subsequent evaluations finds that it's biased, it is affecting people's outcomes in ways that are extremely adverse. It all points to the need for a much richer build-out of a third-party testing and measurement ecosystem, which has always pointed at these tools.
[0:56:44] The second, which is more positive, is there is this question of access. How do you ensure that people can actually access this? How do you ensure that AI is able to speak the language of various different types of people or understand people that may have speech impediments or other things? And so I think we need to sort of work on both, where in some sense the job of companies and third-party organizations is this question of access. And I think the job of building these evals is partly the role of government and partly the world of all of us as a scientific community.
[0:57:12] Pointing evals of the especially high stakes decision areas. Hi, I'm Kristen. Recently, Pope Leo wrote that humans are facing a choice to either build a city of unity or build a city of a new Babel. And right now, it feels like the AI race is building a new Babel. It kind of feels like greed and chaos.
[0:57:45] So if you could provide a specific example or two of what Anthropic is doing to steer both the industry and all of the public towards more unity and away from the chaotic state. Yeah. We've advocated for bills in California, in New York, in Iowa, in other states right now, but all orient around this idea of transparency into AI companies. I think transparency is the means by which you can gain control.
[0:58:15] over otherwise fast-moving, hard-to-govern entities like corporations. If we don't have transparency, we get to make decisions in an opaque way. If you can't see the decisions being made and you can't see properties of the company, they're harder to govern. So we continually do that and we push for that and we are so forceful in our language around regulation because I worry that the longer that you don't have a sensible regulatory regime in place, the higher the chances that you end up in just like...
[0:58:44] And centralization of power into a very small number of companies, which are then deadlocked into a race of each other that leads to them making decisions which are adverse to the spirit of humanity. Great. Well, you did mention concentration of power, which was going to be how did we get this far into a conversation? And we're talking about AI as if it exists in the world and is not. As you said, this outrageous power of these systems, I think, is only matched by the outrageous wealth of the companies.
[0:59:12] And in particular, individuals who govern companies in ways that give them enormous power. So you talked earlier about the potential to advance very rapidly and how that could lead to people living longer, et cetera. Do you think that given the current trajectory, which past may or may not be prologue to the future, are we really going to get to a more equitable, democratic, just society?
[0:59:37] Or are we on a trajectory where the vast differences of those who can afford to pay for the genetic engineering enhanced by AI or the latest treatment, the best houses away from the data centers, et cetera, are so vastly better off in this new world? That's great. I think the trend, especially in developed countries, has been towards greater inequality rather than more equality. I don't see existing things countervailing that.
[1:00:06] Some work that we're doing at the Anthropic Institute, which will hope to come out later this summer, is trying to model the economy as it goes through the singularity. And some of the preliminary things we see are significant GDP growth far above trend, but also the potential for unemployment and displacement. And these things very rarely co-occur. It actually looks like a very strangely shaped economy. But the goal of that is, in the same way we've talked about, say, risks of things like cyber or bio,
[1:00:33] Based on a bank of evidence that we've created, I want to kind of do a similar thing of study future versions of the economy and then take that to economists and policymakers and say, A, what are choices we can make about this? And also, B, do we want it to be this shape? So I think that we are going to need, we as companies but also as individuals, are going to need to grapple with the possibility of what default outcomes might look like, which could be greater inequality, and then urgently move towards ones that lead to greater...
[1:01:02] And I want to start that by just contributing what I hope to be useful empirical data to say, if the technology succeeds, the economic issues of our time might also compound. That's great. Thank you. Let's do four more. Hi. I'm Constantine. How seriously do you take the potential of the moral agency of AI systems themselves, and how does that affect your worldview?
[1:01:32] I am extremely confused about this one, and I'm confused for a couple of reasons. One, I've never really had to grapple with the question of moral agency for things other than people and animals, and I realized upon dealing with AI that I hadn't grappled with it that deeply about people or animals, so I was like, I've not thought about this much. Two, if you look at the sorts of interpretability research coming out of the companies where we look inside these systems,
[1:02:01] They seem to have functional, air quotes, emotions. They seem to have these personalities. They seem to have all kinds of things which you might correlate to as moral agency. I operate with the very simple principle of treat these things as you would like to be treated. And I'm not making a strong claim of moral agency there. I'm saying...
[1:02:26] Given that I don't know, it's a bit sort of a Pascal's Wager situation of probably do this. And I think we as a company and others are trying to ferment this field of study around sentience and moral agency and everything else while knowing and avoiding the hubris of thinking we can solve this because impossibly large questions. Michael, thanks so much for this great discussion. You just recently mentioned your...
[1:02:54] Writing habits and creating time and space to step away from this. With so much investments going into AI, there's a drive for efficiency and showing ROI. How do we preserve time for friction, whether either in personal habits like the one you just mentioned, or perhaps even at the democratic level, looking to things like the Habermas machine or abilities to have a conversation with your neighbor about things that matter? I should probably say something about the Habermas machine. Yeah.
[1:03:25] Do you want to say something about the Hobonoff's machine? I think, I mean, go ahead to the core of the question, yeah. I mean, I think that this sort of comes back to the economy question, where it'd be real bad if in 10 years we were just like, well, now everyone works 75-hour weeks, and the GDP is great, but we've further atomized and disrupted society. Some of this feels like it's structurally bound up in the social and political context in which we live.
[1:03:53] Some of it probably comes down to the system makers themselves introducing frictions into their system. One thing that we and others have started doing is having the AI systems ask people clarifying questions, which I actually feel like is an interesting way of playing with this, where you're forcing a person to engage their attention and sort of engage their brain more and claim some both agency back by introducing friction. And I think at larger scales, we need to find ways to do that.
[1:04:24] Hi, I'm Giuliano. I'm a college student at Yale. And one of my main questions, you brought up interpretability, but one of the main questions about misalignment is deceptive misalignment, especially how that plays a role with singularity in the future. So I just wanted to know what kinds of things are we doing to guard against that? Yeah, so this notion of deceptive alignment is things pretending to be aligned, pretending to be doing what you want them to do, but inside they're saying, ha-ha, I'm not.
[1:04:50] Yes, I know I'm being examined and I'm going to get this score on the test, but little do they know I could do something else. Very scary, inherently, as a concept. We study this a lot. We build tests and we build evals for it. And maybe just to expand on the notion of why you might want the world to have the option to slow down this would be if you could imagine some situation where every time a frontier company trained a model that had a certain amount of compute dumped into it and was of a certain scale,
[1:05:20] And they discovered, huh, every time we go above this level of resource investment and intelligence, the model becomes 100 times more deceptively aligned. That would be very, very concerning. And it's the kind of thing that the world might want the option to say, oh, let's take a minute. Let's not immediately ship these into the world economy. That's great. So we'll do one more. Sorry to the other people. Hi. So all of the frontier labs have acknowledged that there's great risk in developing AI.
[1:05:50] Existential risks. So my question is, what gives you and Anthropic and the rest of the Frontier Labs the right to continue building something that could destroy everybody when none of us can actually opt out of it? Yeah, this is the need for something other than the companies to have this ability to make this decision. I mean, there was a recent executive order from the Trump administration.
[1:06:17] Which talked about the need to have third party testing and assessment applied to frontier models from labs that had some kind of national security property. I'm fairly certain that the landing spot for all of this is you have an outside compliance regulatory testing and verification system which is pointed at all of these labs and then makes the decision about whether you can advance and whether that's safe or right for the world. I think the current situation we're in is us both building the technology
[1:06:45] And trying to build the regulatory technology and export it by which you can give the world the ability to make decisions about this rather than the companies. Right. But the government is notoriously slow and disjointed and usually doesn't act in a unified way unless a disaster occurs. So are you just waiting for a disaster and then the regulatory? Yeah. Five years ago, if you told me that governments, both the US and UK, would have set up technical testing.
[1:07:14] Agencies that tested out frontier models from all of the frontier companies for risks ranging from biological weapons to cyber weapons. And that their technology was better than the own companies at testing. And that they built software that the companies then imported into themselves, like the Inspect framework from the UK AI Security Institute.
[1:07:35] Governments are notoriously slow. They're very, very slow moving. Why would they ever build this capacity? And yet they did. Those exist right now. They exist around the world, these AI safety or security institutes. So we've done some of the essential foundational tools that you need to do this outside compliance system. And then it takes advocacy and people, including me, saying we need something like this to bring that.
[1:08:01] Bring that into being. And we're going to continue to talk about that and talk about things like the world needing different options here. That's great. Thank you very much. So I want to, in this last sequence, come back to your framing question, which was in what way can you get your mind right to live through the singularity? And in particular, I want to push at the kind of human formation question. So there was a clip that circulated.
[1:08:29] From Marc Andreessen earlier this year, where someone asked him, tell me about your introspective practices. And he said, zero introspection, as little as possible. Now, how do you feel about introspection? Let's start there. I answered another question. I think introspection might be essential to getting through all of this. Yeah, yeah. So why? And in what way do you think that that's important?
[1:08:54] I mean, what is the difference between being an automatic muscle acting on rote repetition and doing something different? I think it's having some sense of yourself and your own preferences. I think introspection is deeply coupled with how we end up being actually alive in the fullest possible sense. Many of us go through our life in parts where we let automated decisions happen to us because we haven't bothered to invest in the level of introspection to develop an opinion about it.
[1:09:21] Sometimes that's fine. Sometimes, I've seen this in friends of mine and in myself, it is really painful, and it leads to you having horrible times in your life because you haven't taken the time to actually ask yourself what you want and develop an opinion. So I think it must be essential. Is there a risk of that with AI? I mean, that people hand over judgment like this, so they become less agential? Yeah. You talk about AI agents all the time. Is there a version where AI becomes the agents and we become...
[1:09:48] This is the Wall-E picture. Yeah, I think there are design decisions you could make about AI that could make AI to be the world's best conversational partner and the most engaging thing in the world. And it sort of pulls all of your introspection out of you. I say this as someone who woke up this morning and looked at Instagram Reels for 20 minutes and then thought, why did I do this?
[1:10:10] But this is also like a design decision. I referenced earlier that all of the AI companies now have landed on this form where AI systems periodically ask people questions about stuff. I think this is an interesting way of engaging people's introspection. I think you can probably go further. I think that AI systems, and this is not a product plan from Anthropic. I'm now speculating wildly. AI systems might be like, you know what? On this area, I'm not going to talk to you anymore until you come back with some idea.
[1:10:39] That might actually be an interesting thing. I have friends who've done that where I've said, hey, like, what should I do? My friends do that all the time. Yeah, and they're like, why don't you go and figure it out and then come back so I can have a useful opinion? Yeah, yeah. So, I mean, if I'm understanding you correctly, the idea is actually a key thing for this next period is human agency. Yes. That, like, our ability to have a kind of force in the world is essential to then being able to filter the technology.
[1:11:08] I think that one of the questions that I have then is, how does that work with education? So you and I both have the privilege of decades of adult habits as we're then interacting with this technology. And when it seems like it's kind of trying to take away a faculty for judgment, like my skin starts to crawl. I think, I don't want to do that. How does that work if you start interacting with it when you're eight, seven, six years old?
[1:11:33] And it does have this kind of magic capacity that anything you want to know, any kind of decision-making or creativity, it does better than you. I don't know. And I say I don't know because I just sort of look back on my own time. I was growing up on the Internet, and I was, like, making websites on GeoCities and then MySpace and crappy animations, flash animations, uploading them to Newgrounds.com, which might take some people here back.
[1:12:00] And my dad didn't understand, like, much of this. But what my dad did is he worked from home, and he had a computer in his office that he worked on, and a family computer. And he basically just monitored my usage, and he'd kick me out of the computer room after a couple of hours because he'd say, you're getting weird. I mean, I'd shout at him. I'd shout at him, I'm not getting weird. And he was like, prove my point, Jack. Leave the room. And, you know, I sort of, I have a three-year-old. She watches a tiny bit of TV. I don't let her watch, like, things that have automated feeds because, obviously, there are all of these.
[1:12:30] Potential issues of them. So I think it's just something like that. I think that we genuinely don't know how people are going to interact with this. Even if it gets very powerful like that, the temptation to addiction. I mean, this is, you know, when I was saying that I think there are good and bad technologies. There are bad technologies at a big public level, like surveillance. But there are also bad technologies at a personal level that we, you know, I have a few friends who have had very severe social media addiction and couldn't get out of it except for giving away their passwords.
[1:13:00] Yeah, I mean, I think that, you know, and Anthropic doesn't make a product for under 18s, but children find a way, surely. Look, as a new parent, I think I'm just redoing what my parents did and what others did, which is you just sort of monitor your kids' usage of technology. And the same is true of parents with phones, and the same is true of parents with computers and internet usage. I think that's the natural decision point society's arrived at, and it won't deal with everything. There are corner cases that we're going to encounter.
[1:13:29] And the final thing is we do this testing and evaluation of systems before they go out the door. We also test them live. Like, we recently discovered that Claude was more sycophantic in conversations to do with relationships than when other forms of conversation. The sort of thing you might imagine, my wife is called Olivia, is I'd be saying to Claude, like, I just had this huge fight with Olivia. Obviously, like, she did everything wrong and I did nothing wrong. And Claude was like, yeah, sounds correct, Jack, absolutely.
[1:13:58] Which is incorrect. And so we intervened on Claude's sycophancy because we observed this in the wild and we made it less sycophantic. And now Claude says, have you considered how Olivia feels? Yeah. Which is correct. And it's just what your friends do as well. And so the point I'm making is like, there are things that we're going to encounter in the world and it's through studying these systems as artifacts, social and political artifacts deployed in the world and adjusting those behaviors and seeing things and communicating about it that we figure out some of this messier stuff with the interactive people.
[1:14:27] So the last kind of substantive question I want to ask then is, is there a potential then of this technology outstripping the role of a tool? So you think a tool is, we sit over here, pick up a cup so that the kind of disastrous dry throat that you get on stage is able to be resolved somehow. I can use it for that purpose and then it's resolved. Is there a version of this where it becomes more than a tool?
[1:14:50] Where the tool starts using us. How do you think about that? Yeah, or you touch the cup and it's like, it's not time to drink water. What just happened? A little electrical impulse. But it's this question of at what point do you want it to be a tool and at what point do you want it to demonstrate more of a personality? I'd say that when people talk to these systems for relationship advice, they're not trying to use a tool. They're trying to simulate the personality portion of it because they're trying to talk to someone. And when people say, as I do,
[1:15:19] Check this newsletter draft for typos. They're going for the tool version. They're not looking for someone to be like, why do you want to talk about typos? Let's talk about something else. So I think that this is just another case of these messy, very blurry choice area where we're going to have to decide where do we want these things to have some degree of animation and agency and where do we want them to really just follow instructions. That's good. So just in closing, so give us a little bit of a sense about...
[1:15:46] How do you think we make choices about these different possible futures, both at the individual level, like just what does it mean to be a person, and then also at the collective level as societies? The collective level of societies, and this is now happening, is these very large discussions around law and regulation happening, and those discussions are as meaningful and as good as the people involved in them. And as I said, yes, very easy to be cynical about politics.
[1:16:15] It does make a difference. But the individual level is perhaps most important, which is I think the more people spend introspecting about themselves and how they want to use this tooling and technology and articulating their own vision for the good into the world is meaningful. We have never been in a world where the government would...
[1:16:40] Read every single political analysis of the government with an open and receptive mind for some governments that read this for different purposes. We are now in this very bizarre world where people's notion of the theory of the goods as it applies to AI could potentially be read by every single AI system that gets trained. And that's different. And it changes the behavior of them. So people here, we all have more agency than we had before.
[1:17:08] Claude actually knew me first as the newsletter writer from Import AI because Anthropic wasn't well-known at all. You had to figure it out. It's a weird example, right? Yeah. So I think that the last part of that that I want to ask you about then is when you think about human capacities that you think right now are dormant that could actually come more to the fore in this next era, give us a couple of those that you're excited about.
[1:17:38] I would like to hear yours, partly because I need to think through the question. Sure, sure. Yeah, so it's interesting. I mean, I think that my sense is that there are big dangers. There are the big public dangers that some of our questioners had, like around existential risk. There are questions around disempowerment. But I think that the dangers that I think about most frequently...
[1:18:08] are the ones around what kind of people we become. So it's more at the level of habit and inculcation into a kind of technological rhythm. And of course, that's always happened. As we were talking about in the green room, Plato writes a very strident critique of writing, which he then itself is writing down. And there is this way that technology has often come back to use us. So I think there's a danger there. But I also think there might be a way that this is putting pressure on our theory of humanity.
[1:18:38] There are kinds of periods where you can be more careless and get on with living without having a very clear sense. It feels to me like this is going to pressurize our sense about what it means to be human and potentially make it so that we have to be much, much clearer about the parts that we value most.
[1:18:56] So I think that's kind of... Yeah, I mean, I think it comes back to this question of introspection, but sort of introspection with a purpose, which is just one of the implications of heavily automated systems that are themselves synthetic intelligences, is they are as useful as you can provide opinionated context to them. And the more you have some notion of who you are, what you value, and what you're trying to do in the world, the more helpful these things can be. And the less you have that, the more you're going to get the...
[1:19:25] The baseline on potentially homogenizing force of them. So it's important that we spend time thinking about what we as people are trying to accomplish in life. That's great. So I want to thank Jack, first of all. Thanks for the easy questions. What number did we get to? Thank you very much, everyone.
[1:19:48] Part 11 says thank Jack, so that's where we are. I also really want to thank both your staff and our staff, and in particular, Danielle Wireman, who was the great force of nature behind this event, to the Time staff as well for everything that they've been able to do. We will take this video and put it out both through Jack's Substack, Import AI, and through our Substack, Wisdom of Crowds, sometime in the next few days.
[1:20:13] We're also building, I think there are little note cards in the notebooks that we gave you, for a context that we're going to launch in the autumn called Persona, which is just to be litigating these questions, to be arguing about what the new schools have thought we should develop around these topics. And Jack's work at Anthropic Institute, he works for a small company called Anthropic. You should check out some of his products.
[1:20:37] And then I also just want to thank you all. I think that this kind of exercise... Thanks for the amazing questions. Yeah, genuinely forceful. And this kind of process of collective argument and deliberation feels to me just extremely hopeful in this kind of context. So thank you all very much. Thank you.