[0:00:00] All right, welcome back. And Jared, welcome. Thank you. I feel so fortunate to have you here at the Tech Summit. I mean, Anthropic is one of the companies that's really defining this new wave of generative AI that's sparking this flood of talent and capital into generative AI companies, along with OpenAI, of course, and Google's BARD and Bing. But I know that...
[0:00:26] Anthropic makes an argument that your language model and chatbot Claude is different than the others, is differentiated. So tell us, give us the argument. Why is Anthropic a different kind of generative AI company? Yeah, I'm glad you asked. So there are a few things that make Claude different from other chatbots that are available now. One of them is that it's trained with a technique called constitutional AI that allows
[0:00:55] And then the AI system is able to train itself to try to follow those principles. Another interesting feature about Claude is that it has a very, very, very long context length.
[0:01:16] What that means is that Claude can read many, many scientific papers, even something as long as a short book, and talk to you about it, answer questions about it. And that really makes it possible to engage with AI in a new way.
[0:01:31] Okay, I want to start out by posing a poll question to the audience and we'll see the results, but perhaps we can keep the poll open and maybe you could even make your argument and sway them a little bit. So the poll question is, would you be more or less likely to use an AI assistant that has been trained with and may be constrained by?
[0:01:52] A SET OF STRICT MORAL VALUES. SO THE KIND OF CONSTITUTIONAL AI THAT JARED'S TALKING ABOUT, MORE LIKELY AI SHOULD BE ETHICAL, LESS LIKELY, YOU DON'T NEED AN AI NANNY, GIVE ME SOMETHING UNCONSTRAINED OR C, IT WON'T IMPACT MY CHOICES. I MEAN, JARED, WHY, SO YOU'VE DEFINED CONSTITUTIONAL AI, WHY SHOULD PEOPLE CARE, RIGHT?
[0:02:18] Google with guardrails, you want to search the web. So why would you want AI that's constrained? It's a great question. And of course, any individual person, if they're playing with an AI, they know they're a moral person. They're going to do the right thing. So why should they care? I think there are a couple of reasons. I think the main reason why we're interested in this is that
[0:02:42] We think that AI has risks and problems when used at scale already. It's not that it's just one person playing with an AI by themselves. It's millions of people and companies deploying these systems at scale. And so we're very concerned about issues like bias and misinformation, problematic content. We're also concerned about people misusing AI to, say, generate misinformation ahead of elections. So that's the kind of thing we're very concerned about. The other thing that's on our minds is that
[0:03:11] We think AI is going to get much, much more capable very soon. We think that systems that we have now, like Claude, will be replaced with systems in just the next few years that are much smarter, able to do much more complex tasks. And so as AI systems effectively become more powerful,
[0:03:32] The risks become more serious and so it's very important to us to get ahead of that and do the research today to make systems more interpretable, more reliable and safer so that we're prepared for even more powerful AI systems tomorrow.
[0:03:47] And now those concerns about reliability and safety, they're shared by your colleagues in the industry, and in particular, your former colleagues at OpenAI. And I just want to revisit the founding story of Anthropic because it's so curious to me. I mean, you're a theoretical physicist. You're a professor at Johns Hopkins. You were a consultant at OpenAI when you and a bunch of your colleagues, Dario, the CEO, Daniela,
[0:04:14] An executive and his sister, my former colleague, Jack Clark, broke off to start Anthropic. So to the extent that you want to flashback a little bit, tell us in 2021 what you saw at OpenAI that made you guys think that to really pursue these values, you had to do it outside OpenAI with your own company.
[0:04:36] For a very long time, so I've actually been friends with Dario and Daniela and Crisola, some of the other founders for more than a decade. And they've been talking to me about AI for quite a while, telling me that it was going to be absolutely transformative. And of course, I thought that they were crazy. But over time, I sort of read more and more about the AI literature. I spent time with some other people. So I come from physics, other physicists reading and learning about the subject.
[0:05:05] And I was really struck by the pace of change. And this is something that Dario had been thinking about for a long time. He wrote this paper back in 2016 before any of the AI systems people are excited about today even existed, before the technology for them existed.
[0:05:22] Arguing that we could already start working on kind of concrete problems to make these systems safer in case they did in fact become really powerful and transformative for society. And so I think for a long time many of us had kind of a vision of building an organization that put this kind of safety research at the center of what we did and we just wanted to be able to pursue that vision on our own. And was there a feeling that
[0:05:50] Your former colleagues at OpenAI didn't have the same values or were moving too fast? No, I mean, I still keep in touch with those people. I mean, I think I had dinner with people from OpenAI just a few months ago. I mean, we're all still friends. We still talk. But we just thought, I mean, maybe it's a sort of Silicon Valley sense, but, like, if you have an idea, you start a company. And yet there really is a sort of foot race.
[0:06:18] In AI happening now before us all in terms of raising money, partnerships. Anthropic, of course, recently announced a partnership with Amazon, has raised money from Google. There's a war for talent going on. There's a feature race with the latest products. And I guess I wonder, does the old fashioned Silicon Valley requirements of speed and competitiveness, is there tension with the more cautious?
[0:06:47] Constitutional AI approach. Yeah, I think that's a great question. And something that we'd really like to see, and that's sort of part of the reason why we're excited to have started Anthropic, is we think that there should be kind of a race to the top on safer AI, more ethical AI in preparation for the fact that we believe there will be more powerful systems.
[0:07:10] So I think the goal is for there to be competition in the direction of safer and more reliable systems. So to sort of make that concrete, we recently made a set of commitments, a responsible scaling policy, responsible scaling commitments, about basically standards that we'll hold ourselves to as we build more powerful systems. So the current systems we have now, like Claude 2, we have...
[0:07:39] We have information security. We have constitutional AI. We think that that's sufficient. But we're imagining systems that are effectively able to operate autonomously, a system that might be able to sort of install itself and run itself on new computers all on its own, or a system potentially that might be used by bad actors to, say, build or operate cyber weapons, other kinds of weapons. We all know that
[0:08:06] Current AI systems are really helpful to software engineers for coding, but if you take that a step further, more powerful equivalent systems could be used.
[0:08:14] So we're very concerned about those possibilities and even more speculative possibilities over the next few years. And so we made a set of commitments about the sort of information security and the level of safety and robustness to what's called red teaming where people try to break these models to get them to violate their principles and do something bad. So we made a set of commitments for evaluation and testing and then
[0:08:42] Effectively, we can't release these systems if we can't meet certain safety benchmarks. And so our hope is that that kind of thinking, maybe something very similar, maybe something else equivalent, will be embraced by other competitors and that we'll all be sort of racing to develop technology that's more reliable and more steerable, controllable, et cetera. Yeah. And tell us about Claude2 and who's using it.
[0:09:10] And is the proposition of constitutional AI and accountability and perhaps a little bit more transparency, is it having an impact in the marketplace? Absolutely, yeah. So I mean, I think there's sort of a very simple bottom line sense in which no large company, no enterprise wants to use an AI system that might insult their customers or give incorrect information to
[0:09:37] To their associates, their customers, et cetera. So I think that the sort of safety brand and trustworthiness really is something that a lot of companies are excited to work with Claude because of that reliability. And how about with consumers? I mean, maybe it's early, but of course we see OpenAI integrating with Bing. We see Google making BARD an increasingly prominent part of every search. Where in this emerging environment does Claude factor in?
[0:10:07] Yeah, so you can go to claude.ai and talk to Claude for free. You can also sign up for a Claude Pro account and then get a lot more usage. You can give Claude lots of material. You can take advantage of its long context to talk to books, things like that. So we're definitely seeing a lot of engagement from consumers as well.
[0:10:32] Okay, of course there is a big raging and somewhat endless debate in the AI world about AGI and this idea of AI that achieves a semblance of human cognition. And of course lots of debates over when, if, how, why, what happens. What is your personal view of how far away we are from that point? It's a great question.
[0:10:57] I don't know that we're actually that far away. I certainly, I mean, as you mentioned, I was working as a physicist before I moved to AI. And I started working on AI because I just thought that the promise of the technology was incredible. And so I definitely expect that level of AI system in my lifetime. And I think it's quite plausible in the next five to 10 years. So I think it could be quite imminent. Now, I mean, we might be wrong. I was always very skeptical of these statements.
[0:11:26] Maybe 10 years ago, but I've slowly become convinced. So why is that? Well, if you play with Claude or ChatGPT, you're sort of seeing a snapshot in time of what AI was capable of. What you haven't done is played with GPT, I imagine, is played with GPT-1, which could barely write a coherent sentence, or GPT-2, which couldn't really write a coherent paragraph, or maybe you've played with GPT-3, which was sort of one of the first.
[0:11:52] Models that was made commercial by OpenAI, which can write a few paragraphs, but it's really hard to control. It's really hard to get what you want out of that model. It certainly couldn't do math. It couldn't really write all that much computer code, et cetera. And so if you see the rate of progress viscerally
[0:12:09] firsthand over just the last five years. That's all it took to get from GPT-1 to current models. It's not so crazy to imagine that the next set of models, Claude 3, Claude 4, GPT-5, et cetera, are going to be capable of even more incredible things that
[0:12:28] That maybe people aren't anticipating right now. OK, so you're talking about really exponential growth. Do you foresee any bottlenecks, either the availability or the expense of compute, or even just the availability of data that there might be to crawl? I mean, right now, obviously, everybody with copyright owners are starting to put restrictions on the way in which their material can be crawled. So when you develop Claude 3, are there bottlenecks? It's a great question.
[0:12:57] I think it's possible there are bottlenecks, but there are a lot of people who said that large language models weren't really capable of much just a few years ago, and that AI was going to run into a wall.
[0:13:11] The meme is more that AI is just bashing through every wall that we see. So I think there could be bottlenecks with data. There could be bottlenecks with compute. But my guess is that innovation is going to find a way to keep improving AI systems. And do you need more content to crawl to develop the next version of Claude? Or is there a point at which it can kind of train itself and learn to reason and doesn't need to ingest as much data?
[0:13:38] That's a great question. I think it's very plausible that at some point it won't need more data. If you look at constitutional AI, the thing that's cool about constitutional AI technically is that AI is training itself. You don't need people to label data to train a safe model. You can just have the AI model learn to abide by a set of principles.
[0:14:01] By supervising itself. And so that's an example already where basically 100% of the data is synthetic that Claude's using to train itself. And so I can't imagine these kinds of patterns continuing. And so people getting around data bottlenecks in that way. There is a big kind of raging debate, even among the sort of founding fathers of AI, about
[0:14:23] The threat to humanity. And on one side, you've got the great Canadian AI researcher, Yoshua Bengio, who is saying that there's kind of real danger here as these systems develop. And then you have Yann LeCun from Meta saying, actually, there's a quiet minority of AI scientists who believe that progress is going to do more good than harm and that the threats are overstated. Lots of self-serving voices here.
[0:14:53] On the spectrum of alarmism, where is Jared Kaplan? Yeah. My expectation, I think my default guess is that AI is going to be incredibly beneficial. It's going to make our lives easier. It's going to give us access to better medicine, better ideas. It's going to make productivity easier. That's my default assumption. But I think my view also, I think in a common sense way,
[0:15:21] is that if you have a technology that's becoming more and more powerful every year in really dramatic ways, that there's reason to be wary and reason to be sort of thoughtful about that progress. Because any technology that makes companies, governments, and individuals much, much, much more powerful, gives them access to a lot of new information, could also potentially be misused in harmful ways. And furthermore, I think
[0:15:50] None of the AI models we have now are perfect. So Claude is definitely not perfect. Constitutional AI is not perfect. It's possible to jailbreak Claude and get around at safety guardrails. That's true for Bard, for GPT, for basically every model out there. And so we already have these systems that are fairly powerful, but maybe just beginning to be useful, that we can't
[0:16:17] 100% control. I can't give you 100% guarantee or anything like that that these systems are controllable or safe. So if you imagine that we get much, much more powerful systems without learning anything new about how to make them safer, and I think it really begs the question of sort of where are we headed? Are these systems really going to be controllable when we have Claude 4, et cetera? And so I think that both for misuse reasons and because
[0:16:45] The actual technology to make these systems controllable and do what we want. For both of these reasons, I'm concerned and I think regulators should be as well. You know, it's interesting. I mean, on that question, one of the most surprising things of the last year has been AI researchers like Jeffrey Hinton stepping out of their day-to-day roles and really expressing some kind of existential doubt about their role in the industry, about the direction of the technology. Has that...
[0:17:15] Self-doubt infected you at all? Have you ever stopped to think, OK, what role am I playing? And how much good am I doing here, considering the long-term questions? Absolutely. So I mean, I got involved in AI mostly because I was becoming convinced that there was going to be this rapid rate of progress. And therefore, there were benefits, but also risks. And I really wanted to work on maximizing the benefits, minimizing the risks.
[0:17:42] At every step in the research that I've done and as we've deployed the technology, I find it daunting how widely used and how this technology is already. Basically, at the rate of change, rate of progress, the way that it's not just a bunch of nerds at a lab somewhere who are excited, but the whole world has woken up and is embracing this technology. So I find that daunting.
[0:18:12] I'm concerned about technology just because it's sort of dual use. So maybe to give the most surprising example, when we first developed constitutional AI and were writing a paper about it, I think in some ways that paper was almost one of the scariest that I've written because I felt like, well, anyone can write this constitution, any AI developer. So although we're trying to develop this research to make AI
[0:18:37] And so I think really kind of drove home to me this idea that the technology is dual use, that it can be used for good or ill. So next week, Dario, your CEO, is coming to London and is going to participate in Rishi Sunak's AI Summit. And in addition to a lot of other AI leaders from the US, I wonder what you think, you know,
[0:19:04] Global leaders like the Prime Minister are trying to accomplish there. What can reasonably be accomplished? Well, yeah, so I think probably a lot can be accomplished. I think the danger is that the technology is moving so quickly that it's very difficult to know exactly what kind of actions governments can take to improve AI. And of course, I think we're going to a large extent to listen and to help in whatever way we can. But I think...
[0:19:33] There are a few things. So for one thing, you might imagine that it's easy to determine is an AI system beneficial, is it safe? And we don't think it's easy at all. We think that evaluating these systems because they're so general purpose is really, really challenging. And so we think that investing in that is really important. And then we also think that something perhaps inspired by vaguely similar to our responsible scaling policy where we avoid crying wolf and saying,
[0:20:01] That every single AI system needs to be regulated. Every single AI system is dangerous. But we really take a sort of forward-looking approach and say, look, at a certain point, these AI systems could become so powerful that they are really dangerous and really do merit regulation. And trying to find a way to do that in a way that's sort of sensitive to the way the technology will change over time.
[0:20:23] I think it's really important and I hope Anthropic can help contribute to that mission. And what does it mean practically? Do you give access to governments like the UK government, access to the foundational model? Do you allow governments to test Claude? Yeah, I mean, I think that it could mean many things. I think, first of all, governments, of course, are funders of academic research. A lot of that research...
[0:20:50] does focus on safety and continue to, and in particular on evaluations for these systems. And then as the science of evaluation advances, then it should be possible to also create some well-informed regulations on
[0:21:08] On systems that are sufficiently powerful or sufficiently problematic. We've got two minutes left and I want to hit two more points. You recently engaged in an experiment called collective constitutional AI, essentially allowing your users to suggest the rules. And I'm curious practically where that experiment starts and stuff, particularly considering you might get two wildly different sets of rules in different parts of the US and different regions of the world.
[0:21:37] Yeah, so that's a great question that you raised. So what we did is we used this platform, Polis, to basically sort of source potential constitutional principles. Then we allowed people participating, around 1,000 people, to basically accept or reject or modify these principles and vote on kind of what principles seemed best. The list of principles we got, and this was in the US, was actually quite similar to the kind of constitution that
[0:22:07] We created initially as kind of a rough draft for Claude. Differed in a few ways. I think there was sort of more of an insistence that the model kind of at every stage acknowledged that it was an AI, that it not be confused with a person. But to the broader question, yeah, I mean, this is a procedure you could try to follow.
[0:22:29] around the world in order to sort of source information about global values. Yeah. Okay, leave us with this. As we look back at Claude 2 in 10 years or GPT-4, what are we going to make of these models that we're now using in 2023? I think that in 10 years, Claude 2 will look extraordinarily limited and primitive and very, very confined in the sense that with, say, Claude 2, all you can do is talk to it. It can't...
[0:22:57] Use any other modalities. I think that systems in 10 years will look much, much, much more powerful, and I hope they are also much safer and more beneficial. All right, we'll leave it there. Jared Kaplan, thank you very much.