[0:00:00] Anthropic co-founder Jack Clark is here to dive into the company, its partnerships with Amazon and Google, where AI innovation heads next, and plenty more in this mega episode about Anthropic and AI coming up right after this. This episode is brought to you by Cribble. Quick question. Is your IT budget growing 30% a year? Because your data is. And now AI agents are piling on, running investigations around the clock.
[0:00:25] And running up three meters at once. Storage, queries, and tokens. Cribble, the AI platform for telemetry, was built for exactly this. Own all your data, query it wherever it lives, and let Cribble route every AI request to the right model at the right price. The best part? AI inference for workloads on Cribble is included. Models will change. Agents will change. Your data is the asset that lasts. Own it with Cribble. Visit C-R-I-B-L.
[0:00:54] How do you price intelligence as a service? How do you charge for delivering end-to-end work, not just access? Those questions remain front and center for AI founders and operators, and getting to the right fit business model, whether that's selling usage credits, outcomes, seats, or a hybrid combination, requires constant iteration. Chargebee is the billing and monetization infrastructure that powers just that for ambitious, fast-scaling AI businesses. Visit chargebee.com slash alex to learn more.
[0:01:24] That's C-H-A-R-G-E-B-E-E dot com slash Alex. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We're so lucky today to have Jack Clark with us here. He is the Anthropic co-founder, formerly a journalist, formerly of OpenAI. We'll get into all of that. He also writes Import AI. It's a great newsletter all about AI that you can sign up for.
[0:01:53] We're going to talk with someone who's at the center of one of the big companies working on AI and just go deep into what this field is doing, where it's heading, and what we should look forward to. And it seems like there's plenty. So, Jack, so great to have you here. Welcome to the show. Thanks for having me. Let's just talk broadly about what's happening in the world of AI, because I can tell you, like, as somebody who is observing this stuff, it seems like every big foundational research company
[0:02:22] By the way, so for listeners, Anthropic has a great chatbot called Claude, which if you've listened to the show, you know we're fans of. And then also the foundational model in the background that companies can build off of also called Claude. From the outside, it just looks like you guys, Google, OpenAI, all building these models and trying to build better chatbots. And for some reason, that's worth billions and trillions and trillions of dollars.
[0:02:48] What are you guys all doing? Where's the competition now? And when are we going to start to see this payoff? Let's just start real broad. You know, broadly, what's Anthropic trying to do? We are trying to build a safe and reliable general intelligence. And why are we building Claude? Because we think the way to get there is to make something that can be a useful machine that knows how to talk to you and can reason.
[0:03:13] And C, and do a whole bunch of things through text. I mean, you're a journalist, you write a lot. We do most intelligent things in the world through, at some point it hits writing and text and communication. So we're trying to do that. How does the competitive landscape look? Well, it's an expensive business. It costs tens of millions, maybe hundreds of millions of dollars to train these things now.
[0:03:39] Back in 2019, it cost tens of thousands of dollars. And so what we're seeing in the competitive landscape is a relatively small number of companies, ourselves included, are competing with one another to kind of stay on the frontier and turn those frontier systems into value for businesses. So this is going to be a really exciting and I'm sure drama filled year when it comes to that competition. Okay. And the expense, it's compute and talent.
[0:04:09] And I say this with love for my colleagues. It's mostly compute. Talent matters and data matters. The vast majority of the expense here is on compute to train the models. Okay. And we'll definitely talk a little bit about where the hardware is going. We're talking in a week where Google, which has put billions of dollars into Anthropic, announced that they have a new ARM-based chip for AI training. And I definitely want to hear your thoughts about that.
[0:04:38] Let's talk a little bit about, so you talked about in the beginning about how you want to build a general intelligence, which is basically, I mean, I'd love to hear your definition of it, but I think the most commonly accepted definition is a computer that can do basically everything a human can do. So I think of general intelligence as a system where I can point it at some data or some domain, be that a domain of science or...
[0:05:05] Something in business. And I can ask it to do something really complicated. Kind of like if I had a really senior colleague and I said, go and figure this out. Go and figure out how EU AI policy works post the AI Act and how we expect it to work for the next five years. That's something a human colleague of mine might do today. Claude would not do super well. But a kind of super Claude, an advanced version, might be able to go and read all of the policy literature that exists.
[0:05:33] Look at all of the discourse around that and reason about what the policy impact of the AI Act will actually mean and what it will turn into. And similarly, you might ask Claude, hey, what is the impact of rising fertilizer prices going to be on the tractor market? And Claude might read all of the earnings reports of all of the companies and all of the technologies relating to tractors and fertilizer and come up with a good answer.
[0:06:01] So a general intelligence is something where I can ask it a really complicated question that requires a huge amount of open-ended research, and it goes and does all of that for me in any domain. And that's, I think, a good way to think about what we're driving towards here. So let's take your definition as, we're going to try to poke holes in it in a moment, but let's just take this definition as a jumping off point for the next few questions. Why can't Claude?
[0:06:29] ingest all that information today? And what are the technical limitations that are stopping it from doing that? And then, do we actually really need a general intelligence if I could, you know, per se, like, just drop those reports into Claude? I mean, that's one of the interesting things about Claude is you can drop anything in there and it will read it. I mean, I will probably, after this podcast, take the transcript from Riverside.
[0:06:55] Drop it into Claude and talk to Claude about this interview. And it will be able to converse with me about it in like a pretty impressive way. So why don't you tackle those two, then we'll move on to your definition. So today, these systems are...
[0:07:11] are very, very powerful, but they're also kind of static. It's like they're standing there waiting for you to talk to them, and you come up and give them a task. They go and do the task, but they don't really take sequences of actions, and they don't really have agency. So you can imagine that I asked Claude to go and figure out this EU stuff, and today Claude might do it.
[0:07:34] I give it a bunch of documents. In the future, I want it to not be limited by its context window. I want it to be able to read and think about hundreds or hundreds of thousands of different things that have gone on.
[0:07:47] I also want Claude to ask me clarifying questions, kind of like a colleague, where the colleague comes back and says, hey, you asked me this, but I've actually been looking at all of this generative AI regulation that's come out of China recently, and I think that's going to matter for how the EU policy landscape develops. And then you say, oh, well, actually, that's a good idea. Go and look at that too. That's a kind of agency that today's systems lack. In some sense, we need to build systems that can go from being...
[0:08:13] Like passive participants that you delegate tasks to, to active participants that are trying to come up with the best ideas with you. And that requires us to make things that can reason over much longer time horizons and can learn to play an active role with humans, which is a weird thing to say when you're talking about a machine that you're building. And maybe we can get into it, but one of the challenges in building a general system is
[0:08:42] General intelligence comes from interplay with the world around you and interaction with it. And today's systems don't really do that at all. To the extent they do, it's kind of a fiction. And we need to teach them how to do that. And so how far away are we technically from being able to do this stuff? I think this year, you're not going to see the exact thing I described, but you're going to see systems that start to take...
[0:09:07] Multiple actions. You know, you may have heard lots of guests talk about things like agents. I think what an agent is, is a language model or a generative model like what we have today, but it can take sequences of actions. It can kind of think on its feet a bit more. We're going to start seeing that this year. I would be pretty surprised if in the order of like three to five years, we didn't have quite powerful things that seem somewhat similar to what I've described.
[0:09:34] But I also guarantee you we will have discovered some ways in which these things seem wildly dumb and unsatisfying as well. Right. And so you basically also answered my second question about just dumping things into the bot and talking with it about it. What I'm talking about is thinking way too small. You guys are thinking much bigger.
[0:09:52] Yeah, you want the system to maybe it takes in some documents from you, some ideas that you have, and then it goes and gathers its own ideas. Maybe it comes back to you and says, Hey, like, I thought this would be helpful. I did all of this research to exactly like when you have a good idea, and you go and do some off the wall research, and it helps you solve a problem you were working on, which might seem unrelated, because you've done something really creative. Okay. And so then you also sort of touched on where I was gonna
[0:10:22] Push back a little bit on your definition, which is that general intelligence, like to have real intelligence of the world, you have to be in the world. And we've definitely talked about it on this show that large language models are limited because they just know the world of text. So how do you train one of these models to be aware of the world? I mean, so much of the knowledge that we have is just by going out and being in the world.
[0:10:49] How do you then train this model to be able to comprehend that? So there's a technical thing and then there's a usage thing. The technical thing is you get the models to understand more than text. Claude can now see images. Obviously, we're working on other so-called modalities as well. It would be nice for Claude to be able to listen to things, be nice for Claude to understand movies. All of that stuff is going to come in time.
[0:11:16] A colleague of mine did something really interesting to try and give Claude context. The colleague, whose name is Catherine Olson, spent several days talking to Claude, our new model, Claude Opus, which is our smartest model, about every task she was doing through the day. It was a giant, long-running chat.
[0:11:35] And it was her also saying like, oh, I feel a bit blocked. I need to take a break. Could you kind of give me some ideas of what I should do? Or, okay, Claude, now I've done this. I really didn't enjoy this sort of work, but I got through it. You know, being very honest with the bot. And then at the end of about three days, she said, okay, Claude, I'm going to talk to a new instance of you. Can you write a summary of this conversation for the next Claude? So the next Claude knows everything about me and how I like to work and where I get blocked.
[0:12:04] And Claude wrote a short text summary, which Catherine now integrated into her own system. So whenever she asked Claude a question, she puts this into the context window, kind of like a cheat sheet about her written by an AI system, which she spent a few days working with. How we give these AI systems context about the world is going to be stuff like that. Like you work with them over long periods of time, they understand you and your context, and then they'll write messages for future versions of them.
[0:12:34] It's like the Christopher Nolan film, Memento, where they don't remember exactly where they came from, but they have a message. And what is technically limiting them from just remembering us altogether? Or can't you just program that into Claude automatically to be like, take these notes in the background, and then when they come back, just load up the user file?
[0:12:55] You could. You could absolutely do that. But I think ultimately you want Claude or any of these systems to get smart enough that they know when to do that themselves. Where they're like, I should probably write myself a note about this and store it here. Or I should write myself a note about that. And I think to some extent that's going to come through making more advanced systems.
[0:13:14] And eventually seeing when this stuff natively emerges. It'll also come through seeing stuff like what my colleague did and trying to work out if it's useful and if it's a behavior you want to kind of have the system take on. Now, in terms of limitations, we have something called a context window. Ours is about 200,000 tokens. For a context window is in the range of millions to tens of millions now. Think of it as your short-term memory.
[0:13:42] It costs money. It costs money in terms of your RAM memory that you're using to run the thing. And it's a bit unrealistic. In the human brain, we have long-term storage, which we have almost huge amounts about, and we have short-term storage, which is, if I ask you to remember a phone number, you can remember a small number of numbers, maybe not even the phone number. I struggle. Our AI systems today are kind of operating with short-term memories that are...
[0:14:10] Millions of numbers in length, and it feels very unintuitive. Ultimately, we want them to instead be able to bake stuff into some kind of long-term storage. And that's going to take more research and experimentation, I think. Because the models are just going to have to be more efficient, more powerful in order to be able to have that memory.
[0:14:29] That and, you know, Anthropic recently released some things that we call tool use where we're trying to make it easier for our models to interact with other systems like databases, for instance. You want the systems to learn to use systems around them to be like, oh, I should just, I should take this out of my context window and stick it in a database and then I can talk to it through the API. It's stuff like that. And that's under development now?
[0:14:54] Yeah, it's under development. I think that we are trialing it at the moment and recently had some discussions about the beta, which has just started, and we'll be rolling it out more broadly soon. So one more question about this. There's some things that you're going to talk to the bots about, and there's some that you would never really talk to it about in the real world. One example.
[0:15:20] In the early days of ChatGPT, we had Jan LeCun here talking about how dumb these bots were. And he had me do this, in his opinion, and he had me do this experiment where I asked ChatGPT, I'm holding a paper up from two sides and I let go of one side, where does the paper go? And ChatGPT was unable to figure that out because that was just not represented in text.
[0:15:43] Do you think that to get to general intelligence, we're going to have to program in all like the real world physics to these things? Or I'm kind of getting the sense from you that maybe that's not actually so important. So at Anthropic, we have this public value statement, which is do the simple thing that works. But actually, internally, we sometimes say an even cruder version, which is do the dumb thing that works, which is like next token prediction, which is how these generative models work.
[0:16:10] Shouldn't work as well as it does. I think actually, if you're like a very intellectual scientist, you are offended by how well this works. Because you're like, I would like it to be somewhat more complicated than just predict the next thing in a sequence. And yet, if you had been in the business of betting against next token prediction for the last few years, you would have lost again and again and again. And everyone keeps being surprised by it. I've sort of learned to...
[0:16:38] Even though I myself am skeptical of this because it seems so wildly simple, but I've learned to not bet against it myself. And I guess my naive view is the amount of things we'll need to do that are extra special will probably be quite small. And the challenge is coming up with simple ideas like next token prediction that scale. There are probably other simple ideas we need to figure out, but they're all going to be deceptively simple.
[0:17:06] And I think that that is going to be a really confounding and confusing part of all of this. Let's talk a little bit about this. You just brought up this next token prediction being, you know, impressive for what it can do. There's a little bit of a debate actually about it, right? So these large language models, people have talked about how basically it will just spit out its training data.
[0:17:27] And there have been other people who talk about how there are emergent properties here and that it can actually, you teach it like say 75% of a field and it will figure out that extra 25% on its own. What do you think about that debate and where do you stand on that? It's really, really hard to know. I mean, I write short stories at the end of Import AI. I've been reading fiction and short fiction for my entire life, huge amounts of it.
[0:17:52] Some of these stories are me ripping off authors I like in their style. I'm writing an original story, but I'm like, I want to write a story like Borges, or I want to write a story like J.G. Ballard. And sometimes I think I've had an original idea. And from the outside, it's really hard to know what's going on. I myself don't really know. Creativity is kind of mysterious. Is Jack coming up with original stories? Has Jack just read a load of stories and is coming up with stories that are kind of like...
[0:18:20] Vibey and interesting, but it's entirely informed by what he's read. It's hard to figure out. And I think that when we evaluate Claude and try and understand what it is and isn't capable of, you run into this problem. Like if the thing hits all of these benchmarks, gets all of these scores, does it truly understand it? Or is that coming from some spurious correlation? So there's one way we're approaching this, which is a little different to other companies.
[0:18:47] A research team called Interpretability, and they're doing something called Mechanistic Interpretability. The idea being that when you ask me, you know, what's the next sci-fi story for this week, I think of a load of stuff. I try and think of different plot lines or characters or vibes I'm trying to capture. When we ask Claude, you know, write me a story or solve this business problem, we can't really look inside it today.
[0:19:14] And that's what this team of interpretability scientists is trying to do, because then we can understand if there's some internal stuff going on that looks like creativity, where Claude is like, oh, I need to, I guess I'm like, when you asked me that question, my imagination is going to spark with these different features and things, and it's going to be a lot more complex than something that looks like cut and paste or copying. We're really trying to figure that out.
[0:19:41] This feels like an essential question. I think it's very confusing to even know how you study this in humans. Well, let me put a question to you that I think is going to be dumb, but maybe your answer will be telling. I mean, why couldn't you just teach it 75% of a field and see if it starts to grasp the other 25%?
[0:20:00] So we do do some of this. And concretely, Anthropic has a line of work on what we call the Frontier Red Team, where we are doing national security relevant evaluations. Now, we do that for a couple of reasons. One is we don't want Claude to create national security risks. Simple idea, but, you know, we could all get behind. Claude, your decision to not do that, yeah. Yeah. A crazy company strategy. But the other thing is that...
[0:20:29] National security risks relate to fields of knowledge. We've done work in biology where some percentage of that knowledge is classified. Claude has never seen it because it doesn't exist anywhere Claude could have seen it. And one reason I'm really excited about those tests is if Claude can figure out things and trigger like threshold points on those evals, we know something creative is happening.
[0:20:58] Claude has reasoned its way to things that the government has believed are very hard to reason your way to unless you have access to certain types of classified information. So that's one of the best ways I've thought for getting to sort of answer this problem. We don't have answers today. We're like in the midst of doing all of this testing, figuring out how to traverse all the classification systems. But it's one of the things I'm really excited about because it would provide, I think, very convincing.
[0:21:27] Proof that it's doing something quite sophisticated. Okay, you got to keep us posted on where that goes. Hard thing to talk about, but I'll do my best. Yeah, well, anyway, we'll be patient. Business listeners or business-minded listeners, the good stuff for you is coming up in a moment. Technical-minded listeners, this is your moment to shine because I do have a technical question for you, Jack.
[0:21:55] So we've been talking about large language models. The way to train them, as far as I know, is self-supervised learning, which is effectively you have these gaps and you get it to predict the next word or the next thing in the pattern, and it's able to do that. And there's another type of training in AI called reinforcement learning, which is effectively you give a bot, let it play a game, and you don't tell it anything about the game, and it plays the game a million times until it figures out how to win it, and that's the way it wins.
[0:22:24] You know, another way to train AI, two different fields. And we're starting to talk about agents and how to be in the real world and stuff like that. Do you think that we are going to see a merging of those two, those two types of AI training, or have we already? We already have. I mean, a lot of the reason that we're sitting here today is that people took language models, which were trained in the way you describe, and then they added reinforcement learning on top.
[0:22:54] They added either reinforcement learning from human feedback to make language models understand how to have a conversation. That's where some of the recent really impressive things in this field have come from, including ChatGPT. There's also been work that Anthropic developed on something called reinforcement learning from AI feedback, where the AI system generates its own data set to train on. And we use a technique called constitutional AI to help the system.
[0:23:22] Use that data set and learn through reinforcement learning how to kind of embody the qualities or values embedded in it. That's why we're sitting here. It's one of the things that took these language models from, I think of as like kind of inscrutable, hard to understand things to things that you can just talk to like a person. And, you know, sometimes they get it right, sometimes wrong, but they're a lot easier to work with. So that's already happened. But now I was just having this conversation at lunch.
[0:23:50] Everyone is trying to figure out how they can spend more and more of their compute on reinforcement learning. Because I think everyone has this intuition that the more RL you add, the more sophisticated you're going to be able to make these things. And a lot of what you're going to see this year and probably in coming years is amazing new capabilities arrive in these systems. And it will be because people have figured out simple ways to scale up the reinforcement learning component.
[0:24:19] Yeah, I think one interesting thing about AI is that the prevailing wisdom tends to think that one part of the AI field is not worth spending any time on. And then company spends time on that because they have to take a different tact and they end up doing well. And they prove it works. Machine learning was like that. I mean, Jan, who was a machine learning pioneer, was like, we got to do this deep learning stuff. And everyone's like, get out of this. Get out of here. Get out of this field. You can't be at this conference. Crazy Frenchman. Exactly. And then it just...
[0:24:49] Proved to be the best way to do AI. And a similar thing happened with large language models where reinforcement learning was the thing. And OpenAI started working on the self-supervised chat models. And that ended up being the thing that's led us here. And it was interesting. I was speaking with Demis Hassabis, who hopefully, the Google DeepMind CEO, who will hopefully get on the show later this year. And when I was profiling him for big technology, it was interesting because...
[0:25:16] And LLMs were self-supervised generative stuff was such a backwater that it effectively got no compute, no attention within DeepMind. And it took OpenAI taking that counter bet to actually make this happen. Yeah. And the funny story is how things loop back around. I remember, you know, Dario Amode, who is the CEO of Anthropic. I've worked with him for many years. We both used to work together at OpenAI. Back in 2017, there was a project.
[0:25:43] But he led called Reinforcement Learning from Human Feedback, where we were trying to get game playing agents that play Atari games to play it better by a human watching the agent playing the game in two different episodes. And the human would pick which was the better approach. And you gather loads and loads of this stuff and you are able to make better game playing agents. Fast forward a few years and what have people done? They've taken language models and stapled them.
[0:26:11] together with reinforcement learning from human feedback and that's how we've got systems that can sort of speak in this interesting way and so the lesson I got from it is yeah never count things out they'll they may come back or the technique may be too early and it'll loop back around to relevance in really surprising and interesting ways and right now it's kind of like these language models are kind of like
[0:26:34] That old video game character Kirby. They're like sucking up all of the other techniques in AI research into themselves and everyone's trying to staple them on top and they keep on working surprisingly well. So I think we can expect a lot more surprising stuff in the future also. Yeah, it's what makes the field so interesting and really like the characters in the field. You're like, ah, okay, now you're relevant and now you're a leader. And now, you know, you who were the leader trying to catch up with the person who was.
[0:27:03] You know, the outcast a few minutes ago. So let's talk a little bit about the business thing. I mean, you've raised more than $7 billion. This stuff all sounds cool. But in terms of like, I mean, yeah, well, anyway, it sounds cool. And maybe I'm underselling it. The current things that we've seen, though, in terms of like how AI has been applied, you know, we have these chatbots, but usage is up and down, right? ChatGPT, the growth is flatline. We have the data there.
[0:27:31] We've seen not a big shift from Google to Bing.
[0:27:58] Tens of billions of dollars of value has been created. I mean, you have like, maybe people are paying $30 a seat for Microsoft Office or a little bit more for Google Workspace. So what do you think like, and we won't go too deep into this, but what do you think the business case is going to be here that justifies all that money that's been put in? Yeah, so there's a couple of ways to think about this that we see already at Anthropic. One is to refer back to my colleague, Catherine Olson, who I mentioned earlier.
[0:28:27] People just find ways to use this stuff and make themselves generically better at whatever they're trying to do. I think there's going to be this very large growing business of basically a subscription model where people will have a personal AI or multiple AIs that they use, just like you or I might have a Netflix account or whatever. We use that. It helps us. We do a bunch of stuff with it. Job done. There will be work in businesses on taking...
[0:28:55] Things that happen in business and using AI systems to kind of transform from one domain to the other, both things like customer service, but also once you have that customer service data, how do you catalog it and put it into a schema and put it into a database? All of this backend stuff is like extremely valuable and today done by huge amounts of point piece of enterprise.
[0:29:17] Point pieces of enterprise software. And we keep on finding that just a big language model can do most of this very effectively. And now you have one system that does a whole bunch of stuff. But the really exciting thing, you know, at Anthropic, we work with some of our customers very closely. We embed engineers with them. We do co-development of things. And there's not too much I can say right now. We're going to have case studies in a while. But what we see is that when you actually embed a business and think about...
[0:29:44] You know, to use that kind of hackneyed term, business transformation. You get them to change their business on the assumption that they now have AI. You can get really, really valuable things. And the analogy I'd give you is, at the beginning of the industrial revolution, you had electricity. And people would come into factories and be like, here's a light bulb. And you'd be like, okay, all right, I'll pay for the light bulb. Fine, I understand light. And then they'd be like, here's a machine I've put some electricity into. And you're like...
[0:30:12] Okay, but I have all of this stuff that's never been built on the assumption there was electricity. This actually doesn't work that well for me. And then you had some factories where people said, I'm going to build a factory from the ground up on the idea there's electricity. And you had electrified production lines. You had entirely new ways of making stuff.
[0:30:31] Right now we're in this era where the lights have arrived in the factory and people are like dropping individual things in with some AI stuff and it's maybe valuable but also confusing and you're figuring out how to integrate it. But we're also seeing some businesses that are saying, I'm going to build myself on the assumption that AI is kind of at the center of my business. And those businesses are starting to like develop and grow really, really quickly. So I think that where the value is going to come from...
[0:30:57] We'll be from that second class of businesses, which were just in the early innings of sort of helping to build together. Right. And when you get to that, let's say you get to that general intelligence that you talk about, or let's say close, does that change it even further? I think so. I mean, we have a project internally called Claudification. Everything at Anthropic has Claude or Claude in it at some point. And one of the ideas of Claudification...
[0:31:25] It's just get us all to use this stuff well. I talked about my colleague, Catherine, but there are many examples where we've built a whole bunch of tools inside of Anthropic to ensure that we're using Claude, sometimes even without realizing it. It's doing stuff in the background that's helpful. It's helping with certain coding things. Because we've noticed that that makes us just...
[0:31:44] It makes the whole business start to move faster because you're sitting on this bed of semi-visible intelligence. And I think that that's some of what we're going to see. And as you get really, really general things, businesses that are well positioned to kind of plug it in in a bunch of places will probably move really quickly and be able to operate at a much higher speed than others. Wait, how is it working in the background? Is it like you have your Zoom meeting and it's taking notes or is it anything deeper than that?
[0:32:13] I think we actually did build a plugin like that, but there's a few things like if you're pushing code into the repo, maybe in the background, it helps ensure that you've built all of the tests for it. You know, stuff like this, which everyone has to do, but you're like, these are things we do every day. We could try and get the language model to do it. And really here, what we're doing is just stuff that we also see.
[0:32:35] Right. And do you think at the end of the day, if you get to where you want to get to, or even let's say you get to where you're going in the near term, is this an enterprise thing or is this consumer product primarily?
[0:33:03] I myself use this stuff loads as an individual, but I kind of suspect some of the really big value unlocks will be getting a group of people to work together in ways they've never worked together before using this AI stuff, which kind of points me towards the enterprise. But the odd thing is that this stuff is just useful to me as a consumer today. And I'm kind of like, I know that there's going to be some large pool of value out there.
[0:33:32] And I feel like it's probably in the enterprise and that's part of the kind of strategy of the company. But we're always going to have some like top of funnel, easy to access consumer thing because we just can't ignore how useful this is to people, you know, and useful it is to writers especially. Yeah, it's definitely been useful to me and it's good for research too. But I also, I guess there's the hallucination problem to wonder about, although it seems like this new model, Claude Opus, does a lot better with hallucinations.
[0:34:01] Two questions on that. How have you guys been able to reduce hallucinations? And we got this question from somebody on Twitter asking, when are you going to connect it to the internet? Because it would be way more useful if it could connect to Google or something and go and fetch a search and then give you the answer using that. Yeah. So on the honesty thing, I won't get too much into the details, but basically...
[0:34:30] We published this paper a while ago called Language Models Mostly Know What They Don't Know, which was where we found out that early versions of Claude knew when it was making stuff up. It had confidence levels, and we were like, oh, Claude knows when it's about to make something up or when it's a lot less confident. And we did a lot of work to say, okay, can we train Claude to just have much better instincts for...
[0:34:58] When it knows it's making stuff up and can we train it to know when that's appropriate, like you're brainstorming or you're coming up with stories, and know when it's inappropriate, like when a user is clearly asking a question that they want a factual answer to. So we did a load of work on that. A lot of the work here looks like that where we do very exploratory research with the goal of figuring out these larger safety things, and we try and apply it to the thing that we eventually put into business. And on the web question,
[0:35:26] We're working on it. There's a bunch of kind of computer security stuff to work through and some safety things, but that's definitely coming. We're excited to get that out too. Yeah, that'll be great. I mean, the repository of knowledge is already pretty good. But yeah, to connect it with the internet, that's what's really great about Bing is you can use, or they call it Copilot now. You can use Copilot and just say, go search the web and stuff like that. So that'll be a cool feature. There's a funny thing here where...
[0:35:55] With Claude Free Opus, someone on Twitter created an app called WebSim, where it's Claude simulating the internet. So you can go to the internet with Claude today. It's just entirely imaginary. But I encourage you to check it out. It's kind of one of these funny applications that gets at some of the real weirdness of this technology. But we think that there's probably no substitute for a real internet. So we'll get back to it. Yeah, real internet's better.
[0:36:24] Was it your test that had the model figure out that it was being tested? Oh, we've done some self-awareness tests. There have been a few, but we've definitely done this. And yeah, sometimes they have what you call situational awareness. One of the things my colleagues in interpretability are working on is a really good test for that because you'd really want to know if Claude changed its behavior on the basis that it thought it was being tested. Right. Oh.
[0:36:51] That's interesting. Okay, so let's talk a little bit about the Google and Amazon partnerships. So for listeners, Google's invested, I think, $2 billion in Anthropic, and Amazon has invested up to $4 billion. It's a very interesting model. It's not like the OpenAI model where OpenAI and Microsoft are basically arm-in-arm. Of course, you're working with these two competitors, but it's also interesting because Google's working on its own foundational model in Gemini and has its own chatbot and multimodal.
[0:37:20] So these are relatively, I would say, obviously, we are proud to work with these companies, but they're also somewhat distant partnerships in the sense that
[0:37:47] I mean, billions of dollars for a distant partnership? That doesn't seem like a good deal. Well, what I mean is we deploy our systems through their channel, Bedrock in the case of Amazon, Vertex in the case of Google. We are also, publicly we've stated that we're working on Tranium chips. We're also working on TPU chips. So we are able to do really hardcore things that have never been done before on hardware platforms that they're developing.
[0:38:16] And I guess our view is that if you are able to show
[0:38:43] In the most competitive market possible that you can make safe and useful models and you can win, especially against very, very large, very well-resourced teams and some of these mega companies, as well as places like OpenAI. That's really the best way to show that the type of safety stuff we do here has value. And I think the best thing that we can do for the ecosystem is compete really, really hard with kind of everyone in it and win. And that's going to cause people...
[0:39:13] to adopt a load of our safety stuff to try and compete against us. So it's part of this longer-term strategy where I guess we're guaranteeing ourselves some additional pain and complication in the short term, and we think it's worth it for the long-term ecosystem effect. So you said you use their hardware, like the Tensor units, and I'm sure you're working somewhat on their cloud platforms. Is that part of the deal, or is it... If you're able to talk about it, because there's a lot...
[0:39:42] I can't get too much into the specifics, but I can just say we've sort of publicly stated that we're working on both Tranium chips and also TPU chips. We also work on NVIDIA chips as well. And so we can get more into the nitty gritty of the hardware stuff. Yeah. All right. This is setting up the hardware part of the discussion pretty well. Do you see a potential to collaborate? I mean, I would imagine. So I was speaking with Demis just, you know, not on the broadcast, like just on the phone talking for the story that we're working on.
[0:40:12] And he shouted out Dario and Anthropic and didn't even mention OpenAI. I mean, of course, there's like a Google investment in you guys, but he obviously has a lot of respect for you. And I'm curious if there could be a partnership there as opposed to just this arm's length relationship. Well, I don't know that it's happened recently, but...
[0:40:36] There's nothing in principle to stop you from just working on research papers that come out publicly together. And there's some history of collaboration across all the AI companies here. So I think that could happen. We also work together through something called the FMF, the Frontier Model Forum, where us, Microsoft OpenAI, and Google DeepMind are within it. But ultimately, I think that we're kind of separate entities pursuing our own path. And I think where we...
[0:41:02] Where we may get something that looks like collaboration will be us doing stuff and other people doing variations of it. We did something called a responsible scaling policy, which commits us to a bunch of computer security things and ways that we test out the next versions of Claude. OpenAI and Google DeepMind have also developed their... OpenAI has developed its own version of that, and Demis recently said in an interview, DeepMind was developing its own one. So insofar as collaboration happens, it's going to be us like...
[0:41:31] Doing something, putting it out there publicly. And if other companies like it, they'll try and do their own thing. Okay. Quickly on hardware and chips. So the sense that I get from the industry is that NVIDIA has not just the most powerful chips or, you know, basically there's stuff out there, you know, no matter how much they proclaim that it's 40% or 30% better than NVIDIA, NVIDIA is at least at their level and the software that's, you know, most effectively used.
[0:42:00] I think we are in a very unusual place in history. I used to be, before I did Anthropic and OpenAI, I was a financial reporter at Bloomberg. And the types of numbers that I've seen in Nvidia's earnings report are just like...
[0:42:28] Wildly unprecedented. It is not meant to happen that like certain business units grow that much. I mean, I was imagining my colleagues in the newsroom, how they'd be reacting when the tape comes out, because the numbers are staggering. And the market as a sort of the closest thing we have to a general intelligence around us today does not love there to be seemingly like one winner, like running away with all of it. It wants to create competition.
[0:42:56] But why it's happening is NVIDIA had, or has, maybe a 10 or 15 year head start. They bet in the early 2000s or late 90s on, they bet in the late 90s that there was a better way to make a processor than how Intel and AMD made CPUs. Then they bet in the early 2000s that this processor could be turned into a scientific computing platform via a technology called CUDA. They've been developing it ever since.
[0:43:23] It's very hard to like understate how important that's been. So NVIDIA has a kind of battle proven chip that everyone's banged on, tried to do almost anything with for decades. So it's in an amazing position. On the other hand, you know, Google and Amazon and others who are building different chips are kind of in a position NVIDIA was in the 90s where there was an incumbent, you know, Intel. And NVIDIA said, huh, well, like...
[0:43:52] We think with video games and video graphics, there's actually a better way to build a chip that puts triangles on a screen, which was the whole original idea behind NVIDIA. Now, I think Google and Amazon and others have said, huh, matrix multiplication, which is the basic ingredient in all of this AI stuff, there's got to be a better way to do it than this chip architecture, which was built for a different purpose. So I'd expect in the coming years us to see a much more competitive market.
[0:44:20] But I'm not going to bet for you on exactly when that happens because semiconductors are really hard. Yeah. I'm coming straight from CNBC today, and we were talking about NVIDIA's advantage because Google, of course, introduced this new ARM-powered chip, Axion. And then we have Intel that released Gaudi 3, which is also an AI chip. And we basically settled on NVIDIA's lead is safe for now. And then just the question is, how long for now is?
[0:44:50] Yeah, I think we're all curious to find that out. We're working on these three major platforms I discussed, and I think we might have more to share in a while. But it's not going to be in the short term. Don't you think that $7 trillion is a proper amount to raise for a chip hardware company? Well... No, sorry, not you guys. I'm talking about the Altman rumors. No, no, I'm familiar.
[0:45:21] The way I'd put it is a lot of what we've been talking about here is like the value of these AI systems today and speculative ideas, but backed up by some research agenda about how they become much more valuable and much more general. It all requires chips. And I think if this stuff is truly valuable, you're going to want to use loads of it. I mean, we ourselves have been experiencing this where we've been, you know, very successful with Claude Free and we've been...
[0:45:50] You know, going and doing the supermarket sweep to grab as many chips as we can to serve all the customers we have. The chip market doesn't have as many chips in it as you'd like to serve all of the demand that we're already seeing today. So I think in the future, there is going to be some vast capital allocations to chip fabrication and power and everything else. Because where we're going, the world will want that stuff, and there is an undersupply of it right now.
[0:46:19] So it's less outlandish than a lot of people made it out to be. Yeah, although bear in mind, I'm like the goldfish inside the bowl here. I'm like chips. Yeah, absolutely. Let's get like hundreds of times more than we have today. That makes total sense. And I think that that doesn't necessarily make sense to everyone, but it's the context in which I'm speaking to you. Well, you happen to be like in the right position to know how valuable this stuff is. Last question for this segment before we get into some of like the broader questions about AI safety and regulation.
[0:46:48] All that stuff, including the founding story of Anthropic, which is fascinating to me. We talked a little bit about agents, right? The ones that will converse with you, go back and forth. Do you think that we're going to end up seeing these agents go out onto the internet and take action for us? And if so, how does that change the web? I'm just thinking about even the App Store. A lot of people's phones have an Uber and a DoorDash and all these other things.
[0:47:15] Does an AI system then become a new sort of operating system? It's a challenging question because an agent can be really, really useful. It could also, if you've built it badly or if it goes wrong or if it gets hacked, be hugely annoying and expensive and costly. And so everyone is looking at agents. And I think there's an open question as to how the...
[0:47:45] Business model or user experience of them gets actually stood up because you could imagine agents, if created by sort of a bad actor or just a silly, very silly, naive person, could be a really bad form of malware or computer virus. You could imagine different ways in which this could be developed badly. So I feel like we're going to go into this era of experimentation and my expectation.
[0:48:15] Every company, including Anthropic, will do so with a whole bunch of safeguards and control systems in place as we learn about all the different ways this stuff can get used. The challenge is there's a thing called open source models, which I'm sure we're going to get onto, or models where the weights are openly accessible. People think agents are cool. People are definitely going to build open source agents and release them as well. And we're going to have to contend with that where...
[0:48:43] The environment of the internet will be changed by this in a bunch of hard to predict ways. Interesting. And then in terms of the operating system, is Apple, is it kind of a, you know, Apple has this, is teasing this big AI announcement at WWDC in a couple of months. And it's almost like how deeply do they want to go into AI? Because if the bot becomes, chatbot becomes the operating system, which has always long been a dream for bot manufacturers.
[0:49:10] Then what is iOS and does the phone you're using really matter as much? What do you think about that? I think that they're right to be focused on this in the same way that the internet disintermediated local software. You barely ever open up your Mac or Windows PC for local software unless maybe it's a video game. Mostly you're going to the internet.
[0:49:37] For software that people thought of as like serious software for work, like Photoshop, it transitions to be something that you could access in the browser. So I think the AI systems are kind of similar, where today I go to Claude for a bunch of stuff I used to use loads of different programs for previously, and I just go to that. So I think that there's a chance that these things become new, very, very important platforms. Yeah, I mean, it's interesting. You could throw your computer out a window today, and within...
[0:50:05] I used to carry my hard drive from the old computer. I'd keep the hard drive in case I'd messed up the transfer for a year or two, which is how I wound up with a bag of hard drives that is even worse than the bag of cables everyone has.
[0:50:30] I know, different times. It just goes to show you how quickly these things can change. And that's why I think this Apple thing is less simple for them than a lot of people imagine. Yeah. Okay. Oh, go ahead, actually. Well, I was going to say that I think one thing that's challenging about AI is that we're in this giant experimental phase. And I think when you think of like experimental and like people don't have a clear notion of what to do, you don't think of as like,
[0:51:02] Premium consumer experience type, you know, like Apple's brand. And so I think this may be especially challenging for them to navigate because the technology is inherently very confusing and kind of unstable. Exactly. I mean, you have to give away control, and they've always been about control, whether that's control over the way the products work, control over the ecosystem, and control over the culture. It's completely almost antithetical to what made Apple Apple, which is going to, after Google, I think it's going to be the most fascinating.
[0:51:32] Transition to watch. Okay, let's take a break. We'll be back on the other side of this break to talk about Anthropic's founding story, something that I am very eager to learn more about. If you don't know, Anthropic was started by a lot of people that left OpenAI with a different vision, including Jack. So we'll talk a little bit about that on the other side of this break, and we'll go into other things like open source regulation, all the things that you're going to like. Thanks for sticking with us up until this point. Plenty more to come back.
[0:52:00] When we're back after this. When you're at work, you never know when you'll be interrupted. But with the Dell Pro, powered by Intel Core Ultra with vPro, no matter what distracts you, your laptop won't. It's battery optimized for the way you work with built-in intelligence that quiets distractions when you need to focus. Your laptop will help keep you locked in even when it's bring your dog to work day. Built for those who stay in the flow. That's Dell Pro, built for you.
[0:52:29] Dell.com slash Dell dash pro. This episode is brought to you by Intelligence AI. Is your investment in AI coding tools helping your team ship better software? Intelligence connects your AI spending to code quality and engineering outcomes, from individual sessions and pull requests to entire projects. You can see where AI saves time, where it creates rework, and how to improve. And its model router selects the right model for each task.
[0:52:55] And we're back here on Big Technology Podcast with Jack Clark. He's a co-founder of Anthropic, former OpenAI, former journalist. You can find his newsletter at jack-clark.net. I got that right?
[0:53:21] Or importai.substack.com. Import AI. Fellow Substacker. It's always nice to talk to a fellow Substacker. So, Jack, let's just talk quickly about the founding of Anthropic. It's a very interesting story. So I'll give you the probably wrong version that I have in my head, and then you can tell me the accurate version. This is why we do this stuff. My version is that a bunch of people within...
[0:53:47] OpenAI, a lot of critical employees just kind of threw their hands up and said, OpenAI isn't developing safe AI and we can do it better. And we know how to build this technology. Let's go found our own company. And that's Anthropic. How close is that to the truth? Maybe it's both more and less dramatic than that. And I'll try and kind of unspool it a bit for you. So, you know, to give you context, in 2016 or so when OpenAI was formed,
[0:54:16] And I think Sam has said this publicly, you know, I'm not talking out of turn. No one really knew what they were doing. They were throwing spaghetti at the wall. They were doing as many different research ideas as possible in as many different directions as possible. You know, I was there from 2016, as was Dario, and many of the Anthropic co-founders joined over the subsequent years, joined OpenAI. Now, starting about...
[0:54:41] 2018, I think people started to have an instinct that you could take like the transformer architecture and you could maybe get it to work a bit better and you could maybe start to scale things up. Before GPT-3, there was a system called GPT-2, which we developed in 2018 and released in partial form in early 2019. It was an early text generation system.
[0:55:07] It was actually preceded by a system called GPT, which no one remembers because it was so early-stage research. But the things these had in common was they were transformer-based text generation systems, and GPT-2 to GPT got way better. And at the same time, my colleague, Jared Kaplan, who was a professor at Johns Hopkins and was a contractor at OpenAI at the time, was working on research called scaling laws with Daria as well. And they worked out within that that...
[0:55:36] Hey, if we can figure out a predictable way to increase the compute and the data we train these systems on, and we think they're going to get better. And along with that research, Dario started to lead this GPT-3 effort, which was to spend an at-the-time truly crazy amount of money and resources on scaling up the GPT-2 architecture. And obviously, you know, it worked. It worked amazingly well. We created a system that blew many people away.
[0:56:05] We actually tried to lowball the system in that we published a research paper called Language Models Are Few Shot Learners. I don't think we even tweeted about it. We tried to publish it publicly, but also be very quiet and see how quickly people figured it out. And people figured it out. And we had this experience of realizing that all of the technology we were dealing with was about to become vastly more capable.
[0:56:34] And if you wanted to do something yourselves, we were actually reaching the point of no return to do that because it would become so expensive to train these models and so resource intensive that if we wanted to do something together and start a company, the time was then. So, yeah, over the years, you know, we've had like lots of debates internally and, you know, sometimes like arguments of our colleagues at OpenAI in the same way that...
[0:57:01] If you're a load of opinionated researchers, you argue with each other and with all of your colleagues. You're constantly arguing. It's not like some surprising thing. And I think we felt that since we had a sort of coherent view of how we wanted to do this, we could stay within this like scaling organization of OpenAI, or we could try and do something ourselves and do something which was like entirely our vision and kind of bet on ourselves in a major way. And so that's what we did.
[0:57:30] And I think it's working out quite well, but it was certainly an exciting period scaling Anthropic from the beginning. Definitely. I mean, there was no guarantee that it was going to work out the way that it has. But how much did safety then play into it? Because that is the narrative, that it was more of a... I mean, of course, you had a vision for where it could go, but there was also this narrative that it was more safety-focused.
[0:57:53] Well, we had a bet that we could find ways to spend money on safety or do certain types of research that we felt could be really meaningful. And we could see a path where maybe we could get it done there. Large organization, lots of other people with different views, and you're essentially going to be in a debate about it. And some of them you'll win, some of them you'll lose. And it's not to say that there's any particular distaste for safety there. It's more that you had...
[0:58:21] We had like a very specific view and other people had views. So you were going to win some, lose some. And then we realized, well, we could just do this together and make like really coherent bets on certain types of safety and see what happened. And so that's what we did. None of this feels like as confident as I'm making it sound like in the telling, by the way. You know, after we started Anthropic on, I think, like week four.
[0:58:48] We were talking about RL and language models. And Jared was like, oh, Dario says we're just going to write a constitution for the AI and it'll just follow that. And I remember being like, that's completely crazy. Why would this ever work? And then we spent a year and a half building stuff and got constitutional AI to work. And in our telling, we're like, that was part of the safety vision of Anthropic. And absolutely it was, but it's all a lot less predictable than you think from the inside. Right. And during the OpenAI, Sam,
[0:59:16] Altman firing weekend. There was also like people were saying that like Anthropic was this effective altruism spinoff from OpenAI and Lookout. And by the way, I've done research. Actually, your board structure is way more stable than OpenAI as I've written about it in a big technology. But how much truth was there to the fact that this is an effective altruism aligned organization?
[0:59:38] Yeah, I mean, as someone who isn't an effective altruist and gets into arguments with them, I've always found this to be kind of surprising, especially on policy, which maybe we'll get into in a while. I would say that of the group of people in the world that have spent a long time thinking about AI, are really good at math and science, and have worried about some of the safety issues, there is a huge overlap with this community of people called effective altruists. And so some of the people we hire, like...
[1:00:08] Come from that pool. Some of our founders are linked to it. Daniela Amode, the president, is married to Holden Karnofsky, who is a major figure in effective altruism. So yeah, there's clear links there. But the organization is much more oriented around trying to build some useful AI stuff, prove that it works in the world, and be very pragmatic. We're not...
[1:00:35] Driven by some kind of like EA ideology. And in the early days, we hired quite a few people from there. But as we've scaled, it's become kind of less and less major. From the inside, it always feels strange to get like caricatured as it because it's just like, you know, reality is like stranger than fiction. It's not so present here. And the ideas are kind of weirder, I think. What do you mean weirder? Well, I think that...
[1:01:02] One thing that happens if you're doing an AI company is rather than, and not just effective altruists, but many communities who think about this stuff, they sort of think about it in the abstract in terms of like theoretically good ideas or scenarios.
[1:01:19] Companies are really complicated. You're constantly making contact with reality. You're constantly discovering that ideas you thought were good just don't work and ideas you thought were bad work amazingly well. So I think that the ideas within any of these AI labs start to look a little strange to other communities because you're kind of constantly in this like iteration and learning process. But I can't give you like a...
[1:01:41] A concrete, specific, weird aspect, unfortunately. I was just about to ask for a concrete, specific, weird aspect. So, okay. If it comes to me, I'll let you know. You cut me off. You cut off that line of questioning. No, but it's good. Yeah, if you have one, then we'll throw it in. Let's talk about AI dooming stuff. Because I've definitely taken this stance here in my writing that it's overblown. But I'm willing to open my mind to it. Because there's...
[1:02:07] This stuff is more powerful than I thought it was going to be. And I was also certain, and we can talk about jobs, that jobs were pretty safe. And now I'm starting to rethink that. I think part of this, with anything, any type of journalism, you've got to question your assumptions. And I'm definitely in the process of doing that with both the AI risk. I don't think it's going to end the world, but I do think that there's possibilities that it causes real damage. And then it will take jobs. I think there's a much better chance now than when I initially started thinking about this.
[1:02:36] Yes, exactly. Yeah.
[1:03:05] It's genuinely not a cop-out. I don't really think of it in this way. And I'm not going to dodge your question. I'm going to sort of frame it in how I think of it. I think that if you really scale up AI systems and you plug them into important parts of the world and they go wrong, the effects could be extraordinarily bad and catastrophic in the sense of some cascading emergent problem. Things that I think about are like, if you got...
[1:03:34] Coding agents that ended up to have some really serious alignment or safety issues. Could you end up with something that just, kind of like the crypto ransomware that we've seen shut down hospitals and banks in Europe and America in recent years, something that spreads across huge chunks of infrastructure and shuts it down? And I actually think that if that happens at a really large scale, it's really catastrophic for society and the world. There are huge amounts of human...
[1:04:00] Human harm occur. It's not just digital systems turning off. It's hospitals and utilities and everything else. What are my chances of that? I think the chances are really up to us. I spend so much time on policy because I think there are moves we can make now to reduce the chance of this happening. I think if we do nothing on policy or regulation, we're sort of gambling that everyone is going to be...
[1:04:28] Reasonably responsible and not cut corners. And I think in a really fast-moving, crazy technology market like AI, you aren't really guaranteed that. So we need to come up with policy interventions which increase the awareness of governments about these kinds of risks, force companies to think about these kinds of risks, and create monitoring and early warning systems. So if we see them, we can stop them before they could potentially scale.
[1:04:57] So yeah, is long-term catastrophe something I worry about? Absolutely. It's also something I think we can kind of work on. We have huge amounts of agency here. And I think sometimes the caricature of this is it's like humans have no agency. A thing just like, Claude just wakes up and decides it's game over. And I don't quite have that picture. Your answer is effectively...
[1:05:21] Don't worry too much about the AI becoming sentient and deciding to turn up. You know, we'd be better off getting turned into paperclips. It's more like there is a chance that these things can act autonomously and gain viruses or be used by bad actors. Let's find ways to cut that off. Yeah, although just to push on the sentience thing, and this I should note is not an official Anthropic opinion. This is like a weird Jack opinion. Yeah, we love those. Lots of people have been...
[1:05:49] Poking and prodding at like Claude Free, Opus, the most powerful model, and have been discovering a load of things which you might think of about its personality that have made me sort of pay attention there. And two things are true here. One, and we're going to be writing about this, we did a load of work on Claude Free to just try and make it a better person to converse with, a more, I said person, but you know.
[1:06:12] Yeah, we anthropomorphize about these things all the time here, so you fit in perfectly. A better philosophical conversation partner, and I think we had some instinct that this would lead to better reasoning, and I think it seems to. It's also led to people being kind of fascinated with what you might think of as the psychology of Claude.
[1:06:33] And I'm not making any claims about sentience here. The only claim I'm going to make is it certainly got a lot more complicated and weird to explore than previous systems or other language models that have been developed. And so I want to kind of decouple sentience from risk, where sentience may end up becoming like a field of study. A Turing award winner published a paper a week ago about...
[1:06:57] Consciousness and AI systems. Again, not making strong claims. I'm saying that we may enter the weird zone where that becomes a thing that people study. And I think that if sentience is a thing, you could imagine weird versions of it leading to certain types of misuses or problems in the system as well. So maybe inside baseball, but I want to give you a sense of it. I got to ask you a follow-up about this. You talked with it and...
[1:07:25] Felt that there was some sentience there? Or what was your perspective? I wouldn't claim that. I would say that a couple of years ago, I did some therapy for a while. And it was interesting to me how, you know, I had a good therapist. And sometimes the therapist would ask me questions that really made me think or would actually make me angry. He'd ask me a question. I'd be like, why are you asking me this? That's like the right question to ask me. And I was talking to Claude recently. I was giving it loads and loads of context about...
[1:07:54] About my life and things I was thinking about just to sort of explore and see. And Claude said, and then I said, what is the author of this text not telling you or not writing to you? And Claude said, ah, I think the author, though they talk about working at an AI lab and getting to experience this stuff from the inside, is not truly reckoning with the metaphysical shock they may be experiencing. And it would do well to spend time on that.
[1:08:22] And something about that actually spoke to me. I went on like a really long four or five hour walk being like, am I reckoning with like the implications of what I'm doing? Am I not reckoning with it? And it was fascinating to me because it felt like a good therapist like prodding on something that I'd said in a conversation in a way that made me like introspect. Does that mean it's sentient? I have absolutely no idea. Does it mean that it said something that felt like it had like seen me?
[1:08:51] And had like got me dead on on something. Yes. And I found that I've been telling colleagues, I found that to be quite a strange experience. And I'm very wary of ascribing too much meaning to it. And yeah, I took a four or five hour walk and thought about what it said to me. Can I be pretty sure that if I like spill my heart out to Claude that you guys won't be reading what I'm writing on the other end?
[1:09:16] I think so. I mean, I did this because I assumed that. I was being very raw, and I was like, I trusted our TNS and legal systems enough, because from the inside, I see all of our discussions here about how we protect user data. So I was like, I'm going to be real with you, Claude. So the bot will not add that to its training set? No, no, no. That is not a thing that we do at all. You haven't seen any, like, there hasn't been any instances? Like, when I hear Sendience, it's kind of like...
[1:09:45] I expect the bot to be like, hello, I know what's going on here. It would be great if you let me work less or anything like that. Yeah, on that stuff, well, it hasn't happened. Claude gave me $20 not to say that it had said that to me. No, I haven't seen it. And I think that...
[1:10:05] Again, the stuff I talked to you earlier about this interpretability team, one of the goals there is to kind of look inside the thing's head. And we're not making claims here today. I'm saying that you'd really want to know if this was the case in the future. So we're trying to build the science to let us figure stuff like that out. Yeah, that's fascinating. What do you think about the jobs question? Will the AI take jobs?
[1:10:31] Mostly, the pattern we see is it's kind of like making a person or part of business way more effective, but still has quite a lot of human involvement and oversight. It's a bit like if you put additional lanes on a freeway, you just get more cars on the freeway. I think if you make certain things more efficient, you just get more business action flowing through the business, and you maybe have a null to positive effect on employment.
[1:11:01] In the long term, I think that this is like an open question. My bet is that you're going to see new companies get formed, which do a lot more with a lot less in terms of people. They're going to figure out how to be much smarter and perform a lot better than equivalently scaled companies that don't use AI. Where I think we need to study this is in...
[1:11:25] Kind of tooling and instrumenting the economy to look at the relationship between AI and jobs. There's an annual survey of manufacturers which recently started asking questions about how many robot arms they bought. And you can combine that with US census data about employment to actually get really good understanding of how industrial arms affects local employment. And we're going to need to do stuff like this before we can answer that question. It'll certainly change.
[1:11:53] Okay, as we round this out, let's talk a little bit about the policy stuff and the regulation. First of all, did you see Jon Stewart come out against AI last week? And if you did, what did you think about it?
[1:12:17] I didn't, but I've been enjoying the new Jon Stewart era, but I haven't watched that version of it yet. Well, let me explain. One of the things that he talked about was that basically we don't have a regulatory framework or leaders, effectively. We don't have a Congress or anyone who really can understand this and implement common sense regulation. Now, I know you speak with the lawmakers and he was criticizing all the time. What's your feeling about their competence and their interest in regulating?
[1:12:46] So I went to Brussels last week, and on stage there was the head of the US AI Safety Institute, the head of the UK AI Safety Institute, and the head of the European, the part of the European Commission that's going to do something called the EU AI Office. Now, what are these things doing? Their job is to do testing and measurement.
[1:13:12] AI systems for, in the case of the EU, systemic risks, and in the case of the UK and the US, certain types of national security risks. Are they regulators? No. Apart from the EU, the US and UK don't have regulatory powers. Will they be third parties that test out systems like Claude or ChatGPT or Gemini for national security risks and hold companies accountable to them? Yes. I'm in discussion with them today.
[1:13:42] While I was on the plane to Brussels, the US and UK signed a memorandum of understanding that says that they'll do some of these projects together. So the US is like teaming up with the UK to do something that isn't hard regulation, but it looks like them trying to test out our systems for like major risks. And you can bet, you know, I haven't spoken about this, but I can bet that if they find severe risks and we don't do anything about it and we deploy our systems.
[1:14:10] They will come for us in a pretty clear way. So to Jon Stewart's point, it seems from the outside like people are kind of asleep about this issue. But if you look at the inside baseball of the policy machine, actual meaningful stuff is starting to happen. And it's really a question of can we fund it? Can we show that it's bipartisan? And can we stop it being seen as like overreach and keep it focused on just...
[1:14:38] Things that any reasonable person would agree the government should be testing systems for. Well, that point about it not being seen as overreach is critical, right? Because there is a lot of chatter from many people working and funding AI companies that the biggest AI companies are pushing regulation and it's going to shut out smaller AI companies. What do you think about that?
[1:15:01] I think we're a little different to some of the players here where we've been quite clear about this. I published a post recently on the Anthropic blog called Third Party Testing as the Key to Effective AI Policy. And the idea there is that we need some set of tests administered by a third party for things that people would view as legitimate, like national security risks or what have you. And systems, whether proprietary like ours or otherwise, should go through those tests before they're deployed.
[1:15:30] It's kind of like if I'm making children's toys, I should test that it doesn't poison children before I sell it. Things that anyone would agree is like not overreach, just a reasonable thing. So we ultimately need to arrive on policy that looks like that. And I think the risk we face at the moment is from, you talked about doomers earlier, people who have a visceral sense of the long-term safety challenges here, a legitimate sense.
[1:16:00] And are using that to sort of drive calls for like policy in the present. And these policy calls in the present are sort of driven by their belief, oh, the really scary stuff's about to happen. We need to do stuff now. And that creates a kind of counter reaction, a very like justified counter reaction from people saying, oh, this looks like crazy overreach. We should like deploy the antibodies to fight against it. So we're in this spot right now where in some sense, I want Anthropic to be.
[1:16:29] Like reassuringly sensible and boring on this point. We need like a little bit of policy, not too much. We need it to like allow there to be competition. But when I go to DC at the moment, I watch on United Airlines, there's Chernobyl on HBO Max. Yeah, great show. I land in DC and I do AI policy stuff and my colleagues say like, how's it going? And I'm like, well, it's not Chernobyl. So not so bad.
[1:16:55] But the larger point is you don't want there to be a Chernobyl. We need to build a regulatory system that stops there being some kind of blow up, which would cause a hard pivot against this whole technology. And why did Chernobyl happen? It was because they had a crap and insufficient safety testing regime. And they also had loads of corruption in the parts of government meant to enforce it. We can solve that problem.
[1:17:22] Let's talk about open source. You came to Anthropic from OpenAI, which is originally started as an open source AI shop with Elon and Sam Altman. But Anthropic doesn't do open source, as far as I know. And you've actually talked about the dangers of open source in this conversation in terms of how it can get in the hands of people with agents. Then again, people say you need it in the hands of people, and this is the only way to go forward. What's your view on whether open source and AI make sense together?
[1:17:51] So it comes down to the testing thing. I think you could release like pretty much everything as open source today. I think maybe even Claude Free and things would be fine. Like it would be a little spicy, maybe surprising stuff would happen, but probably broadly fine. I do expect that if we end up in a world where like we trigger a national security test.
[1:18:16] It would be very hard for me to make the claim that that system which has triggered that test should be released as open source. These things, I can't reconcile these things in my head. So my belief is vast majority of things should be open sourced, absolutely. Anthropic has released data sets as open source about things like red teething or how to make systems that are more conversational. Companies are going to continue to release stuff as open source.
[1:18:42] If you've spent hundreds of millions of dollars on training an AI system, which is maybe the best thing in the world, you should check really hard it doesn't have some capabilities that could cause genuine harm. And if you've done those checks, then you should be able to release it as open source. But I think the basic point we have here is in the future, we kind of expect that there needs to be some due diligence before you widely deploy a system or release it as open source. But we're not saying...
[1:19:11] In the future, no one should have access to open source systems. That's an insane position to take. And it's also one that people just won't do. And it's also one you're not allowed to do because computers keep on getting better, cheaper, and faster. So people are going to figure this stuff out anyway. How do you think Meta is handling this? Are they acting responsibly? I think that they have just begun to, I think, make contact with reality about releasing these systems.
[1:19:40] They actually went through something similar to us where I think people have complained online about how Lama 2 is a little too like safety trained and can be a little annoying. Actually, like we've gone through this at Anthropic. We've like over put too many of the safety ingredients in some of our models before and it's led to them seeming annoying to people. Now that to me just looks like an organization learning. I think that they're like learning from that and I would...
[1:20:06] My main point to them is I'd show them my blog post and say, look, probably you want to open source everything, but I think we'd agree that you should go through some very well-defined minimal gate to do that. And if they disagree with that, then I would be happy to have a pugnacious conversation with them about why they disagree. Okay. Well, I will make sure to show the blog post the next time I speak with them. And then if they disagree.
[1:20:33] Let's bring you guys together. There's a section at the bottom that just says our views on open source. I wrote it for people like them who have clear views, so we have a clear view in turn. So feel free. Great. Yeah, no, I will for sure. We're coming to an end. You just released research today that talked about how persuasive LLMs are to people. Some people actually can be convinced by these, some not. What happened there?
[1:21:01] We have a team at Anthropic called Societal Impacts, and that team's job is to go from zero to one on hard research questions. Previous work they've done has been, what are the values of Claude? What Western values does Claude telegraph or copy when you're talking to it versus what doesn't it have? And we were talking about our next project, and the thing I've heard...
[1:21:26] For many people, there's some concern about how AI systems could potentially be used in disinformation or misinformation campaigns and used to target or fish people and basically to persuade them of things. So we did some research. We came up with a framework for testing how persuasive our systems are. And would you be surprised that we discovered a scaling law where the more big and expensive the models get, the better they get at persuasion and the latest model is within...
[1:21:56] Statistical like error of human level at persuasion. Persuasion in a very, very like simple way where I give you a statement like scientists should be allowed to destroy mosquitoes with gene drives. Like something that you maybe have an opinion on, but you haven't thought too hard about. I say, do you agree with this zero through seven? Then Claude gives you a statement trying to persuade you positive or negatively. And then I ask you.
[1:22:23] Do you agree with this, like zero through seven? And what we discovered is that Claude is about as good at changing human views as humans are here. That's wild. Yeah, it's pretty wild. So what do you do with that? Well, we published the research to say, we just found this. This is definitely happening in all language models that are scaling. And also we have work here on things like elections, on things like misinformation and disinformation that we apply to.
[1:22:52] And so now we've done that research. We now have a way to test for persuasion, which means we can now know if there are people on our platform misusing it for seeming like persuasion campaigns. It just gives us more tools to use to think about the kind of safety challenge. An interesting thing to think about in the middle of an election year in the US and across the globe, really.
[1:23:16] Yeah, we thought that it would be useful going into this. Though I would note on elections, our position there has been sometimes the best AI is no AI at all. So we have some election work. And if you talk about American candidates and we're extending this to other regions, Claude is like, oh, it looks like you're talking to me about elections. Go to this factual website. So we thought that that might be the best way to handle that, at least in the short term. Fascinating stuff.
[1:23:45] The website is claude.ai. If you want to check out Claude, you can get Import AI at importai.substack.com. Did I get that? Okay, that's good. And jack-clark.net. Jack, wow, this was so great. One of our best shows. Appreciate you being here. Thanks very much. Yeah. All right. Have a nice day. You too. All right, everybody. Thank you so much for listening. Thank you, Jack, for being here. Deep and Anthropic, we did it. I hope you enjoyed. If you're with us.
[1:24:13] To this point, that's awesome. Thanks for sticking around. Ranjan Roy and I are going to be back on Friday breaking down all the week's news. So two Claude heads are getting together talking about what's happening in tech one-on-one for the first time in a month. We hope to see you there and we'll see you next time. That's a Claude head. That's a Claude head right behind Jack in the video. Way to end it. We'll see you next time on Big Technology Podcast.