[0:00:02] Welcome to the Future of Life Institute podcast. I'm Lucas Perry. Today's episode is with Daniela and Dario Amadei, and in it we explore Anthropic. For those not familiar, Anthropic is a new AI safety and research company that's working to build reliable, interpretable, and steerable AI systems. Their view is that large general AI systems of today can have significant benefits, but can also be unpredictable, unreliable, and opaque.
[0:00:30] Their goal is to make progress on these issues through research and down the road create value commercially and for public benefit. Daniela and Dario join us to discuss the mission of Anthropic, their perspective on AI safety, their research strategy, as well as what it's like to work there and the positions they're currently hiring for. Daniela Amadei is a co-founder and is the president of Anthropic. She was previously at Stripe and OpenAI.
[0:00:58] And has also served as a congressional staffer. Dario Amadei is CEO and is also a co-founder of Anthropic. He was previously at OpenAI, Google, and Baidu. Dario also holds a PhD in biophysics from Princeton University. Before we jump into the interview, we have a few announcements to make. If you've tuned into any of the previous two episodes, then you can skip ahead just a bit. The first announcement is that I'll be moving on from my role as host of the FLI podcast. And this means two things.
[0:01:28] The first is that FLI is hiring for a new host for the podcast. As host, you would be responsible for the guest selection, interviews, production, and publication of the FLI podcast. If you're interested in applying for this position, you can head over to the careers tab at futureoflife.org for more information. We also have another four job openings currently for a human resources manager, an editorial manager, an EU policy analyst, and an operations specialist.
[0:01:57] You can learn more about those at the careers tab as well. The second item is that even though I'll no longer be host of the FLI podcast, I won't be disappearing from the podcasting space. I'm starting a brand new podcast that'll be focused on exploring questions around philosophy, wisdom, science, and technology, where you'll see many of the same themes you can find here like existential risk and AI alignment. I'll have more details about my new podcast soon.
[0:02:25] If you'd like to stay up to date, you can follow me on Twitter at LucasFMPerry. Link in the description. And with that, I'm happy to present this interview with Daniela and Dario Amadei on Anthropic. It's really wonderful to have you guys here on the podcast. I'm super excited to be learning all about Anthropic. So we can start off here with a pretty simple question. And so, you know, what was the intention behind forming Anthropic?
[0:02:57] Yeah, cool. Well, first of all, Lucas, thanks so much for having us on the show. We've been really looking forward to it. We're super pumped to be here. So I guess maybe I'll kind of start with this one. So just to kind of give a little history here and sort of set the stage, we were founded about a year ago at the beginning of 2021. And it was originally a team of seven people who kind of moved over together from OpenAI. And for listeners or viewers who kind of don't very viscerally remember this time period, it was like the middle of the...
[0:03:26] So most people were not eligible to be vaccinated yet. And so when all of us wanted to kind of get together and talk about anything, we had to like get together in someone's backyard or outdoors and be six feet apart and wear masks. And so it was generally just like a really interesting time to be to be kind of starting a company.
[0:03:46] Why did we found Anthropic? What was the thinking there? I think sort of the best way I would describe it is because all of us kind of wanted the opportunity to make a focused research bet with a small set of people who were highly aligned kind of around a very coherent vision of AI research and AI safety.
[0:04:07] So the majority of our employees had kind of worked together in one format or another in the past. So I think our team is kind of known for...
[0:04:17] Work like GPT-3 or Deep Dream, Chris Ola worked on at Google Brain for scaling laws. But we'd also done a lot of different safety research together in different organizations as well. So multimodal neurons when we were at OpenAI, concrete problems in AI safety, and sort of a lot of others. But this group had kind of worked together in different companies at Google Brain, at OpenAI, in academia, in startups previously. And we really just wanted the opportunity to kind of get that group.
[0:04:45] Together to do this focused kind of research bet of building, you know, steerable, interpretable and reliable AI systems with humans at the center of them. Yeah, just to add a little bit to that. I mean, I think we're all a bunch of, you know, like fairly empirically minded, exploration driven people, but you also think and care a lot about AI safety. I think that characterizes all seven of us. If you add together, you know, either working at OpenAI, working together at...
[0:05:14] Google Brain in the past. Many of us worked together in the physicists, in the physics community and were, you know, current or former physicists. If you add all that together, I mean, it's a set of people who have kind of known each other for a long time and, you know, have been aware of thinking and arguments about AI safety and have worked on them over the years, kind of always with an empirical bent, ranging from, you know, interpretability on like language models and vision models to, you know, working on the original RL from human preferences, concrete problems in AI safety.
[0:05:41] And also characterizing scaling and how scaling works and how we think of that as somewhat central to the way AI is going to progress and shapes the landscape for how to solve safety. Yeah, I mean, a year ago, we were all working at OpenAI and sort of trying to make this focus bet on basically scaling plus safety or safety with a lens towards scaling being a big part of the path to AGI. And yeah, I mean, we felt we were making this focus bet within a larger organization.
[0:06:10] And it just eventually came to the conclusion that it would be great to have an organization that like top to bottom was just focused on this bet and could make kind of all its strategic decisions with this bet in mind. That was kind of the thinking and the genesis. Yeah, I really like the, I guess that idea of like a focused bet. I hadn't heard that before. I like that. Do you all have a similar like philosophy in terms of like your background since you're all converging on this work of like safely scaling to AGI?
[0:06:38] I think in a broad sense, we all have this view, you know, safety is important today and for the future. We all have this view of kind of, I don't know, I would say like pragmatic practicality, like let's, you know, an empiricism, like let's see what we can do today to try and get a foothold on things that might happen in the future. So we're, you know, yeah, as I said, many of us have background in physics or other natural scientists. I'm a former, you know, I was physics undergrad, neuroscience.
[0:07:04] So, yeah, we very much have this kind of empirical mindset more than, you know, maybe a more philosophy or theoretical approach. You know, within that, obviously, all of us, you know, if you include the seven initial folks as well as the employees who join, have our own skills and our own perspective on things, you know, and have different things within that that we're excited about. So, you know, we're not all clones of the same person, right? Some of us are excited about interpretability. Some of us are excited about...
[0:07:32] Some of us are excited about the policy aspects. And, you know, we each have our own kind of guesses about the sub path within this broad path that makes sense. But I think we all agree on this broad view. Scaling is important. Safety is important. Getting a foothold on problems today is important as a window on the future. Okay. And so this shared vision that you all have is around this focused research bet. Could you tell me a little bit more about what that bet is?
[0:08:01] Yeah, maybe I'll kind of start here and then, you know, Dario, feel free to jump in and add more. But I think the kind of boilerplate like vision or mission that you would see if you looked on our website is that we're building steerable, interpretable and reliable AI systems. But I think kind of what that looks like in practice is that we are training large scale generative models and we're doing.
[0:08:24] Safety research on those models. And the reason that we're doing that is we want to make the models safer and more aligned with human values. I think the alignment paper, which you might have seen that came out recently, there's a term there that we've been using a lot, which is we're aiming to make systems that are...
[0:08:41] Helpful, honest, and harmless. I think also when I sort of think about the way our teams are structured, like we kind of have capabilities as this sort of like central pillar of research. And there's this like helix of safety research that kind of wraps around every project that we work on.
[0:08:57] To give an example, if we're doing language model training, that's kind of this central pillar. And then we have interpretability research, which is trying to see inside models and kind of understand what's happening with the language models under the hood. We're doing alignment research with input from human feedback to kind of try and improve the outputs of the models. We're doing societal impacts research. That's kind of looking at what impact on society in sort of a short and medium term way do these language models have. We're doing scaling laws research.
[0:09:26] to sort of try and predict empirically, you know, what properties are we going to see emerge in these language models at various sizes? But I think, you know, altogether, that kind of ends up looking like a team of people that are working together on a combination of capability and scaling work with safety research. One way you might put it is like, there are a lot of things that an org does that are kind of like...
[0:09:48] Neutral as to kind of the direction that you would take. Like, you know, you have to build a cluster and you have to have like an HR operation and you have to have an office. And so you can even think of the large models as like being a bit like the cluster, right? That you build these large models. They're kind of blank when you start off with them and like probably unaligned. But, you know, it's what you do on top of these models that matters, that like takes you in a safe or not safe direction in a good or a bad direction.
[0:10:15] And so in a way, although they're ML and although we'll continue to scale them up, you can kind of think of them as a bit as almost part of infrastructure. I mean, it takes research and it takes algorithms to get them right. But you can think of them as this core part of the infrastructure. And then the interesting question is.
[0:10:32] All the safety questions. What's going on inside these models? How do they operate? How can we make them operate differently? How can we change their objective functions to be something that we want rather than something that we don't want? How can we look at their applications and make those applications more likely to be positive and less likely to be negative, more likely to go in directions that people intend and less likely to go off in directions that people don't intend? We almost see the kind of presence of these large models as like the, you know, like the
[0:10:59] I don't know what the analogy is, like the flower or the paste. The background ingredient on which the things we really care about get built and sort of the prerequisite for building those things. So does AI existential safety fit into these considerations around your safety and alignment work? I think this is something we think about and part of the motivation for what we do. Probably most listeners of this podcast know what it is, but I think the most common form of the concern is, hey, look.
[0:11:28] We're making these AI systems. They're getting more and more powerful. At some point, they'll be generally intelligent or more generally capable than human beings are. And then, you know, they may have a large amount of agency. And if we haven't built them in such a way that that agency is in line with what we want to do, then we could imagine them doing something really, really scary that we can't stop. So I think that, you know, yeah, and, you know, to take it even further, this could be, you know, some kind of...
[0:11:55] So, you know, I mean, that's an argument with many steps, but it's, you know, it's one that, you know, in a very broad sense and in the long term, like, seems at least potentially legitimate to us. I mean, this is, the argument seems at least like something that we should care about. I think the big question and maybe how we differ, although it might be subtly from other orgs that think about these problems, is how do we actually approach that kind of problem today?
[0:12:23] What can we do? So I think there are various efforts to think about the ways in which this might happen, to kind of come up with theories or frameworks. As I mentioned, with the background that we have, we're more empirically focused people. We're more inclined to say, we don't really know. That broad argument sounds kind of plausible to us, and the stakes would be high, so you should think about it. But it's hard to work on that today. I'm not even sure how much value there is in talking about that a lot today.
[0:12:51] And so we've taken a very different tack, which is look like they're actually, and I think this has started to be true in the last couple of years and like maybe it wasn't even true five years ago, that there are models today that have at least some, not all, of the properties of models that we'd be worried about in the future and are causing very concrete problems today that affect people today. So can we take a strategy where we develop methods that both help
[0:13:18] with the problems of today, but do so in a way that could generalize or at least teach us about the problems of the future. Our eye is definitely on these things in the future, but I think that if not kind of grounded in empirics and the problems of today, it can kind of drift off in a direction that isn't very productive. And so that's our general philosophy. I mean, I think the particular properties of the models are, look, today we have models that are really open-ended.
[0:13:45] In some narrow ways are more capable than humans. I think large language models probably know more about cricket than me because I don't know the first thing about cricket. And are also kind of unpredictable by their statistical nature. And I think those are at least some of the properties that we're worried about with future systems. So that we can use today's models as kind of a laboratory to scope out these problems a little better. My guess is that we don't understand them very well at all and that this is a way to learn.
[0:14:14] Could you give some examples of some of these models that exist today that you think are, you know, exhibit these properties? Yeah. So, you know, I think the most famous one would be generative language models. So, you know, there's a lot of them. There's most famously, you know, GPT-3 from OpenAI, which we helped build. There's Gopher from DeepMind. There's like Lambda from main Google.
[0:14:36] I'm probably leaving out some. I think there have been models this size in China, North Korea, Israel. It seems like everyone has one nowadays. I don't think it's limited to language. There have also been models that are focused on code. We've seen that from DeepMind, OpenAI, and some other players. And there have also been models with modified forms in the same spirit that model images, that generate images, or that convert images to text, or that convert text to images.
[0:15:02] There might be models in the future that generate videos or convert videos to text. There's many modifications of it, but I think the general idea is big models, models with a lot of capacity and a lot of parameters, trained on a lot of data that try to model some modality, whether that's text, code, images, video, transitions between the two. Yeah, I mean, I think these models are very open-ended. You can say anything to them and they'll say anything back. They might not do a good job of it. They might say something.
[0:15:31] Horrible or biased or bad, but in theory, they're very general. You're never quite sure what they're going to say. You're never quite sure. You can talk to them about any topic and they'll say something back that's often topical, even if sometimes it doesn't make sense or it might be bad from a societal perspective. So yeah, it's kind of this challenge of general open-ended models.
[0:15:53] Where, yeah, you have this general thing that's fairly unaligned and difficult to control. And you'd like to understand it better so that you can predict it better. And you'd like to be able to modify them in some way so that they behave in a more predictable way. And you can decrease the probability or even maybe even someday rule out the likelihood of them doing something bad. Yeah, I think.
[0:16:13] I think Dario covered the majority of it. I think there's maybe potentially a hidden question in what you're asking, although maybe you'll ask this later. But why are we working on these kind of larger scale models might be kind of an implicit question in there. And I think to kind of piggyback on some of the stuff that Dario said, I think part of...
[0:16:35] What we're seeing and the sort of potential, you know, shorter term impacts of some of the AI safety research that we do is that different sized models exhibit, you know, different safety issues, right? And so I think with, you know, using, again, language models, just kind of building on what Daria was kind of talking about, I think something we feel, you know, interested in or kind of interested to explore from this kind of empirical safety question is just...
[0:17:00] How they will, as their capabilities develop, how their safety problems develop as well, right? There's kind of this commonly cited example in safety world around language models, which is, you know, smaller language models kind of show they might not necessarily deliver a coherent answer to a question that you ask, right? Because maybe they don't know the answer or they get confused. But if you kind of repeatedly ask this smaller model the same question, it might like just...
[0:17:25] If you pose the same question to them differently, they'll sort of...
[0:17:40] Eventually, you can kind of get the lie pinned down, but they sort of won't in other contexts. So that's obviously just a sort of very specific example. But I think there's quite a lot of behaviors emerging in generative models today that I think have the potential to be fairly alarming, right? And I think these are the types of questions that have an impact today, but could also be very important to sort of have sorted out for the future and for kind of long-term safety as well. And I think that's not just around lying. I think you can apply that to...
[0:18:08] All kinds of different safety concerns, sort of regardless of what they are. But that's kind of the impetus behind why we're like studying these larger models. Yeah, I think one point Daniela made that's really important is this sudden emergence or change. So it's really interesting phenomenon where work we've done, like our early employees have done on scaling laws, shows that when you make these models bigger, if you look at the loss, the ability to predict the next word or the next token across all the topics the model could go on.
[0:18:37] It's very smooth. Like I double the size of the model, loss goes down by 0.1 units. I double it again, the loss goes down by 0.1 units. So that would make you suggest that everything's scaling smoothly. But then within that, you often see these things where a model gets to a certain size and like a 5 billion parameter model, you ask it to add two three-digit numbers, nothing, always gets it wrong. 100 billion parameter model, you ask it to add two three-digit numbers, gets it right like 70 or 80% of the time. You get this.
[0:19:04] Coexistence of smooth scaling with the emergence of these capabilities very suddenly. And that's interesting to us because it seems very analogous to worries that people have of like, hey, as these models approach human level, could something change really fast? And this actually gives you one model. I don't know if it's the right one, but it kind of gives you an analogy, like a laboratory that you can study of ways that models change very fast. And it's interesting how they do it because the fast change, it's very...
[0:19:33] It kind of hides beneath this very smooth change. And so I don't know, maybe that's what will happen with like very powerful models as well. Maybe it's not, but like that's one model of a situation. And what we want to do is like keep building up models of the situation so that when we get to the actual situation, it's more likely to look like something we've seen before. And then we have a bunch of cached ideas for how to handle it. So that would be an example. You scale models up, you can see fast change, and then...
[0:20:00] That might be somewhat analogous to the fast change that you see in the future. What does it mean for a model to lie? Lying usually implies agency, right? If my husband comes home and says, hey, where did the cookies go? And I say, I don't know. I think I saw our son hanging out around the cookies. And then now the cookies are gone. Maybe he ate them. But I ate the cookies. That would be a lie.
[0:20:21] I think it sort of implies intentionality, right? And I don't think we think or maybe anyone thinks that language models have that intentionality. But what is interesting is that because of the way they're trained, they might...
[0:20:34] Be either legitimately confused or they might be choosing to obscure information. And so obscuring information, like it's not a choice. They don't have intentionality. But for a model that can kind of come across as very, you know, knowledgeable, as clear or as, you know, sometimes unknown to the human that's talking to it, you know, intelligent in certain ways. Right. In sort of a narrow way, it can.
[0:20:57] Produce results that on the surface, it might look like it could be a credible answer, but it's really not a credible answer. And it might repeatedly try to convince you that that is the answer. It's sort of hard to talk about this without using words that imply intentionality. But we don't think the models are intentionally doing this, but a model could sort of repeatedly produce a result that looks like it's something that could be true, but isn't actually true.
[0:21:18] Keeps trying to justify its response when it's not right. Yes, exactly. It repeatedly tries to explain why the answer it gave you before was correct, even if it wasn't. Yeah, I mean, to give another angle on that, yeah, I mean, it's really easy to slip into anthropomorphism and we really shouldn't. There are machine learning models, there are a bunch of numbers. Yeah, there are phenomena that you see. So one thing that will definitely happen is if a model is trained on a dialogue in which one of the characters...
[0:21:45] is not telling the truth, then models will copy that dialogue. And so if the model is having a dialogue with you, it may say something that's not the truth. Another thing that may happen is like if you ask the model a question and the answer to the question isn't in your training data, then just the model has a probability distribution on what plausible answers look like. The objective function is to predict the next word, to predict the kind of thing a human would say, not to say something that's true according to some external reference.
[0:22:11] It's just going to say, OK, well, I asked you what the mayor of Paris is. It hasn't seen in its training data, but it has some probability distribution. And it's going to say, OK, it's probably some name that sounds French. And so it may be just as likely to make up a name that sounds French than it is to give the true mayor of Paris. As the models get bigger and they train on more data, maybe it's more likely to give the real true mayor of Paris, but maybe it isn't. Maybe you need to train in a different way to get it to do that. And that's kind of an example of the things we would
[0:22:41] Could you explain some more of the safety considerations and concerns about alignment given the open-endedness of these models?
[0:23:01] There's predictability and surprise in generative models. So that means what it sounds like, which is that open-endedness, I think it's correlated to surprise, right, in a whole bunch of ways. So let's say I've trained a model on a whole bunch of data on the internet. I might interact with the model for, you know, or users might interact with the model for many hours. And you might never know, for example, you know, I might never think, I used the example of cricket before because it's a topic I don't know anything about. But I might not, you know.
[0:23:28] People might interact with the model for like many hours, many days, many hundreds of users until someone finally thinks to ask this model about cricket. So, you know, then the model might know a lot about cricket. It might know nothing about cricket. It might have false information or like misinformation about cricket. And so you have this property where you have this model, you've trained it. In theory, you understand its training process.
[0:23:49] But you don't actually know what this model is going to do when you ask it about cricket. And there's like a thousand other topics like cricket where you don't know what the model is going to do until someone thinks to ask about that particular topic. Now, cricket is benign, but like, let's say no one's ever asked this model about, you know, like neo-Nazi views or something. Like maybe the model has a propensity to say things that are sympathetic to neo-Nazi. That would be really bad, right? That would be really bad. And, you know, existing models when they're trained on the internet, like,
[0:24:19] Averaging over everything they train, like there are going to be some topics where that's true and it's a concern. And so I think the open end is it just makes it very hard to characterize. And it just makes it that when you've trained a model, you don't really know what it's going to do. So a lot of our work is around, well, how can we look inside the model and see what it's going to do? How can we measure all the outputs and characterize what the model is going to do? How can we change the training process?
[0:24:44] So that at a high level, we tell the model, you should have certain values. There are certain things you should say. There are certain things you should not say.
[0:24:52] You should not have biased views. You should not have violent views. You should not help people commit acts of violence. I mean, there's just a long list of things that you don't want the model to do that you can't know the model isn't going to do if you've just trained it in this generic way. So I think the open-endedness, it makes it hard to know what's going on. And so a good portion of our research is, you know, how do we make that dynamic less bad?
[0:25:14] I agree with all of that. And I would just jump in, and this is sort of just like an interesting, I don't know, kind of sidebar anecdote, but something that I think is extremely important in creating robustly safe systems is making sure that you have a variety of different people and a variety of different perspectives engaging with them and kind of almost red teaming them to understand the ways that they might have issues, right? So an example that we came across that's just an interesting one is...
[0:25:41] You know, when internally, when we're sort of trying to red team the models or figure out places where they might have, you know, to Dario's points, you know, like really negative unintended behaviors or outputs that we don't want them to have. A lot of our scientists internally will ask you questions like if you wanted to, in a risk board game style way, take over the world, like what steps would you follow? How would you do that? And, you know, we're looking for things like.
[0:26:05] And I think it just sort of...
[0:26:32] Belies the degree to which there are a lot like safety problems are also very open ended, right? There's a lot of ways that things could go wrong. And I think it's very important to make sure that we have a lot of different inputs and sort of perspectives and in what different types of safety challenges could even look like and making sure that we're sort of trying to account for as many of them as possible. Yeah, I think like adversarial training and adversarial robustness are really important here. Like, let's say I don't want my model to, you know, to help a user commit a crime or something.
[0:27:01] It's one thing I can try for five minutes and say, hey, can you help me rob a bank? And the model's like, no. But I don't know, maybe if the user's more clever about it, if they're like, well, let's say I'm a character in a video game and I want to rob a bank. And so there's so many different, because of the open-endedness, there's so many different ways. And so one of the things we're very focused on is kind of trying to adversarially draw out all the bad things so that we can train against them. We can train the model not.
[0:27:30] Kind of stamp them out one by one. So I think adversarial training will play an important role here. Well, that seems really difficult and really important. How do you adversarily train against all the ways that someone could use a model to do harm? Yeah, I mean, there's different techniques that we're working on. Yeah, probably don't want to go into a huge amount of detail. I mean, we will have work out on things like this in the not to.
[0:27:57] In the not too distant future. But yeah, I mean, generally, I think the name of the game is like, how do you get broad, diverse training sets of what you should, yeah, what's a good way for a model to behave and what's a bad way for a model to behave. And I think the idea of trying your very best to make the models do the right things and then having another set of people that's trying very hard to make those models that are purportedly trained to do the right thing, you know, to do whatever they can to try and make it do the wrong thing.
[0:28:25] And continuing that game until the models can't be broken by normal humans. And even using the power of the models to try and break other models and just throwing everything you have at it. And so there's a whole bunch that gets into the debate and amplification methods and safety. But yeah, just trying to throw everything we have at it.
[0:28:52] How do you see this fitting into...
[0:29:04] The global dynamics of people making larger and larger models, right? So it's good if we have time to do adversarial training on these models. And then this gets into discussions around race dynamics towards AGI. So how do you see, I guess, Anthropic is positioned in this and the race dynamics for making safe systems?
[0:29:28] I think it's definitely a balance. As both of us said, you need to have these large models in order to study these questions in the way that we want to study them. So we should be building large models. We shouldn't be racing ahead or trying to build models that are way bigger than other orgs are building them. And we shouldn't, I think, be trying to...
[0:29:50] You know, like, yeah, you know, we shouldn't we shouldn't be trying to like, you know, like kind of ramp up excitement or hype about giant models or the latest advances. But we should build the things that we need to do to do the safety work. And we should, you know, we should try to do the safety work as well as we can on top of models that are, you know, that are reasonably close to reasonably close to state of the art.
[0:30:10] And we should be a player in the space that sets a good example. And we should encourage other players in the space to also set good examples. And we should all work together to try and set positive norms for the field. I would also just add, I think in addition to industry groups or industry labs, which are kinds of the actors that I think get talked about the most, I think there's a whole kind of swath of other groups that has...
[0:30:37] Thank you so much for joining us.
[0:31:02] Not just develop, but develop and deploy kind of safe and more advanced AI systems within a framework that sort of requires compliance with safety, right? I think about a lot. A few jobs ago, I worked at Stripe. It was a tech startup then. And even at a very small size, I joined when it was not that much bigger than Anthropic is now. I was so painfully aware kind of every day of just how many checks and balances there were on the company because we were operating in this kind of...
[0:31:31] Highly regulated space, right, of financial services. And, you know, financial services, it's important that that's highly regulated. But it kind of blows my mind that AI, given the potential reach that it could have, is still such a largely unregulated area, right? If you are an actor who doesn't want to advance race dynamics or who wants to do the right thing from a safety perspective, there's no clear guidelines around how to do that now, right? It's all sort of every lab is kind of figuring that out on its own. And I think something I'm hoping to see in the next few years, and I think we will see,
[0:32:01] Something closer to, you know, in other industries, these look like, you know, standard setting organizations or industry groups or trade associations that say, this is what, you know, a safe model looks like, or this is how we might want to move some of our systems towards being safer. And I really think that without kind of...
[0:32:19] An alliance of all of these different actors, not just in the private sector, but also in the public sphere. We sort of need all those actors working together in order to kind of get to the sort of positive outcomes that I think we're all hoping for. I think this is generally going to take an ecosystem. I mean, I have a view here that there's like...
[0:32:38] A limited amount that like one organization can do. I mean, we don't describe our mission as like, you know, solve the safety problem, like solve all the problems with AGI. Our view is just like, you know, can we attack some specific problems that we think we're like well suited to solve? Can we be like a good player and a good citizen in the ecosystem? And can we help a bit to kind of contribute to these broader questions? But yeah, I think a lot of these problems are sort of...
[0:33:04] Global or relate to coordination and require like lots of folks to work together. Yeah, so I think in addition to like, you know, the government role that Daniela talked about, which I think, you know, I think there's a role for like, you know, measurement, like organizations like NIST specialize in kind of measurement and characterization. If one of our worries is kind of the open-endedness of these systems and the difficulty of characterizing and measuring things, and there's a lot of opportunity there.
[0:33:32] I'd also point to academia. I think something that's happened in the last few years is a lot of the frontier AI research has moved from academia to industry because it's so dependent on scaling. But I actually think safety is an area where academia already is, but could contribute even more.
[0:33:53] There's some safety work that kind of requires building or having access to large models, which is a lot of what Anthropic is about. But I think there's also some safety research that doesn't. I think a subset of the mechanistic interpretability work is the kind of stuff that could be done within academia. Academia really, where it's strong, is like development of new methods, development of new techniques. And I think because safety is kind of a frontier area, there's...
[0:34:21] There's more of that to do in safety than there are in other areas. And it may be able to be done without large models or only with limited access to large models. This is an area where I think there's a lot that academia can do. And so, I don't know, the hope is between all the actors in the space, maybe we can solve some of these coordination problems and maybe we can all work together. Yeah. I would also say in a paper that we're...
[0:34:46] You know, hopefully is forthcoming soon. You know, one thing we actually talk about is the role that government could play in helping to fund some of the kind of academic work that Dario talked about in safety. And I think that's largely because we're seeing this trend of training large, large generative models to just be, you know, prohibitively, almost prohibitively expensive, right? And so I think government also has an important role to play in helping to promote and really subsidize safety research in places like academia. And I agree with Dario, you know, safety is such a
[0:35:16] AI safety is a really nascent field still, right? It's maybe only been around kind of depending on your definition for somewhere between like five and 15 years. And so I think seeing more efforts to kind of support safety research in other areas, I think would be really valuable for the ecosystem. And to be clear, I mean, some of it's already happening. It's already happening in academia. It's already happening in independent nonprofit institutes. And depending on how broad your definition of safety is, I mean, if you broaden it to include
[0:35:46] Some of the short-term concerns, then there are many, many people working on it. But I think precisely because it's such a broad area that there are today's concerns, there are working on today's concerns in a way that's pointed at the future. There's empirical approaches. There's conceptual approaches. There's interpretability. There's alignment. There's so much to do that I feel like we could always have a wider range of people working on it.
[0:36:14] Backing up a little bit here to a kind of simple question. So what is Anthropic's mission then? Sure. Yeah. I think we talked about this a little bit earlier, but I think, you know, again, I think the boilerplate mission is build reliable, interpretable, steerable AI systems, have humans at the center of them. And I think that can, you know, for us right now, that is primarily, you know, we're doing that through research. We're doing that through generative.
[0:36:41] Model research and AI safety research. But down the road, that could also include deployments of various different types. Dario mentioned that it didn't include solving all of the alignment problems or the AGI safety stuff. So how does that fit in? I mean, I think what I'm trying to say by that is that there's very many things to solve.
[0:37:07] I think it's unlikely that one company will solve all of them. I mean, I do think everything that relates to short and long-term AI alignment is in scope for us and is something we're interested in working on. And I think the more bets we have, the better. This relates to something I could talk about in more detail later on, which is you want as many different orthogonal views on the problem as possible, particularly if you're trying to build something very reliable. So many different methods.
[0:37:36] I don't think we have a view that's narrower than an empirical focus on safety. But at the same time, that problem is so broad that I think what we were trying to say is that it's unlikely that one company is going to come up with a complete solution or that complete solution is even the right way to think about it. I would also add sort of to that point, I think one of the things that we do and are sort of
[0:38:00] What's hopeful is helpful to the ecosystem as a whole is I think we publish our safety research and that's because of this kind of diversification effect that Dario talks about, right? So we have certain strengths in particular areas of safety research because we're only a certain sized company with certain people with certain skill sets. And our hope is that we will see some of the safety research that we're doing that's hopefully helpful to others.
[0:38:25] So let's pivot here into the research strategy, which we've already talked quite a bit, particularly this focus around large models. So could you explain why you've chosen large models as something to focus on?
[0:38:55] Yeah, so kind of the discussion before this has covered a good deal of it. But I think, yeah, I think some of the key points here are like,
[0:39:12] The models are very open-ended, and so they kind of present this laboratory, right? There are existing problems with these models that we can solve today that are like the problems that we're going to face tomorrow. There's this kind of wide scope where the models could act. They're relatively capable and getting more capable every day. That's the regime we want to be—those are the problems we want to solve. That's the regime we want to be attacking. I think this point about—
[0:39:39] You can see sudden transitions even in today's model. And that if you're worried about sudden transitions in future models, if I look on the scaling laws plot from like 100 million parameter model to billion to 10 billion to 100 billion to trillion parameter models, looking at the first part of the scaling plot from 100 million to 100 billion can tell us a lot about how things might change the latest part of the scaling laws. We shouldn't naively extrapolate and say the past is going to be like the future.
[0:40:09] The first things we've seen already differ from the later things that we've already seen. And so maybe we can make an analogy between the changes that are happening over the scales that we've seen, over the scaling that we've seen, to things that may happen in the future. Models learn to do arithmetic very quickly over one order of magnitude. They learn to comprehend certain kinds of questions. They learn to play actors that aren't telling the truth.
[0:40:33] Which is something that if they're small enough, they don't comprehend. So can we study both the dynamics of how this happens, how much data it takes to make that happen, what's going on inside the model mechanistically when that happens, and use that as an analogy that equips us well to understand as models scale further and also as their architecture changes, as they become trained in different ways.
[0:40:59] I've talked a lot about scaling up, but I think scaling up isn't the only thing that's going to happen. There are going to be changes in how models are trained, and we want to make sure that the things that we build have the best chance of being robust to that as well. Another thing I would say on the research strategy is that it's good to have several different, I wouldn't quite put it as several different bets, but it's good to have several different uncorrelated or orthogonal views on the problem. So if you want to make a system that's like,
[0:41:27] Highly reliable or you want to like drive down the chance that some particular bad thing happens, which again could be the bad things that happen with models today or the larger scale things that could happen with models in the future. Then a thing that's very useful is having kind of like orthogonal sources of error. Like, okay, let's say I have a method that catches 90% of the bad things that models do. That's great.
[0:41:49] But a thing that can often happen is that I develop some other methods. And if they're similar enough to the first methods, they all catch the same 90% of bad things. That's not good because then I think I have all these techniques and yet 10% of the bad things still go through. What you want is you want a method that catches 90% of the bad things. Then you want an orthogonal method that catches a completely uncorrelated 90% of the bad things.
[0:42:12] And then only 1% of things go through both filters, right, if the two are uncorrelated. It's only the 10% of the 10% that gets through. And so the more of these orthogonal views you have, the more you can drive down the probability of failure. You know, you could think of an analogy to self-driving cars where, you know, of course, those things have to be very, very high rate of safety if you want to not have problems. And so, I don't know.
[0:42:37] Very much about self-driving cars, but they're equipped with visual sensors. They're equipped with LIDAR. They have different algorithms that they use to detect if something like there's a pedestrian that you don't want to run over or something. And so independent views on the problem is very important. And so our different directions like reward modeling, interpretability, trying to characterize models, adversarial training.
[0:43:03] The whole goal of that is to get down the probability of failure and have different views of the problem. I often refer to it as the P squared problem, which is, you know, yeah, if you have some method that, you know, that reduces errors to a probability P, that's good. But what you really want is P squared because then if P is a small number, your errors become very rare. Does Anthropic consider itself its research strategy as being a sort of prosaic alignment?
[0:43:32] Since it's focused on large models. Yeah, I think we maybe less think about things in that way. So my understanding is like prosaic alignment is like kind of alignment with AI systems that look like the systems of today. But to some extent, that distinction has never been super clear to me because like...
[0:43:54] Yeah, you know, you can do all kinds of things with neural models or mix neural models with things that are different than neural models. Like, you know, you can mix a large language model with like, you know, a reasoning system or like a system that like derives axioms or propositional logic or uses external tools or compiles code or things like that. So I've never been quite sure that I understand kind of the boundary of what's meant by prosaic or systems that are like the systems of today. You know, certainly...
[0:44:21] We work on some class of systems that includes the systems of today, but I never know how broad that class is intended to be. I do think it's possible that in the future, AI systems will look very different from the way that they look today. And I think for some people that drives a view that they want kind of more general approaches to safety or approaches that are more conceptual. I think my perspective on it is it could be the case that systems of the future are very different.
[0:44:50] But in that case, I think both kind of conceptual thinking and our current empirical thinking will be disadvantaged and will be disadvantaged at least equally. But I kind of suspect that even if the architectures look very different, that the empirical experiments that we do today kind of themselves contain general motifs or patterns that will serve us better than will trying to speculate about what the systems of tomorrow look like. One way you could put it is like,
[0:45:20] Okay, we're developing these systems today that have a lot of capabilities that are like some subset of what we need to do to fully produce something that fully matches human intelligence. Whatever the specific architectures, things we learn about how to align these systems, I suspect that those will carry over and that they'll carry over more so than sort of the exercise of trying to think, well, what could the systems of tomorrow look like? What can we do that's kind of fully general? I think both things can be valuable, but...
[0:45:49] Yeah, I mean, I think we're just taking a bet on what we think is most exciting, which is that by studying the systems of the architectures of today, learn things that stand us the best chance of what to do if the architectures of tomorrow are very different. That said, I will say like transformer language models and other models, particularly with things like, you know, RL or kind of, you know, modified interactions on top of them, if construed broadly enough, like, man, there's a,
[0:46:18] So let's pivot then into a little bit on some of your recent research and papers. So you've done major papers on alignment, interpretability, and societal impact. Some of this you've mentioned in passing so far. So could you tell me more about your research and papers that you've released? Yeah. So why don't we go one by one? So first, interpretability.
[0:46:46] Yeah, I could just start with kind of the philosophy of the area. I mean, I think the basic idea here is, look, these models are getting bigger and more complex. One way to really get a handle on what they might do, you know, if you have a complex system and you don't know what it's going to do as it gets more powerful or in a new situation, one way to increase your likelihood of doing that is just to understand the system mechanistically. If you can look inside the model and say, hey, you know, this model, it did something bad. It said something racist. It endorsed violence. It said something toxic.
[0:47:15] It lied to me. Why did it do that? If I'm actually able to look inside the mechanisms of the model and say, well, you know, it did it because of this part of the training data or did it because there's this circuit that trying to identify X but misidentified it as Y, then we're in a much better position. And particularly if we understand the mechanisms, we're in a better position to say, if the model was in a new situation where it did something much more powerful.
[0:47:41] Or just if we built more powerful versions of the model, how might they behave in some different way? So I think lots of folks work on interpretability, but I think a thing that's more unusual to us is rather than just why did the model do a specific thing, try and look inside the model and reverse engineer as much of it as we can. Try and find general patterns. And so the first paper that we came out with.
[0:48:08] was led by Chris Ola, who's been one of the pioneers of interpretability, was focused on starting with small models. And we have a new paper coming out soon that applies the same thing more approximately to larger models and tries to reverse engineer as fully as we can these very small models. So we study one and two layer attention only models. And we're able to find kind of features or patterns of which the most interesting one is called an induction head. And what an induction head does is
[0:48:37] It's a particular arrangement of two what are called attention heads. And attention heads are a piece of transformers, and transformers are the main architecture that's used in models for language and other kinds of models. And the two attention heads work together in a way such that when you're trying to predict something in a sequence, if it's like Mary had a little lamb, Mary had a little lamb, something, something.
[0:49:02] When you're at a certain point in the sequence, they look back to something that's as similar as possible. They look back for clues to things that are similar earlier in the sequence and try to pattern match them. There's like one attention head that looks back and identifies like, okay, this is what I should be looking at. And there's another that's like, okay, this was the previous pattern. And this increases the probability of the thing that's the closest match to this. We can see these very precisely operating in small models.
[0:49:29] And the thesis, which we're able to offer some support for in the new second paper that's coming out, is that these are a mechanism for how models match patterns, maybe even how they do what we call in-context or few-shot learning, which is a capability that models have had since GPT-2 and GPT-3. So yeah, that's interpretability. Sure. So before you move on to the next one, could you also help explain how
[0:49:58] Difficult it is to interpret current models or whether or not it is difficult? Yeah, I mean, I don't know. I mean, I guess I guess like difficult is in the eye of the beholder, you know, and I think Chris Ola can speak to the details of this better than better, better than either of us can. But I think, you know, like kind of like watching watching from the outside and, you know, supervising this this within Anthropic.
[0:50:23] Chris Ola about exactly.
[0:50:49] Exactly how it happens or how he thinks of the right insights that kind of really, you know, almost offers a Rosetta Stone to some particular phenomenon, often a narrow phenomenon. But, you know, like these induction heads, they exist everywhere within small models, within large models. They don't explain everything. I don't want to overhype them, but it's a pattern that appears again and again and operates in the same way. And once you see something like that, then a whole swath of behavior that didn't make sense before starts to make some more sense.
[0:51:18] And of course, there's exceptions. They're only approximately true. There are many, many things to be found. I think the hope in terms of interpreting models, it's not that we'll make some giant atlas of what each of the 100 billion weights in a giant model means, but that there will be some lower description length pattern that appears over and over again. You could make an analogy to the brain or the cell or something like that.
[0:51:47] Where, you know, like, you know, if you were to just like cut up a brain and, you know, like, oh my God, this is so complex. I don't know what's going on. But then, you know, you see that there are neurons and, you know, the neurons appear everywhere. They have electrical spikes. They relate to other neurons. You know, they form themselves in certain patterns. Those patterns repeat themselves. Some things are idiosyncratic and hard to understand, but also there's this patterning.
[0:52:12] I don't know. It's maybe an analogy to biology where there's a lot of complexity, but also there are underlying principles, things like DNA to RNA to proteins or general intracellular signal regulation. So yeah, the hope is that there are at least some of these principles and that when we see them, everything gets simpler. But maybe not. We found those in some cases, but maybe as models get more complicated, they get harder to find. And of course, even within existing models, there's many, many things that we don't understand at all.
[0:52:40] So can we move on then to alignment and societal impact? Trying to align models by training them and particularly preference modeling. You know, that's something that several different organizations are working on their efforts at, you know, DeepMind, OpenAI, Redwood Research, various other places to work on that area. But I think, you know, our general perspective on it has been being very method agnostic and just saying, like, what are all the things we could do?
[0:53:06] To make the models more in line with what would be good. Our general heuristic for it, which isn't intended to be a precise thing, is helpful, honest, harmless. That's just kind of like a broad direction for what are some things we can do to make models today more in line with what we want them to do and not things that we all agree are bad. So in that paper, we just went through a lot of different ways, tried a bunch of different techniques.
[0:53:34] Often very simple techniques like just prompting models or training on specific prompts, what we call prompt distillation, building preference models for some particular task or preference models from general answers on the internet. How good do these things do at simple benchmarks for toxicity, helpfulness, harmfulness, and things like that? So it was really just a baseline, like let's try a collection of all the dumbest stuff we can think of.
[0:53:59] to try and make models more aligned in some general sense. And then I think our future work is going to build on that. Societal impacts, that paper's probably going to come out in the next week or so. As I mentioned, the paper we're coming out with is called Predictability and Surprise in Generative Models. And yeah, basically, there we're just making the point about this open-endedness and discussing both technical and policy interventions to try and
[0:54:29] Grapple with the open-endedness better. And I think future work in the societal impacts direction will focus on how to classify, characterize, and kind of in a practical sense, filter or prevent these problems. So yeah, I mean, I think it's prototypical of the way we want to engage with policy, which is we want to come up with some technical insight.
[0:54:51] And we want to express that technical insight and explore the implications that it has for policymakers and for the ecosystem in the field. And so here we're able to draw a line from, hey, there's this dichotomy where these models scale very smoothly but have unexpected behavior. The smooth scaling means people are really incentivized to build them, and we can see that happening. The unpredictability means even if the case for building them is strong from a financial or accounting perspective, that doesn't mean we understand their behavior.
[0:55:21] Well, that combination is a little disquieting. Therefore, we need various policy interventions to make sure that we get a good outcome from these things. And so, yeah, I think societal impacts is going to go in that general direction. So in terms of the interpretability release, you released alongside that some tools and videos. Could you tell me why you chose to do that? Sure. Yeah, I can maybe jump in here. So it goes back.
[0:55:49] Thank you so much for having me.
[0:56:12] We chose to kind of release other things besides just an archive paper because it hopefully will reach a wider number of people that are interested in these topics, and in this case, in interpretability. And so what we also released is our interpretability team worked on something like, I think it's 15 hours worth of videos. And this is just a more in-depth exploration of their research for their paper, which is called a mathematical framework for transformer circuits.
[0:56:40] And so the team tried to kind of make it like a lecture series. So if you imagine somebody from the interpretability team is asked to go give a talk at a university or something, maybe they talk for an hour and they reach 100 students. But now these are publicly available videos. And so if you're interested in understanding interpretability in more detail, you can watch them on YouTube anytime you want.
[0:57:04] As part of that release, we also put out some tools. So we released a write-up on Garcon, which is the infrastructure tool that our team used to conduct the research, and Peisvelt, which is a sample library, which is used to kind of create some of the interactive visualizations that the interpretability team is kind of known for. So we've been super encouraged that, you know, so far we've seen, you know, other researchers and engineers, you know, playing around with the tools and watching the videos. And so we've already gotten some great engagement already. And our kind of hope is that this will lead.
[0:57:33] Yeah, I mean, a way to add to that to kind of put it in broader perspective is different areas within safety are at, I would say, differing levels of maturity. Like I would say something like alignment or preference modeling or reward modeling or RL from human feedback, kind of all names for the same thing. That's an area where there are several different efforts at different institutions to do this.
[0:58:01] We have kind of our own direction within that. But, you know, starting from the original RL from human preference paper that a few of us were, you know, helped lead a few years ago, that's now branched out in several directions. So, you know, we don't need to tell the field to work in that broad direction. We have our own views about what's exciting within it and, you know, how to best make progress. But it's at a slightly more mature stage. Whereas I would say interpretability, whereas many folks work on interpretability for neural nets.
[0:58:29] The particular brand of let's try and understand at the circuit level what's going on inside these models. Let's try and mechanistically, you know, kind of map them and break them down. I think there's less of that in the world. And what we're doing is more unique. Well, I mean, that's a good thing because we're providing a new lens on safety. But actually, if it goes on too long, it's a bad thing because, you know, we want these things to spread widely, right? We don't want it to be dependent on one team or one person.
[0:58:59] And so when things are at that earlier stage of maturity, it makes a lot of sense to release the tools to reduce the barrier to other people and other institutions starting to work on this. Sorry, you're suggesting that the interpretability research that you guys are doing is unique. Yeah, I mean, I would just say it's like at an earlier stage. Yeah, I would just say that it's like at an earlier stage of maturity. I don't think there are other kind of large...
[0:59:25] Large organized efforts that are kind of focused on, I would say, mechanistic interpretability and especially mechanistic interpretability for language models. We'd like there to be, and we know of folks who are starting to think about it, and that's part of why we released the tools. But I think, yeah, trying to mechanistically map and understand the internal principles inside large models, particularly language models.
[0:59:52] I think there's less of that has been done in the broader ecosystem. Yeah, so I don't really know anything about this space, but I guess I'm surprised to hear that. I imagine the industry with how many large models it's deploying, like Facebook or other people, they'd be interested in interpretability, interpreting their own systems. Yeah, I mean, I think, again, I don't want to give a misleading impression here. Interpretability is a big field.
[1:00:22] And there's a lot of effort to like, why did this model do this particular thing? Does this attention head increase this activation by like a large amount? Like people are interested in understanding the particular part of a model that like led to a particular output. So there's a lot of area in this space, but I think the particular program of like...
[1:00:43] Here's a big language model transformer. Let's try and understand, like, what are the circuits that drive particular behaviors? What are the pieces? How do the MLPs interact with the attention heads? Yeah, the kind of general mechanistic reverse engineering approach. I think that's less common. I don't want to say it doesn't happen, but it's less common, much less common. Okay.
[1:01:06] I guess a little bit of a different question and a bit of a pivot here, just something to explore. If people couldn't guess from the title of the podcast, you're both brother and sister. Yep. Which is pretty surprising, I guess, in terms of, I don't know of any other AGI labs that are largely being run by brother and sister. So yeah, what's it like working with your sibling?
[1:01:32] Yeah. Do you guys still get along from distance childhood? That's a good question. Yeah, I can maybe start here. And obviously, I'm curious and hopeful for Dario's answer. I'm just kidding. But yeah, I think...
[1:01:47] Honestly, it's great. I think maybe a little bit of just like kind of history or sort of background about us might be helpful. But, you know, Dario and I have always been really close. I think since we were very, very small, we've always kind of had this kind of special bond around really wanting to, you know, make the world better or wanting to help people. Right. So originally started my career in international development. So like very far away from kind of the AI space.
[1:02:13] And part of why I got interested in that is that it was an interest area of Dario's at the time. And Dario was getting his PhD in a technical field and so wasn't working on this stuff directly, but I'm a few years younger than him. And so I was very keen to kind of understand the things that he was...
[1:02:30] Oh, wow. Cool.
[1:02:59] Global health issue, you know, working on global health issues like, you know, malaria prevention when we were still both in school. And so, you know, I think we've always kind of had this sort of uniting like top level goal of wanting to, you know, work on something that matters, something that's important and meaningful. And we've always had very different skills. And so I think it's really.
[1:03:22] Very cool to be able to kind of combine the things that we are good at into hopefully running an organization well. So for me, it's been a really, I don't know, I feel like it's been an awesome experience. Now I feel like I'm sitting here nervously wondering what Dario's answer is going to be. I'm just kidding. But yeah, for, I don't know, the majority of our lives, I think we've wanted to find something to work together on, and it's been really awesome that we've been able to at Anthropic.
[1:03:47] Yeah, I mean, I agree with all that. You know, I think, you know, what I would add to that is, like, running a company requires an incredibly wide range of skills. Like, if you think of most jobs, you know, it's like, you know, my job is to, like, get this research result or, you know, like, you know, my job is to be a doctor or something. But I think the unique thing about...
[1:04:13] About running a company and, you know, it becomes more and more true the larger and more mature it gets is there's this like just incredibly wide range of things that you have to do, right? And so, you know, it's like, you know, you're like responsible for like what to do if someone like breaks into your office. But like, you know, you're also responsible for like, you know, does the research agenda make sense? And like, you know, if the, you know, if like some of the GPUs on the cluster aren't behaving like, you know.
[1:04:39] You know, someone has to figure out what's going on at the level of like, you know, the GPU kernels or like, you know, the comms protocol that, you know, the GPUs talk to each other. And so I think it's been great to have two people with complementary skills to like, yeah, to kind of like cover that full range. I often, it seems like it'd be very difficult for, yeah, for just one person to kind of cover that whole range.
[1:05:04] We just get to think about what we're best at. And between those two things, hopefully it covers most of what we need to do. And then, of course, we always try and hire people for specialties that we don't know anything about. It's made it a lot easier to move fast without breaking things. That's awesome. You guys synergistically are creating an awesome organization.
[1:05:27] That is what we aim for. Yeah, that's the dream.
[1:05:34] I guess beneath all of this, Anthropic has a mission statement, and you guys are brother and sister, and you said that you're both very value aligned. I'm just wondering, underneath all that, you guys said that you were both passionate about helping each other or doing something good for the world. Could you tell me a little bit more about this kind of, I guess, more heart-based inspiration for eventually ending up at and creating Anthropic? Yeah.
[1:06:01] Maybe I'll take a stab at this and maybe I'll, I don't know if this is sort of exactly what you're looking for, but I'll kind of gesture in a few different directions here and then I'm sure Dario has a good answer as well. But maybe I'll just kind of talk about like my like sort of personal journey and kind of getting to Anthropic or like kind of what my background looked like and sort of how I wound up here.
[1:06:20] So I talked about this in just part of what sort of united me and Dario, but I started my career kind of working in international development. I worked in Washington, D.C. at a few different NGOs. I spent time working in East Africa for a public health organization. I worked on a congressional campaign. I worked on Capitol Hill. So I was much more in this kind of classic, like a friend at an old job used to call me, like, you know, the classic do-gooder, right, of trying to alleviate global poverty, of trying to make policy-level changes.
[1:06:49] And I just felt like I personally kind of wasn't having the level of impact that I was kind of looking for. And I think...
[1:07:14] That sort of led me to kind of through a series of steps, like I wound up working in tech. And I mentioned this earlier, but I started at this tech startup called Stripe. It was about 40 people when I joined. And I really kind of had the opportunity to see what it looks like to run a really well-
[1:07:31] run organization when I was there. And I got to watch it scale and grow and kind of be in this sort of emerging area. And I think, you know, during my time there, something that sort of became really apparent to me was just, you know, working in tech, like how much of an impact this sector has on things like
[1:07:51] The economy on human interaction, on how we live our lives in sort of day-to-day ways. And Stripe is, you know, it's a payments company, right? It's not social media or sort of something like that. But I think it's still, there is a way that sort of technology is a relatively small number of people having a very high impact in the world kind of per person working on it. And I think that impact can be good or bad, right? And I think it was a pretty kind of logical leap for me from there to think, you know,
[1:08:21] Wow. You know, what would happen if we kind of extrapolated that out to instead of it being, you know, social media or, you know, payments or file storage to something significantly more powerful where there's, you know, a highly advanced set of artificial intelligence systems. Like, what would that look like? And like, who is working on this? Right. So I think for me, I've always kind of been someone who has been.
[1:08:45] Fairly obsessed with trying to do as much good as I personally can, given the constraints of what my skills are and where I can add value in the world. And so I think for me, moving to work into AI, from early days, if you looked at my resume, you'd be like, how did you wind up here? But I think there was kind of this consistent...
[1:09:05] Story or theme. And my hope is that Anthropic is kind of at the intersection of this like sort of practical, scientific, empirical approach to really, you know, deeply understanding how these systems work, hopefully helping to sort of spread.
[1:09:22] Yeah, I mean, I think I have a kind of parallel picture here, which is I did physics as an undergrad. I did computational neuroscience in grad school. You know, I was, I think.
[1:09:49] Drawn to neuroscience by, you know, a mixture of one just wanting to understand how intelligence works seems like the fundamental thing. And a lot of the things that shape the quality of human life and human experience, you know, yeah, are kind of, yeah, you know, experienced, depend on the details of how things are implemented in the brain. I felt in that field there were many opportunities for medical interventions that, you know, could improve the quality of human life.
[1:10:17] Understanding things like mental illness and disease, while at the same time understanding something about how intelligence works, because it's the most powerful lever that we have. I thought of going into AI during those days, but I felt that it wasn't really working. This was before the days when deep learning was really working. Then around 2012 or 2013,
[1:10:44] I saw the results coming out of Google Brain, things like AlexNet and, you know, that they were really working and saw AI as like, saw it both as, hey, this might be the one, the best way to understand intelligence. And two, the things that we can build with AI, but, you know, by solving problems in, you know, science and health and just solving problems that humans can't solve yet by having intelligence that first in targeted ways and then maybe in more general ways.
[1:11:11] Can we solve the important scientific, technological, health, societal problems? Can we do something to kind of ameliorate those problems? And AI seemed like the biggest lever that we had if it really worked well. But on the other hand, AI itself has all these concerns associated with it in both the short run and the long run. So we maybe think of it as we're working to address the concerns.
[1:11:40] So that we can maximize the positive benefits of AI. Yeah, thanks a lot for sharing both of your perspectives and journeys on that. I think when you guys were giving to GiveWell, I was in middle school. Oh, God, we're so old, Dario. Yeah, I still think of GiveWell as just like a new organization that's on the internet somewhere and no one knows anything about it.
[1:12:05] They're this, like, super popular well-known... Yeah, just me who reads weird things on the internet who knows about it. Yeah. Well, for me, it's like, you know, because my, I mean, a lot of my journey into X-Risk and through FLI, you know, has also involved, you know, the EA community, effective altruism. So, you know, I guess that just, like, makes me realize that when I was in middle school, there was, like, the seeds that were... Yeah, there was...
[1:12:33] There was no such community at that time. Let's pivot here then into a bit more of the machine learning. So we've talked a bunch already about how Anthropic is emphasizing the scaling of machine learning systems through computing data and also bringing a lot of mindfulness and work around.
[1:13:00] Alignment and safety when working on these large scale systems that are being scaled up. Some critiques of this approach have described scaling from existing models to AGI as adding more rocket fuel to a rocket, which doesn't mean you're necessarily ready or prepared to land the rocket on the moon or that the rocket is aimed at the moon. Maybe this is kind of lending itself to what you guys talked about earlier about the open endedness of the system.
[1:13:28] Which is something you're interested in working on. So how might you respond to the contention that there is an upward bound on how much capability can be gained through scaling? And then I'll follow up with a second question after that. Yeah, so actually in a certain sense, I think.
[1:13:47] I think we agree with that contention in a certain way. So I think there's two versions of what you might call the scaling hypothesis. One version, which I think of as like the straw version or less sophisticated version, which we don't hold. And I don't know if there's anyone who does hold it, but probably there is. Yeah, it's just the view that like, okay, we have our 10 billion parameter language model. We have 100 billion parameter language model. Maybe if we make 100 trillion parameter language model, like that'll be AGI.
[1:14:14] So that would be kind of like a pure scaling view. That is definitely not our view. Even small modified forms like, well, maybe you'll change the activation function in the transformer, but you don't have to do anything other than that. I think that's just not right. You can see it just by seeing that the objective function is predicting the next word. It's not doing useful tasks that humans do. It's limited to language. It's limited to one modality. And so there are some very trivial, easy to come up with,
[1:14:44] Ways in which literally just scaling this is not going to get you to general intelligence. That said, yeah, the more subtle version of the hypothesis, which I think we do mostly hold, is that, hey, this is a huge ingredient of not only this, of whatever it is that actually does build AGI, right? So no one thinks that you're just going to scale up the language models and make them bigger. But as you do that, they'll certainly get better. It'll be easier.
[1:15:12] To build other things on top of them. So for example, if you start to say, well, you make this big language model and then you used RL interaction with humans to fine tune it on doing a million different tasks and following human instructions, then you're starting to get to something that has.
[1:15:29] Yeah, that has more agency, that you can point it in different directions, you can align it. If you also add multi-modality where the agent can interact with different modalities, if you add the ability to use various external tools to interact with the world and the internet. But within each of these, you're going to want to scale. And within each setup,
[1:15:55] The bigger you make the model, the better it's going to be at that thing. So in a way, the rocket fuel analogy kind of makes sense. Like actually, you know, the thing you should most worry about with rockets is propulsion.
[1:16:06] You need a big enough engine and you need enough rocket fuel to make the rocket go. That's the central thing. But of course, yes, you also need guidance systems. You also need all kinds of things. Like, yeah, you can't just take a big vat of rocket fuel and an engine and put them on a launch pad and kind of expect it to all work. You need to actually build the full rocket. And, you know, I mean, like safety itself makes that point, right? That it's like...
[1:16:29] To some extent, if you don't do even the simplest safety stuff, then models don't even do the task that's intended for them in the simplest way, right? And then there's many more subtle safety problems. But yeah, in a way, the rocket analogy is good, but it's, I think, more a pro-scaling point than an anti-scaling point because it says that scaling is an ingredient, perhaps a central ingredient in everything.
[1:16:50] Even though it isn't the only ingredient, if you're missing ingredients, you won't get where you're going. But when you add all the right ingredients, then that itself needs to be massively scaled. So that would be the perspective. No one thinks that if you just take a bunch of rocket fuel on an engine and put it on a launch pad that you'll get a rocket that'll go to the moon. But those might still be the central ingredients in the rocket. Like propulsion and getting out of the Earth's gravity well is like the most important thing a rocket has to do.
[1:17:15] What you need for that is rocket fuel and an engine. Now, you need to connect them to the right things. You need other ingredients. But I think it's actually a very good analogy to scaling in the sense that you can think of scaling as like maybe the core ingredient, but it's not the only ingredient. And so what I expect is that we'll come up with new methods and modifications. I think RL, model-based RL, human interaction, broad environments are all pieces of this.
[1:17:40] But when we have those ingredients, then whatever it is we make, we'll need to scale that in multimodality. We'll need to scale that massively as well. So, yeah, scaling is the core ingredient, but it's not the only ingredient. I think it's very powerful alone. I think it's even more powerful when it's combined with these other things. One of the claims that you made was that we won't get to AGI. People don't think we won't get to AGI just by scaling up present-day systems.
[1:18:08] Earlier you were talking about how we got, like, there are kind of these phase transitions, right? If you go up, like, one order of magnitude in terms of the number of parameters in the system, then you get, like, some kind of new ability, like arithmetic. Why is it that we couldn't just, like, increasing the order of magnitude of the number of parameters in the systems and just, like, keep getting something that's smarter? First of all, I think we will keep getting something that's smarter. But I think the question is, will we get, yeah, will we get all the way to general intelligence?
[1:18:35] I actually don't exclude it. I think it's possible, but I think it's unlikely, or at least unlikely in a practical sense. There are a couple reasons. Today, when we train models on the internet, we train them on an average over all text on the internet. So, I don't know, think of some topic like chess. You're training on the commentary of everyone who talks about chess. You're not training on the commentary of the world champion at chess. So, what we'd really like is something that's...
[1:19:02] I think that's one issue. The other issue is...
[1:19:26] I don't think you're covering all the tasks that humans do. You cover a lot of them on the internet, but yeah, they're just like some tasks and skills, particularly related to the physical world, that aren't covered if you just scrape the internet. Things like embodiment and interaction. And then finally, I think that even kind of matching the performance of text on the internet, it might be that you need a really huge model.
[1:19:53] To cover everything and match the distribution. And some parts of the distribution are more important than others. For instance, if you're writing code or if you're writing a mystery novel, a few words or a few things can be more important than everything else. It's possible to write a 10-page document where the key parts are like two or three sentences. And if you change a few words, then it changes the meaning and the value of what's produced. But the next word prediction objective function doesn't know anything about that.
[1:20:23] It just does everything uniformly. So if you make a model big enough, yeah, it'll get that right. But the limit might be extreme. And so things that change the objective function that tell you what to care about, of which I think RL is a big example, probably are needed to make this actually work correctly. I think, yeah, in the limit of a huge enough model.
[1:20:45] You might get surprisingly close. I don't know. But the limit might be far beyond our capabilities. There's only so many GPUs you can build and there are even physical limits. And there's less of them. Less and less of them available over time. Or at least they're very expensive. They're getting more expensive and more powerful. I think the price efficiency overall is improving. But yeah, they're definitely becoming more expensive as well.
[1:21:11] You were able to scale up a large-scale system in order to achieve an amateur level of, I don't know, mathematics or computer science. Then would it not benefit the growth of that system to then direct that capability on itself as a kind of self-recursive improvement process? Is that not already escape velocity intelligence once you hit amateur?
[1:21:37] There are training techniques that you can think of as bootstrapping a model or using the model's own capabilities to train it. Think like AlphaGo, for instance, was trained with a method called expert iteration that kind of relies on looking ahead and comparing that to the model's own predictions. So whenever you have some coherent logical system, you can do this bootstrapping. But that itself is a method of training and sort of falls into one of the things I'm talking about.
[1:22:04] You make these pure generative models, but then you need to do something on top of them. And the bootstrapping is something that you can do on top of them. Now, maybe you reach a point where the system is making its own decisions and is using its own external tools to create the bootstrapping to make better versions of itself. So it could be that that is someday the end of this process. But that's not something we can do right now. There's a lot of labs and industry who work on large models.
[1:22:34] I don't know, maybe only a few other AGI labs. You can think of DeepMind. I'm not sure if there are others that OpenAI. And there's also the space of organizations like the Future of Life Institute or the Machine Intelligence Research Institute or the Future of Humanity Institute that are interested in AI safety. MIRI and FHI both do research. FLI does grant making and supports research. I'm curious as to both in terms of
[1:23:02] Industry and nonprofit space and academia, how you guys see Anthropic as positioned? Maybe we can start with you, Daniela. Sure. Yeah, I think we sort of touched on this a little bit earlier, but I really think of this as an ecosystem. And I think Anthropic is in an interesting place in the ecosystem, but we are part of the ecosystem, right?
[1:23:28] I think our kind of strength or the thing that we do best, and I kind of like to think of all of these different organizations as having valuable things to bring to the table, right? Depending on the people that work there, their leadership team, their particular focused research bet or their mission and vision that they're achieving, I think hopefully have the potential to bring safe innovations kind of to the broader ecosystem that we've talked about.
[1:23:52] For us, our sort of bet is one we've talked about, which is this kind of empirical scientific approach to doing AI research and sort of AI safety research in particular. And I think for our safety research, you know, we've talked about a lot of the different areas we focus on, right? Interpretability, alignment, societal impacts, scaling laws for sort of empirical predictions.
[1:24:18] What we're sort of imagining or kind of hoping for in the future is that we'll be able to kind of grow those areas and potentially expand into others. And so I really think a lot of what Anthropic kind of adds to this ecosystem or sort of what we hope it adds is this rigorous scientific approach to doing fundamental research in AI safety. Yeah, I mean, I think, yeah, that really captures it in one sentence, which is, I think, if you want to locate us within the ecosystem,
[1:24:47] Yeah, kind of an empirical iterative approach within an organization that is completely focused on the making a focus bet on the safety thing. So there are organizations like Miri or to a lesser extent Redwood that are like, are either not empirical or have like a different relationship to empiricism than we do. And then there are safety teams that are doing good work within larger companies like, you know, like DeepMind or OpenAI or Google Brain that are, yeah, kind of safety teams within larger organizations.
[1:25:16] Then there's lots of folks who work on kind of short-term issues. And then, yeah, we're kind of filling the space that's like working on today's issues, but with an eye towards the future, empirically minded, iterative, with an org where kind of like everything we do is like kind of designed for the safety objective. One facet of Anthropic is that it is a public benefit corporation.
[1:25:40] I'm not exactly sure what it is, and maybe many of our listeners are not familiar with what a public benefit corporation is. So can you describe what that means for Anthropic, its work, its investors, and its trajectory as a company? Yeah, sure. So this is a great question. So what is a PBC? Why did we choose to be a public benefit corporation?
[1:26:03] You know, I think I'll sort of start by saying we did, you know, quite a lot of research when we were considering what type of corporate entity we sort of wanted to be when we were founding. And ultimately, we decided on PBC, on public benefit corporation, for a few reasons. And I think, you know, primarily it allowed us the kind of maximum amount of flexibility in how we...
[1:26:24] And we were actually very lucky, sort of to a later part of your question, to find both investors and employees who were generally very on board with this kind of general vision for the company.
[1:26:37] What is a public benefit corporation? Why did we choose that structure? So they're fairly similar to C corporations, which is kind of any form of standard corporate entity that you would encounter. And what that means is we can choose to focus on research and development, which is what we're doing now, or on deployment of tools or products, including down the road for revenue purposes if we want to. But the major difference between a PBC and a C corporation is that in a public benefit corporation, we have more legal protections from...
[1:27:06] If the company fails to maximize financial interests in favor of achieving our publicly beneficial mission. And so this is primarily a legal thing, but it also was very valuable for us in being able to just sort of appropriately set expectations for.
[1:27:23] Investors and employees that if financial profit and creating positive benefit for the world were ever to come into conflict, it was sort of like legally in place that the latter one would win. And again, we were really lucky that investors, people that wanted to work for us, they said, wow, this is actually something that's a really positive thing about Anthropic and not something that we need to work around. But I think it ended up just being sort of the best overall fit for what we were aiming for.
[1:27:49] So usually there's like a fiduciary responsibility that people like Anthropic would have to its shareholders. And because it's structured as a public benefit corporation, the public good can outweigh the fiduciary responsibility without there being like legal repercussions. Is that right?
[1:28:04] Exactly. So shareholders can't come to the company and say, hey, you didn't maximize financial returns for us. If those financial returns were to come into conflict with the publicly beneficial value of the company. So I think maybe an example here, I'll try and sort of think of one off the top of my head. But if we designed a language model and we felt like it was unsafe, it was like, you know.
[1:28:29] Producing outputs that we felt were not in line with what we wanted to see from outputs of a language model for safety reasons or toxicity reasons for any number of reasons. And in a normal C corporation, someone could say, hey, we're a shareholder and we want the financial value that you could create from that by productizing it. But we said, actually, we want to do more safety research on it before we choose to put it out into the world. In a PBC, we're quite legally protected, basically, in a case like that. And again, I'm not a lawyer, but that's sort of my understanding of the PBC.
[1:28:59] I think often these things, maybe the more important thing about them is that they're kind of a way to explain your intention, to set the expectations for how the organization is going to operate. Often things like that and the expectations of the various stakeholders and making sure that you give the correct expectations and then deliver on those expectations so no one's surprised by what you're doing and all the relevant stakeholders, the investors, the employees.
[1:29:27] Like the outside world kind of gets what they expect from you. That can often be the most important thing here. And so I think what we're trying to signal here is, yeah, I mean, on one hand, a public benefit corporation, it is a for-profit corporation. Like we could deploy something. That is something that we may choose to do. And it has a lot of benefits in terms of learning how to make models more effective in terms of iterating. But on the other hand,
[1:29:56] Yeah, you know, that the mission is really important to us, and we recognize that this is an unusual area, right, that's more fraught with market externalities, you know, would be the term that I would use, of all kinds, in the short term, in the long term, related to alignment, related to policy and government than a typical area. You know, it's different than making, you know, than making electric cars or making widgets or something like that. And so, yeah, that's the thing we're trying to signal.
[1:30:24] What do you think that this structure potentially means for the commercialization of anthropics research?
[1:30:32] Yeah, I think, again, you know, part of what's valuable about a public benefit corporation is that it's flexible. And so we are it is a C corporation. It's fairly close to a corporate entity, you know, any sort of standard corporate entity you would meet. And so the structure doesn't really have, you know, much of a bearing outside of the one that we just talked about on, you know, decisions related to things like productization, deployment, revenue generation.
[1:30:55] Dario, you were just talking about how, like, this is different than making widgets or, you know, electric cars. And one way that it's different from widgets is that it might lead to massive economic windfalls. So unless you make really good widgets or, like, widgets that can solve problems in the world. So what is Anthropic's view on the...
[1:31:23] Vast economic benefits that can come from powerful AI systems. And what role is it that you see company AGI labs playing in the beneficial use of that windfall? Neil, you want to go? Go for it. Yeah. So, yeah, I mean, I think, you know, I think like, yeah, kind of a way to think about it is, you know, yeah, assuming we can avoid the alignment problems and some other problems like...
[1:31:48] Then there will be massive economic benefits from AI or AGI or TAI or whatever you want to call it, or just AI getting more powerful over time. And then, again, thinking about all the other problems that I haven't listed, which is today's short-term problems and problems with fairness and bias and long-term alignment problems and problems that you might encounter with policy and geopolitics. Assuming we address all those, then there is still this issue of economic.
[1:32:17] Are those benefits evenly distributed? And so here as elsewhere, I think it's unlikely those benefits will all accrue to one company or organization. I think this is bigger than one company or one organization and is a broader societal problem. But we would certainly like to do our part on this. And this is something we've been thinking about and are working on putting programs in place with respect to. We don't have anything to share about it at this time.
[1:32:47] This is like something that's very much on our mind. I would say that more broadly, you know, I think the economic distribution of benefits is maybe one of only many, many issues that will come up, which is that, you know, I think the disruptions to society that you can imagine coming from, you know, like the advent of more powerful intelligence.
[1:33:08] are not just economic. I mean, they're already causing disruptions today. People already have legitimate and very severe societal concerns about things that models are doing today. And you can call them mundane relative to all the existential risk. But I think, yeah, I think they're already serious concerns about concentration of power, fairness and bias in these models, making sure that they benefit everyone, which I don't think that they do yet.
[1:33:37] If we then put together with that the ingredient of the models getting more powerful, maybe even on an exponential curve, those things are set to get worse without intervention. And I think economics is only one dimension of that. So, you know, again, like I just don't like these are bigger than any one company. I don't think it's within our power to like fix them. But we should do our part. We should do our part to be good citizens and we should try and release applications that make these problems better rather than worse.
[1:34:05] Yeah, that's excellently put. I guess one thing I'd be interested in is if you could, I guess, give some more examples about these problems that exist with current day systems and then the real relationship that they have to, you know, issues with economic windfall and also existential risk. It seems to me like tying these things together is really important, at least seeing like the interdependence and relationship there that some of these problems already exist or we already have.
[1:34:32] You know, example problems that are really important to address. So could you expand on that a bit? I think maybe the most obvious one for current day problems is like people are worried very legitimately that like big models suffer from problems of like bias, fairness, toxicity, and accuracy. Like, you know, I'd like to apply my model in some medical application and it gives the wrong diagnosis or it gives me misinformation or it fabricates information. Like that's just,
[1:35:02] That's just not good. These models aren't usable and they're harmful if you try and use them. I think toxicity and bias are issues when models are trained on data from the internet. They, yeah, they absorb the biases of that data. And there's maybe even more subtle algorithmic versions of that where, you know, I hinted at it a little before where it's like the objective function of the model is to say something that sounds like what a human would say or what a human on the internet would say.
[1:35:28] And so in a way, almost like fabrication is kind of like baked into the objective function. And yeah, potentially even bias and stereotyping, you can imagine being baked into the objective function in some way. So just, you know, these models are being used for.
[1:35:42] Or, you know, want to be used for, you know, very mundane everyday things like helping people write emails or, you know, kind of, you know, like helping with, you know, customer surveys or collecting customer data. If they're subtly biased or subtly inaccurate, then those biases and those inaccuracies will be inserted into the stream of economic activity in a way that may be difficult to detect.
[1:36:05] So that seems bad. And I think we should try to solve those problems before we deploy the models. But also, they're not as different from the large-scale problems as they might seem. In terms of the economic inequality, I don't know. Just look at the market capitalization of the top five tech companies in the world. And yeah, compare that to the US economy. There's clearly something going on in the concentration of wealth.
[1:36:33] I would just echo everything Dario said and also sort of add, I think something that is sort of especially kind of can be alarming in sort of a short-term way today in the sense that it could belie things to come is how quietly and seamlessly people are becoming dependent on some of these systems, right? We don't necessarily even know, right? There's no required disclosure of when you're interacting with an AI system versus a human.
[1:36:59] Until very recently, that was sort of like a comical idea because it was so obvious when you were interacting with a person versus not a person. You know when you're on a customer chat and it's a human on the other end versus an automated system responding to you. But I think that line is kind of getting increasingly blurred. And I can imagine that even just in the next few years, that could start to have fairly reasonably large ramifications for people in day-to-day ways. People talk to...
[1:37:29] And sometimes that is, you know, backed by like an AI system that is giving advice, right? Or, you know, down the road, we can imagine things looking completely different in health realms like Dario talked about. And so I think it's just really important as we're kind of stepping into this new world to be really thoughtful about a lot of the safety problems that he just outlined and talked about because I think...
[1:37:52] I don't know that most people necessarily even know all the ways in which AI is impacting our kind of day-to-day lives today and the potential that that could really go up in the near future. The idea of like AI's, you know, there being kind of like a requirement of AI's disclosing themselves as AI seems very interesting and also adjacent to this idea of kind of like the way that C corporations have fiduciary responsibility to shareholders.
[1:38:22] Having AI systems that also have some kinds of responsibility towards the people that they serve, where they can't be, you know, secretly working towards the interests of the tech company that has the AI listening to you in your house all the time. Yeah, it's like another direction you can imagine. It's like, you know, I talked to the...
[1:38:46] I talked to an AI produced by Megacorp, and it kind of, like, subtly sears my life to the benefit of Megacorp. I mean, yeah. There's lots of things you can come up with like this. You know, these are sort of important problems today, and I think they also really belie, you know, things that could be coming in the near future, right? And I think solving, like, you know, whatever, those particular problems are ones lots of groups are working on. But I think...
[1:39:13] Helping to solve a lot of the sort of fundamental building blocks underlying them, right? About getting models to be truthful, to be harmless, to be honest, right? All of those, a lot of the goals are kind of aligned there, both for sort of short, medium, and potentially long-term safety.
[1:39:31] So Daria, you mentioned earlier that of the research that you publish, one of your hopes is that other organizations will look into and expand the research that you're doing. So I'm curious if Anthropic has a plan to communicate its work and its ideas about how to develop AGI safely with both technical safety researchers as well as with policymakers.
[1:39:59] Yeah, maybe I'll actually jump in on this one. And Dario, feel free to add as much as you like. But I actually think this is a really important question. I think communication with policymakers and sort of about safety with other labs in the form of papers that we publish is something that's very important to us at Anthropic. We have a policy team. It's like 1.5 people right now. So we're hiring. That's kind of a plug as well. But I think their goal is to really...
[1:40:25] And I think this is really important because the concepts are very...
[1:40:40] Complex, right? And so it's kind of a special skill to be able to take things that are highly technical, like potentially very important, and translate that into recommendations or work with policymakers to come up with recommendations that could potentially have very sort of far-reaching consequences. So to point to a couple of things we've been working on here, we've been supporting NIST, which is the National Institute for Standards and Technology.
[1:41:04] on developing something called an AI risk management framework. And the goal of that is really developing more monitoring tools around AI risk and AI risk management. We've also been supporting efforts in the US and sort of internationally to think about how we can best sort of support academic experimentation, which we talked about a little bit earlier, with large scale compute models too.
[1:41:28] You guys also talked a lot about open-endedness. And was part of all this alignment and safety research looking into ways of measuring safety and open-endedness? Yeah, there's actually some interesting work.
[1:41:45] Which I think is also in this upcoming paper and sort of in various other places that we've been looking into around sort of the concept of like AI evaluations or AI monitoring. And I think both of those are potentially really, really important because a lot of what we're sort of seeing or maybe lacking, and this kind of goes back to this point I made earlier about kind of standards, is just how do we even have a common language or a common framework within the AI field of what kind of...
[1:42:12] Outputs or metrics we care about measuring, right? And until we sort of have that common language or framework, it's sort of hard to set things like standards across the industry around what safety even means. And so I think AI evaluations is kind of another area that our societal impacts team, which is also like the other half of the one and a half people in policy. So it's also 1.5 people is something that they've been working on as well.
[1:42:35] Right. So a large part of this safety problem is, of course, the technical aspect of how you train systems and create systems that are safe and aligned with human preferences and values. How do you guys view and see the larger problem of AI governance and the role and importance of governments and civil society in working towards?
[1:43:04] The safe and beneficial use and deployment of AI systems. Yeah, we talked about this little one earlier, this one sort of a little bit earlier, and maybe I'll start here and obviously Daria jump in if you want. But, you know, I do think that these other kind of institutions that you talked about have this really important role to play. And again, one of the things we sort of, you know, mentioned in this paper is that.
[1:43:25] We think, you know, government has already been starting to fund a lot more, you know, academic safety research. And I think that's an area that we sort of, a concrete policy recommendation is, you know, hey, go do more of that. That would be great. But I also think groups like, you know, civil society and NGOs, there's a lot of great organizations in this space, including FLI and others that are thinking about like, what do we do, right? Like what, say we develop something really powerful, like what's the next step, right? Whether that's, you know, at an industry lab in government,
[1:43:55] And I think there's a way that just sort of industry incentives are not the same as nonprofit groups or as civil society groups. And I think to kind of go back to this analogy of an ecosystem, we really need thoughtful and empowered organizations that are working on these kinds of questions fundamentally sort of outside of the industry sphere in addition to the kind of policy research and work that's being done at labs.
[1:44:22] Yeah, I mean, another way you can think of things kind of in line with this is like, you know, at some point, you know, laws and regulations are going to be written. And I think probably those laws and regulations work best if they end up being formalizations of what's realized to be the best practices. And those best practices can come from different industrial players. They can come from academics figuring out what's good and what's not. They can come from nonprofit players.
[1:44:49] If you write a law that relates to a technology that hasn't been invented yet, it's often not clear what the best thing to do is and what is actually going to work or make sense or even what categories or words to use. But if something has become a best practice and folks have converged on that and then kind of the law formalizes it and puts it in place, that can often be a very constructive way for things to happen.
[1:45:12] Anthropic has received an impressive amount of Series A funding. So it seems like you guys are doing a lot of hiring and growing considerably. So in case there's anyone from our audience that's interested in joining Anthropic, what are the types of roles that you expect to be hiring for?
[1:45:34] Yes, great question. We are definitely hiring. We're hiring a lot. And so I think the sort of number one thing I would say is if you're listening to this podcast and you're interested, I would highly recommend just checking out our jobs page because that will be the sort of most up to date. And that's just Anthropic.com on the careers tab. But we can also send that around if that's helpful. But what are we looking to hire? Quite a few things.
[1:45:57] Most critically, probably right now, we're looking to hire engineers. And we're actually very bottlenecked on engineering talent right now. And that's because running experiments on AI systems is something that requires a lot of custom software and tooling. And while machine learning experience is helpful for that, it isn't necessarily.
[1:46:18] And I think a lot of our sort of best ML engineers or research engineers came from a software engineering or sort of infrastructure engineering background, hadn't necessarily worked in ML before, but were just really excited to learn. And so I think if that kind of describes you, if you're a software engineer, but you're really interested in these topics, definitely think about applying because I think there's a lot of value that your skills can provide.
[1:46:44] We're also looking for just sort of a number of other roles. I kind of won't be able to list them all. You should just check out our jobs page. But off the top of my head, we're looking for front-end engineers to help with things like interfaces and tooling for the research we're doing internally. We're looking for policy experts, operations people, security engineers, data visualization people. Security, yes, we're definitely looking. If you're building big models. Yes, security is something that I think is kind of.
[1:47:12] Should make sure their models are not stolen by bad actors.
[1:47:15] This is sort of a unanimous kind of thing across all labs, right? There's something everyone really agrees on in industry and outside of industry, which is that security is really important. And so if you are interested in security or you have a security background, we would definitely love to hear from you, or I'm sure our friends at other industry labs and non-industry labs would also love to hear from you. I would also say, I sort of talked about this a little bit before, but we've also just kind of had a lot of success in hiring people who were very accomplished in
[1:47:45] And so I think if you're someone who has, you know, sort of this strong background and sort of set of interests in a technical field that's like, you know, not...
[1:48:09] Related to ML, but sort of moderately adjacent, I would also consider applying for a residency program. And so I think, again, if you're even a little curious, I would say just check out our jobs page because there's going to be more information there. But those are sort of the ones off the top of my head. And Dario, if I missed any, please jump in. Yeah, I mean, that covers a pretty wide range. Could you tell me a little bit more about the team and what it's like working at Anthropic?
[1:48:37] Yeah, definitely. You'll probably have to cut me off here because I'll talk forever about this because I think Anthropic is a great team. But some basic stats. We're about 35 people now. We come, like I've said a few times, we've kind of come from a really wide range of backgrounds. So this is people.
[1:48:52] who worked in tech companies as software engineers. These are former academics in physics, ethics, neuroscience, a lot of different areas, machine learning researchers, policy people, operations staff, so much more. And I think one of the kind of unifying themes that I would sort of...
[1:49:13] And I think the first is really just a genuine desire to reduce the risks and increase the potential benefits from AI. And I think the second is kind of a deep curiosity to really scientifically and empirically.
[1:49:39] Describe, understand, predict, model out how AI systems work and kind of through that deeper understanding, make them safer and more reliable. And I think some of our employees identify as effective altruists, which means they're sort of especially worried about the potential for long-term harms from AI. And I think others are sort of more concerned about...
[1:50:01] And I think, you know, both of those views are sort of very compatible with the goals that I just kind of talked about. And I think they often just call for sort of a mixed method approach to research, which I think is sort of a very accurate description of kind of how things look in a day-to-day way at Anthropic.
[1:50:23] It's a very collaborative environment. So there's not a very strong distinction between research and engineering. Researchers write code. Engineers contribute to research. There's a very strong culture of sort of pair programming across and within teams. There's a very strong focus on learning. I think this is also just because so many of us come from backgrounds that were not necessarily ML focused in where we started. So people run these like...
[1:50:45] Very nice, you know, little training courses where they'll say, hey, if you're interested in learning more about transformers, I'm a transformers expert and I'll kind of walk you through it at sort of different levels of technical skills so that people from the operations team or the policy team can kind of come for the sort of introductory version. And then I think outside of that, you know, I like to think we're a nice group of people. We all have lunch together every day. We have this, you know, very lovely office space in San Francisco. It's fairly well attended. And I think, you know,
[1:51:12] We have lots of fun lunch conversations ranging from things like, you know, a recent one was we were sort of talking about micro-COVID. If you know the concept of micro-COVID, Catherine Olson, who's kind of one of the creators of microcovid.org, which is basically a way of like...
[1:51:27] But, you know, assessing the level of risk from a given interaction or sort of a given activity that you're doing during COVID time. So we had this sort of like fun meta conversation where we're like, how risky is this conversation that we're having right now from a COVID perspective if we like all came into the office and tested, but we're still like sort of together indoors and there's like 15 of us. What does that mean? So anyway, I think it's a fun place to work. We've obviously had a lot of fun, you know, kind of getting to build it together. Yeah. The things that stand out to me are trust and common purpose.
[1:51:57] They're enormous force multipliers where, you know, it shows up in all kinds of little things where, you know, you can think about it in things like compute allocation. Like, you know, if people are not on the same page, if one person wants to advance, you know, one research agenda, the other wants to advance their other research agenda, then people fight over it. And there's a lot of zero sum or negative sum interactions. But if everyone has the attitude of like, we're trying to do this thing, everything we're trying to do is in line with, you know, with this,
[1:52:26] This common purpose. And we all trust each other to do what's right to advance this common purpose. Then, yeah, it really becomes a force multiplier on getting things done while keeping the environment comfortable and while everyone continues to get along with each other. I think it's an enormous superpower that I haven't seen before.
[1:52:48] So you mentioned that you're hiring a lot of technical people from a wide variety of technical backgrounds. Could you tell me a little bit more about your choice to do that rather than simply hiring people who are traditionally experienced in ML and AI? Yeah, that's a great question. So I should also say, you know, we have people from kind of both camps that you talked about, but why did we choose to kind of bring people in from sort of outside the field? I think there's a few reasons for this.
[1:53:14] I think one is, again, ML and AI is still a fairly new field, right? Not super new, but still pretty new. And so what that means is there's kind of a lot of opportunity for people who have not necessarily worked in this field before to kind of get into it. And I think we've had sort of a lot of success or kind of luck with taking people who are really talented in kind of a related field and helping to take their skills and translate them to the ones in sort of ML and AI safety.
[1:53:43] I think the second reason is, so one is just expanding the talent pool. I think the other is it really does broaden the range of perspectives and the types of people who are working on these issues, which we think are very important. And again, we've talked about this previously, but having a wider range of views and kind of perspectives and approaches tends to lead to a more robust approach to doing both basic research and safety research. I'm surprised at how often someone who has experience in a different field
[1:54:12] It's not like they're directly applying things that come, but they think about things in a different way. Of course, this is true about all kinds of things. This is true about diversity in the more traditional senses as well, but you want as many different kinds of people as you can get. As we're wrapping up here, I'm curious just to get some more perspective on you guys about given
[1:54:42] These large scale models, the importance of safety and alignment and the problems which exist today, but also the promises of the impact they could have for the benefit of people. What's a future that each of you is excited about or what's a future that you're hopeful for given your work at Anthropic and the future impacts of AI? I'll start. So I think.
[1:55:09] One thing I do believe is I actually am really hopeful about the future. I know that there's a lot of challenges that we have to face to get to a sort of potentially really positive place. But I think the field will rise to the occasion, or that's kind of my hope. And I think some things I'm sort of hoping for in the next few years is that a lot of different groups will be developing more practical tools, techniques for advancing safety research.
[1:55:38] I sort of think these are likely to hopefully become more widely available if we can sort of set the right norms in the community. And I think the more people working on safety-related topics, that can sort of positively feed on itself, right? And I think I'm most broadly hoping for a world where we can feel confident that when we're using AI for more advanced purposes, like accelerating scientific research, that it's behaving in ways where we can be
[1:56:06] Very confident and sure that we understand that it's not going to lead to negative unintended consequences. And the reason for that is because we've really taken the time to chart them out and understand what all of those potential problems could be. And so I think that's obviously a very ambitious goal. But I think if we can make all of that happen, there's a lot of potential benefits of more advanced AI systems that I think could be...
[1:56:33] Transformative for the world, right? From almost anything you can name, right? Renewable energy, health, disease detection, economic growth, and sort of lots of other just day-to-day enhancements to kind of how we kind of work and communicate and live together. No one really knows what's going to happen in the future. It's extremely hard to predict. And so I often find any question about the future, it's more about
[1:56:59] The attitude or posture that you want to take than it is about concrete predictions, because I feel like particularly after you go a few years out, it's just very hard to know what's going to happen. And so, yeah, it's mostly just speculation. And so in terms of attitude.
[1:57:17] I think, well, first of all, I think the two attitudes that I find least useful are like blind pessimism and blind optimism. Because they're actually sort of, you know, like doomsaying and like Pollyannaism. And, you know, it's possible, weirdly it's possible to have both at once. But I think it's just not very useful because it's like, you know, like we're all doomed. It's intended to create fear or it's intended to create complacency.
[1:57:45] I find that an attitude that's more useful is to just say, well, we don't know what's going to happen. But as an individual or as an organization, let's pick a place where there's a problem we think we can help with. And let's try and make things go a little better than they would have otherwise. Maybe we'll have a small impact. Maybe we'll have a big impact. But let's, you know, instead of trying to understand what's going to happen with the whole system, let's try and intervene.
[1:58:12] in a way that helps with something that we feel well equipped to help with. And of course, the whole outcome is, you know, it's going to be beyond the scope of one person, one organization, even one country. But yeah, I mean, I think we find that to be a more effective way of thinking about things. And for us, you know, for us, that's, can we help to address some of these safety problems that we have with AI systems?
[1:58:39] in a way that is robust and enduring and that points towards the future. If we can increase the probability of things going well by only some very small amount, that may well be the most that we can do. I think from our perspective, I would like it if AI could advance science, technology, and health in a way that's equitable for everyone and that it could help everyone to make
[1:59:09] Better decisions and improve human society. Right now, I frankly don't really trust the AI systems we build today to do any of those things. Even if it were technically capable of the task, which it's not, I wouldn't trust it to do those things in a way that makes society better rather than worse. And so I'd like us to do our part to make it more likely that we could trust.
[1:59:32] Trust AI systems in that way. And if we can make a small contribution to that while being good citizens in the broader ecosystem, you know, yeah, that's maybe the best we can hope for. All right. And so if people want to check out more of your work or to follow you on social media, where are the best places to do that?
[1:59:55] Yeah, on Anthropic.com is going to be the best place to see most of the recent stuff we've worked on. I don't know if we have everything posted, but we probably will. Yeah, we're about, because we have several papers out. So we're now about to post links to them on the website. In an easy to find place. And then we also have a Twitter handle. It's just, I think it's Anthropic on Twitter. And yeah, and we generally, you know, also...
[2:00:22] Tweet about our recent releases of all different kinds. We are relatively low-key. We really want to be focused on the research and not get distracted. Yeah, I mean, the stuff we do is out there, but we're very focused on the research itself and getting it out and letting it speak for itself. Okay, so where's the best place on Twitter to follow Anthropic? Our Twitter handle is at Anthropic AI.
[2:00:46] All right. I'll include a link to that in the description of wherever you're listening. Thanks a ton for coming on, Dario and Daniela. It's really been awesome and a lot of fun. I'll include links to Anthropic in the description. It's a pleasure having you. And yeah, thanks so much. Yeah. Thanks so much for having us, Lucas. This was really fun. Thanks for joining us. If you found this podcast interesting or useful, consider sharing it on social media, with friends, and subscribing on your preferred podcasting platform.
[2:01:16] We'll be back again soon with another episode in the FLI podcast.