[0:00:00] Okay, so this has been a great two days. Thanks, everyone, for your engagement, your question. Thanks to all the speakers, and of course, thanks to the organizers. My name is Jesse Thaler. I'm the director of iFi. But actually, I play no role in the organization of this effort. I certainly do the effort of Phil, Fiala, Sid, Gaia, and Marisa, who have brought this together. And again, thanks to all the speakers, and thanks to the organizers.
[0:00:25] Oftentimes in the machine learning world, there's a tradition of having fireside chats with various people from academia and industry. And we're really happy to have Jared Kaplan doing one of these chats this afternoon. And I've known Jared since graduate school. We overlapped at Harvard. He had the same PhD advisor. He went on to an academic career as a faculty member at Johns Hopkins.
[0:00:50] And then he caught the machine learning bug and eventually became co-founder of Anthropic. And Anthropic, I think it was this month, that released kind of eye-popping results with Claude Triobus. And it's really kind of fascinating to see someone who comes from the same theoretical physics background that I have, thinking about quantum field theories, now taking some of those tools and other things.
[0:01:18] Understanding things like scaling and AI systems, but then also going to actually create an actual commercial product at his company. So Jared, welcome and looking forward to having this conversation. Great, thanks. Yeah, thanks for having me. Good to see you again. Yeah, Jesse was kind of like my big brother in grad school because I think you were like two or three years ahead of me anyway. So yeah, we work together a lot. And yeah, it's always fun to chat.
[0:01:48] Yeah, and so I think I remember even some of our first conversations when you were getting into the machine learning direction. But maybe you can just tell us a little bit about your path that got you from physics to industry. What was it that drew you into the machine learning world, first from, I guess, the more academic side, but then kind of led you into the Anthropic direction? Yeah, so I guess it was like many strands. I mean, I always, growing up, sort of wanted to be a physicist, and I didn't really want to do anything else.
[0:02:18] So that was always kind of my passion. I thought like AI was interesting, but I mean, back when we were in school, it really wasn't making any obvious progress. Like I actually had a chance when I was in undergrad at Stanford to take the machine learning class that Andrew Ang, who was kind of a famous ML person, gave. And I remember I took the class and it was just sort of a bunch of linear algebra and there wasn't really almost any neural network.
[0:02:46] And I think actually Andrew asked me, oh, do you want to do summer research? And I said, no, of course not. I'm doing physics summer research. I don't care about this stuff. So yeah, I was always interested. When I was a grad student, I became friends with Dario Amadai, who is the CEO of Anthropic and another founder. And so he and I have been friends since we were in grad school together. And I think that was...
[0:03:11] One of the things that made my transition to ML easy. I think having smart friends who do diverse things is useful. And actually, Dario himself also started in theoretical cosmology, but then moved to neuroscience and bioinformatics, and then finally AI. So a lot of the people that I work with in AI didn't necessarily start there. But yeah, I was always interested in AI. I worked in many different areas in theoretical physics. I mean, first LHC physics and cosmology, then...
[0:03:39] Things that were sort of more bordering on string theory and quantum gravity, like the formal bootstrap. And yeah, I kind of meandered around. And around 2017, 2018, AI was starting to seem a lot more exciting. Well, it started getting exciting maybe back in 2012, but it was sort of slowly ramping up. And with a bunch of other physicists, I basically just had a journal club, just kind of reading AI papers and learning about them.
[0:04:06] And I was on sabbatical. And so I figured I eventually got up the courage to sort of ask Dario, oh, hey, could I like volunteer at OpenAI since it was a nonprofit legitimately at that point? So you could be a volunteer there. He was like, oh, yeah, sure. Like, maybe we can get you a laptop and you can sort of like collaborate. And I helped OpenAI hire Sam McCandlish, who also was a physicist. And so Sam and I basically worked together. And yeah, it was a lot of fun.
[0:04:35] I guess the other motivation was just like Dario and many of the other people I hung out with were concerned about sort of AI becoming powerful quickly and there being kind of safety concerns there. So I think that was that was another sort of constraint. But I mean, ultimately, like I think I was I was like I wouldn't have worked on AI if it hadn't been interesting. And so, yeah, we started working together and then I started
[0:04:58] As a faculty member, you usually can do something where you work part-time doing something else like 20% time. So I sort of did that while I was a Hopkins professor for a couple of years. And I just did more and more AI research and therefore comparatively less physics. And so then we decided to found Anthropic. And I think for me, there wasn't any particular desire to found a company, be in industry, et cetera. All of the motivation was just that.
[0:05:28] We seem to be learning that AI systems could become sort of predictably more capable with more computational resources. And it was hard to get those resources to sort of scale quickly as an academic, especially as some random physicist who just written a few ML papers. Like I wasn't an authority or anything. So for me, that was sort of like the main reason why.
[0:05:52] Why we sort of founded a company rather than just sort of sticking around as a faculty member. And yeah, the sort of scale has continued. So I don't know, that's sort of a quick version. Yeah, and so now the conversation is not just about AI or ML. Now the conversation is about generative AI. And this workshop is a symposium about generative AI in the physical sciences.
[0:06:18] Did you actually see a distinction with that extra word generative? Of course, that's the way that we describe chatbots. That's the way that I would describe to some of what Claude is doing. But in your mind, is there really a distinction between just maybe guard rights, statistics, machine learning, AI, generative AI, other types of computational strategies? How do you draw the line? And is using that word generative actually important for communicating, let's say, the stakeholders about what you're doing?
[0:06:43] Yeah, I don't think so. I mean, I think there are a lot of buzzwords. My attitude towards AI, especially coming from physics, has just been that AI is like, the reason why there's so much opportunity is that it's all like really, really kind of simple stuff. For a while, I would give talks about like, what is AI to physicists?
[0:07:04] It's just curve fitting with a lot of parameters. It's like, I mean, everyone I'm sure in the audience already knows all this stuff, so I don't need to rehash it. But I think the word generative just comes from the idea that you're just kind of trying to directly model the probability distribution of the data itself. I mean, maybe that even is too restrictive. But I mean, I think it's related to this paradigm that has existed now for three or four years, but is relatively new.
[0:07:31] If you train a generative model to just model the distribution of the data itself via text or images or videos or some kind of scientific data, that model will sort of inevitably learn a lot of interesting features that describe that data just in the process of fitting it. And then you can fine tune those models or train them in other ways to do more specific tasks. So I mean, large language models are generative models that just try to generate
[0:08:00] Text in the distribution that humans write it in, and then we can fine tune them to be useful assistants and kind of perform tasks when asked. So let's just like kind of putting your industry hat firmly on, you know, so, you know, either with plot tree opus or with the next generation of things that are coming, you know, just from a purely technical perspective, you know, what are you really excited by? Where do you see things heading in in this space? Yeah. So
[0:08:31] I mean, it's a slightly longer answer, but I think that my core expectation is that AI systems will become broadly more capable over time. I think that's sort of a predictable outcome of scaling laws and increasing investment in AI and improvements in sort of the number of
[0:08:58] Floating point operations per dollar in improvements in chips and clusters and improvements in algorithms. I mean, I think broadly, I just see AI getting more and more capable quite quickly. I guess I tend to think about this in sort of two axes. One axis is kind of the generality of the environment that the AI can operate in. And the other is like, how complex is the task the AI system is doing? So, I mean,
[0:09:27] Back when I was in high school, I think like Deep Blue came out and it was better than humans at chess. But chess is a very, very narrow regime. And of course, then AlphaGo came out more recently that better than humans at Go. But obviously with LLMs and then multimodal models, we have systems that can sort of operate in the space of general text interactions, general code.
[0:09:51] They can now see the world. So I think it's very plausible we're going to see more and more AI systems operating robots, using computers in the same way people do, et cetera. So I think the systems are going to very rapidly become capable of sort of operating in more and more different environments that are kind of useful and relevant. And then the other direction is just like, how long does it take a person to do a task? And I think a lot of AI systems
[0:10:16] Like the current state of their systems can do things that take a person a few minutes or an hour maybe. But I think we're going to start to see systems that fix their own mistakes, do long, what we call long horizon tasks, and therefore can do things that might take a person a day or a week, et cetera. And I think then kind of the sky's the limit. Can you say a little bit more about kind of generality? It sounds like I don't have a good intuition for the fact that it seems like these general tools.
[0:10:47] When fine-tuned, do better on specific tasks. And there's kind of this sense of, you know, as a scientist, I would have naively thought that if I had a specific task in mind, that I would target that one specific task, that I would do better than if I tried to grill the more generic tool. Do you have an understanding of that? Or, you know, this axis you described of moving in this more general context direction?
[0:11:08] Do you have a sense of why that's actually a good direction to go and maybe thinking about that in scientific applications? Why is it that if I'm trying to do more things in a more general context, that might actually be beneficial even if I had a specific goal in mind? Yeah. So, I mean, I think that if you have an infinite amount of data that's of reasonably high quality in one specific task, then I think my experience is that you do just win by training entirely on that task.
[0:11:37] If you really have an infinite amount of Mathematica notebooks written by physicists, and you wanted to train a model that was just really good at writing Mathematica code, you'd probably want to train pretty much entirely on those Mathematica notebooks. But I think that the data sets involved in training state of the art language models these days are so large that there is not an infinite amount of data of any particular type for any particular task you want. And so you're able to do better.
[0:12:05] by training on a diverse set of data. I mean, it's a little bit more complicated than this. Presumably, if you were making a bot to write Mathematica code, you'd want to be able to instruct it in English. And so you might want to include some text or some data where there is sort of a lot of comments in the Mathematica code. But I think broadly, you do generally do better if you have all of that data. And I mean, of course, if you really want a specialized
[0:12:29] AI bot, you can probably fine tune it at the end on sort of mostly the data distribution you want to focus on and get some gains. But I think it's really a question of just sort of how much data you have. And currently, it sort of seems to be better. Well, I mean, there are many reasons. I mean, there's a question of like, what is the goal? But I mean, it seems like you can do better by training a very general purpose AI that understands a lot of aspects of the world and then asking it to do things versus just...
[0:12:58] Focusing on only a single task. So eventually I want to talk about kind of more on the science side, but again, just like I'm staying in the industry world, just so we can get a context or a sense of it. So we've talked about some of the things that you're looking forward to, you know, being able to do these long time horizon, at least human long time horizon tasks. What are you most apprehensive about for the types of systems that you're developing? Yeah, so I mean, I think, I think
[0:13:24] I mean, generally, I'm pretty optimistic, but I mean, I think there are a lot of things to be sort of somewhat apprehensive about. I think for me, all of the apprehensiveness pretty much comes like 80 to 90% of it just comes from kind of like relatively common sense seeming things. So there's sort of a baseline question of like, do you actually believe that AI systems are going to keep getting better? And I think that...
[0:13:50] It certainly has been very reasonable to not believe that. Maybe it's becoming less reasonable over time, but still, I very much understand skepticism. But if you really believe that AI is just going to keep getting better, then I think that we're very, very rapidly building a much, much more powerful technology that's different than anything we've had before.
[0:14:16] We don't really understand how it works very well because it's so new. As indicated by some of the questions you're asking, there are very basic things that you might think that you should understand that we don't. Also, if it's really going to be capable of a lot of the cognitive tasks people can do, then it can easily be misused. I think broadly, I just think that the way that I most empathize with people who don't
[0:14:46] I mean, the way that I think about this basically is that there are
[0:15:13] Prospects for misuse of AI. There are prospects of AI systems getting used maybe in ways that should be beneficial, but the AI systems themselves sort of not doing what we expected them to do in high stakes situations and sort of not being sufficiently safe or aligned. And I think then there's just prospects for sort of disruption in society if we're rolling out this really powerful technology very quickly.
[0:15:40] It's in your company's name, Anthropic. I mean, there's a sense of kind of human-centeredness, things like constitutional AI, or again, you mentioned the term alignment. Of course, there's a messaging at the kind of corporate level. On the technical side, what strategies do you see to kind of mitigate some of these models?
[0:15:59] Yeah, so there's a lot that I think can be done. And of course, it's partly technical, but it's partly sort of a question of policy and sort of how the ecosystem works. But on the technical level, there are a few things. So we do a lot of work on interpretability, which is in some ways like the longest term, most ambitious bet where maybe we can reverse engineer how neural networks work.
[0:16:27] Or at the very least, kind of like do something like a brain scan on neural networks and understand what the sort of features are that are that that are causing them to behave a certain way and try to understand what they're doing better and then therefore maybe be able to detect problematic behaviors or uses or situations. The second direction, which is sort of more sort of standard ML, is just like there's a lot of things you can do to try to
[0:16:55] Either train systems to be sort of safer and to sort of behave more in accord with sort of human preferences and values and also to evaluate these systems. So I'm kind of the training side. I mean, we develop things like constitutional AI. We're working on all sorts of other sort of more elaborate techniques.
[0:17:18] And the idea there is, I mean, with constitutional AI, the idea is that you're able to list some set of principles and have the AI kind of evaluate its own behavior based on those principles and train itself to abide by those principles. It's kind of similar to the laws of robotics or something. There are all kinds of other more complex ideas for how to do this.
[0:17:40] There's sort of like a slogan that we say of like, well, we want our AI systems to be helpful, honest, and harmless. And so sort of that's the goal. There's all sorts of other projects. There's some work that we've done recently that's kind of tantalizing of like looking at whether you can train systems that are in some way deceptive or have some behavior that gets triggered out of distribution in a problematic way.
[0:18:10] We're sort of experimenting with that because basically we want to make empirical toy models of things that could go wrong and then understand when do these things actually go wrong, are they easy to mitigate, et cetera. So there's all kinds of stuff like that. There's work on red teaming models and then automating that. So having AI systems red team other AI systems in order to train them to sort of not do bad things.
[0:18:32] And then there's all sorts of evaluations. Of course, there's the standard evaluations in the literature. We also, with respect to misuse and more advanced issues that could come up, we have national security relevant type evaluations. Are AI systems useful to a terrorist who's trying to build a bioweapon? Can we evaluate?
[0:18:57] Are the systems capable of actually moving the needle compared to Google? I mean, obviously, we want to be pragmatic. I mean, you can learn a lot from Google, but our system is actually more, do they know more than Google knows? And do sort of safety measures prevent that kind of thing? So that's some sampling of things that we're doing. I mean, there's all sorts of more detail I could go into.
[0:19:19] And maybe I think we'll use the three H's of both, helpful, honest, and harmless. The Biden administration, their executive order was talking about safe, secure, and trustworthy. Do you have any comments about that? Or maybe how do you actually convince stakeholders that your product actually has those features? Yeah, so I mean, the products are definitely not perfect. And I mean, that's another reason why you should be a bit apprehensive. It's not like we have completely solved
[0:19:48] Like these three H's or something and can guarantee that our systems never make mistakes. I mean, they're probabilistic models. There's some probability of sampling sort of problematic things kind of no matter what. And furthermore, like, I mean, these systems can be jailbroken. They make mistakes. So they're certainly not perfect.
[0:20:13] So, I mean, you mentioned sort of the Biden order. I mean, I think these things kind of map, but I think security is important in the sense that like information security of advanced AI systems is important. If we're going to build systems that are very capable, very powerful, then we don't want them to sort of be stolen by...
[0:20:31] By bad actors. So sort of information security is another pillar that's technical. It's not a research. It's not exactly a research question necessarily, but that's also important. The other way that we think about this, again, like kind of being practical is that we have this system that we're devising and it's really a work in progress of sort of AI safety levels, or you could say they're sort of like AI danger levels in a sense of where
[0:20:57] We kind of rate current systems, including, say, Claude 3, as not all that dangerous in the system, which is kind of analogous to biosafety levels. The current systems are kind of ASL 1 or ASL 2. But ASL 3 is kind of defined to be a system that would.
[0:21:16] For example, be helpful to terrorists if it was jailbroken and they were able to ask it, like, how do I build a bioweapon? It would have like very, very useful, detailed knowledge. Maybe it would know as much about building bioweapons as like a really good grad student in biology or something might know. And so I think it's probably not, it probably will be possible to build systems that are sort of at that capability level soon. And we're trying to test our systems for those level of capabilities.
[0:21:45] Very carefully. And then if we sort of reach that capability level, we have a bunch of commitments that trigger for both information security and kind of like robustness and harmlessness. And so for systems that are not very dangerous, we sort of don't want to kind of tie our hands behind our back. We don't want sort of caring about safety to be impractical. But for more powerful systems, I think there's sort of.
[0:22:11] New constraints and and we're sort of thinking about, like, how will this ratchet up over time? So I don't want to transition. And I think we talked beforehand about how, you know, you're a bit out of the loop of what's actually going on in physics these days. But nevertheless, I'm going to poke you a bit about about speculating and trying to kind of pick your brain.
[0:22:34] About any lessons from your journey or your experience that might be relevant for our discussion of gendered AI in the physical sciences or this very fancy curve fitting in the physical sciences. So maybe just kind of make your first move and end it. Do you have any thoughts? Yeah. So I mean, like the kinds of systems that I mostly am developing and playing around with that are very general, like
[0:23:01] I think we're on the cusp of being useful to scientists for science. A lot of scientists these days write code. I think our models and models in the last few years have been very useful for that. I think they're probably getting better and better, such that you can ask for code to write a simulation or do something. I think we're at a point where that's actually pretty helpful. Some people anecdotally like to play with these systems.
[0:23:30] Brainstorm or even like, I mean, I've seen people on Twitter say, oh, like I was about to write the scientific paper and I dumped like the first half of it in to this model and asked for a bunch of ideas and it came up with a bunch of the ideas that we're actually working on. So like, I think we're like near to a point where these systems are sort of maybe useful as kind of like, like research assistants on some level. I mean, I, I personally play with the models in this way and like, I think they're like pretty good, but.
[0:23:57] But maybe not there yet. I don't know. With Claude 3 Opus, when we were developing it, I was asking it to do various calculations using the Virasoro algebra. And it would get the simple calculations right some of the time. But some of the time, it would make mistakes. And so it's not obvious. But yeah, I think that we're headed in a direction where these systems will be genuinely useful. And I'm sure people will find more creative ways of capturing that.
[0:24:25] So, Bob, we made the conversation in various directions, and maybe this is a question not really for you, but you stepping back a little bit to think about the AI industry more broadly. What is the perspective of that industry on some of the things that we're doing, which are, let's say, curiosity-driven research? So there's an aspect of the physical sciences.
[0:24:48] We've had this at this workshop and talking about things like materials discovery, where there there's an application and you say, you know, you can you can evaluate whether you've been successful because it actually creates a substance or a product that you didn't have before. But then there's a kind of a kind of curiosity driven angle where you're just trying to do exploration. I think your own experience going into machine learning wasn't that my kind of curiosity driven direction, but kind of thinking about the industry as a whole. What's the perspective on those type of things that many of us in this room are interested about?
[0:25:16] Yeah, I mean, obviously that's sort of very natively relevant to me. I mean, I guess there's a bunch of different directions we could go in discussing this. I mean, I think that, I mean, hopefully the system's going to be useful for that. I think that certainly I'm very curious about where things are headed, but at the same time, like, it feels like there are pretty clear directions to go in. So I guess.
[0:25:45] For AI research itself, it feels for me very, very different from theoretical physics in that like with theoretical physics, the field is kind of like 400 years old. A lot of giants have come before and it's like really hard to kind of like make progress. Like I think I spent a lot of time being frustrated. And I think that like really like in AI research.
[0:26:12] It's really easy to have ideas that people just haven't explored before. And so there's a lot of like curiosity driven stuff you can do. In terms of like applying AI to physics, I would expect exactly the same. And it's really just a question of like, are you applying a narrow AI system to a narrow domain? Or are you like hoping that more general purpose AI systems will become capable enough soon that they're actually useful? And anyway, there's a lot of different questions. What are you most interested in chatting about?
[0:26:40] So let's actually go to the generalist system. So for a specific problem, very narrow, I can imagine myself and a small computing cluster being able to churn through a reasonably sized data set and be able to create something that I could use. And in fact, part of what we'll be discussing in this workshop is even taking those trained models and making them even smaller, for example, real-time use of the Large Hadron Collider. But if I'm thinking about it as a scientist building a generalist tool,
[0:27:08] I certainly don't have, you mentioned this already, I certainly don't have access to the computer resources that Anthropic would have. And there's a question of how can I either convince you that you should build this tool for us or figure out what is a computational strategy that would allow me to have that kind of generalist tool that's more appropriate for scientific applications? Yeah, I mean, we definitely want our models to be useful to scientists.
[0:27:37] I mean, we definitely like one of the ways that we train these models is with this idea of like reinforcement learning from human feedback, where basically you just get people to interact with the model and sort of choose what kinds of outputs, what kind of behavior is most useful. And so, I mean, we definitely have human feedback from people in the sciences, in medicine, like obviously computer programmers, all sorts of different areas.
[0:28:06] I mean, we are trying to make our systems better for this. And certainly we like want as much high quality sort of scientific data as we can get. So, I mean, I hope the models are useful. I mean, maybe you have to ask for it, but I mean, you should be able to get the models like output LaTeX and things like that. So I think we're doing all we can, I think, to sort of move in that direction. I mean, I think that like...
[0:28:35] Of course, this direction is associated with some of the scary things in the sense that if you can really automate scientific research, that's a big disruption, that's a big change. But I think in the near term, we're very excited about making systems better for scientists. I think it's possible in some cases to fine-tune, like it will be in the near future, to fine-tune Claude for different use cases. My guess is it'll be hard to...
[0:29:04] Really move the needle from the general purpose model to be like way better at one scientific area that way. But but it's the kind of thing that's that's that's definitely possible. It's actually this is something which I don't know the answer to. Is there already or will there ever be kind of an open source version of the of your products? Because, you know, I'm thinking about I don't know that I want to send you
[0:29:27] Or even if you want to take a massive amount of data from the LHC and be able to process something as a kind of multimodal way to actually solve some of the problems that we're thinking about. But certainly you can imagine if there was some kind of open source reduced complexity system that we could play around with that. I actually don't know whether there's anything open source coming out of Anthroper or whether that's in the plans or how you feel about kind of the open source culture that we tend to have in the scientific community.
[0:29:55] Yeah, I mean, I think open source is great. I mean, in terms of like very, very frontier systems, like I think that sort of open source models should kind of be under whatever the constraints are that closed source models are in terms of like if they're very capable. We don't plan on open sourcing our models, but I mean, there are many open source models out there from other providers. So, I mean.
[0:30:24] Definitely be possible to train. I think we would be very happy if someone sent us a bunch of scientific data in some reasonable format. I'm guessing we would be happy to include it in training of models in order to make models better at that kind of thing. I think we would be open to that.
[0:30:50] But yeah, aside from yeah, we don't we don't have like a specific program for it. But I mean, you could you could shoot me an email. Yeah, so I think this is kind of getting so I think there's two aspects of it. One is that there's this incredible resource within industry and computer resources, and there is an opportunity for alignment of interests between between academic research and industry research. So like thinking about collaborative possibilities.
[0:31:17] The diversity of data set that we deal with in the physical sciences is vast. It goes beyond just text or image or video. And there's all sorts of different types of data that we play with and work with, including mathematical structures. So can you talk about other collaborative possibilities in the multimodal domain or progress you can imagine in multimodal interfaces with data sets? Yeah, no, I think that seems really cool.
[0:31:46] We definitely train our models on lots of like specially formatted data like JSON or XML or other formats like that so that models sort of understand those formats so that they understand code. And so, I mean, that's one thing like CSV files. So, I mean, models definitely see that and hopefully ideally see that surrounded by...
[0:32:10] Explanatory text so they can sort of associate it with concepts. In terms of multimodal data, I think that would be awesome. I mean, we always want to improve our models' multimodal capabilities and understanding like charts and graphs and like also, I mean, any other sort of data. So I think, I mean, I think it would be interesting. Like, I think if there were like, say, astronomical phenomena associated with like descriptions or discussion of like what those phenomena were, like, or LHC events, like.
[0:32:40] I think we'd be very happy to include that kind of information in training. I think probably you need a lot of it for it to really move the needle. But I mean, I think there's definitely space for collaboration there. I mean, if the data is trivially available publicly, maybe we already have it. But yeah, we could definitely talk about this if it feels like it would really be helpful.
[0:33:09] Four kind of scientific use cases. Yes, I wrote a paper with Sid that if you're last night in the archive with the high school intern, we're indeed taking a whole database and kind of doing that type of text image association. And actually, I have no idea whether Claude has ever seen that type of data set and what benefit there would be about trying to develop those associations.
[0:33:33] So another topic that has come up besides availability of compute and that difference with industry are career pathways. So some folks in this room, like a graduate student or postdoctoral stage, want to go on the academic direction. Other are curious about going in industry directions. Do you have advice or thoughts or observations about that in terms of career paths for junior researchers? Yeah.
[0:34:02] Many. I think that, I mean, I always sort of like, I don't know that I feel that strongly about this, but I mean, I always sort of like wanted to be in academia because I just wanted kind of total freedom. And that felt like it was important to me. And I sort of think that is, that's definitely valuable. And so I think that like.
[0:34:24] That is a plus of like being in academia. And of course, like as an academic, like I'm sort of used to just like telling everyone what I think about it, about kind of anything. And so I think those things are sort of like real benefits of academia and also ones that if you go into academia, you should really take advantage of. Like, I think I often see people go into academia, but like then kind of feel like, oh, like I have to sort of conform to my community or conform to like this tenure process or something. And it's like, I don't know, like.
[0:34:53] You could probably get paid way more. If someone's going to tell you what to do and tell you what you can and can't do, you can get paid way more in industry to have a boss who will tell you what to do. I think it's important to take advantage of the freedom you have in academia to pursue what seems meaningful and important to you. I do think that the whatever industry
[0:35:20] Also has interesting advantages. I mean, I never for a moment considered working in industry when I was younger. I mean, in undergrad and grad school, I just wanted to be an academic. I just wanted to do research. I generally regarded industry with a fair amount of cynicism. And so I very much understand that perspective. I think, though, there definitely are organizations.
[0:35:48] There's just a huge number of different organizations you can work with or for. I think many of them are genuinely pretty mission oriented and have great people at them. And so I think it can be great. And then I think it is just the case that if you have the kind of skills to get a PhD in physics, probably you can be well compensated in industry.
[0:36:12] In terms of specifically AI and Anthropic, I mean, Anthropic was founded by a lot of people who are associated with physics. I mean, Dario, as I mentioned, was originally in physics. I was. Sam McCandlish was. And so we've hired a lot of physicists. And also, as I've said, I kind of think that AI and ML, it's kind of like a five or 10-year-old field. So there's very little that you need to know.
[0:36:40] As a physicist, you probably know linear algebra and basic information theory and obviously calculus. So I mean, if you kind of know those things, then you can, and probably if you're in this room, you've already done work in ML. So I think it's really easy to kind of pick up. And so we hire a lot of people from physics who spend three to six months kind of spitting up and learning programming and basic ML, and they're extremely productive. And so we're sort of always hiring people like that.
[0:37:08] At any given time. And I don't know that other organizations sort of realize how well people like that do. But I mean, there are other places for physicists and industry. So I want to make sure that we leave time for questions from the audience. So think about the questions for those of you who are in the room. But I am curious and I just kind of need to know. So if you pull back your theoretical physicist hat and you say, OK, great.
[0:37:39] You know, I have, what do you say, this 400-year-old field, there hasn't been any progress, and it's really hard to make progress. Can you imagine you yourself using a cloud-like system?
[0:37:52] for scientific discovery? And if so, how? And there's various different provocative forms of that type of question. But I'm just curious, like, could anything get you back into the research sphere and with the physics research sphere? And in what sense would that be kind of AI enabled? Yeah, I mean, my guess is that, like, I'm going to keep working on AI and AI safety for a while. I mean,
[0:38:21] I'm just being honest. I think that there's a very significant chance that we'll be able to build AI systems that are just better at theoretical physics than ourselves. And so I'm very excited to talk to those systems about what they think. But I mean, I think...
[0:38:40] I think that is sort of the most natural, obvious sort of direction for me. I mean, we've sort of gone from systems like GBD2 and GBD3 that could kind of barely write text. And I remember when working on GBD3 thinking, oh, maybe like logical reasoning is just kind of a bottleneck here. These systems are really, really bad at it. And I think that like we're seeing them get better and better. And I think we're at the point where these models are kind of like quite good high school students at like
[0:39:09] Math and math contest problems, except they know way more. So they can get easy graduate level qualification exam questions, things like that. I think they're improving very rapidly. And so I think that this kind of thing is, to me, quite plausible and seems like the most obvious way that these systems could really dramatically move the needle on theoretical physics.
[0:39:37] Maybe this will turn out to be crazy and these systems will plateau before they reach that kind of level of performance. I look forward to asking Plot 7 for a solution to the hierarchy problem. Exactly, yeah. It's just the Anthropic principle, Jesse. You may be entirely correct.
[0:39:59] Well, I want to open things up to questions from the audience. And Guy, you have the microphone. And I'm sorry that we can't show you the faces of the people who are asking the questions. You can send it right now, yeah. You can also say your name also, just for . Hi, my name's Brent Penprez. I'm a professor at a liberal arts college, really interested in using AI for teaching in physics and astronomy. And I asked a question earlier about this, but I'm wondering if
[0:40:36] You guys are working kind of intentionally toward creating environments within the AI that are very easily used by teachers and maybe even one that interacts with groups of students rather than having a one-on-one type mode of conversation. Are there any thought about having AI as sort of a general purpose kind of TA, teacher, chat moderator kind of groups of people?
[0:41:04] Yeah, it's a great question. I mean, this is actually something that I know people who've been working on AI for a long time have used for a long time. I mean, like one of the people at Anthropic, even going back to GPT-3, used to sort of prompt it and set it up to sort of like teach them about various things. And I think, I mean, basically the answer is like, I think systems are getting better and better at this. You can sort of tell Claude like,
[0:41:32] Let's role play, and you'll be a teacher. You'll follow the Socratic method, and so you'll ask me a lot of questions about this thing, or you'll sometimes explain it. And I think that you can get pretty good results there, and I think you would get some moderation. Probably a place where the models just nail it at this point would be, say, teaching you a foreign language. I think the models are very, very fluent in most languages. And so if you want to be
[0:42:00] taught how to speak Spanish or Hebrew or something, then I think you can interact with the models very well in that way. With respect to physics, I think you'd go pretty far. But I mean, eventually, at least if you're working through physics problems, I think the models still make some mistakes. So you probably need some environment where that's OK. But I think with that caveat, you definitely can set these systems up to
[0:42:29] To do that. And then, of course, like students can just interact directly with the system and ask it questions. And again, as long as they're they're not going to be too fooled by by by subtle errors. Yeah, I think I think it's very possible. Kind of picking up on part of this question is kind of the the way that the chatbots are structured right now is very much geared towards one on one interaction where it's a human interacting with the bot going back and forth.
[0:42:59] Of course, we can imagine having more group-style learning. Are there any, like, thinking about, you know, having, well, how one would structure that if you had multiple students engaging in some way? I think that was part of the question, yeah. Yeah, yeah, yeah. Thinking about. Yeah, so, I mean, I think it's really, I think this is really just kind of like some mix of, like, a bit of software engineering and or a bit of prompting in the sense that.
[0:43:26] It's really just a question of what you're looking for. You can have a group chat where the AI just sees different people who talk in the group chat and responds. If you don't want the AI to respond unless it has something to say, you could set up some sort of prompt via, say, the API where you say, oh, these people have said these things. Do you have anything very useful to add or not?
[0:43:54] And if the model says, yes, I do, then you can have it respond. And if not, then you can sort of hold off. So I think there are a lot of opportunities like that that are pretty simple. I know like, and I'm not sure if they use it for group chat, but I mean like Slack, like the sort of app for kind of collaboration and discussion, like uses Claude for like sort of various AI features. And so I think a lot of these things are possible. And it's just a question of kind of tuning it for kind of like the workflow that you're interested in.
[0:44:34] My name is Sujit. I used to be a Fino a long time ago. But I have a question about AI safety, which I think you're interested in. In LLMs, it seems like, as opposed to traditional system design, the safety controls are inexplicably linked with the input mechanism, the data, it's all kind of mixed together.
[0:45:03] And this makes it vulnerable to really trivial like social engineering attacks, which are, you know, like these LLMs are very intelligent, but they're also very naive in some ways with comparison to a human. So what do you think is the future of this for having a more systematic way of having access controls or
[0:45:29] There are a couple of things. One is that we don't just think about safety in terms of the LLM by itself. We have a lot of other systems that sort of surround the LLM.
[0:45:56] And can intercede to change the behavior, change the outputs, add additional instructions, and generally detect problems. So there are all sorts of AI systems that, I don't know, social media uses for detecting problematic behavior. We have systems that are based on AI that look at prompts that are being submitted and look at completions.
[0:46:24] And I mean, it's up to us. There are all sorts of thresholds. There's lots and lots and lots of kind of product and trust and safety questions here about like exactly what do we do with this. But we do use other systems that are outside the LLM where we can also respond to changes more quickly than say updating the LLM for safety. I think the other is just sort of a question of like how, like I think there's kind of like a frontier, like a Pareto frontier of like how.
[0:46:52] How harmless is the system versus how helpful is the system in a certain sense? So we could train a system that will just basically follow any instruction to the best of its ability that you like. And we can train systems that I think are quite difficult to sort of jailbreak or attack, will therefore be quite uncooperative and suspicious and may refuse to do what various users say.
[0:47:19] There's really just sort of a trade-off there, and it's a question of kind of like where you want to live on that trade-off curve. I think that trade-off curve improves with model intelligence to a large extent, although there could be exceptions. So I think that's kind of the situation. There's sort of these external monitors, and then there's sort of the training of the system itself. And yeah, there are new techniques that we can employ to sort of improve things like automated red teaming to sort of generate lots of attacks and train systems.
[0:47:48] To avoid them. But that's kind of the current situation. It's possible there could be some interpretability-based tools or interventions that would give us better control or better insights into sort of how all of this works. But I think that's kind of the situation as it stands right now. I think if you have a system that's like extremely powerful and you're thinking mostly about misuse, there are also discussions of just sort of like...
[0:48:16] Either having detectors just for those narrow ranges, like, I don't know, bioweapons is one that we talked about a lot. So, like, maybe you have some very specific bioweapon targeted safety mechanisms. Maybe you even consider removing from the training data information that could be most relevant to that. So there's lots of things that kind of worth thinking about in those directions. My name is Sean Park. I'm an undergraduate student at Carnegie Mellon. I'm 12 years old, and I'm really excited about the whole advancement.
[0:48:51] Things that are coming along in AI. But I have this irrational fear of somewhat like what would happen if all the advancement or somewhat contribution that I'll make in the future. And one day Claude 10 comes along and all of a sudden it beats everything and all sorts of metrics. So how could one future-proof my career? And what would some advice be for young students?
[0:49:20] Yeah, it's a great question. I don't think there are any kind of easy answers, which is, I think, why, like, I mean, Anthropic also does research and is very connected to sort of like policymakers. I think that I don't really know how to future-proof one's career. I do think a really weird thing that's plausible with AI is that
[0:49:47] AI might be better at knowledge work than it is at, say, what one might think of historically as lower-skilled or lower-paid labor. It might be the case that we can build pretty good automated software engineers long before we can build automated plumbers or et cetera. So I think it's pretty unclear.
[0:50:13] The sort of optimistic vision is just that AI systems will be very empowering. People will be able to do more with less much more easily. You'll be able to build all sorts of spectacular things or learn spectacular things much more cheaply. And maybe also just the general point of view that if the GDP is growing, then there's some way of kind of like...
[0:50:42] Sharing these benefits. But I think it's really thorny, and I think this is a thing to take seriously. And I think it's one of the many reasons to be apprehensive and cautious and thoughtful about the whole process. One more thing is, what are some other things besides foundation models that excites you recently with robotics and figure and all these stuff coming along? What is the next thing for Anthropic?
[0:51:12] Yeah, I mean, I think robotics is pretty exciting. I think that sort of the combination of these AI systems with robotics could make it possible for these systems to do a lot of work and interesting tasks in the physical world. Yeah, I mean, I think for me, the bottlenecks that I'm most excited about and are focused on are kind of things related to honesty and self-correction and having systems like...
[0:51:42] Retrospectively identify their own mistakes and like backtrack and fix them and learn from them and things like that. Hi, Phil Harris, Energy Physics at MIT. So I'd like to, a lot of the problems we're dealing with these days, we're trying to approach with kind of machine learning in like energy physics domain is often trying to kind of extrapolate out of the training domain to kind of new.
[0:52:24] Base space, new parameter space. At some point, there's lots of interesting literature, but I'd like to understand how this context or how relevant this is to the work that's being done in Anthropic. The level of interest there is there because it's something that we're struggling with a lot in smaller networks we're dealing with.
[0:52:54] Yeah, it's a good question. So, I mean, we did some work on generalization with influence functions for LLMs. So, I mean, like, I just was kind of motivated by safety in the sense that, like, how these systems generalize might be a key question for, like, whether or not they're sort of safe to deploy in kind of the real world. I think we, like, learned a lot of interesting things there. I'm not sure if it's directly applicable. I mean...
[0:53:23] I think that like qualitatively the way generalization works seems to be sort of different for larger and larger models in the sense that like a canonical example is like larger models will like learn information in say Chinese and then know that information and be able to use it and apply it in English. Whereas smaller models will sort of be like mostly learning just the individual patterns of letters and words in
[0:53:52] In each language and there won't be very much generalization across languages. I mean, I don't really know anything. I mean, again, like I sort of fall back on like this field is very new and we don't really know much. I think that like a slogan that I often say, which I don't know is true, but like it sort of seems intuitive is like for these larger models, like extrapolation is just like interpolation in some sort of more abstract space. So like.
[0:54:19] When you ask how do humans actually extrapolate, I don't think that we have some sort of magic extrapolation capabilities that are special to the human brain. Most likely, I sort of just think we've seen a lot of patterns, and so there are patterns among patterns. And so it's like, well, I see this curve, and I know that I'm going to guess that it looks like it's kind of similar to...
[0:54:43] Like an exponential. And then I'm going to try to model it that way and sort of see how it works and look at the difference. And so I guess I tend to think of extrapolation as being something where if you have a lot of data on how other things have extrapolated, you can then learn how to extrapolate. So then I would guess that with very small models like
[0:55:02] It's really kind of up to the designer of the model, sort of like how to teach it, how to extrapolate. Maybe you need something like a larger model or meta-learning or learning over many different kinds of data sets. But I mean, like, but yeah, I really don't know like sort of what the best way to think about it is. I guess I'm always curious, like with state-of-the-art models, like you could just give them the data set and ask them either by writing code or by literally looking at the data points.
[0:55:31] Can they model the data themselves in some way? But probably what you get from that is more primitive than what a scientist can figure out. Other questions in the audience before we are coming up on the hour? Hi, I'm Georgi. So basically, my question is that recently there's been some work on designing automated theorem provers using, say, LLMs.
[0:56:12] Things that are essentially based on neural networks. I'm just curious, what are your thoughts about using things like LLMs, which are inherently prone to making approximations and not are inherently precise in thinking? And what are your thoughts about using these sort of things or things which require precision and rigor? Yeah. So I mean, I think that kind of work is pretty exciting. To me, I mean, basically what I see is that LLMs, as they get
[0:56:42] Better are becoming better at logical reasoning and it kind of like knowing what what mistakes they're making and fixing them. I think the question for sort of that direction of research is is in my mind really like kind of like who's going to get there first. So I think it's really powerful to like take natural language mathematics and translate it into some formal language and then check it automatically.
[0:57:10] And maybe translate back. And I think a lot of LLMs are really good at translating back and forth in these kinds of ways. And so you can get some sort of training loop where models like maybe go back and forth in this way and therefore become better at reasoning. On the other hand, I sort of see just the pure models themselves getting better quickly enough at math and reasoning that like doing it entirely in natural language seems very plausible. And I sort of expect to see a lot of progress on that.
[0:57:40] So for me, it's kind of like a question of will translating back and forth into formal languages end up giving us way, way better performance way more quickly? Or will we be able to kind of approach reasoning in a very general way in natural language? And I don't know the answer. I think both are kind of plausible.
[0:58:04] I will say, you know, I got interested in the physical sciences in part because the language was the language of mathematics and logic and reason. And whereas natural language seems all kind of slippery. But here, he seemed to be very comfortable, you know, talking about things in the natural language world as both a major base, but actually what you just described here was actually like maybe, but could be just as powerful as a formal system.
[0:58:29] Is this a change in your own mindset coming from your work, or have you always thought that way about the human interfaces to other scientific data sets or just human interfaces of the world? Yeah, I mean, I guess like I mean, I guess I think of a mathematical proof written in English as like a natural language object where like there aren't like extremely sharp rules about exactly how you can write the proof, et cetera. I mean, maybe maybe in some cases we
[0:58:58] I remember when I was learning proofs in geometry class, I think, in junior high or something, we actually did kind of have step-by-step, line-by-line things that you could and couldn't use. But I mean, when we in advanced math classes and in physics write solutions or proofs, we just use English. So I think I just mean that. I guess I've always enjoyed the ambiguity in a certain sense.
[0:59:26] One reason why I like physics and I'm a physicist rather than a mathematician is that I do think that often in physics, you make assumptions and there's some art to making the right assumption that's reasonable but allows you to then solve the problem. Then maybe you can go back and justify the assumption, but not everything is completely rigorous, but things can be very convincing and can really be very insightful without necessarily being rigorous.
[0:59:55] I don't know. I think I have always enjoyed that kind of ambiguity. And on that note, now I'm wondering, should I be excited or scared that we now have an AI agent that's able to work with this not quite rigorous but nevertheless insightful toolkit? So Jared, thanks so much for joining us here. Let's thank our speaker. Thanks for having me. Yeah, it's a lot of fun. And obviously, I...
[1:00:24] I miss physics, so it's good to be back.