[0:00:00] I definitely agree that continual learning is really the thing. It's really the thing that we're building, but I don't really think this is like a problem that's ignored and off the path of what we're doing currently. I think it is what we're working toward. What are like the other research areas within alignment that you're paying attention to or that you think are promising? A lot of the like longer term challenge with alignment is about generalization. What are the values that the model falls back on? What are the things that you need to figure out to be able to really make models work well in some of these other spaces? I come back to this reality of... Ako Payoki is the chief scientist of OpenAI.
[0:00:29] I think literally one of the most important people on the planet. And today on Unsupervised Learning, I got to ask him literally everything that I've been thinking about. And I know a bunch of people in the ecosystem have too. We talked a lot about model progress, what's required to make long-running agents work, as well as the really interesting work OpenAI has done in the AI for science world and the progress he sees in that over the next years. We talked a lot about how companies should be thinking about model building in this moment.
[0:00:54] When they should be doing reinforcement learning, how they should be thinking about the evolution of harnesses and the impact that will have. We hit on a lot of his really interesting research, including the work he's done around alignment, the work that OpenAI broadly has done around math competitions. And we also talked about this focusing moment at OpenAI and what it means for the research organization and how he runs his team. Literally just such an awesome opportunity to talk to someone who is driving so much of the change that has revolutionized the space in the world.
[0:01:20] I hope folks enjoy this wide ranging conversation as much as I did.
[0:01:26] I feel like you are the perfect person to talk to you about all the questions everyone has in the ecosystem. What's happening with model progress? A lot of companies are thinking about how they should be building things based on what's happening with the models. A lot of people at a societal level are thinking about the impact AI is going to have on science and broader society. And you've been at the forefront of the space for pretty much every generation of improvement these past years. And so really excited to have you on the podcast.
[0:01:52] Happy to be here. I think I'll start with one of the juiciest things you said, which is, you know, four months ago, I think you and the OpenAI team talked about aiming for a system with research-level interim capabilities by September of this year. So coming up, I think that's, what, six months from now? And then a more fully automated AI researcher by March 2028. And so I guess, you know, checking in four months later, how are you feeling about those timelines? Yeah, I think, you know, over the last months, I think, like,
[0:02:20] The change that really happened is we've seen this explosive growth of coding tools. It's an understatement. Yeah, we've definitely really kind of gone to a place in OpenAI where we use codecs for the majority of actual coding. And so I think for most people, the act of programming has changed quite a bit.
[0:02:48] So I definitely see this as a signal that something here is on track. The other kind of very interesting update over the last few months, to me, has been the progress on the math research capabilities, also the results we've seen in physics, in other fields. I think this kind of level of capability, this level of ability to provide insight, when combined with ability to access infrastructure, ability to use maybe more compute at test time, that's something that Coddick is using currently.
[0:03:19] And very strong improvement in general intelligence, which I also expect over the next couple months. Yeah, it's something we're still very much planning for and very focused on. And how do you know when you've gotten there? What's a workflow you might look to to say, hey, OK, I think we've got these research intern-level capabilities? The way I would distinguish a research intern from full automated researcher is the span of time that
[0:03:48] We would have it work mostly autonomously, or the specificity of the task that it has to be given. So I don't expect we'll have systems where you just tell them, oh, go improve your model capability. Go solve alignment. That would be nice. And they will do it. Not this year. I think we might get there at some point. But I think for more specific technical ideas, I have this particular idea how to improve the models, how to run this evaluation differently.
[0:04:15] I think we have the pieces that we mostly just need to put together. Andrej Karpathy released a pretty viral version of using some of these models to improve some of his obviously way less complex models than what you guys are building here. But did that feel like generally in the spirit of some of what these tools might look like? Yeah, I think it's in the spirit. Yeah, I mean, I expect it to look like a pretty continual evolution from kind of where Codex is now.
[0:04:45] I think towards a bit more autonomy running for a longer time. But yeah, I think we'll see a lot of this sort of application. I think in general we'll see more autonomous and higher compute use of these models for different things. You mentioned kind of like the math and physics side, and obviously you've had these really impressive breakthroughs in math on some interesting different kinds of competition problems.
[0:05:09] Maybe I think for our listeners, it intuitively makes sense how progress in coding directly translates to something like helping with AI research. How does math and physics progress also tie into this? The biggest draw that focusing on these math benchmarks has played for us as a general benchmark and a north star for how to improve the psychology. Math is very measurable. It's much easier to tell whether you've actually solved the math problem and whether you've
[0:05:39] Even produced a good piece of software. And also, it can get very hard, right? So you can have things where it's very definite whether you solve them, but it can be arbitrarily pretty much hard to actually solve them. I would say up until not too long ago, my perspective has been like, well, OK, our models are not maybe able to solve simple math problems. OK, our models are able to solve simple math problems, but are not able to solve IML-level problems. So clearly, there is just a gap in just this intelligence of these models that
[0:06:08] That is very measurable, very easy to run at. It's very clear what we need to do. And this has been kind of our raw start for reasoning models and so far. Now, of course, that is changing quite a bit, right? And we have kind of reached these milestones that we've been working towards. So like, yeah, IMO goals level, solving IMO problem six, and making forest into research level mathematics.
[0:06:37] From this point, I think there still is utility continuing to measure progress on this. I think there's also definitely transfer that you can get from getting better at mathematical reasoning to getting better at AI research. A lot of our best researchers are mathematicians we're training or from other theoretical fields. But definitely, we are very much...
[0:07:06] Changing how we think about, you know, these nine stars. And we are very focused on how the models, the next models that we're producing are actually useful in the real world. You know, useful, you know, especially for AI research, but also for other kind of economically valuable activities and for other fields of science. And especially maybe more applied sciences. And the reason for this shift is because we believe the models are now capable enough. Not as smart as people in all ways, but capable enough.
[0:07:35] To actually materially change the economy, change how things are done. And so, yeah, we feel a lot of urgency about that. In the early days, picking a domain like math that is so hard to solve, but then easily to verify whether you did it, like it's kind of the perfect place to get started. And I think code obviously shares a lot of attributes to that, you know, possible to check and verify and great for reinforcement learning. I think one question that a lot of people are thinking about is, okay, we've seen
[0:08:03] Reinforcement learning work incredibly well in these domains where you can verify it rather easily. A lot of valuable tasks in the world, medicine, law, finance, there's some level of the ability to do that, but it's certainly not to the same extent that math and code are. And so I think a lot of people are trying to figure out, are we going to see similar improvements? Obviously, code and math, the rates of improvement have been so astronomical and shocking. Yeah, I definitely expect so. I think an interesting duality that we think about a lot is
[0:08:32] For these more general tasks, for these tasks that are kind of harder to evaluate, they share a lot of commonalities with just longer horizon tasks. Because if you think about even a very well-specified math or coding problem, again, if it's something that you need to work on for a year, then even if it's very clear what the criteria of success are in the long term, what to do on your first day of working on it is a pretty open-ended problem.
[0:09:02] I kind of believe these difficulties coincide, and they're very clearly the next frontier for how these systems develop. And I think we've definitely seen very encouraging signs, both on just our ability to scale RL on these more general domains, and I think also we can scale RL methods that show a lot of promise. In these other domains, it feels like one of the hardest things to know is just what was success in a task, right?
[0:09:31] You can imagine, you know, there's going to be, you know, whatever the problems you are that are facing in coded math that are short-term tasks and then longer-term tasks, it feels like it will be amplified in a space that is, you know, outside of those, right? Where a short-term legal task or medical task may be harder to run thousands of iterations on, right, and figure out, you know, was that done correctly? And then those longer-term tasks, like, even harder. I'm curious, like, how you even conceptualize that research challenge. Like, what are the things that need to be, that you need to figure out to be able to really make?
[0:10:00] Models work well in some of these other spaces. Yeah, I think I come back to this duality of how do we make the models work for a very long time, and how do we teach them to evaluate partial progress. I mean, I think if you look at even outside of RL, where that sort of progress on longer horizons is coming from, as the models become more consistent from just pure supervision and pre-training,
[0:10:29] Again, some idea of what does a good partial artifact here look like. And so I think even if we weren't scaling RL very meaningfully, we would see an elongation of these horizons over time. Yeah, it's definitely a research challenge to figure out how to leverage these new ideas from RL and so forth to apply this to general domains, but I'm quite optimistic about that.
[0:10:52] Yeah, and it sounds like part of your mental model is like the models themselves being able to check progress at some sort of cadence that is, you know, reliable enough. From the outside, at least, it's not totally clear if we've seen like generalization in RL yet. It feels like we, yeah, clearly you seem to have some techniques that really optimize models around whatever we choose to focus on, but it's like almost feels like an older school version of ML of like one thing at a time. Is that like, you know, I guess would you agree with that characterization? And like, you know, how do you kind of see this current climate?
[0:11:22] We are buying a lot of compute, right? Because we don't... I mean, we still believe the beta or less, and we believe it more than ever to some degree. Yeah, we've seen new techniques and I think new ways to scale, but that is kind of the lens through which we've been doing things. Yeah, I think there is a certain amount of complexity that we need to grapple with and kind of everyone needs to grapple with because...
[0:11:53] You know, we're no longer really like purely building like, you know, brain in the sky that's completely isolated from the real world, right? Like if you actually, you know, if you want this model to do like medical research, if you want it to cure cancer at some point, it needs to like learn about the real world in some meaningful way, you know, maybe conduct some experiment and learn from its results. And for that, you need to figure out how to actually connect it, right? And that is going to involve something that is.
[0:12:21] Yeah, that goes in the direction you described. But I don't think that goes counter to actually scaling, finding and scaling the simple algorithms that we've been developing. I feel like I talk to a lot of companies. And one of the main questions everyone seems to be asking these days is, should we be doing our own reinforcement learning? Take an open source model and we have some data on a task that people do. We have evals because we know our domain pretty well. Is this something that makes sense for us to do?
[0:12:51] Should we just wait for the models to continue to get better at some of these things? What advice would you give for the many builders that listen to the podcast as they think through the extent to which they invest on the reinforcement learning side? JAREK WILKIEWICZ- Reinforcement learning definitely can be a very data efficient way to really improve the model at some sort of task. There is a much more data efficient way of learning that.
[0:13:12] We know, right? We're just learning in context, right? And this is maybe the most fundamental way that people teach these models. You just prompt them with examples, with instructions for what you want. I expect that learning is going to get much better over time. And so I think it definitely really matters that the models can adapt to your context. They can adapt to the kind of tasks you care about. So I think that will be very important. I'm not sure if replicating the kind of current URL pipelines is going to be like...
[0:13:41] The right way to go about it. But yeah, it's definitely a problem that we're thinking about. Yeah. So it's almost like, yeah, you still have to do the work. You still should figure out what the evals are that matter, gather the data, the examples. But it may just turn out in the future you're far better off just feeding that into this context than trying to do anything on your own model. Yeah. I think that's quite plausible. And I think that obviously people have seen the success of tools like Codex, which I know you've obviously been a key part of and wondered.
[0:14:09] You know, hey, do we need to build like our own kind of, you know, should we build our own harnesses or our own ways of using these things or, you know, for our own domains, whether it's like, you know, legal or finance or healthcare? Or do we kind of just like take the harnesses that the large models do and kind of use them within, you know, with the context that we have? Any thoughts around like that?
[0:14:32] The implementation of the harness shouldn't really be a limitation for a very long time. I think we'll be able to get much more general harnesses that people can use for all sorts of other domains. I mean, I think Codex is pretty good, actually, if you try using it for things beyond coding. MARK MIRCHANDANI- That's so interesting, a much more general harness being something that's almost adaptive to or just works across whatever the specific set of tools you have in your domain or specific set of things you want to expose to the model.
[0:14:59] Yeah, I mean, I think it's also worth thinking about what is the ultimate interface that we want to interact with the model with. So the models give some new affordances. They can build their own UIs. They can do things that people would find very time-consuming. But yeah, I definitely think there is also just a lot of space to enable the models to access the current interfaces that we use for people.
[0:15:29] We want to have AIs on Slack, for example, that are plugged into our context and are able to learn from it and able to utilize these existing things. So definitely, there is some meat in the middle here. But definitely, I believe long term, by default, the AI should meet you where you are. And if not,
[0:15:57] That would be because it has new abilities, not because it has limitations. MARK MIRCHANDANI- It's an interesting point that basically today it feels like these harnesses are so bespoke to certain environments. But over time, as you add more and more skills and tools and models can navigate across those effectively, it would just be a general way humans have. That makes a tremendous amount of sense. I guess I'm curious. Obviously, I'm sure every day you see crazy stuff on the research side.
[0:16:26] What are the milestones that are still meaningful to you? As you think about, it would be pretty crazy if I did a run one day and saw X or Y. What are the things you're paying most attention to? Yeah. I mean, at this point, it really is about research. It is about, can the model discover new things? Can it execute on a longer horizon research problem?
[0:16:54] It's almost like looking for some sort of insight that you're like, oh, someone on my team had come up with that. I've been pretty intrigued by it. Yeah. We've actually had some minor, but I think quite impactful, ideas come from GPT-5.2 Pro that we're using entirely. But I think it's still very, very small compared to what I expected it to be. Yeah. I mean, it seems like almost inevitably these models are going to get better. They will be used in research. They'll be used in science more generally. You're like one of the first people
[0:17:23] Interacting directly with these models as like research partners almost at this stage. Anything like you've learned around the right way to do that? Or do you think about like what a research organization, you know, as these models continue to get better might look like? Yeah, I think we're definitely kind of at a transition point where kind of the short term immediate quality of the model is about to be a quite determining factor for the pace of our research progress because the models are going to drive a lot of that.
[0:17:52] And so that definitely requires rewiring some intuitions about how to run a research organization. Normally, you try to not be too focused on immediate quality. You try to be much more focused on the longer term. I think we have a lot of very exciting stuff queued up that we are working towards. But I feel a lot of urgency to actually execute on it and to actually use these advances in model intelligence to
[0:18:21] Accelerate research on the AI and especially AI alignment. Yeah, it's such a fascinating point because I've heard you talk before about running a research organization. And I feel like in the past, it was like giving people the space to pursue a lot of things that weren't directly, hey, this is for a month or two months of progress. But it's like, what are the ideas that are really going to drive things forward?
[0:18:40] It makes total sense that we're in a time now where you're like, look, everything we do will be so much better if we just focus on this in the short term and make it better. It must be fascinating to navigate that and these maybe further off research ideas at the same time and running an organization. Yeah, it's definitely something we spend a lot of time on with Mark nowadays.
[0:19:01] Right now you have a ton of compute as a company, but obviously you have great scaling laws on the pre-training side. You have great scaling on the RL side. You have probably lots of experiments going on that have nothing to do with either of those vectors, but are interesting new ways. How do you even think about allocating compute across all of this stuff? Yeah, it's going to be very complicated, right? Because there's so many things that we need to do. One thing we've been, one kind of discipline we've started keeping is we...
[0:19:29] We try to make sure we just explicitly budget a large chunk of our compute to the most scalable methods, to the things that we believe are the most responsible for driving general model intelligence. And even if it's not the most efficient allocation of compute at all times, because if you're allocating so much compute to one experiment or one set of experiments, there are so many things you could accelerate for a little bit of that compute elsewhere. But I think it's easy to kind of...
[0:19:53] With all the interesting and important things that we're doing, I think it would be very easy to kind of portion all of it and not really end up doing the things that we believe are most important. You definitely want to understand the kind of empirical evidence. You definitely want to make sure your evaluations are in order and the kind of experimental rigor is there. And then you also want to apply some regulation based on like, OK, do we understand this method? Do we actually expect it to scale? Do we expect this is something you can actually build on in the future? Is this kind of a one-off? And I think based on that.
[0:20:22] I'll determine the price. MARK MANDELMANN- Yeah. It's so interesting. You probably find all the ways that you know you could improve things, but they feel maybe off a little bit to the side of where you think the overall arc of progress is. And so you end up leaving some of these low-hanging fruits to some extent. Because really, the most important thing is finding the future area direction, and then the scaling within that, and devoting compute toward that. Obviously, the place where we talked about Codex a lot, and the success of coding, and it feels like last year was the year of just incredible hill climbing on coding.
[0:20:51] I'm curious. Obviously, Codex has been a super successful product. In many ways, Anthropic was first to this market. Claude Code was a dominant product there. Reflecting on that, what do you make of the success Anthropic's had in this space? I think it's a matter of really focusing your product direction on where you believe the next application of the technology is.
[0:21:20] The kind of prioritization we've had on the product of the AI. I mean, we have been working on cutting price, but they have kind of been a secondary thing compared to our main priorities. An interesting thing is that it's not very reflective of the priorities of the research organization within OpenAI. I think given that we've kind of had this.
[0:21:42] You know, explosive success of ChatGPT. You know, ChatGPT, as it was, you know, I think ChatGPT is evolving quite a bit and it's going to evolve quite a bit. But as it was in 2023, right, it's this particular, you know, product that's maybe not, you know, I think it's definitely quite aligned with our vision of, like, where AGI is going. But, like, it's not really, like, the representative of, like, everything that it enables. And so the majority of, like, our work in research has been focused on, like, that future thing. And I think increasingly...
[0:22:11] It has decoupled from our short-term product strategies, right? Yeah, I'm very confident about the things we've been building and the things we are building on the research, on the model intelligence side. A lot of our prioritization and increased focus on the product side is about actually getting to deploy them and the belief that actually they are.
[0:22:36] The thing that really matters now. Yeah. And now it feels like, you know, clearly the whole company priority, you know, is so locked in and focused on this. And you've seen just incredible improvement in Codex in recent months. For all the developers that listen to the podcast, like, again, it's almost like hard to comprehend, like, what the world looks like as these models keep hill climbing on longer and longer tasks. Like, what do you think will look different in their lives? Or, like, how will they be using Codex in, you know, three, six months?
[0:23:02] I realize three months and six months are very different timelines in this world, but take whatever in-between point you'd like. I would expect just a gradual increase in just the level of autonomy you feel comfortable funding the model, just the fakeness of description that can work with the level of supervision it needs. I think we're not very far from models that can work autonomously for a couple of days.
[0:23:27] Maybe use quite a bit more computer than they're using now and produce much higher quality artifacts on their own. MARK MANDELMANN, Do you have a gut instinct on what, like, you know, there's always been this question of, like, will the world, you know, do you need that software engineering skill set to supervise these models running for a few days? Or, like, hey, does it turn out at some point of, like, being able to run for a while, you know, anybody can use coding agents and supervise them to some sort of output? I mean, I think definitely for, like, a lot of outputs, you already don't need much experience, right? I think still the distinction I would draw between, like, you know,
[0:23:56] An intern here and really an autonomous researcher or software engineer would be that if you want to build something bigger, you probably still want to apply supervision. You still want to have an overarching thing. You want to recognize what building blocks fit in and which don't. But yeah, I definitely expect that desired skill set to shift quite a bit over time towards this more general vision setting.
[0:24:23] You know, I guess on the research side, I feel like there's been, you know, maybe like a month ago, I feel like all anyone could talk about was continual learning. And there was just, you know, it was in the zeitgeist. There's all these neolabs starting to go focus on continual learning. Some folks left OpenAI to go focus on that. I'm curious, like, you know, I think maybe part behind that is a belief that, like, you know, R all alone, you know, either won't get us there or will get us to, like.
[0:24:47] Some level of very inefficient scaling, and it's kind of different than the way humans learn. I think even I've heard you say before that RL is still very different today than the way that humans learn. What's your take on that whole movement? Yeah, I am a little bit confused by it because in my mind, the whole kind of excitement that we've had, I mean, even if you look at the titles of the GPT-3 paper, it is that, oh,
[0:25:18] This class of models is actually capable of continual learning. It's capable of learning to learn in context. That has been really the driving force behind the excitement to scale these GPT models further. That has been the premise for why we really need to teach them with RLs, learn in context more efficiently. And so I definitely agree that continual learning is
[0:25:44] Really the thing, right? Like it's really the thing that we're building. But I don't really think this is like a problem that's like, oh, you know, it's kind of ignored and off the path of what we're doing currently. I think it is what we're working towards. Yeah. Like in your mind, this is like the single best path to get there is to continue to kind of scale the pre-training in a row. I think that is kind of how we've made the most progress on this problem so far. And, you know, I think there are, I think that there definitely are like more ideas, more steps. I think also a lot of improvement that will just come from scale. Yeah.
[0:26:12] And I guess we have a lot of folks listening that maybe have been able to do a lot of simpler things with these models. And then they try to do some of these more complex, I don't know, call it 100-step or longer-term tasks. And they're like, oh, the models don't work for this yet. And I think it's harder. You on the inside constantly feel this improvement. But for them, it feels like, hey, this is night and day away from being able to do this much longer thing.
[0:26:34] How do you kind of articulate to them, I guess, the set of things that need to be true for these like much longer steps to happen? Is it around kind of checking in more often as you were talking about before? Or I feel like there's just this belief among the research community of like, oh, all of these tasks will be solved in the next year or two. And then in the wild, a lot of people may be not totally rocking that like improvement line that we've been seeing. Yeah, I mean, definitely a lot of that prediction comes from just looking at like historical improvement lines, right?
[0:27:05] I think increasingly we can roughly see the shape here. I don't think a lot of this is about just the models becoming intelligent enough to recognize whether they're making progress. I think some of this is this very pragmatic work of are the models actually... Can they actually access all the context, all the files, all the infrastructure they need to do the work you want them to do? I remember in the past when we were discussing the roadmap.
[0:27:35] That we're taking with RL. I definitely feel like, OK, we just need to teach the model to kind of reason with its own tokens as kind of the priority. And then, of course, we'll need it to use tools to interact with the environment. At some point, we definitely need to teach it to see. At some point, we need to teach it to use a physical body. But yeah, I mean, I think we're definitely well into the stage where it really needs to interact with the environment and it really needs to see. And someday soon, we'll really talk about robots.
[0:28:02] Yeah. I mean, it does feel like a lot of the times when I hear people complain about, oh, a model can't do X or Y, it's like literally just because you haven't fed, you know, or connected to the systems or fed enough context into it. Actually, I do wonder if, like, context was universally applicable and able to flow into these things. Like, I feel like a lot of these problems would actually just be solved with today's models. You know, I want to talk about some of the AI for science stuff that you guys have been working on. And one thing in particular, you know, I feel like the coding stuff is something that everyone feels very viscerally, you know, in every company. They're using these tools and getting tons of productivity.
[0:28:31] You know, the math side, not all of us competed in IMO competitions and necessarily have as much of like an intuitive feel for some of these breakthroughs. And so one of them I know that was really interesting that you guys did is use some compelling work around like first proof, right? And I think these are like very different problems than kind of traditional competition math. I wonder if you just speak a little bit to that because I think it's just a space that our listeners might be less familiar with and kind of less familiar with understanding the implications of models being able to do pretty cool work here.
[0:28:57] Yeah, I mean, you know, I think, yeah, I was very excited with the first proof challenge. And, you know, I kind of view that particular one as kind of a benchmark, right? It's like a couple, you know, respected mathematicians, theoretical computer scientists releasing problems that, like, they believe are, like, representative of their day-to-day work but haven't been published anywhere so that we can really have our mouths take a crack. We were so excited about this challenge, but, you know, it was kind of dropped without any advance warning.
[0:29:27] It was like a week-long deadline to actually execute. We had a very exciting model of training at the time. And so one of the people in charge of training, James Lee, kind of started prompting that model just by hand. And yeah, and actually kind of seeing, oh, OK, it's actually solving these problems. It was really fascinating things to see.
[0:29:57] It's from a domain that I did my PhD in. And yeah, seeing the model kind of come up with these ideas, which I would be quite proud to come up with in a week or two, seeing it come up with them in like an hour or so, that was very, yeah, it's a very weird feeling, right? Like, yeah, I think like in the past, when I felt like that was like when watching our Dota bot like play just like very interesting Dota games infinitely, right? And it feels like just there's some sort of magic.
[0:30:26] One thing that's fascinating, too, is the idea that you're trading these models and it's like...
[0:30:52] You throw these problems in and it's like nobody knows how good will they be at solving them. And I think it must just be fascinating to see something that you know so well in a space that you spend so much time in and realizing, hey, probably the previous generation of models wouldn't have been able to do that. And you wouldn't have even thought necessarily that this was the benchmark to do. But it's just generally showing the general purpose capabilities and improvements of the models. I mean, it is at a stage where we need to like...
[0:31:18] Seek out experts in the particular domains to be able to tell us whether these particular proofs are correct or not. But it's still much easier to tell whether you've actually made progress than for something like even coding. Because sure, competitive programming, you can evaluate. But most programming is not competitive programming. And it's about, oh, are the abstractions right? Are you handling all the cases? I feel like there was this maybe common criticism a year ago, and I don't know if it's as strident now, that
[0:31:47] Okay, these models are like pattern matchers, but like you really want AI for science, like we're not going to get new ideas or like, you know, entirely novel things out of pattern matching. Feels like we continue to like chip away at that narrative. Are we getting closer to kind of fundamentally disproving that? I believe so, yeah. I mean, I think kind of on schedule we're starting to see like minor advancements, right? Like not huge things, right? Like a small idea here or there. I mean, maybe some like bigger papers in collaboration with.
[0:32:15] With scientists, right? But, you know, was AlphaZero a pattern match or AlphaGo a pattern match? Or, you know, our data bot a pattern match? Or, like, they did kind of come up with new strategies for their respective games. Yeah. It's funny that there's counter examples to it all the way back to, you know, 2016, 2017. Right, right? And, you know, and you can say, like, well, I guess you can always pull two flaws in that, which I think is interesting. Like, AlphaGo can be beaten with some strategy. Our data bots could have been beaten with some strategy.
[0:32:45] I think there will be a lot of definitions for a while of these models. But I think also they are able to discover new things because they have a lot of these capabilities. It's taken a couple of years to go from these very tiny game environments to this much more general scientific research. It requires going through a decent approximation of
[0:33:14] All human knowledge in the meantime. And, you know, learning all the human languages and so forth. But I think the basic principle is very similar. Yeah. You know, it's funny. I think, like, when you guys had these first proof results, I remember, like, the organizers said, you know, they were commenting on these AI solutions, and they were like, this feels like, you know, 19th century mathematics of, like, brute force, you know, computation-heavy approaches rather than these, like, elegant modern techniques, which I'm not sure is a feature or a bug of, you know, obviously the way these models work. But, like...
[0:33:43] Hearing that, does that concern you, excite you? It doesn't concern me. I mean, I think it's expected. I'm not sure. I fought for at least one of the problems. Actually, our models produced a pretty nice preview that was quite a bit shorter than the intended one. But I think in general, you would expect, yeah, these models can produce so much more reasoning in a short time than a person can, just in terms of the raw number of tokens or fonts. I don't expect that to be a long-term feature.
[0:34:12] It feels like there's so much momentum behind AI for science right now. And you mentioned, obviously, at some point you do have to connect these models to the physical world. And you guys released some cool stuff with Ginkgo and some of these other things you've been experimenting with. I'm sure you've thought a lot about AI for a bunch of different areas of science. As you've kind of dug into some of this stuff,
[0:34:32] Have you dealt with any intuition for, as you think about three years from now, the spaces of science where you're like, oh, there's going to be crazy progress there versus the ones that might prove a little more resistant to immediate change? A tempting answer would be that, oh, it's really about whether the things require some manual work, where the models are not quite plugged in. The ecosystem or the different laboratories will also kind of
[0:35:01] Within those STEM fields, obviously, you know, I feel like there's a question of, is it like an LLM with access to the physical world? Or you've obviously had companies that have been started specifically around these domains, right? Like an isomorphic in biology or periodic in material sciences or physical intelligence and robotics. What's your kind of gut instinct on the extent to which it makes sense to pursue some of these things like independent with different model architectures versus like all within the context of one place?
[0:35:30] Yeah, I think it's kind of similar to my answer with the UI for codecs, which I would build around the capabilities of a technology and not around its limitations so much. So you definitely, if you have something that can suddenly design a huge amount of interesting chemical or biological experiments, yeah, it makes sense to build labs that enable that.
[0:35:55] I think if we did get to a place where the model is very capable of designing high-quality experiments, it would also make sense to have it work with humans in a loop. We shouldn't think of it as always either you automate it fully and you have this fun thing using some tools on the side. We will get to a world where it's just very natural to be collaborating with.
[0:36:13] AI scientists that are working hard on a problem. Yeah. It's so interesting. It's almost like a different vision. It's like one world where this works is like, hey, you just train a model to basically run these end-to-end tasks and be the automated biologist or chemist or whatever it is. And there's another one which is like, well, you're building really tools to both propose, run, kind of work in tandem with a bunch of human researchers. I mean, I wouldn't necessarily categorize it as, I mean, of course there are tools in some sense, but I think like,
[0:36:42] We'll get to a point where they're driving a lot of the design and ideation for the whole process. With an LLM architecture, but just being able to figure out the right kinds of experiments to run and then actually design it. When it comes to different architectures, for sure, natural language reasoning, the kind of things that we're prioritizing, that gives you a lot of generality. There are things that are...
[0:37:08] You want to train a different model to model. I think even if you want to create a very good model, I don't think models are the most efficient way to go about this, although they might result in the best model eventually. I think it's similar for protein folding or other tasks of this kind. So you think it makes sense to have some independent efforts around that, but obviously the
[0:37:33] That will end up being paired with a core, really good researcher, large language model that is helping drive a bunch of this stuff. I want to also make sure just to talk about AI safety, because I think that's an area that you've done a lot of really pioneering work on. And I'm not sure all our listeners will be familiar with, you actually did some really interesting work across the labs, right? And were focused on chain of thought monitoring. And so maybe to start, just tell us a little bit about that work and what you found. Yeah, so this is...
[0:38:02] A realization that actually we had around the time we actually saw like the first reasoning models of kind of the current crop. We realized that like, okay, like, well, this works, right? And we were pretty, you know, we were thinking a lot about what this means. We kind of were like, okay, like probably that worked.
[0:38:22] We really changed this over the next, I don't know, year or two or three. We were thinking what this means for safety and for our ability to understand what these models are doing. And we realized that because of the way we train these models and because we don't supervise the reasoning process directly, it's not like ChartGPT is trained to kind of be polite and nice. It always tells me I have great ideas. Yeah. Well, that's a separate issue.
[0:38:50] Even assuming it's aligned exactly in the way we would want it to, which is definitely not sycophantic, it's still kind of not going to be... There are still some things it's not going to reveal about its motivations and time because maybe it would be unsafe or maybe it would be unkind. Or maybe it's actually not aligned the way we think, but it wants to hide that, right?
[0:39:17] And the way we train the reasoning models, the chain of thought doesn't have any of that. It's not optimized to be in any particular way because it's just not directly graded. It's only graded in how it relates to producing a high-quality output. And really, this is actually a very powerful paradigm for being able to interpret what the model is doing. It's actually not a very different idea from...
[0:39:44] Mechanistic interoperability, right? Because in mechanistic interoperability, the idea is, again, you kind of have this model. You have these activations of the model that are not directly supervised to predict any label. They're kind of indirectly supervised. But the model has never been trained with any sort of inspection of these activations. And so these activations might reveal something about these inner workings.
[0:40:08] But the big advantage of the chains of thought is that by default, they are in English. And so it's so much easier to understand what is going on, especially as the concepts get more advanced. And the other interesting thing is we were just talking about how we believe in the future where we go, well, these models work for a very long time. They work autonomously. And so there is much more of this reasoning.
[0:40:33] And so if this is a big axis of how the capability of these models increases, our ability to supervise them will scale commensurately. Yeah, this really comes down to this principle, though, that you're not supposed to supervise the chain of thought. And so this is actually something, when we were using the OWN preview model, we made this decision to hide the chains of thought. Yeah, I remember.
[0:41:03] For me, that was the primary motivation. That was the reason I didn't really even want to consider releasing it in different ways. There definitely was a bit of internal discussion about this, but the reason I felt very strongly that we should just hide it is because of this. Then there was this other concern that I didn't initially think about, but I think was also very valid of, well, these bottles can be distilled to some extent, blah, blah, blah. And that's definitely also been a big factor here. But I actually think that this
[0:41:33] You know, allowing the models some sort of private space. Oh, and by the way, like, why do I think it's important that we don't, like, you know, show these chains of thought in product? You know, if I'm saying, like, the important thing is not to supervise them during training. Well, I think if we did show them product, if we, like, established a paradigm where, like, oh, you just show these chains of thought in product, eventually you'd kind of have to train them, right? Like, you have to train them for the same reasons you have to train, like, whatever models you ship.
[0:42:01] And I just think that would be like- We might not all want to know what the chain of thought our model has that gets to a response for- Right. I mean, you know, I think it would be useful to some extent. And we are trying to capture most of that value, you know, either with like chain of thought summaries, which I think are kind of like a little bit of a stopgap. I think the longer term solution here is having the model actually talk to you in real time, which, you know, the latest version of Codex kind of do, the latest version of the reasoning GPT models kind of do, but I think that will get much better. Yeah, but-
[0:42:31] Yeah, I think there's something very exciting here about just not having the training signal fight against us. Yes, because I think if you want to be able to understand what the model does in the long term, but you're scaling a method that is going directly against that, you're probably not going to have a good time, unless the others have a better lesson. And so this decoupling, I think, is a very...
[0:43:00] It's an idea that gives me a lot of hope for our ability to at least understand how these models' motivations and generalization evolve as they get better and as they work for longer. Yeah, I don't think it's a complete solution to AI safety and alignment by a long shot. I think it's just another tool in our toolbox.
[0:43:20] But I am hopeful that building our toolbox with technical tools like this, we can actually continue chipping away at the fundamental problems here. Yeah. It seems like almost over the medium term, it's something that's going to be incredibly helpful, probably not the catch-all solution for long-term alignment. Yeah. I mean, I think it's a tool that can help us understand. I think it's actually very useful to build understanding of long-term alignment. So for example, there has been this very exciting work from
[0:43:48] From a planning collaboration with other labs on model scheming, where they investigate depending on what environment you put the model in, how you train it. Is it prone to start having hidden objectives that it pursues? And what enables that whole line of work is chain of thought monitoring. It's this notion of, oh, you can actually inspect what the most motivations are. And I think from that, that might take us in a completely different direction in terms of
[0:44:17] Yeah, it's almost like foundational for any further area of research. What are the other research areas within alignment that you're paying attention to or that you think are promising areas to focus on?
[0:44:44] A lot of the longer-term challenge with alignment is about generalization. We can train our models to do well, or at least mostly to some extent. We can mostly control their behavior in the things that are in distribution that we train for.
[0:45:04] But the things that are worrisome is like, well, what happens when the model is asked to do something very, very different, or it finds itself in a very different situation, or it's much smarter than it ever was before, and it has all these capabilities? We haven't really thought about how to train for it. And so I think the study of this longer-term value alignment is really a study of generalization. What are the values that the model falls back on? One line of research I am very excited about here, and something that we're
[0:45:34] I will speak to that.
[0:46:02] The longer term challenges of like value alignment, right? Or like what happens when you have very smart models. The way my thinking about the problem has evolved over the past few years is definitely kind of gone from, you know, always this like very nebulous problem that like it's just like very hard to even grapple with or define to like, oh, you know, I think we can actually make progress at it by very concrete technical solutions and technical insights. And this is why we've really been viewing alignment as like just a core part of research and really...
[0:46:34] Making sure that we are designing our reasoning models, thinking about this, and we are conducting our alignment research with these reasoning models in mind and so forth. I think my general belief that there is a research path here that actually gets us to an extremely happy world has increased quite a lot. At the same time, I think
[0:47:03] My timelines to very capable models have definitely decreased a lot. I think we're not that far, right? And again, I don't think these are models that are smarter than humans in some of the ways, but I think these are models that are just very transformative. And so I'm quite optimistic that we can keep a good grip on how we're doing on the alignment problem, how to roughly evaluate the risks of our models or the problems with them. But I do think we have to be, as an industry, really prepared to take trade-offs.
[0:47:32] Possibly slow down development, depending on what we see. It's already interesting to see a lot of this work happening across the major labs. The fact that you did this in collaboration with, I think, Anthropic and DeepMind. And it seems like, has that just come up organically? Or I imagine, is there a lot of alignment talk between the major players, given, I guess, the three of you are really at the forefront of all this?
[0:47:54] There's definitely some. I mean, there's definitely like shared interest in these topics, yeah. I want to shift a little bit to going inside OpenAI. I feel like no company probably or the world has been more interested in over the last two, three years. And, you know, I think particularly what it's like to run a research organization. You know, we talked a little bit about this previously, but you talked before about how it's, you know, important.
[0:48:14] Part of your job is giving researchers, you know, to kind of have comfort and space to, you know, almost be cave dwellers, right? And think about what the models will look like in a few years. You know, we were kind of alluding to it earlier. We're also in a time where it feels like there's just massive competitive race. And, you know, it's certainly, you know, everyone's going really gung-ho on these coding models. I'm wondering, like, how do you actually operationalize this balance today? And, you know, anything you've kind of changed in your thinking, you know, overseeing this organization around the right way to do this?
[0:48:44] I focus on high-quality experiments, recognizing are we actually making progress, being honest with ourselves, and promoting honesty about the results. I don't think that has changed. And even though our work will evolve a lot, I believe we still have quite a lot of work left to do. And so I don't think it's like, oh, we need to wrap up all our projects very quickly. So yeah, I don't think those fundamentals change.
[0:49:11] I think what that's changed is a level of urgency to really kind of bring some of these things that we think are most promising to fruition. And then obviously, you know, I feel like there's been, you know, some very public internal moments of OpenAI over the years. You've been here for a long time. As you kind of reflect back, like, what were some of the difficult decisions that you guys made that maybe were like 51-49 that really, you know, defined the company?
[0:49:34] Do you think back on the movie of the last seven, eight years of your life, the key moments that kind of stick out to you? Well, yeah. I mean, there's certainly a number of dramatic moments like this. I think the ways the company underwent the most change is not really just snap changes, snap decisions, but more just shifts in how it operates. I would say OpenAI has gone through a couple of phases. When I joined at the start of 2017, very much kind of...
[0:50:04] It felt like a very academic lab pursuing a lot of different ideas, not so scaling-pilled in practice. I think that was the first big change with the data product with GPT. We've moved to, okay, we actually are going to have to buy big computers. We're actually going to have to scale things. We're going to have to develop the science of scaling, and we'll have to develop the infrastructure for it. That started the second phase of, okay, now we're scaling.
[0:50:33] We're still going to pursue a lot of these basic research ideas, but we are going to evaluate them. Are they scalable? Then there was this interesting period that I talked about earlier where you kind of have chat GPT as this big thing. Complete surprise, right? Yeah. I mean, I thought it would look a little bit differently. I was actually surprised that text models are actually kind of the first thing. I thought we would be in a world where, like,
[0:51:05] It's more the kind of like video style users of AI are kind of like the first big thing to take off. And we'll have to trade off pursuing the kind of longer term text-based research. So yeah. But I think definitely we anticipated that this sort of tension would arise, right? Where you have a thing that is kind of popular now, but you believe it's going to evolve quite a lot before you get to where you're going.
[0:51:34] And so I think that's kind of the phase we've been in for a while. And yeah, I think now we're like, well, yeah, I mean, we believe we are kind of starting to be in this phase where we're actually deploying AGI or, you know, deploying models that are actually very economic transformative.
[0:51:53] No, it certainly seems that way. Well, I guess we always like to end our interviews with a standard set of quickfire questions, which are basically me just stuffing all my overly broad questions I couldn't fit anywhere else. So you'll shamelessly indulge me. I guess to kick it off, what's one thing you've changed your mind on in the AI world in the last year? Yeah, I mean, I think it's really starting to reconcile this tension between...
[0:52:18] The AI that you build ultimately is something that affects the world. But until you get pretty close, it's a pretty theoretical thing that you're just training and developing algorithms for. And so recognizing that, OK, now we really need to make a lot of progress and focus on how actually we're deploying this technology in a while. This is definitely something I've been thinking about.
[0:52:49] A lot lately. Yeah, it's interesting. Basically, like, you know, outside of ChatGPT, it was almost like more in the abstract or research hill climbing, you know, with some usage in the real world. And then in this last year, we've obviously seen, you know, primarily via coding agents, just, you know, it trickling, you know, in a pretty massive way. Yeah, I think, I believe ChatGPT is kind of going in the same direction as like the coding models for like, it's actually going to be something, you know, very useful. It's going to be something that's like a meaningful.
[0:53:17] Part of people's lives. When you say going in the same way, you mean just like executing longer-term tasks or more like, you know, the... I think that's part of it, right? But also just, you know, coming to become like a dependable, trustworthy assistant or compliant. Yeah. I mean, it's amazing to watch the way younger people use chat. I'd argue it's already pretty much there for the way a lot of folks in high school and college and, you know, seem increasingly, you know, comfortable using it.
[0:53:44] You know, I wouldn't be a shameless podcaster if I didn't ask a top researcher, you know, timelines for a few things. I think particularly actually interesting is the stuff outside of the core alum world. And so I think there's a lot of buzz around robotics these days. Do you have any, like, I mean, obviously it's hard to pinpoint, like, a moment robotics, quote, works. But I think, you know, whether it's finding scaling laws or finding some sort of, like, ChachiBT-esque moment for robotics. Yeah, I mean, I definitely think there are, like, very promising algorithmic ideas there that I believe are going to work that are...
[0:54:11] Not too dissimilar from the LLM space of ideas. So I'm quite optimistic about timelines there. Although I do think they're longer than the virtual AI. Obviously, I'm sure you think a lot about, because you're always thinking about the next frontier for what these models can do, just the impact on society as a whole. As you think about this pace of continued model improvement, what's maybe one thing that you think we're underthinking right now as a society in terms of the impact of these models? Yeah, I think...
[0:54:40] Getting to a point where so much intellectual work can be automated, I think, comes with pretty big problems that I don't think have obvious solutions. One, actually, is the question of jobs and concentration of wealth, and I suspect this requires real policymaker involvement. Yeah, I've heard some kind of optimistic takes on how this is resolved, but I think at that...
[0:55:08] At a fundamental level, it does seem like some things that used to be very valuable, used to cost a lot, and used to provide something, now can be done pretty cheaply. And in the long term, it should be a good thing. But I think it can happen quite quickly. And there is a related question of, if you actually have an automated research laboratory or an automated company that can do so many things,
[0:55:39] It can be controlled by a very small number of people. It can do a lot. And this gets even more crazy when you have robots, but you don't need to have robots. And I think figuring out what does governance of such things look like? What are these organizations that are so powerful and yet maybe made of only a couple of people? How to think about these things, I think, is a new question we have to grapple with as a society. I'm going to give other new questions. One thing that's very top of mind for me, I recently had a kid, and I've been thinking a lot about, like, you know,
[0:56:08] What is his life going to look like in 10 years? You're really close to this stuff. How has your work on AI changed the way you think about the way in which this next generation should be raised? A task for all of us is to build the AI, build the world in a way where at the end of the day, humans have the agency. Humans set the direction. And maybe a lot of the technical challenges that we cherish right now will become more of a
[0:56:38] You know, past time, that's something that you really kind of like needs to do in order to make progress. And the challenges will be more and like figuring out like what are the things that are important? What are the things we should go do? You know, I think that that will still be, you know, I think, you know, in that world, like people can end up with, you know, more things to do and definitely more exciting things to do. And, you know, I think you still want like to have an understanding of, you know, of like, you know, some understanding of like, you know.
[0:57:07] Technology, all the kind of basic education, however you want to acquire it, for the sake of being able to think about these problems. MARK MANDELMANN Well, this has been fascinating. I really appreciate you sitting down and talking about so many different things. I want to make sure to leave the last word to you. Anything you want to point our listeners to, whether it's research you're doing or products you're excited about, or really anything you'd like to plug the floor is yours. I'm sure there's tons of threads people want to pull out of this conversation.
[0:57:36] Problems we just discussed, right? And also the questions around alignment, monitorability. I think those are growing to be very urgent challenges. And I don't think there are challenges only for AI researchers, right? I think there are challenges for policymakers, but also just things we have to think through as a society. And yeah, I'm happy to see some discourse starting to arise. And I think we need more of it.
[0:58:05] Yeah. Well, I thought I could talk to you for hours more, but I'd be doing the world a great disservice by keeping you from your actual work of continuing to improve these models. Thank you so much for doing this. This was a ton of fun. I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project. In addition to my day job as an investor at Redpoint, but our ability to get these incredible guests on,
[0:58:34] Really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.