[0:00:00] Patrick, over to you. All right. Thank you, Graham. And thank you, Sam, for being with us. Last year, I actually interviewed Sam Bankman-Fried, which was clearly the wrong Sam to be interviewing. So it's good to correct it this year with the right Sam. So we'll start out with the topic on everyone's mind. So when will we all get our world coin? I think.
[0:00:30] If you're not in the US, you can get one in a few weeks. If you're in the US, maybe never. I don't know. It depends how truly dedicated the US government is to banning crypto. So WorldCoin launched around a year ago or so? Actually, it's been in beta for maybe a year, but it will go live relatively soon outside of the US. And in the US, you just won't be able to do it maybe ever. I don't know. All right. Which is a crazy thing to think about.
[0:00:58] You know, think whatever you want about crypto and the ups and downs. But the fact that the US is the worst country in the world to have a crypto company in, or you just can't offer it at all, is sort of a big statement, like historically big statement. Yes, yes, it is. Yeah, it's hard to think of the last technology for which that was the case. Maybe the Europeans are supposed to do this, not us. Yes, supersonic air travel or something. Something like that.
[0:01:29] All right. So I presume almost everyone in the audience is a ChatGPT user. What is your most common ChatGPT use case? Like not when you're testing something, just you actually want to get, you know, where ChatGPT is purely an instrumental tool for you. Summarization by far. I've gotten like, I don't know how I would still keep, I wouldn't still keep up with email and Slack without it.
[0:01:55] But posting a bunch of email or Slack messages into it hopefully will build some better plugins for this over time. But even doing it the manual way works pretty well. Have any plugins become part of your workflow yet? Browsing and the code interpreter once in a while. But honestly, for me personally, they have not yet tipped into a daily habit. So obviously, it seems very plausible that we're on a trend of
[0:02:25] Super linear, you know, realized returns in terms of the capabilities of these models. But who knows? Maybe we'll asymptote soon. Not saying it's likely, but it's at least a possibility. If we end up in the world where we asymptote soon, what do you think kind of ex post we will, you know, look back on the reason as having been too little data, not enough compute? What's the most likely bottleneck? So, yeah, look, I really don't think it's going to happen. But if it does, I think it would be.
[0:02:57] I think it'd be that there's something fundamental about our current architectures that limits us in a way that is not obvious today. So maybe we can never get the systems to be very robust, and thus we can never get them to reliably stay on track and reason and understand when they're making mistakes. And thus they can't really figure out new knowledge very well at scale. But I don't have any reason to believe that's the case. And some people have made the case that
[0:03:27] We're now training on kind of order of all of the internet's tokens and you can't grow that, you know, another two orders of magnitude. I guess you could counter with, yeah, with the synthetic data generation. Do you think data bottlenecks matter at all? I think you just touched on it. Like as long as you can get to like over the synthetic data event horizon where the model is smart enough to make good synthetic data, I think it should be all right.
[0:03:55] We will need new techniques for sure. I don't want to pretend otherwise in any way. But naive plan of just scale up a transformer with pre-training tokens from the internet, that will run out. But that's not the plan. So one of the big breakthroughs in, I guess, GPT 3.5 and 4 is RLHF. If you, Sam, personally sat down and did RL...
[0:04:25] All of the RLHF, would the model be significantly smarter? Does it matter who's giving the feedback? I think we are getting to the phase where you really do want smart experts giving the feedback in certain areas to get the model to be as generally smart as possible. So will this create a crazy battle for the smartest grad students? I think so.
[0:04:53] I don't know how crazy of a battle it'll be because there's like a lot of smart grad students in the world, but smart grad students I think will be very important. And how many, like how should one think about the question of how many smart grad students one needs? Like is one enough or do you need like 10,000? We're studying this right now. We really don't know. Like how much leverage you can get out of one really smart person where kind of the model can help and the model can like do some of its own RLEing. Like this is...
[0:05:22] We're deeply interested in this, but it's a very open question. Should nuclear secrets be classified? Probably, yes. I don't know how effective we've been there. I think the reason that we have avoided nuclear disaster is not solely attributable to the fact that we classified the secrets, but that we did something. We did a number of smart things. And we got lucky. You know, the amount of energy needed, at least...
[0:05:53] for a long time was like huge and sort of required the power of nations. And we made the IAEA, which I think was a good decision on the whole and a whole bunch of other things, too. So like, yeah, I think probably anything you can do there to increase probability of a good outcome is worth doing. Classification of nuclear secrets probably helps. Doesn't seem to make a lot of sense to not classify it. On the other hand, I don't think it'd be a complete solution.
[0:06:22] What's the biggest lesson we should take from our experience with nuclear non-proliferation in the broadest sense as we think about all the AI safety considerations that are now central? So first of all, I think it is always a mistake to draw too much inspiration from a previous technology. Everybody wants the analogy. Everybody wants to say, oh, it's like this, or it's like that, or we did it like this, so we're going to do it like that again. And the shape of every technology is just different. However, I think nuclear materials and
[0:06:50] AI supercomputers do have some similarities, and this is a place where we can draw more than usual parallels and inspiration, but I would caution people to overlearn the lessons of the last thing. I think something like an IAEA for AI, and I realize how naive this sounds and how difficult it is to do, but getting a global regulatory agency that everybody signs up for, for extremely powerful
[0:07:19] AI training systems seems to me like a very important thing to do. So I think that's one lesson we could learn. And if it's established, it exists tomorrow, what's the first thing it should do? Any systems over... The easiest way to implement this would be a compute threshold. The best way to implement this would be a capabilities threshold, but that's harder to measure. Any system over that threshold, I think, should submit to audits.
[0:07:48] Full visibility to that organization. Be required to pass certain safety evals before releasing systems. That would be the first thing. And some people on the, I don't know how one would characterize the side, but let's say the more pugilistic side would say, that all sounds great, but China is not going to do that.
[0:08:15] And therefore, we'll just be handicapping ourselves. And consequently, it's a less good idea than it seems on the surface. There are a lot of people who make incredibly strong statements about what China will or won't do that have never been to China, never spoken to someone who has worked on diplomacy with China in the past, really kind of know nothing about complex, high stakes international relations.
[0:08:42] I think it is obviously super hard, but also I think no one wants to destroy the whole world, and there is reason to at least try here. Also, I think there's like a bunch of unusual things. This is why it's dangerous to learn from any technological analogy of the past. There's a bunch of unusual things here. There's, of course, the energy signature and the amount of energy needed, but there aren't that many people that are making the most capable GPUs, and you could require them all to put in some sort of monitoring.
[0:09:12] So one of the big surprises for me this year has been the progress in the open source models. And it's been this kind of frenzy pace the last 60 days or something. How good do you think the open source models will be in a year, say? Well, actually, I'll just ask that first. Yeah, good. I mean, I think there's going to be two.
[0:09:41] Thrusts to development here. There will be the hyperscalers best closed source models. And there will be the progress that the open source community makes. And it'll be a few years behind or whatever, a couple years behind maybe. But I think we're going to be in a world where there's very capable open source models. And people use them for all sorts of things. And the creative power of the whole community is going to impress all of us. And then there will be the frontier of what.
[0:10:10] David Luan made the case to me that the set of economically useful activities is clearly a subset of all possible activities and that
[0:10:40] Pretty good models might be sufficient to address most of that first set. And so maybe the super large models will be very scientifically interesting, and maybe you'll need them to do things like generate further AI progress or something. But for most practical day-to-day cases, maybe an open source model will be sufficient. How likely do you think that future is?
[0:11:01] I think for many super economically valuable things, yes, the smaller open source model will be sufficient. But you actually just touched on the one thing I would say, which is like help us invent super intelligence. That's a pretty economically valuable activity. So is like cure all cancer or discover new physics or whatever else. And that will happen with the biggest models first. Should Facebook open source Lama? At this point, probably.
[0:11:26] Should they adopt a strategy of open sourcing their foundation models slash LLMs or just LLM in particular? I think Facebook's AI strategy has been confused at best for some time, but I think they're now getting really serious and they have extremely competent people and I expect a more cohesive strategy from them soon. I think they'll be a surprising new real player here.
[0:11:56] Is there any new discovery that could be made that would meaningfully change your PDoom probability, either by elevating it or by decreasing it? Yeah, I mean, a lot. I think that's most of the new work between here and superintelligence will move that probability up or down. Okay, is there anything you're particularly paying attention to? Any kind of contingent fact you'd love to know? If we could...
[0:12:28] So first of all, I don't think RLHF is the right long-term solution. I don't think we can rely on that. I think it's helpful. It certainly makes these models easier to use. But what you really want is to understand what's happening in the internals of the models and be able to align that, you know, say like exactly here is the circuit or the set of artificial neurons where something is happening and tweak that in a way that then gives a robust change to the performance of the model.
[0:12:57] And if we can get that... The mechanistic interpretability stuff. Yeah. Well, that and then beyond. There's a whole bunch of things beyond that, but that direction. If we can get that to reliably work, I think everybody's PDoom would go down a lot. And do you think sufficient interpretability work is happening? No. Why not? A lot of people say they're very worried about AI safety, so it seems superficially surprising.
[0:13:25] Most of the people who say they're really worried about AI safety just seem to spend their days on Twitter saying they're really worried about AI safety or any number of other things. There are people who are very worried about AI safety and doing great technical work there. But we need a lot more of them. We're certainly shifting a lot more effort inside, a lot more technical people inside OpenAI to work on that. But what the world needs is not more...
[0:13:54] AI safety people who like post on Twitter and write long philosophical diatribes. It needs more people who are like going to do the technical work to make these systems safe and reliably aligned. And I think that's happening. It'll be a combination of people that have good ML researchers shifting their focus and new people coming into the field. A lot of people on this call are active philanthropists and most of them don't post very much on Twitter.
[0:14:23] They hear this exchange like, oh, maybe I should help fund something in the interpretability space. If they're having that thought, what's the next step? One strategy that I think has not happened enough is grants to single people or small groups of people that are very technical, that want to push forward a technical solution and are maybe in grad school or just out or an undergrad or whatever. I think that is well worth trying. They need access.
[0:14:53] Two fairly powerful models and OpenAI is trying to figure out programs to support independent alignment researchers. But I think giving those people financial support is a very good step. To what degree, in addition to being somewhat capital bottlenecked, is the field skill bottlenecked, where there are people who maybe have the intrinsic characteristics required but don't have the four years of learning or something like that that are also productive for their being effective?
[0:15:21] I think if you have a smart person who has learned to do good research and has the right sort of mindset, it only takes about six months to take a smart physics researcher and make them into a productive AI researcher. So we don't have enough talent in the field yet, but it's coming soon. We have a program at OpenAI that does exactly this, and I'm astonished how well it works. It seems that pretty soon we'll have agents that you can converse with.
[0:15:51] In very natural form, low latency, full duplex, you can interrupt them like the whole thing. And obviously we're already seeing with things like character and replica that even nascent products in this direction are getting pretty remarkable traction. It seems to me that these are likely to be a huge deal and maybe we're substantially underestimating it. Again, especially once you can converse through voice. A, do you think that's right? And then B, if that's right,
[0:16:21] What do you think the likely consequences are? Yeah, I do think it's right for sure. A thing someone said to me recently that has stuck with them is that they're pretty sure their kids are going to have more AI friends than human friends. And I don't know what the consequences are going to be. One thing that I think is important is that we establish a societal norm soon that you know if you're talking to an AI or a human or sort of like weird AI-assisted human situation.
[0:16:52] But people seem to have a hard time kind of differentiating in their head. Even with these very early week systems like Replica that you mentioned, it's whatever the circuits in our brain are that crave social interaction seem satisfiable with like, for some people in some cases, with an AI friend. And so figuring out how to handle that I think is tricky. Someone recently told me that a frequent topic of discussion on the Replica subreddit is
[0:17:23] How to handle the emotional challenges and trauma of upgrades to the replica models because suddenly your friend becomes somewhat lobotomized or at least a somewhat different person. And presumably these interlocutors all know that the replica is in fact an AI, but somehow to your point, our emotional response doesn't necessarily seem all that different. What I think we're heading to is a society like...
[0:17:51] I think what most people assume that we're heading to is a society with one sort of supreme being, super intelligence, you know, floating in the sky or whatever. And I think what we're heading to, which is sort of less scary, but in some senses still as weird, is a society that just has a lot of AIs integrated along with humans. And there have been movies about this for like a long time. Like there's like, you know, C-3PO or whatever you want in Star Wars.
[0:18:17] People know it's an AI. It's still useful. They still interact with it. It's kind of like cute and person-like, although you know it's not a person. And in that world where we just have like a lot of AIs that are contributing to the societal infrastructure we all build up together, that feels manageable and less scary to me than the sort of single big superintelligence. Yeah, yeah. Well, this is a financial event.
[0:18:46] How will, you know, well, there's kind of a debate in economics as to whether changes in the working age population push real interest rates up or down because, you know, you have a whole bunch of countervailing effects and, you know, yeah, they're more productive, but you also need capital investment to kind of make them productive and so forth. How will AI change real interest rates?
[0:19:17] I try not to make macro predictions. I'll say I think they're going to change a lot. Okay. Well, how will it change measured economic growth? I think it should lead to a massive increase in real economic growth, and I presume we'll be able to measure that reasonably well.
[0:19:44] And will at least the early stages of that be an incredibly capital intensive period because we now know which cancer curing factories or pharma companies we should build and what exactly the right reactor designs are and so forth? I would take the other side of that. Again, we don't know, but I would say that human capital allocation is so horrible that if we know exactly what to do,
[0:20:13] Even if it's expensive. You mean like the present day capital allocation done by humans? Yeah. Or you mean like the allocation of the actual people themselves across society into different roles? No, I meant the way that we allocate. Capital allocation done by humans. Yeah, done by humans. How much do you think we spend on cancer research today? How much do we spend on cancer research a year? I don't know. Probably, well, it depends if you count the pharma companies, but it's probably about like...
[0:20:42] Eight-ish, 9 billion from the NIH, and then not how much the drug companies spend, but I don't know, probably some small multiple of that again. But if it's like globally... Under 50 billion. Okay, I was going to guess, total guess, between 50 and 100 billion per year. And if an AI could tell us exactly what to do and spend like $500 million a year for one single project, which would be huge for a single project, but it was the right answer, that would be a great efficiency gain. Okay, so...
[0:21:11] We will actually become significantly more capital efficient once this technology. That's my guess. Yeah. Interesting. For OpenAI, obviously, you guys want to be and are a preeminent research organization. But with respect to commercialization, is it more important to be a consumer company or an infrastructure company? I am a believer as a business strategy in platform plus killer app.
[0:21:43] I think that's like worked for a bunch of businesses over time for good reason. I think the fact that we're doing a consumer product is helping us make our platform much better. And I hope over time that we figure out how to like have the platform make the consumer app much better too. So I think it's like a good cohesive strategy to do them together. But as you pointed out, really what we're about, we'd like to be the best research org in the world. And that is more important to us than any productization.
[0:22:12] And building the org that can make these repeated breakthroughs. They don't all work. We've gone down some bad paths, but we have figured out more than our fair share of the paradigm shifts. And I think we'll have the next big ones will come from here too. And that's really kind of what is important to us to build. Which breakthrough are you most proud of OpenAI having made? The whole GPT paradigm, I think. I think that was a kind of thing that...
[0:22:46] That has been transformative and an important contribution back to the world and comes from the sort of work, the multiple kinds of work that OpenAI is good at combining. Google I.O. is tomorrow, I think, or starts tomorrow. If you were CEO of Google, how would you do? I think Google's doing a good job. I think they have had quite a lot of focus and intensity recently and are really trying to figure out,
[0:23:21] How they can move to really remake a lot of the company for this new technology. So I've been impressed. Are these models and their attendant capabilities actually a threat to search? Or is that just a sort of superficial response that is a bit too hasty? I suspect that they mean search is going to change in some big ways, but not.
[0:23:52] Not a threat to the existence of search. So I think it would be like a threat to Google if Google did nothing, but Google is clearly not going to do nothing. How much important ML research comes out of China? I would love to know the answer to that question. Like how much does it come out of China that we get to see? Not very much. Yes, yes. I mean from the published literature. Non-zero, but not a giant amount.
[0:24:28] Do you have any sense as to why? Because the number of published papers is very large and there are a lot of Chinese researchers in the US who do fantastic work. And so why is the kind of per paper impact on the Chinese stuff relatively low? I mean, what a lot of people suspect is they're just not publishing the stuff that is most important. Do you think that's likely to be true? I have...
[0:24:58] I don't trust my intuitions here. I just feel confused. Would you prefer OpenAI to figure out a 10x improvement to training efficiency or to inference efficiency? It's a good question. It sort of depends on how important synthetic data turns out to be. I mean, I guess if forced to choose, I would choose inference efficiency.
[0:25:38] I think the right metric is to think about all the compute that will ever be spent on a model plus all inference and try to optimize that. And you say inference efficiency because that is likely the dominant term in that equation. Probably. I mean, if we're doing our jobs right. When GPT-2 came out, only a very small number of people noticed that that had happened and really understood what it signified, to your point about the importance of the breakthrough.
[0:26:09] Is there a GPT-2 moment happening now? There's a lot of things we're working on that I think will be GPT-2-like moments if they come together, but there's nothing like release that I could point to yet and with high confidence say this is the GPT-2 of 2023. But I hope by the end of this year, by next year, that will change. What's the...
[0:26:38] What's the best non-OpenAI AI product that you use? Honestly, the only product that I think of is really, I don't use a lot of things. I kind of have a very narrow view of the world, but ChatGPT is the only AI product I use daily. Is there an AI product that you wish existed and that you think that our current capabilities make possible or will very soon make possible that you're looking forward to?
[0:27:16] I would like a copilot-like product that controls my entire computer. So they can look at my Slack and my email and Zoom and iMessages and my massive to-do list documents and just do most of my work. Some kind of 3++ sort of thing. And you mentioned curing cancer. Is there an obvious application of these techniques and technologies to science that, again, you think we...
[0:27:46] Have or are just having capabilities for that you don't see people obviously pursuing today? There's a boring one and an exciting one. The boring answer is that if you can just make really good tools like that one I just mentioned and accelerate individual scientists each by a factor of three or five or 10 or whatever, that probably increases the rate of scientific discovery by a lot, even though it's like not directly doing science itself. The more exciting one is I do think that same, a similar,
[0:28:18] The system could go off and start to read all of the literature, think of new ideas, do some limited tests in simulation, email a scientist and say, hey, can you run this for me in the wet lab? And probably make real progress. And that's, I don't know how exactly the ontology works here, but you can imagine
[0:28:46] Building these better sort of general purpose models that are kind of like a human. We'll go read a lot of literature, et cetera. Maybe smarter than a human, better memory, who knows what. And then you can imagine models trained on certain data sets that are doing something nothing like what a human does. You're mapping, I don't know, CRISPRs to edit accuracies or something like that. And it really is a special purpose model in some particular domain.
[0:29:15] And do you think that the scientifically useful application of these models will come more from the first category where we're kind of creating better humans or from the second category where we're creating these predictive architectures for problem domains that we that are not currently easy to work with? I really don't know. I this is like, you know, most areas I am willing to like kind of give some rough opinion in this one. I never have. I don't.
[0:29:45] I feel like I have a deep enough understanding of the process of science and how great scientists actually work to say that I, like, I mean, I guess I would say if we can figure out someday how to build models that are really great at reasoning, then I think they should be able to make some scientific leaps on themselves, but by themselves, but that requires more work. You know, OpenAI has a
[0:30:16] has done a super impressive job of fundraising and has a very unusual capital structure for the nonprofit and the Microsoft deal and like all the things. Are weird capital structures underrated? Like, should organizations and companies and founders be thinking more expansively about, you know, people default or at least historically defaulted like, all right, we're just like a Delaware C Corp. OpenAI, as you pointed out, broke all the rules. Should people be breaking more corporate structure rules?
[0:30:43] I suspect not. I suspect it's a horrible thing to innovate on. You should innovate on products and science and not corporate structures. The shape of our problem is just so weird that despite our best efforts, we had to do something strange. But it's been an unpleasant experience on TimeSuck on the whole.
[0:31:12] Efforts I'm involved in have always had normal capital structures, and I think that's better. And do we underestimate the extent? So a lot of companies you're involved with are very capital intensive. And maybe OpenAI is perhaps the most capital intensive, although who knows, maybe Helium will overtake it or something. But do we underestimate the extent to which capital is a bottleneck, the bottleneck?
[0:31:39] On realized innovation, like, is that some kind of, you know, common theme running through the various efforts you're involved with? I, yes. I mean, that is my, there's basically like, there's like four companies that I'm, I would say, involved with other than just like having written a check as an investor. And all of them are super. Do I enumerate those for the audience? OpenAI and Helion are the things I spend the most time on. And then also Retro and WorldCoin. But.
[0:32:10] All of them raised minimum nine digits before any product at all. And in OpenAI's case, much more than that. And all have raised the nine digits before as either a first round or before releasing a product. And they all take a long time, many years to get to a release of a product. And I think this is just like...
[0:32:37] There's a lot of value in being willing to do stuff like this. And it fell out of favor in Silicon Valley at some point. And I understand why. It's also great for companies that only ever have to raise a few hundred thousand or million dollars and get to profitability. But I think we over pivoted in that direction. And we have forgotten collectively how to do the high risk, high reward, hugely capital and time intensive bets. And those are also valuable. We should be able to support both.
[0:33:07] And this touches on the question of why aren't there more Elons in that I guess the two most successful hardware companies in the broader sense starting the last 20 years were both started by the same person. That seems like a pretty surprising fact. And obviously Elon is singular in many respects.
[0:33:31] But what's your answer to that question? Do we lack people with his particular set of circumstances? Is it actually a capital story along the lines of what you're saying? If it was your job to cause there to be more SpaceXs and Teslas in the world, and maybe you're trying to do some of that yourself, but if you had to kind of push in that direction systematically, what would you be trying to change?
[0:33:57] I have never met another Elon. I have never met another person that I think can be developed easily into another Elon. He is sort of this strange end of one character. I'm happy he exists in the world, of course, but also a complex person. I don't know how you get more people like that. I don't know what you think about how to make more. I'm curious.
[0:34:32] I don't know. I suspect there's something in the culture on both the founder and the capital side, the kinds of companies the founders want to create, and then the disposition. And to some extent, though, maybe to a lesser extent, the fund structure of the sources of capital. A surprise for me, as I've learned more about this space over the last 15 years, is the extent to which the
[0:35:06] There's a finite or essentially finite set of funding models in the world. And each has a particular set of incentives. And for the most part, a particular sociology. And that's evolved over time. Like venture capital was itself an investment. PE in its modern form was essentially an invention.
[0:35:32] And so I doubt we are done with that process of funding model invention. And I suspect there are models that are at least somewhat different to those that prevail today that are somewhat more amenable to this kind of innovation. Okay. So one thing I'm excited about is I think, and you're a great example of this, but I think all of the people who became tech billionaires in the last cycle are
[0:36:01] Pretty, no, most are pretty interested in putting serious capital into long term projects. And the availability of capital for significant blocks of capital upfront for high risk, high reward, long term, long duration projects that rely on fundamental like science and innovation is going to already has dramatically changed.
[0:36:31] So I think there's going to be a lot more capital available for this. You still need the Elon-like people to do it. And one project I've always been tempted to do is say, OK, we're going to identify the, let's say, 100 most talented people we can find that want to work on these sort of projects. We're going to give them like 250k a year.
[0:36:58] So like enough money for 10 years or something. So it's like, you know, give a 20-year-old tenure or something that feels like tenure. Let them go off and without the kind of pressure that most people feel, have the certainty to go off and explore for a long period of time and like, you know, not feel the like very understandable pressure to make a ton of money first. And put them together with great mentors and a sort of a great peer group.
[0:37:24] And then the financial model would be like if you start a company. If not, that's fine. Like go be a writer, a politician, think whatever. If you start a company, the vehicle gets to invest like on predefined terms. I think that would pay off and someone should do it. That's kind of.
[0:37:42] The university model, I guess, and I don't mean that as like this already exists, you know, you're just reinventing the bus or something. I mean, it may suggest evidence that it can work. And universities are usually not that good at supporting their spin-outs, but it happens to at least some extent. And yes, one of the theses for ARC, in fact, is that by maybe formalizing this somewhat more than it is, by encouraging it somewhat more than it tends to be, that that actually might be a pretty effective model.
[0:38:10] So Silvana, my co-founder at Arc, I've known her since we were teenagers, more than half our lives. And Patrick Hsu, the other co-founder, she did her PhD with, and so she'd known him for a long time. And so to your point about the long-term investment, part of how I was comfortable with it is I'd known this person for, again, a really extended period. As you think about something like, you mentioned Retro or some of these other companies where...
[0:38:40] You didn't. How do you decide whether the person is the kind of person you can undertake this super long-term expedition with? Actually, I had known Joe for a long time. I mean, that's a bad example. I guess that's the question. Is that in fact, do you need to have known the person for a long time? It's super important to me. It doesn't always work, but I try to work with people that I've known for like a decade plus at this point. You don't want to only do that. You want some...
[0:39:07] New energy and volatility in the mix, but having a significant proportion of people that you've known for a long time, worked with for a long time. I think that's really valuable. Like in the case of OpenAI, I had known Greg Rockman for a long time. I met Ilya for maybe only like a year before, even a little bit less than we started the company, but spent a lot of time with him together. And that was like a really good combination. But I derive like great pleasure from having like working relationships with people over decades.
[0:39:37] Through multiple projects. And it's a lot of fun to feel like you're building together towards something over that has a very long arc. Agreed. Which company that is not thought of as an AI company will benefit the most from AI over the next five years? I think some sort of investing vehicle is going to figure out how to use AI to be like an unbelievable investor and just have a crazy outperformance. Like Rentech with these new...
[0:40:13] Technologies. Is there like an operating company that you look at? Well, do you think of Microsoft as an AI company? Let's say no for the purpose of this question. Okay, I think Microsoft will transform themselves across almost every axis with AI. And is that because they're just taking it more seriously or because there's just something about the nature of Microsoft that makes them particularly suited to this? Understood it sooner than others and have been taking it more seriously than others.
[0:40:47] What do you think the likelihood is that we will come to realize that GPT-4 is somehow significantly overfit on the problems in the domains that it was trained on? Or how would we know if it was? Or do you even think about overfitting as a kind of concern? The code forces problems before 21 versus after 21 where it does better on the earlier ones, et cetera.
[0:41:20] I think the base model is not significantly overfit, but we don't understand the RLHF process as well. And we may be doing more brain damage to the model in that than we even realize. Do you think that G, the generalized measure of intelligence, exists in humans as anything other than a statistical artifact?
[0:41:51] If the answer to that is yes, do you think there exists an analogous sort of common factor in models? I think it's a very imprecise notion, but there's clearly something real that it's getting at in humans and from models as well. So I think we probably like over, there's like way too many significant figures when people.
[0:42:23] Try to talk about it. But it's definitely my experience that very smart people can learn, I won't say arbitrary things, but a lot of things very quickly. There's also some people who are just much better at one kind of thing than another. And I don't want to debate the details too much here, but I'll say as a general thing, I believe that model intelligence will also be somewhat fungible. And based on your experience,
[0:42:58] Think of all this AI safety stuff. How, if at all, do you think synthetic biology should be regulated? I mean, I would like to not have another synthetic pathogen cause a global pandemic. I think we can all agree that wasn't a great experience. It wasn't that bad compared to what it could have been. But I'm surprised there has not been more global coordination after that. And I think we should have more. So what do we actually do? Because I think some of the same.
[0:43:30] Challenges apply as an AI. Actually, I think this is a part of it. A production apparatus for synthetic pathogens is not necessarily that large and the observability and telemetry is difficult. No, I think this one's a lot harder than the AI challenge where we do have some of these characteristics like tremendous amounts of energy and lots of GPUs. I haven't thought about this as much. I would ask you what we should do. I think that if someone told me this is a problem, what should we do? I would call you. So what should we do? I don't know that I've...
[0:44:05] Ready prescription. I mean, the easy thing to say, I'm not sure how much it helps, is we need a lot more general observability, wastewater sequencing, things like that. We should do that regardless, even if it doesn't help us against synthetic biology attacks. And the fact that we don't have a giant correlational data set of the pathogens that people are infected with and then sort of longitudinal health outcomes is just like a crazy fact in general. And then...
[0:44:34] Obviously, there's a somewhat innumerable set of infectious diseases, like classes of infectious diseases, that people tend to be most susceptible to, and obviously COVID itself being an example of this. And so I think we could make a lot more progress in pan-variant both treatments and vaccines than we do and than we have.
[0:45:02] I think the particular thing, like if it is true that COVID was engineered, I think, you know, instances of that set of slight modifications to already existing infectious diseases, we can probably significantly improve our protections too. Obviously, the concerning category would be, you know, completely novel pathogens, and that's presumably a sort of an infinite search space. Then you, I think, get into how do you, I mean.
[0:45:29] There's a finite set of ways to enter cells and receptors and so forth. So maybe you can use that to tile the space of possible treatments. And you need to invest in a lot more surplus manufacturing capacity than we have.
[0:45:45] For novel vaccines and hopefully mRNA platforms and similar make it easier to have general purpose manufacturing capabilities there. But as you can tell from this kind of long answer, I don't think there's a silver bullet. And it's, I think, plausible that even if you did everything that I just said well, you still would not have enough. So I think it's hard. Getting way better at rapid response treatment, vaccination, whatever, that all seems like just an obvious thing to do that I would have, again, hoped for more progress on by now. Yeah.
[0:46:15] I very much agree with that. And clinical trials. That was the limiting step in COVID. And I think it's at this point been widely reported and remarked upon that we had the vaccine candidate in January. And everything after that, I mean, some of what happened after that was obviously manufacturing scale up, but much of what happened after that was just like,
[0:46:44] How long it took us to tell that this actually works and this is sufficiently safe and that seems among the lowest hanging fruit in the entire biomedical ecosystem to me. 100%. I guess your investment in TrialSpark is consistent with that observation.
[0:47:07] Ezra Klein and Derek Thompson are writing a book about the idea of an abundance agenda and that so much of the left, of the liberal sensibility is about sort of, you know, forbearance and, you know, some kind of, you know, quasi neo-Puritanism, et cetera. And, you know, they believe and I guess are
[0:47:38] Well, have been making the case in some of their respective public writings thus far, but for the purpose of this book, the argument that actually for society as equal and prosperous and environmentally friendly and so forth to actually realize many of these values we care about, we'll need just like a lot more stuff in many, many different domains. More kind of the Henry Adams curve realized.
[0:48:07] They frequently observe that permitting, in the broadest sense, all sorts of well-intentioned but self-imposed restrictions are the great limiting factor in making this happen, maybe most obviously with the energy transition. Across all the different things that you're involved with, to what degree do you think this dynamic of self-imposed restrictions and strictures
[0:48:35] Is is the relevant variable in the progress that actually ensues? It definitely seems huge, but I don't I think also there's a lot of people who like to say, well, this is the only problem. And if we just could resolve like permitting writ large, we'd all be happy. And I don't think it's quite that simple either. I do think that the current system. So I totally agree that we need much more abundance. And, you know, my personal beliefs are.
[0:49:04] Abundant energy and abundant intelligence are going to be two super important factors there, but there's many others. Certainly, as we start to get closer on being able to deliver a lot of fusion to the world, understanding just how painful the process to go get these things out is disheartening, to say the least. And it's pushing us to look at all sorts of very strange things that we can do sooner rather than wait for
[0:49:36] All of the permitting processes that will need to happen to connect these to the grid. It's much easier to go desalinate water in some country that just has their own nuclear authority or whatever. I think it is a real problem. And I think we don't have that much societal will to fix it, which makes it even worse. But I don't think it's the only problem. If Ezra and Derek interviewed you, and I guess they should, for this book and asked you for your
[0:50:10] Number one diagnosis as to that which is limiting the abundance agenda. What would you nominate? Societal collective belief we can actually make the future better. And the level of effort we put on it. Every additional gate you put on something when these things are fragile anyway, I think makes them tremendously less likely to happen. And so you have like,
[0:50:40] You know, it's like really hard to like start a new company. It's really hard to convince people it's a good thing to do. Like right now in particular, there's just like a lot of skepticism of that. Then you have this like regulatory thing. It's going to take a long, you know, it's going to take a long time. So maybe you don't even try that. And then, you know, it's going to like be way more expensive. So it's just like, there's too much friction and doubt at every stage of the process of idea to like mass deployment in the world. And I think it makes people just try less than they used to or believe less.
[0:51:10] When we first met, whatever it was, 15 or so years ago, Mark Zuckerberg was preeminent in the technology industry and in his 20s. And not that long before then, Mark Andreessen was preeminent in the industry and in his 20s. And not that long before then, Bill Gates and Steve Jobs and so forth. Generally speaking, for most of the history of the software sector,
[0:51:39] And one of the top three people has been in their 20s. And it doesn't seem that that's true to me today. I mean, there's some great people in their 20s, but I'm not sure that you have this problem. Yeah. It's not good. Something has really gone wrong. There's a lot of discussion about what this is, but where are the great founders in their 20s? It's not so obvious.
[0:52:10] There's definitely some. I hope we'll see a bunch. I hope this was just like a weird accident in history. But maybe something's really gone wrong in our educational system or our society or just like how we think about companies and what people do aspire to. But I think it is worth significant concern and study. On that note, I think we're out of time. Thank you so much for doing this interview. Thank you very much.
[0:52:39] And thank you to folks at Selwyn and to Graham for hosting.