[0:00:02] The challenge is like, well, shit, I didn't sign up for this. Like I wanted to do AI research. I didn't want to do like AI research plus societal ethics and geopolitics. That's also not my expertise. I think that's a very reasonable point. Unfortunately, there isn't like another crap team of people hiding behind some wall to entirely shoulder the burden of this. You're listening to Grady Dissent.
[0:00:28] A show where we learn about making machine learning models work in the real world. I'm your host, Lucas Biewold. Jack Clark is the Strategy and Communications Director at OpenAI. Before that, he was the world's only neural network reporter at Bloomberg. He's also one of the smartest people I know thinking about policy and AI and ethics. I really am excited to talk to him.
[0:00:50] I feel like I typically get nervous when people ask me, like, you know, kind of big policy questions about AI. And I never really feel like I have much smart to say. And I think the goal of this podcast is mainly to talk about, you know, people doing AI in production. But then when I started writing down questions I want to ask you, I was like, wait a second, like, I want to ask you all the policy questions and all the weird questions that everyone asks me because I have no idea.
[0:01:20] So this question I seriously want to know because I feel like you think about this a lot. I mean, this is such a cliche question, but I'm actually fascinated by how you're going to answer it, which is what probability do you put on AI apocalypse? Oh, good. So we'll start with the really easy question and go from there. Yeah, yeah, yeah. What's your like, is it like one in 10, like nine out of 10, one in a million? Like what are you, what's your?
[0:01:48] I think the chance of an AI apocalypse is quite high. It's like above 50%. But that's only because the chance of most apocalypses, if they get to the point that they're happening, like say a global pandemic, which is something that we're currently going through, it's quite clear that most of today's governments don't have the capacity or capabilities to deal with these really hard challenges. So if you end up in some scenario where...
[0:02:15] You've got like large autonomous broken machines doing bad stuff. And I think your chance of being able to like right the chip is like relatively low. And you don't have like a super positive sort of outlook. I think the chance that we have to like avert that and get ahead of it is actually quite high. But I think your question is more like if something like wakes up and we enter this very, very weird territory.
[0:02:43] What are our chances? And I think if we don't do anything today, then our chances are extremely poor. No, well, okay. So yeah, I think maybe I agree. Our chances of surviving an AI apocalypse are probably low. But I think my question actually is, what do you think the chances are of actually entering the AI apocalypse? And remember that all apocalypse scenarios, they can't sum to more than one, right? So like, I mean, in a way like the...
[0:03:10] Like a pandemic apocalypse, unless you think they're sort of linked. That should make the AI apocalypse probably actually lower, right? I think that there's... I mean, this is kind of like at the beginning of when you started to do massive amounts of computer trading on the stock market, say, what's the chance we're going to enter into a high-frequency trading apocalypse? And I think how someone would have answered that is, it's really high, we'll have problems.
[0:03:39] But it's fairly unlikely that the whole system will topple over due to high frequency trading. And I think that my answer on AI is pretty similar. Like it's really high, but we're going to have some problems because it's a massively scaled technology that does weird things really, really, really quickly. And we'll do stuff in areas where finance has also deployed huge amounts of capital. So the opportunity for big things going wrong is kind of high. But the chance of a total apocalypse feels,
[0:04:09] A little fantastical to me. And that's partly because I think for a real apocalypse, like a really bad, severe one, you need the ability for AI to take a lot of actions in the world, which means you need robotics. And robotics, as you and I both know, is terrible and actually protects us from a huge amount of many parts of the like ornate apocalypse scenarios. The way that I think about this is you develop a load of radical technology and...
[0:04:38] And some of the greatest risks you face aren't the technology deciding of its own volition to do bad stuff. That very rarely happens. It's even unlikely here. There's a chance that you kind of get black mold for technology. Like somewhere in your house, you have not been cleaning as efficiently and you don't have good systems in place and something problematic starts developing in a kind of emergent way that you barely notice. But that thing...
[0:05:04] Has really adverse effects on you. And it's also hard to diagnose the source of the problem and why it's occurring. Interesting. So, but okay. So that's actually like a little bit less of a, that seems like a much more concrete scenario. Like I guess what, what form might that take? I mean, it sounds like you're mostly worried about sort of the things we're doing now. We get, we get better at doing these bad things and that, that causes big problems. Like what, what are like top of mind as, as like concerns for you?
[0:05:33] Yeah, well, I guess I'd frame my concern as we're currently pretty blind to most of the places this could show up. And we kind of need something that looks a lot like weather forecasting and radar and sensors for looking at evolutions in this domain. The sorts of things that I'd be worried about are scaled up versions of what we already have. Recommendation systems.
[0:05:57] Pushing people towards increasingly odd areas of content or subject matter that we aren't realizing are quietly radicalizing people or making people behave differently with each other. I worry quite a lot about AI capabilities interacting with economics. So you have some economic incentive today to create entertaining disinformation or misinformation. I think we think about...
[0:06:24] What happens when those things collide? You have good AI tools for creating misinformation or disinformation and an economic incentive and stuff starts showing up. I think there are going to be relatively few grand evil plans. I think there are going to be lots and lots of accidental screw-ups that happen at really, really large scales and that happen really, really quickly with self-reinforcing cycles. And I think that that's the challenge is you not only need to spot something, but you're going to need to take...
[0:06:53] And that's something that we're traditionally just really, really bad at doing as people. We can observe bad things happening, but our ability to act against them is quite low. So you do a lot of work on ethics and AI and a lot of thinking about it. But it seems like those scenarios feel... Is AI special there? It seems like there's a lot... That might be just general technology risk, right? Do you think AI makes this different?
[0:07:23] I think the difference is delegation. So like technology allows us to delegate certain things. Technology, up until many sort of practical forms of AI, lets us delegate highly specific things that we can write down in a sort of procedural way. And AI allows us to delegate things which have a bit more inherent freedom in how you choose to approach the problem that's being delegated to you.
[0:07:51] Like, you know, make sure people watch more videos on my website. It's kind of a fuzzy problem. You're giving a larger space for the system to sort of think about it. And so I think the ethics there, they're not something that humans haven't encountered before, but it's a form of ethics which is kind of has a lot in common with military or how you do administrative states in the old days, which is the sort of ethical nature of giving.
[0:08:17] Someone, the ability to delegate increasingly broad tasks to hundreds or thousands of capable people. That's like a classic ethical problem that people have dealt with for hundreds or thousands of years. But with AI, now almost everyone gets to do that delegation. And that really hasn't happened before. We haven't had this scale of delegation and this ease with which people can kind of scale themselves up. And so lots of the ethical challenges are like, okay,
[0:08:43] People now have much greater capabilities to do like good and harm than they did before. They have these automated tools that kind of extend their ability to do this. How do you think about the role of the tool developer in that context? Because sure, you're building just like iterations on previous tools, but the scope of which those tools will be used, the areas in which they'll be used is much, much broader than you've sort of dealt with before. And I think it introduces...
[0:09:11] Ethical considerations for you, but maybe governments only previously dealt with. I see. So in your view, AI kind of allows single individuals to have sort of broader impact. Yeah. And therefore, the tools you actually make available to folks, there's more ethical issues within that. Yeah. Like a good way to think about this is...
[0:09:37] I think language models are interesting. Here's an ethical challenge that I find interesting with language models. You have a big language model that has a generative capability. You want to give that to basically everyone because it's sort of analogous to a new form of paintbrush. It's very general. People are going to do all kinds of stuff with it. Except this paintbrush reflects the implicit biases in the sort of data that it was trained on at massive scale.
[0:10:06] So, okay, it's like a slightly racist paintbrush. The problem is now different to just having a paintbrush. You've got like a paintbrush that has slight tendencies and some of these tendencies seem fine to you, but some of the tendencies seem to reflect things that many people have a strong moral view of as being like bad and in society. What do you do then? And I've actually spoken to lots and lots of artists about this.
[0:10:31] Most artists will just say, give me the paintbrush. Like I want the like crazy funhouse mirror version of society so I can talk about it and make interesting things. That feels fine. But then I wonder about what happens if someone gets given this paintbrush and they just want to write text for a kind of economic purpose. They may not know much about the paintbrush they've been given. They may not know about its traits. And then suddenly they're kind of unwittingly creating.
[0:10:58] Massive scaled-up versions of the biases inherited by the thing you gave them. That seems challenging. And where we as technology developers have a lot of choice, a sort of uncomfortable amount of choice, and a lot of problems which are not easy to fix. You can't fix this. You need to sort of figure out how to talk about it and make people aware of it. Wow, that's a really clever analogy. I have not heard that one before. Yeah, I mean, I think it speaks to the weird scalability.
[0:11:28] If we just have tools that let people scale themselves in various directions, and the directions are increasingly creative areas because we're building these scaled-up curve-fitting systems that can fit really weird curves, including interesting semantic domains, then all the problems of curve-fitting now become weird problems of the production of art and thought, which feels different.
[0:11:56] And challenging. I don't have great answers here. I have more like, oh, dear, this is interesting and feels different. But actually, I mean, it's interesting because you speak of the sort of language model as, oh, just for example, what if you had a language model? But I mean, OpenAI actually had this issue. And I'm curious how you thought about it at the time and how you reflect on that now. I think at the time, so this is...
[0:12:23] GPT-2, which is a language model that we announced and didn't initially release, but subsequently released in full. At the time, I think we made a kind of classic error, which is that if you're developing a technology, you see all of its potential very, very clearly. And you don't just see the technology you're holding in your hands. You see Gen 2, 3, 4, and 5 and the implications thereof. I think we treated some of our worries about...
[0:12:52] The misuse of this technology. We were thinking about later versions of the technology than the one we were actually holding. Because what actually happened is we released it. We observed a huge amount of positive uses and really surprising ones like this game, AI Dungeon, where a language model becomes a kind of dungeon master. And it feels like interesting and actually different, like a different form of gameplay, something we wouldn't have expected. And the misuses were relatively small. And it's actually because...
[0:13:22] It's really hard to make a misuse of a technology. It's probably as hard to make a misuse of a technology as it is to make a positive use. And luckily, most people want to do the positive uses. So your amount of people doing the misuses is a lot smaller. I think that means that the responsibility of technology developers is going to be more about maybe you're still going to trickle things out in stages, but you're ultimately going to release lots of stuff in some form.
[0:13:51] It's about thinking about how you can control some elements of the technology while making other parts accessible. Can you control how you'd expect a big generative system to be used while making it maximally accessible? Because you definitely don't want a big generative model that may have biased tendencies providing generations to people in, say, a mock interview.
[0:14:21] So I think our thinking has become a lot more subtle. And I think we did.
[0:14:46] We did anchor on the future more than the present, and that's been one of the main things that's changed. Interesting. So knowing what you know now, you wouldn't withhold the model? I think you'd still do staged release, but I think that you'd do more research earlier on characterizing the biases of the model and potential malicious uses. Because the thing that we did is we did some of this research, and then we did a lot more.
[0:15:15] After some of the initial models have been released on characterizing subsequent models we are planning to release, what I think is now more helpful is if you have a load of that stuff front loaded. So you're basically saying, here's the context, here are the traits of this thing, which is like going to slowly be released and you should be aware of it. So yeah, I think we would have done stuff slightly differently. And I think that this is, what we're trying to do here is learn how to...
[0:15:43] Behave with these technologies. And some of that is about making yourself more culpable than is traditional for its outcomes. Because as a thinking exercise, it makes you think about different things to do. So I'm glad that part of the goal of GPT-2 is bring a problem that we actually don't get to get wrong in the future, earlier in time to a point where we can do different ways of release. And maybe some of them will be good and some of them will be suboptimal and learn from that.
[0:16:13] Because I think in five, six, seven years, these sorts of capabilities will need to be treated in a sort of standardized way that we've thought about carefully. And getting to that requires lots of experiments now. But it's kind of interesting. I guess there's sort of two kinds of problems. I think my understanding of the worry with GPT-2 is actually malicious uses, which more information probably wouldn't help with. But then there's also, I think, your idea of accidentally racist paintbrush. Yeah.
[0:16:44] That sort of speaks to inadvertently bad uses. I mean, both seem like potential issues, but do you now view malicious uses as kind of less of an issue? Because I really could imagine a very good language model having plenty of malicious uses. And I suppose you could say, well, any interesting technology probably has malicious uses, so should we never release any kind of tool? How do you think about that? Yeah, again.
[0:17:11] It's good, but we're doing really easy questions. I'm glad we're warming up for the easy stuff. Well, there's a couple of things. One other thing we did with GPT-2 was we released a detector system, which was a model trained to detect outputs of GPT-2 models. We also released a big data set of unsupervised generations from the model so other people could build different detector systems. I think that...
[0:17:37] A huge amount of dealing with misuse is just giving yourself awareness. You know, like, why are police forces around the world and security services able to actually deal with organized crime? Well, we can't make organized crime go away because that's a socioeconomic phenomenon. But they can, like, tool up on very specific ways to detect patterns of organized crime. And I think it's similar here where you need to release tools that can help others detect the outputs of the things you're releasing.
[0:18:06] For avoiding malicious uses, I think it's actually kind of challenging. I think that it's a little unclear today how you completely rule that stuff out. I think it's generally challenging to do that with sort of technologies. Some of how we've been approaching it is trying to make prototypes. The idea being if we can make like a prototype use case that's malicious and real, then we should sort of talk to affected people. The extent to which we would...
[0:18:36] I think that we're going to need new forms of control of technology in general at some point. I don't think that's like this year's problem or next year's problem.
[0:19:04] You know, in 2025, you're going to have these embarrassingly capable cognitive services which can be made available to large numbers of people. And I think sort of cloud providers and governments and others are going to need to work together to really characterize what can be just generically available for everyone and what needs some level of care and attention paid to it. And getting to that's going to be incredibly unpleasant and difficult, but feels kind of inevitable.
[0:19:35] But I guess just to be concrete, like if you created, say, like a GPT-3 that was much more powerful, you think that you would probably release it along with the detector would be the sort of compromise now? I think you think about different ways that you can release it because like some capabilities might be fine. Some might, you know, you might want to have some sort of control. So you control the model that people access.
[0:20:00] sort of services around it that could be one way that you do it another way could be just releasing fine-tuned versions of models on specific data sets or specific areas because if you fine-tune a model it's kind of like neural silly putty where you take this big blob of like capability you put it on a new data set it takes on sort of the traits of that data set and in some sense you've restricted it so you can do things like that
[0:20:27] I think the challenge for a lot of developers going forward is going to be in how to deal with the root artifacts themselves, the models themselves. Here's a thing I think about quite regularly is it's not today, it's not next year, it's probably not even 2022, but definitely by 2025, we're going to have conditional video models, like someone in the AI community or some group of people.
[0:20:55] are going to develop research that allows us to generate a model, generate a video that runs for some period of time, you know, a few seconds, probably not like minutes, but they can guide it so that it includes specific people and they do specific things and maybe you also get audio as well. That capability is obviously something that it's like the much harder case of just a language model or just an image model. I think that that capability definitely gets like
[0:21:26] Quite a few controls applied to it and needs systems done for like authentication of real content on the public internet as well. Like it provokes questions about that. Yeah, I think we're heading into a weird era for all of this stuff. I think the advantages you get of releasing all of this stuff just sort of publicly on the internet are pretty huge. But I also think that it's like...
[0:21:54] To some degree, a dereliction of duty by the AI community to not think about the implications of where we are in three, four, five years, because I have quite a high confidence that we can't be in this state of affairs where the norm is to put everything online instantly. Because I think we'll just develop things that are frankly too capable, by we I mean AI researchers writ large, for you to be able to do that and say, this is fine.
[0:22:25] Do you think that... I need to ask, I want to ask you about this. What do you think about this sort of issue? What is the responsibility of sort of technologists and how do we get to a more responsible place and is that even necessary? And maybe you could ask me another question for that. I don't know. I don't know. It's funny. I feel like I really want the...
[0:22:49] I want to reserve the right to change my mind on some of this stuff. So do I, to be clear. I didn't realize we were committing in this conversation. I think I'm kind of reluctant to say things publicly because it seems like actually the ethics really depend on sort of the specifics of how the technology works and stuff.
[0:23:16] You know, I think like on GPT-2, just as an example, it seemed like, you know, I thought OpenAI's decision was intriguing and like different than I think what I would have done or what my instincts would have been. But it was kind of like provocative to say, hey, we're not going to release this model. And I think, you know, I think the good thing about it, maybe was it kind of got everyone kind of like talking and thinking about it. I guess also another thing that I don't really have a strong point of view on.
[0:23:46] But just seems like a little interesting is it seems like every, it seems like at the moment, every AI researcher is sort of asked to be like their own kind of ethicist, you know, on this stuff. Like I see like a lot of like, you know, ethics documents coming out with, you know, like even like open source, you know, ML projects will sort of have like their code of conduct. And on one hand, it seems a little.
[0:24:12] It seems a little almost like highfalutin to me. I feel like I have this instinct of like, come on, should I put out a code of ethics with the toaster that I sell? There's something that seems a little unappealing about it, but I can actually also definitely articulate the other side of it that if you think, I guess to me, it's less like the power of an individual and more of just like if this technology can kind of compound and run amok,
[0:24:42] Maybe it's a case that people really should be thinking about it. But honestly, I don't know. And I don't even know. I guess I'm curious what you think about this because you're in this all the time. Do you think that AI researchers are in the best position to decide this stuff? I mean, if it really affects society as profoundly as you're saying, it seems like kind of everyone should get a say about how this stuff works, right? Yeah. So this is unfair, right?
[0:25:13] What's actually happening here is an unfair thing for AI researchers, which is that they are building powerful technologies. They're releasing them into a world that doesn't have any real notion of technology governance because it hasn't really been developed yet. And they're releasing them into systems that will use the technologies to do great amounts of good and maybe a small amount of harm. And so the challenge is like, well...
[0:25:39] Shit, I didn't sign up for this. I wanted to do AI research. I didn't want to do AI research plus societal ethics and geopolitics. That's also not my expertise. I think that's a very reasonable point. Unfortunately, there isn't another crap team of people hiding behind some wall to entirely shoulder the burden of this. There are ethicists and social scientists and philosophers, members of the public, governments, all of them.
[0:26:09] Have thoughts about this and should be involved. But I think the way to view AI researchers is they're making stuff that's kind of important. They should view themselves as being analogous to engineers, like the people who build buildings and make sure bridges don't fall over. You have a notion of ethics. Chemists, you have a notion of ethics because chemists get trained how to make bombs. And so you kind of want your chemists to have a strong ethical compass so that most of them don't make explosives because until you...
[0:26:39] Have a really, really resilient and stable society. You don't want lots of people able to do this who have sort of no ethical grounding because they might do experiments that lead to literal blow-ups. Or even people like lawyers who have codes of conduct and ethics. It's very strange to look at AI research and sort of more broadly computer science and see a relative lack of this when you see it in other disciplines that are as impactful or maybe even less impactful on our current world.
[0:27:07] I don't think any AI researcher is going to solve this on their own. But I think that a culture of culpability, of thinking that actually to some extent I am like a little responsible here, not a lot, it's not my entire problem, but I have some responsibility, is good. Because how you get systemic change is, you know, millions of people making very small decisions in their own lives. It's not like millions of people making huge, optional decisions, because that doesn't happen.
[0:27:36] Hi, we'd love to take a moment to tell you guys about Weights and Biases. Weights and Biases is a tool that helps you crack and visualize every detail of your machine learning models. We help you debug your machine learning models in real time, collaborate easily, and advance the state of the art in machine learning.
[0:28:05] You can integrate weights and biases into your models with just a few lines of code. With hyperparameter sweeps, you can find the best set of hyperparameters for your models automatically. You can also track and compare how many GPU resources your models are using. With one line of code, you can visualize model predictions in form of images, videos, audio, plotly charts, molecular data, segmentation maps, and 3D point clouds.
[0:28:35] You can save everything you need to reproduce your models days, weeks, or even months after training. Finally, with reports, you can make your models come alive. Reports are like blog posts in which your readers can interact with your model metrics and predictions. Reports serve as a centralized repository of metrics, predictions, hyperparameter stride, and accompanying notes.
[0:29:02] All of this together gives you a bird's eye view of your machine learning workflow. You can use reports to share your model insights, keep your team on the same page, and collaborate effectively remotely. I'll leave a link in the show notes below to help you get started. And now, let's get back to the episode. Well, let me ask you another easy question. What do you think about military applications of AI?
[0:29:33] Well, the military applications of AI aren't special in the sense that it's technology that's going to be used kind of generically in different domains. So it'll get used in military applications. I mostly don't like it because of sort of what I think of as the like AK-47 problem. So, you know, the AK-47 was a technological innovation to make
[0:30:00] This type of rifle, more repeatable, more maintainable, and easier to use by people who had much less knowledge of weaponry than many prior systems. You develop this system, it goes everywhere. It makes the act of taking life or carrying out war cheaper and more repeatable. It's massively cheaper and much more repeatable.
[0:30:26] And so we see a rise in conflict. And we also see that this artifact, this technical artifact, to some extent, like drives conflict. It doesn't create the conditions for conflict, but it gets injected into them and it worsens them because it's cheap and it works. And I think that AI, if applied sort of wrongly or rationally in a military context, does a lot of this. It makes certain things cheaper, certain things more repeatable, and seems really, really bad.
[0:30:54] I think AI for military awareness is much more of a kind of gray area, like lots of some ways in which unsteady peace sort of holds in the world is by different sides who are at war with each other, having lots of awareness of each other, awareness of troop movements, distributions, what you're doing. And they use surveillance technologies to do this. And I think you can make a credible argument that the...
[0:31:20] The advances in computer vision that we're seeing that's being applied massively widely may, if adopted at scale by lots of militaries at the same time, which is kind of what seems to be happening, may provide some diminishment on a certain type of conflict because it means there's generally more weapons. I think stuff like the moral question of lethal autonomous weapons is really, really challenging because...
[0:31:48] Right, right.
[0:32:17] Let's do another easy question. What else you got? Actually, this is maybe a less loaded question, but I'm actually genuinely curious about this. So you recently put out this paper, I think it's called Towards Trustworthy AI Development. And I thought the...
[0:32:41] As someone who builds a system that does a lot of saving of experiments and models and things like that, I thought it was really intriguing that you picked as the subtitle, Mechanisms for Supporting Verifiable Claims. So it seems like you draw this incredibly bright, direct line between trust for the AI development and supporting verifiable claims. And I was wondering if you could tell me why that is so connected.
[0:33:09] Well, it's really easy for us to say things that have a moral or an ethical kind of value and in words, commit an organisation to something. Like we value, you know, the safety of our systems and we value them not making, you know, biased decisions or what have you. But that's an aspiration. And it's very similar to a politician on the election campaign trail.
[0:33:36] Being like, well, if you elect me, I will do such and such for you. I'll give you all this money, or I'll build this thing. But it's not very verifiable. You're sort of needing to believe the organization or believe the politician, and they can't give much proof to you. Because AI is going to be really, really significant in society and is going to play an increasingly large role, people are going to approach it with slightly more skepticism, just as they do with anything else in their life that plays a large role and has effects on them.
[0:34:05] And they're going to want systems of recourse, systems of diagnosis, systems of awareness about it. Now, today, for most of this, we just fall back on people. We fall back on the court system as a way to ensure stuff's verifiable. We have these mechanisms in the law that mean that if I, as a company, make a certain claim, especially one that has a fiduciary component,
[0:34:31] The sort of validation of that claim comes from a load of stuff around my company. And the ability to verify it comes from action and also legal recourse if I'm not doing it. There's tons of stuff like that. But I guess like what, but just before, I feel like you, like just because some people will not have read the paper or listening to this. So like when you say like supporting verifiable claims, like what's an example of like a claim that you might want to verify that would be relevant to trust for the AI development? A claim you might want to verify is that
[0:35:01] Say our system is, we feel that we've identified many of the main biases in our system and have labeled it as such. However, we want the world to validate that our system lacks bias in a critical area. So we're going to use a mechanism, a bias bounty, to get people to compete to try and find bias traits in our system. And so there you've got a thing.
[0:35:31] You're making a claim about it. I believe that it's relatively unbiased or I've taken steps to log the bias in it. But then you're introducing an additional thing, which is a sort of transparent mechanism for other people to go and poke holes in your system and find biases in it. And that's going to make your claim more verifiable over time. And if it turns out that your system had some huge trait that you hadn't spotted, well, at least the mechanism helps you identify it and then you're going to iterate from there. Similarly...
[0:36:00] We think about the creation of like third party auditing organizations, right? So you could have an additional step. You could have, I have a system making some claim about bias, putting a bias bounty out there so I have more people like hitting my system. But if I'm being deployed in a critical area, and what I mean by critical is, you know, a system that makes decisions that affect someone's financial life. So, you know, any of these areas that policymakers really, really care about.
[0:36:29] Then I can say, okay, my system will be audited by a third party when it gets used in these areas. And so now, I'm really not asking you to believe me. I'm asking you to believe the results of my public bounty and the results of this third party auditor. And I think when all of this stuff kind of stacks on itself and gives us the ability to have trust in systems, other things might be, I will just, you know, I will...
[0:36:58] Make a claim about how I value privacy, but the mechanism by which I will be training my models and aggregating data will be using sort of encrypted machine learning techniques. So there I've got this claim. You can really verify it because I have an auditable system that shows you how I'm sort of preserving your privacy while manipulating your data. And so the idea of this report is basically produce a load of mechanisms for we and a bunch of other organizations and people.
[0:37:26] And then the goal over the next year or two is to have organizations who are involved in the report and others who weren't implement some of these mechanisms and try them out. We'll be trying to do this with at least a couple of them. Oh, cool. So I can join the red team too. Yeah. I'm really excited about red team. I think like, so obviously having a, we recommend a shared red team.
[0:37:53] That takes a little bit of unpacking because obviously if you're two proprietary companies, your red teams can't share lots and lots of information about your proprietary products, but they can share the methods they use to like red team AI systems and they can standardize on some of those sort of best practices. That kind of thing feels really, really useful because eventually you're going to want to make claims that you red team the system and it's going to be easier to make a trustworthy claim.
[0:38:21] If you use a kind of industry standard set of techniques that are well documented and many have done, but if you just sort of cowboy it and do it yourself. So yeah, please join the red team. We want lots of people on like some shared red team infrastructure eventually. But the red team infrastructure is actually, it seems like the way you describe it, and I'm sure this comes from security, but I just, I'm not super familiar with the field. It's like you have someone like internal to your organization, right? Like you have an internal team that tries to break or tries to find problems with...
[0:38:51] You have that, and then you're seeking to find ways to have your internal team share insights with other people at other organizations. Now, they can't say, here's the proprietary system I broke and what I did. But they can say, when I sit down and crack my knuckles and try Red Team and ML system, here are the approaches I use, and here's what's effective. Not in Red Team, but we have actually done a little bit of this at OpenAI where...
[0:39:19] In a GPT-2 research paper, we wrote about some of the ways we tried to probe the model for biases because we think that this is an area that's generally useful, especially useful to get standards on. And then since then, we have just been emailing our methodology to lots of people at other companies. These people can't tell us about the models that they're testing for bias, but they can look at the probes we're suggesting and tell us if they seem sensible. And so that shows you how we're like...
[0:39:48] Able to develop some shared knowledge without breaking proprietary stuff. Interesting. One thing I kept kind of thinking as I was reading your paper is like, I use all kinds of technology that I don't think has made verifiable claims. I mean, I feel like I rely on all kinds of things to work. And maybe they're making claims, but I'm certainly not.
[0:40:14] I sort of assume that internet security works. I assume that I now have all these things plugged into my home network. Do you think that it sort of seemed like these might be just sort of best practices for developing any kind of technology? Or do you think there's something really AI-specific within it? And where would you even draw the line where you would sort of call something AI that sort of needs this kind of treatment?
[0:40:44] I think some of it comes down to the... So where do you draw the line? I think AI stuff is basically when you cross from a technology that can easily be sort of audited and analyzed and have the scope of its behavior defined to a technology where you can somewhat audit and analyze it and sort of list out where it'll do well, but you can't fully define its scope. And I think with a lot of like...
[0:41:11] Just sort of once you train in neural net, you have this like big, like probabilistic system that will mostly do certain things, but it actually has a surface area that's inherently hard to characterize fully. It's very, very, very difficult to like fully list it out. And mostly it doesn't make a huge amount of sense to you because only a sort of subset of the area, the surface area of your system is actually going to be used at any one time. So it does have some kind of
[0:41:41] Differences or, you know, bias bounties, right, is a kind of weird thing. It's sort of equivalent to saying, all right, before we elect this, like, mayor or before we appoint this person to an administrative position, we want a load of people to ask them a ton of different questions about quite abstract values that they may or may not have because we want to feel confident that they reflect.
[0:42:08] The values that we'd like someone to have in that position. That feels different, actually. It feels a little different to normal technologies. It would be absurd to expect we get to a world where everyone verifies every claim they make all the time because you have the time. I mostly go through my life depending on my own belief that other people are sticking to the rules of the game. But we all have some cases where...
[0:42:34] We want to go in on something that's happening in our life and audit every single facet of this. And I think the way to think about why you need verifiable claims or ability to make them quite broadly is as governments consider how to sort of govern technology and how to let technology do the most good while minimizing the harm. It's probably going to come down to the ability to verify certain things in certain critical situations.
[0:43:02] So you're kind of building all of this stuff, not for the majority of your life, but for the really narrow edge cases where this has to happen. But necessarily that means you need to build quite general tools for verification and then try and apply them in specific areas. It's interesting that, well, I don't know, it seems like there's been a lot of sort of complaining about AI research recently that a lot of the, just the research claims, which are maybe not so loaded and not so applied and we don't interact with are actually not really verifiable.
[0:43:31] Yeah, I mean, some of these things are just because there is a compute gap. There is a minority of organizations with a large amount of compute. There is a majority of organizations and a huge swath of academia, if not all of academia, that has very real computational limits. And this means that at a really high level, you can't really validate claims made by a subset of the industry because they're doing experiments at scales which you can't hope to meet.
[0:44:01] So some of this is about what are really general tools we can create to just resolve some of these kind of asymmetries of information. Because some issues of verifiability are less about your ability to verify a specific thing at that moment. It's more about having enough kind of cultural understanding of where the other person is coming from that you kind of understand what they're saying and the premise behind it and can trust them, which is less you demanding a certain type of verification.
[0:44:31] But being like, okay, well, you know, you're a complete alien to me. You come from another cultural context or another, you know, political ideology. However, we have this sort of strong shared understanding of this one thing that you're trying to get me to believe you're about. And right now, if like certain organizations wanted to motivate academia to do a certain type of research, it would depend on, I come from this like big compute premise land.
[0:45:00] And I'm asking you to hear me when I list out a concern that only really makes sense if you've done like experimentation at my scale because that's calibrated by intuitions. So we need to find a way to give these people the ability to have the same conversation so that you can sort of improve that stuff as well. So are you going to give them a ton of compute? Like what's your solution there? We basically specifically recommend that governments fund
[0:45:29] Cloud computing, which is a bit different. It's a bit wonky, right? But one thing you need to bear in mind is that today, a lot of the way academic funding works sort of centers usually on the notion of there being some bit of hardware or capital equipment that you're buying. And as we know, like that stuff depreciates faster than cars. It's like the worst thing to buy. If you're a researcher in an academic institution, you'd be much better placed to buy like a cloud computing sort of credit or system.
[0:45:58] A system that lets you access a variety of different clouds. We're generally, when we go and work with governments, pushing this idea that they should fund some kind of credit that backs on to a bunch of different cloud systems. Because you don't want a government saying, all right, all of America is going to run on Amazon's cloud. That's obviously a bad idea. But you can probably create a credit which backs on to...
[0:46:27] The infrastructures of like five or six large cloud entities and deal with the competitive concerns that way. And I think this is surprisingly tractable. It's like some policy ideas are relatively simple because they don't need to be any more complicated. And so we're kind of lobbying, for lack of a better word, governments to do this. I think the other thing to bear in mind is that lots of governments, because they've invested in supercomputers, really want to use supercomputers.
[0:46:56] as their compute solution for academia. And that mostly doesn't work. You actually mostly need a dumber, simpler form of hardware for most forms of experimentation. So you're also saying to governments, like, I know you spent all of this money on this stupid computer and it's wonderful. And yeah, it's great. It's simulation nuclear weapons of weather. We love that. You don't need it for this. Stop trying to use it for this, like exclusively. So that's also where some of that comes from.
[0:47:26] Nice. I actually have not encountered that. That's an interesting observation. If you're like the US, right? You're like, we've spent untold billions on like having the winner of the top 500 list. And we're in some pitched geopolitical war with China. Like, of course we want to use this for AI. And you're like, yeah, dude, but like some people just want like an 8 GPU server.
[0:47:52] Actually, most people are fine with that. And this thing is not easy to multiplex and sample out to people compared to AWS or Microsoft or whatever. Interesting. So we're a little bit running out of time. And I'm curious, we always end with two questions. I'm particularly interested in your point of view on this. So the first one, I mean, and you actually, you really view a lot of...
[0:48:22] Things going on in AI, I mean, from your vantage point at OpenAI and then also the newsletter that you put out. So what would you say is the topic that people don't pay enough attention to? The thing that just matters so much more than people compared to how much people look at it. I think the thing that no one looks at that really matters is advances in just a very niche part of computer vision.
[0:48:52] Which is the problem of re-identification of an object or a person that you've seen previously. What I mean is that our ability to do pedestrian re-identification now is improving significantly. It's stacked on all of these ImageNet innovations. It's stacked on our ability to do rapid feature extractions and video feeds. It's stacked on a load of interesting component innovations.
[0:49:20] And it's creating this stream of technologies that will lead to really, really cheap surveillance that eventually is deployable on edge systems like drones or whatever, by anyone. And I think that we're kind of massively underestimating the effects of that capability because it's not that interesting, it's not that advanced, it doesn't even require massively complex reinforcement learning or any of these things that researchers like to spend time on.
[0:49:49] It's just a sort of basic component. But that is the component that supports surveillance states and authoritarianism. And that is the component that can make it very easy for an otherwise sort of liberal government to slip into a form of surveillance and control that no one would really want to have. And I'm actually thinking about
[0:50:15] Yeah, can I write a survey or something about this? Because it's not helpful for someone like OpenAI to warn of this. It's sort of a wrong message. It's maybe okay for me to write about it occasionally in my newsletter, as I do. But I sort of think about writing an essay like, has anyone noticed this? Because I gather all of the scores, right? I look at all of the graphs and stuff. I've got to speak older a bit. It's all going up. It's all going hockey stick. It's all getting cheap.
[0:50:43] Yeah, so it's not very cheerful, but I think it's important, yeah. Wow, great answer as expected. All right, here's my second question, which we always ask, and normally we're talking to kind of more industry practitioners, but maybe you can apply this to OpenAI. So when you look at the DML projects that you've witnessed, and like OpenAI has actually had some really spectacular ones, what's the part of sort of like conception to complete?
[0:51:13] That looks the hardest to you or maybe the most unexpectedly difficult piece of it. Like sort of watching, you know, like solving DOTA or being the best human at DOTA or like GPT. Like where do things get stuck and why? Good question. I think that there are maybe two parts where projects get like stuck or have interesting traits. One is just data.
[0:51:43] Like I used to really want data to not matter so much and then you just look at it and realize that, you know, whether it's like Dota and how you ingest data from like the game engine there or robotics and how you choose to do like domain randomization in simulation or supervised learning where you're just figuring out what data sets I have and what mixing proportion do I give them during trading and how many runs do I do.
[0:52:10] That just seems very hard, and I think others have talked about this. It's not really a well-documented science. It's something that many people treat with intuition, and it just seems like an easy place to get stuck. And then the other is testing. Once I have a system, how well can I characterize it, and what sort of tests can I use from the existing research literature, and what tests do I need to sort of build myself? We spend a lot of time figuring out new evaluations at OpenAI.
[0:52:39] Because for some systems, you want to do a form of eval that doesn't yet exist to characterize performance in some domain. And figuring out how to test for a performance trait that may not even be present in a system is really hard. It's a really, really difficult question. So those would be the two areas, yeah.
[0:52:58] Okay, I can't help myself. Actually, as you were talking, I thought of one more question. I'm sorry to do this, but I feel like the people that I know or that I've watched closely at OpenAI have been actually spectacularly successful. And they've been part of projects that have really seemed to me have succeeded, like the robot hand doing the Rubik's Cube and Dota. Are there a whole bunch of projects that we don't see that have just totally failed?
[0:53:28] Oh, we've got failures. I don't know if you remember Universe. That was sort of a failure. We tried to build a system which was kind of like OpenAI Gin, but the environments would be every Flash and HTML game that had been published on the internet. Oh, yeah, yeah. Yeah, yeah. So that failed, right? That failed because of network asynchronicity. And so basically you ended up having, because we were...
[0:53:53] Sandboxing the things in a browser and you had a separate game engine that needed to go and talk over the network to them. RL actually isn't really robust enough to that level of time jitter to do useful stuff. So that kind of didn't work. So we have some public failures, which I think is kind of helpful. Yeah, we have some kind of private ones. A lot of it is, you know, some people just spend a year or two on an idea, but it ends up not working out. Some people...
[0:54:22] And I won't name the project because it's public, but they came up with a simple thing that worked really well. And they spent six months trying to come up with what, as a researcher, they thought was the more disciplined or better approach to it. And the simple thing always worked, and all of the other things they tried didn't. So they eventually published the system with the simple thing. And they were like, yeah, it works, but I would much rather let my complex idea work. But we don't.
[0:54:49] Like our big bets, like the HAND or DOTA or GPT, those have tended to go okay, but that's usually because they've come from a place of iteration. Like DOTA came from prior work applying PBO and I think evolutionary algorithms to other systems. The HAND came from prior work on just like block rotation, right? So once you can do block rotation, you can do a Rubix. GPT came from prior work on...
[0:55:19] Scaling up language models just with GPT-1. So a lot of it's just happened sort of iteratively in the public domain. But yeah, we don't have like, we don't have an abnormal lack of failure, nor an abnormal amount of success. I think it's just like, it's pretty in distribution. I hope. Awesome. Well, thanks. That was really fun. Thanks very much. Yeah, yeah. Thanks, Jack. Cool. All right. See you, man.