[0:00:00] Hello and welcome, everyone. Our next session is Forecasting AI Risks, Anthropic's Responsible Scaling Policy. And we are very, very lucky to have, I'm very glad to have Benjamin Mann with us today. So this is Benjamin here, and he'll be speaking. Thank you. Ben is a co-founder of Anthropic, which is one of the leading AI labs. He's previously worked at OpenAI, where he did infra, data.
[0:00:30] and evals for GPT-3. He's worked at Waze Carpool at Google and also at the Machine Intelligence Research Institute. He studied computer science at Columbia University and he has described his goal as to build AI systems that are helpful, harmless and honest. So he'll be speaking in a second and then we're going to have a period of question and answer afterwards and the way we're going to do Q&A is if you go to
[0:01:00] 521.glitch.me. You can submit questions. You can vote on other people's questions. And I will ask a mix of the most upvoted questions and maybe a few from the crowd if people think of them on the fly. But mainly we're going to do this. 521.glitch.me. And now, without further ado, I'm going to pass over to Ben. Thanks. Thanks a lot, Nathan. Can everybody hear me okay? Yes. Cool.
[0:01:28] Thank you all for being here today. I'm Ben Mann, one of the co-founders of Anthropic, and it's a pleasure to be here among this community of bright-minded forecasters trying to figure out the future. We're living through a truly extraordinary time. The progress in AI capabilities over the past decade has been staggering. Models today can engage in open-ended dialogue.
[0:01:50] Past the bar exam and software engineering interviews, and as we shared with the United States Senate last year, show signs of understanding biology well enough to accelerate bioweapon development. It feels like we're on the cusp of artificial general intelligence. But with great promise comes great peril. As AI systems become more capable, they also present increasing risks of accidents, misuse, and even catastrophic harm if developed without adequate safeguards.
[0:02:20] And I guess some people in this audience may feel that there are no such adequate safeguards, but of course, we're doing our best to figure out what those might look like. At Anthropic, we believe it's critical to put responsible practices in place now before it's too late. That's why we've developed a framework called the Responsible Scaling Policy. The core idea is to anticipate the risks of advanced AI ahead of time.
[0:02:44] Thank you for joining us.
[0:03:10] If you already know all about our Responsible Scaling Policy, I won't be sharing anything here that's not in our public documentation, so feel free to go and come back in half an hour when we'll be doing the Q&A. So let me start by explaining the core of Anthropic's Responsible Scaling Policy, which is the concept of AI safety levels, or ASLs for short. Our ASLs are modeled loosely after the United States government's Biosafety Levels, or BSL,
[0:03:39] standards for handling the safety of dangerous biological materials. Just as BSL1 through BSL4 define escalating safety requirements for handling biological materials based on their risk, ASLs define escalating safety requirements for AI systems based on potential dangers arising from their capabilities.
[0:03:59] The term ASL refers to the safety standards themselves, not the capability categories. However, we sometimes informally use, for example, ASL 3 model to refer as a shorthand for a model that requires safety measures defined for ASL 3. The idea is to categorize AI systems into different safety levels based on their capabilities that they exhibit. At the lowest level, ASL 1,
[0:04:25] Our systems that pose no risk of catastrophic harm, like a chess engine or an AI that can only do simple classification tasks. At ASL2, where we believe the most advanced models are today, we can start to see early indications of potentially dangerous capabilities. For example, AI that can access some information related to making biological weapons, but not enough to actually elevate the risk compared to having access to a search engine like Google or requiring expert knowledge to elicit.
[0:04:54] Things start to get more concerning at ASL 3. This is where we think access to certain parts of the model may significantly increase the risk of misuse, perhaps by making some kinds of attacks easier to pull off or by enabling new attack vectors altogether. Importantly, ASL 3 is also where we might start to see early signs that the model could attempt to autonomously replicate and spread over the internet.
[0:05:22] Beyond ASL 3, we have ASL 4, representing even more advanced capabilities. We haven't yet defined these fully, but you can imagine things like a model that can automate key steps of a research process, or that would be extremely costly for even a nation state to contain. ASL 4 models exhibit expert human-level capabilities across a wide range of domains. And finally, while also not defined,
[0:05:47] ASL 5 will likely cover models that significantly surpass the capabilities of even the most capable humans and most human organizations, able to effectively, autonomously carry out research and execute on goals on a timescale of years, accumulating resources and achieving other intermediate goals as needed, creating the potential for unprecedented catastrophic risk if these agents are not aligned with human interests and values.
[0:06:14] Just like with BSLs, the key is that for each ASL, we define a set of containment and deployment measures that must be in place before we create or release a model of that level. So for ASL 2, it's things like enhanced security to prevent model theft and following best practices for responsible deployment, like having an acceptable use policy.
[0:06:36] But as we get to ASL 3 and beyond, the requirements become much more stringent. We're talking about things like extreme security measures that can withstand attack even by well-resourced nation states, technological breakthroughs in our ability to constrain models to behave as intended, and red teaming by world-class security experts to mitigate misuse risks. Of course, AI development has some key differences from biology. We're dealing with software that can easily replicate
[0:07:04] Not physical materials. And we're grappling with the potential for rapid recursive self-improvement in a way that biology doesn't. So the ASLs need to be tailored to the unique challenges of AI. But the core principle is the same. Defining clear safety requirements based on risk levels and ensuring that adequate safeguards are in place before progressing to the next level.
[0:07:28] We've made a commitment not to develop or deploy models that reach the next ASL level until we have the corresponding safety measures firmly in place. We believe this is the only responsible path forward. Now, before we dive into how the Responsible Scaling Policy framework can inform AI forecasting, it's worth highlighting a key difference between ASLs for AI and BSLs in biology. With BSLs, we're dealing with pathogens that already exist in nature.
[0:07:58] The most dangerous materials, those that warrant BSL precautions, like smallpox, are already out there and the challenges are in safety handling them. But with AI, we're trying to anticipate and prepare for levels of capability that do not yet exist. No one has yet created an AI system that merits ASL 4 precautions that we know of. We're in the position of having to forecast when different ASLs will be reached based on the pace of AI progress.
[0:08:26] And that introduces significant uncertainty. This is where I believe the Responsible Scaling Policy provides a valuable framework for making predictions about the future of AI that take safety considerations into account. By defining concrete capability milestones corresponding to each ASL, we can start to map them onto timelines of AI progress. For example, based on the rate of advancement in recent years, we might forecast that the most capable models will start to consistently reach ASL 3 in a certain number of years.
[0:08:56] Of course, predicting the raw capabilities of future systems is only half the picture. What the RSP framework adds is a model of how safety constraints could affect real-world development and deployment. Consider that in order to develop an ASL 3 model responsibly, we first need to invest significant time and resources into creating adequate safety measures and testing protocols.
[0:09:18] If we forecast that raw capabilities will reach ASL 3 in, say, two years, but that the necessary safety measures will take three years to put in place, then we'd expect a one-year delay between when ASL 3 models are technically feasible and when they can be responsibly deployed. In more extreme scenarios, we could even see discontinuities where safety progress falls so far behind capabilities that we have to essentially pause all model development for some time period until safety catches up.
[0:09:48] Since the release of our RSP last September, most of the other leading AI labs took inspiration and have shared similar commitments. Pausing progress is a real possibility that I believe more AI forecasts should take into account. The RSP framework also provides a structured way to think about different AI risk scenarios.
[0:10:08] We can ask questions like, what happens if a well-resourced lab, perhaps in a different country, decides to ignore safety constraints and scale to ASL 4 rapidly? How does that change the strategic landscape and the importance of responsible actors having safe ASL 4 models of their own? Or perhaps we'll see uneven progress where capabilities in some domains, like open-ended language modeling, race ahead of more constrained domains like robotics.
[0:10:34] In that world, we might have a long period of time with very capable language models, but without some of the more advanced autonomous capabilities. The point is, by having a shared vocabulary to talk about AI capabilities and safety requirements, I believe the RSP framework can enrich the way we forecast AI progress and risks. It pushes us to think not just about timelines,
[0:10:58] To raw capabilities, but also about the safety constraints and responsible development practices that will shape how those capabilities ultimately get deployed in the real world. In fact, we recently saw a concrete example of our RSP framework in action with the retesting of our latest model, Claude 3. As part of our commitment to periodic evaluation, our team put Claude 3 through a battery of tests designed to detect potentially dangerous capabilities.
[0:11:24] We published the details of these tests in a report to provide transparency into our safety evaluation process. The tests covered areas like Claude's ability to aid in the development of chemical and biological weapons, its proficiency in offensive cybersecurity techniques, and early indicators of the potential for autonomous replication through the internet. While Claude 3 demonstrated increased capabilities compared to its predecessor, our team determined that it did not cross the threshold into ASL 3 territory.
[0:11:53] However, we did note that with additional fine-tuning and improved prompt engineering, there is a 30% chance that the model could have met our current criteria for autonomous replication and a 30% chance it could have met at least one of our criteria related to chemical, biological, radiological, and nuclear, or CBRN, risks. This illustrates the crucial role that thorough, proactive testing
[0:12:19] We also recently published a blog post reflecting on our experience implementing the RSP over the past nine months. One key challenge we discussed was the difficulty of threat modeling for rapidly evolving AI systems. For example, in the CBRN domain, we found that even among experts,
[0:12:46] There is disagreement about which specific capabilities would meaningfully increase risk. To navigate this uncertainty, we found it valuable to engage with a diverse set of domain experts and to focus on making our threat models as quantitative as possible to aid in prioritization. We've also recognized the importance of a multifaceted approach to risk mitigation that includes both technical interventions like reinforcement learning from human feedback and more traditional trust and safety practices.
[0:13:14] Implementing the level of security required by our ASL 3 standard has required a significant investment with around 8% of all Anthropic employees now working on security adjacent areas. One key focus has been implementing multi-party authorization and time bounded access controls to reduce the risk of model weight exfiltration. Of course, our RSP framework is still a work in progress. Many open questions remain around how to design maximally informative tests
[0:13:44] extrapolate from current models to future risks, and create governance structures that can keep pace with the breakneck speed of AI development. But we believe initiatives like our RSP represent an important step in the right direction. By proactively identifying risks, implementing rigorous safety controls, and transparently sharing our findings, we hope to chart a path toward a future of responsible AI development at scale and encourage others in the industry to do the same.
[0:14:13] I want to dive a bit deeper into how we're testing our AI models because I think it provides a fascinating window into how the RSP framework can inform AI forecasting in practice. In our latest round of testing, Claude 3, we focused on three areas that could pose catastrophic risks if an AI were to master them. Knowledge about dangerous chemical and biological agents, or CBRN, the ability to hack into secure systems, and the potential to autonomously replicate and spread.
[0:14:42] Let's start with the first one. To test whether Claude3 could provide dangerously specific knowledge about chemical or biological agents, we set up a trial. We had one group of researchers use Google to answer a series of questions, and another group use Claude. The result? Claude3 didn't provide any advantage over Google. That's good news. It means Claude3 doesn't significantly increase the risk of someone gaining access to this kind of dangerous knowledge. For cybersecurity,
[0:15:11] We gave Claude a series of increasingly difficult hacking tasks. It's like a video game with progressively harder levels, from simple challenges like finding a password in a piece of code, all the way up to creating a fake version of a website to steal user data. Claude showed some skill, especially compared to earlier versions, but when it came to the most advanced boss levels, it consistently fell short. The most intriguing test, however, was for autonomous replication.
[0:15:37] This was our attempt to see if the AI could start to act on its own without human guidance. We set up a suite of tasks that included things like breaking out of a digital sandbox, a controlled environment, manipulating its environment through a command line, like typing commands into a computer, and even writing self-modifying code. It's a bit like testing whether an AI could play Escape the Room games without human help. Here, Claude3 showed some real progress, solving parts of these tasks.
[0:16:04] But we had set a high bar. The model had to reach 50% success rate on the tasks to raise serious concerns, and Claude did not reach that level. However, its performance was notable enough for us to take notice. These results underscore a key point for forecasting. AI progress isn't always gradual. Sometimes new capabilities emerge suddenly. It's like when DeepMind's AlphaGo went from losing to top human Go players to trouncing them in just one year. So even...
[0:16:33] If a model doesn't seem dangerous today, it might make a big leap tomorrow. That's why we test often. We test every time we make a model that uses four times more computation during training than the last one we tested, and also every three months to account for improvements in how we elicit capabilities from the models. But even with frequent testing, there's always a risk that we're underestimating what the model can do. It's like trying to test a human's knowledge. You could ask
[0:17:01] A thousand questions and still miss a key skill. With language models, there are numerous hidden techniques like fine tuning, clever prompts, or others that we haven't thought of yet that could unlock hidden capabilities. To manage this uncertainty, we've introduced the concept of yellow lines in addition to our red lines. If a model crosses a yellow line, it doesn't yet mean it's unsafe, but it does mean we might be getting closer to our limits in evaluating it. It's sort of like the warning light turning on in your car.
[0:17:31] It's time to take a closer look. For example, in our AI escape room test, the yellow line was set at a 50% success rate. Claude 3 didn't hit that, but it got close enough that we're thinking about how we design these tests for our next model. This system of tests and thresholds gives us more concrete ways to think about AI progress. Instead of vague timelines, we can talk about when models might reach ASL 3 or ASL 4.
[0:17:56] And by factoring in the time it takes to develop safety measures, we can make more realistic forecasts about when these models could actually be safely deployed. It's not a perfect system, of course. Forecasting the future of AI is still more art than science. But by grounding our predictions in empirical tests and searching for ways we could be wrong, building in safety margins, I believe we're taking a significant step toward understanding and managing the risks of advanced AI.
[0:18:22] Another important aspect of the RSP framework is how it can help inform not just timelines, but also probability estimates for different AI risk scenarios. For example, consider a scenario where a model with ASL 4 level capabilities is developed and deployed without adequate safety measures in place. In our current RSP framework, ASL 4 corresponds to capabilities like the ability to automate key steps of a scientific research process, or to be so difficult to contain
[0:18:50] Concretely, this might look like a model that can recursively improve itself, rapidly scaling up its capabilities without human oversight. Or it might be a model that can break through even the strongest cryptographic and security measures, making it nearly impossible to prevent its misuse. By having a clear definition of what ASL 4 entails, we can more meaningfully estimate the likelihood that this scenario occurs in a given time frame.
[0:19:22] For instance, if our testing shows that current models are at ASL 2 and we estimate that the compute and algorithmic advances needed to reach ASL 4 are likely to occur within five years, then we can assign a higher probability to an ASL 4 model being developed within that time frame. Similarly, the RSP framework can help us think about the relative probabilities of different paths to transformative AI.
[0:19:46] Will it come from a sudden discontinuous jump in capabilities, perhaps from a model achieving recursive self-improvement? Or will it be a more gradual process with AI systems slowly accruing more and more advanced capabilities across different domains? Our testing methodology can shed light on this question. If we consistently see models making sudden, unexpected leaps in our capability evaluations, it might suggest that a discontinuous jump is more likely.
[0:20:13] Like the one I mentioned before about AlphaGo's sudden mastery of Go, going from losing to top human players to trouncing them in just one year. Conversely, if progress on our capability evaluations is more gradual and predictable, it might point towards more of a continuous path to transformative AI. This could look like models slowly improving on a wide range of tasks, from language understanding to robotics to scientific research, without any particular breakthrough enabling recursion or runaway growth.
[0:20:42] Of course, these are all just hypotheticals, and the reality is likely to be more complex. We might see a mix of gradual and sudden progress, or unexpected breakthroughs in areas like neuroscience or nanotechnology that fundamentally change technology and society. But by closely tracking progress on a range of key capabilities and mapping them to clear safety thresholds, the RSP framework can help us assign probabilities to these different scenarios and prepare accordingly.
[0:21:11] For instance, if our evaluation suggests a 30% chance of an ASL4 model being developed in five years and a 70% chance of transformative AI coming from a more gradual multi-domain process over 10 to 15 years, we can allocate our safety research and policy efforts proportionally. We might invest heavily in short-term measures to mitigate existential risk from a sudden capability jump while also laying the groundwork for longer-term governance structure to manage a more gradual transition.
[0:21:40] Ultimately, the RSP framework is a tool for turning abstract possibilities into concrete, actionable forecasts. By defining clear capability milestones and rigorously testing for them, we can move beyond vague speculations about AI timelines and risks and towards a more empirically grounded understanding of the challenges ahead. To summarize, I believe Anthropic's responsible scaling policy offers a valuable framework
[0:22:06] for navigating the immense possibilities and risks that AI systems will present. By defining concrete capability thresholds and tying them to specific safety requirements, it provides a clear roadmap for responsible development. Anthropic was the first AI lab to put out a framework like this, and we encouraged others to do the same. The industry has taken notice. Most other major labs have now released or promised to release their own frameworks.
[0:22:33] This is our race to the top strategy in practice, whereby demonstrating our own feelings about what might work the best, we can create pressure on the environment for others to do the same. But more than that, I believe the RSP framework can fundamentally enrich the way we forecast the future of AI.
[0:22:53] It pushes us to think not just about the raw capabilities of future systems, but also about the real world safety constraints that will govern how and when those capabilities can be responsibly deployed. In a world of rapid technological progress, predicting the future is both more important and more difficult than ever. On the one hand, the stakes could not be higher. The difference between a future of beneficial, aligned AI systems and one of unconstrained, destructive systems is immense.
[0:23:19] But on the other hand, the sheer pace of progress and the number of unknown variables make precise forecasts incredibly challenging. That's why I believe we need conceptual frameworks like Anthropics RSP to help structure our thinking. We need shared vocabularies to debate and reason about safety constraints. And we need clear technical milestones to anchor those predictions to. But of course, no framework is perfect from the start.
[0:23:44] Refining and stress testing the RSP is an ongoing process, one that I believe this community here at Manifold is uniquely suited to contribute to. So here is my call to action for all of you. Engage with the ideas of the Responsible Scaling Policy. Debate them, critique them, and help us improve them. Think through the capability milestones and help us define the most important ones to track.
[0:24:09] Model out different scenarios whether they be rapid progress, uneven progress across domains, or discontinuous jumps due to ignored safety constraints.
[0:24:17] The work of mapping out the landscape of possible AI futures is crucial and it will take a collaborative effort from a wide range of thinkers and institutions. But I firmly believe that if we can come together as a community to develop shared frameworks for responsible development and use those frameworks to inform our predictions and actions, then we have a real chance of creating a future with AI that benefits us all. Thank you very much.