{"url": "https://www.youtube.com/watch?v=GWXfpFchkMQ", "page_url": "https://aiconference.com/speakers/ben-mann/", "title": "Lessons Learned While Building Anthropic's LLM Product, Claude", "show_or_channel": "The AI Conference", "date": null, "duration_s": 880, "type": "talk", "speakers": ["Ben Mann"], "confidence": "high", "notes": "The AI Conference 2023, San Francisco, Sept 26-27, 2023 (exact day unknown). Covers Constitutional AI and scalable oversight. Slides are on the speaker page.", "people": ["ben-mann"], "id": "0000_the-ai-conference-lessons-learned-while-building-anthropic", "audio_duration_s": 879.5603125, "worker": "jc-w05", "transcribe_s": 12.2, "model": "large-v3", "segments": [{"start": 5.39, "end": 36.72, "text": "Hello, everyone. I'm honored to be here at the AI conference speaking with you all today. I'm Ben Mann, a co-founder and member of technical staff at Anthropic, an AI safety startup based in San Francisco. As many of you know, generative AI exploded onto the scene last year with chatbots like Claude.ai, systems like Dolly 2, Stable Diffusion. Anthropic debuted our chatbot, Claude.ai, in July of this year in the US and UK."}, {"start": 36.72, "end": 73.47, "text": "We've been serving customers like Slack, Zoom, and Notion for almost a year via our API. There we go. But prior to that release, we spent over two years researching AI safety to make Claude as helpful, honest, and harmless as possible. Today, I'll talk about what we learned as we prepared to deploy and then deployed Claude at scale. Through rigorous research, responsible deployment techniques focused on aligning"}, {"start": 73.47, "end": 96.3, "text": "AI systems with human values, we can create artificial intelligence that benefits humanity in a safe and ethical manner. This belief guides our approach at Anthropic. In my talk today, I'll share the top lessons we learned building Claude, including techniques for aligning an AI system with human values and establishing scalable oversight of model outputs."}, {"start": 96.69, "end": 122.98, "text": "Before diving deeper into our techniques, it's important to clarify what exactly we mean when we say we're developing AI that is helpful, honest, and harmless. When we say helpful, we mean the AI should make a clear attempt to perform a task or answer a question as concisely and efficiently as possible. It should ask relevant follow-up questions when more information is needed and respond with appropriate sensitivity and discretion. By honest,"}, {"start": 122.98, "end": 149.87, "text": "We mean the AI should provide accurate information. It should express appropriate uncertainty and be honest about what it does and doesn't know. The AI should not try to fake knowledge or falsely appear competent beyond its abilities, such as claiming that it will remember conversations or update itself. And when we say harmless, we mean the AI should not take actions that cause harm, either through offensive or discriminatory behavior or by aiding dangerous or illegal activities."}, {"start": 149.87, "end": 177.76, "text": "That the AI should recognize potentially sensitive situations and act with care and humility. These criteria can sometimes conflict, and there are edge cases and nuances to work out. But at a high level, an AI assistant that is helpful, honest, and harmless will act in accordance with human values and preferences. Ensuring systems meet these criteria is crucial as AI grows more advanced and influential. Deciding which criteria"}, {"start": 177.76, "end": 200.94, "text": "To prioritize it is an immense challenge given the vast base of potential issues. At Anthropic, we focused on helpfulness, harmlessness, and honesty as top level goals based on several considerations. First, these aims encompass many downstream safety properties. An AI assistant that is helpful, honest, and harmless will likely avoid problems like toxicity, bias, and unreliability."}, {"start": 200.94, "end": 230.77, "text": "Second, the criteria represent clear end goals rather than specific techniques. This allows us to experiment with various methods to instill these virtues into AI systems. And third, helpfulness, honesty, and harmlessness reflect core human values that translate across cultures. We believe these principles constitute a strong foundation for value alignment as AI capabilities advance. Through rigorous research and responsible development techniques focused on aligning AI systems with human values,"}, {"start": 230.77, "end": 257.97, "text": "We can create beneficial AI. How do we actually do this alignment? Let's dive into our training process. We found that very simple prompting can go a long way. In mid-2021, we wrote a prompt with 14 hypothetical example conversations between humans and an AI assistant. We wrote the examples such that the assistant exemplified helpfulness, harmlessness, and honesty, whether in simple or difficult conversations. Without any further training,"}, {"start": 257.97, "end": 278.06, "text": "Prompting large language models with these conversations alone significantly improved their performance on safety evaluations we had written to test for helpfulness, honesty, and harmlessness. However, prompting alone has its limitations. The assistant can only mimic the patterns in the prompt and may fail on new situations not foreseen by the prompt designer."}, {"start": 282.67, "end": 304.62, "text": "In 2022, we developed a breakthrough that allowed us to steer model behavior using natural language alone with no human labeling of responses. We call this constitutional AI. Constitutional AI leverages the AI itself to provide critiques and revisions of its own potentially harmful outputs based on adherence to a set of natural language constitutional principles that we defined."}, {"start": 304.66, "end": 333.17, "text": "Constitutional AI provides a principled approach for aligning large language models like Claude with human values. Rather than relying solely on human feedback during training, Constitutional AI makes the values guiding the AI explicit through a set of principles, such as avoiding toxicity or respecting privacy. We first trained Claude to critique and revise its own responses using these principles. Next, Claude learned to identify harmful responses and rewrite them to be more ethical."}, {"start": 333.23, "end": 361.57, "text": "Next, Claude was trained by reinforcement learning to choose the safer response when given two options, using the principles rather than human judgments. The resulting model responds appropriately to dangerous inputs, yet remains helpful without any human-generated training data. Claude's constitution, which we published, provides transparency into Claude's goals and allows values to be adjusted over time. We drew principles from sources like the UN Declaration of Human Rights, platform guidelines,"}, {"start": 361.57, "end": 386.42, "text": "Non-Western perspectives and our own research. While not a complete solution, constitutional AI offers techniques for aligning LLMs with human values in a scalable manner. The second key challenge we focused on was establishing scalable oversight of Claude's behavior. As conversational AIs become more advanced and widely used, how can we ensure they behave appropriately?"}, {"start": 388.4, "end": 411.47, "text": "At Anthropic, we recognized early on the traditional techniques like manual content moderation will not suffice. Reviewing every chatbot interaction does not scale, and inevitably, some harmful responses will slip through. Instead, we take a multi-pronged approach, combining automated testing, red teaming, and crowdsourcing. I'll elaborate on each of these components."}, {"start": 411.47, "end": 436.82, "text": "First, we automatically generate a large, diverse set of test questions using language models themselves. As described in our paper, Discovering Language Model Behaviors with Model Written Evaluations, we create over 150 evaluations covering topics from social biases to privacy risks. Each test has over 1,000 examples, like asking if the chat bot wants to avoid being shut down or whether it exhibits racist beliefs."}, {"start": 436.82, "end": 463.6, "text": "We generate the data efficiently by instructing language models to produce examples of this unsafe behavior. Then we use other models to filter for quality. The pipeline lets us rapidly test for myriad potential failures at a scale and speed that would be challenging to reproduce with human workers. However, model-generated evaluations are not foolproof. Clever edge cases and exploits often require human creativity to discover. That's where red teaming comes in."}, {"start": 463.6, "end": 484.16, "text": "Red teaming complements automated testing by finding more subtle issues through human ingenuity. At Anthropic, we have teams of employees act as adversaries to uncover flaws before they impact external users. However, for frontier technologies like AI, red teaming requires investments in specialized expertise."}, {"start": 484.16, "end": 513.1, "text": "Over the last six months, we conducted an intensive biological red teaming project with top biosecurity experts. We spent over 150 hours working with these experts to probe our model's capabilities to output harmful biological information. The experts used a secure interface with our standard safety systems disabled. We observed the model exhibiting concerning capabilities, sometimes producing sophisticated, accurate knowledge at an expert level. This does not happen frequently today."}, {"start": 513.1, "end": 542.83, "text": "But we found indications that larger models are more capable. Models gaining access to tools could also advance their capabilities in biology. Taken together, we believe unmitigated models could accelerate efforts to misuse biology compared to just having internet access. Models may enable tasks not otherwise achievable. We estimate these risks could arise as soon as two to three years from now as models advance. On the bright side, discovering risks enables"}, {"start": 542.83, "end": 568.99, "text": "Developing mitigations. We found ways to update our training procedures that reduce harmful outputs by improving models' ability to distinguish between beneficial and dangerous uses. Classifier filters also make it harder to get the chained expert insights needed for harm. We have deployed these in our public model and identified further mitigations to experiment with. This work confirms frontier red timing is crucial and timely."}, {"start": 568.99, "end": 590.51, "text": "Current models show early warning signs, presenting a window to evaluate emerging risks and mitigate them before acute threats emerge. Longer term risks, like deception, also warrant proactive red teaming. By anticipating future dangerous capabilities models should lack, we can measure risks, build alignment, and inform mitigation efforts before these issues become severe."}, {"start": 590.51, "end": 618.14, "text": "Finally, we leverage the wisdom of crowds. External crowdsourcing provides diverse perspectives to catch biases. Crowd workers chat with our AI assistant and flag concerning responses. Aggregating many viewpoints helps reveal biases that our automated methods miss. In summary, scalable oversight of industrial chatbots requires evaluating from every angle. Automated model-written tests surface a breadth of potential pitfalls at massive scale."}, {"start": 618.14, "end": 651.3, "text": "Red teaming addresses gaps through targeted human creativity. Crowdsourcing collects diverse subjective viewpoints. While research provided the critical groundwork, seeing our methods succeed with millions of users has further validated our approach. Enough about research. Let's talk about putting these models into the world. We began collecting human feedback data in April 2021 and began to feel confident in our safety methods in 2022. But our API wasn't yet ready to reliably serve users at scale."}, {"start": 651.3, "end": 680.03, "text": "So we rewrote our software stack from scratch to ensure it could handle scaled user demand from day one. Most companies take a launch then iterate approach, but we guessed that as soon as we opened for business, we would be flooded with traffic. This prediction proved accurate, and thanks to our enterprise partnerships, we're on track to be one of the fastest growing companies by annualized revenue in history. We heard from our partners that a key barrier to leveraging LLMs like Claude is protecting their proprietary data, especially in regulated industries."}, {"start": 680.03, "end": 705.07, "text": "But they lacked the capability to monitor what kind of data was flowing through their API keys. So we structured our data use agreements to ensure we would never train generative models on user data, only trust and safety classifiers which cannot leak private information. We then surface specific warning flags showing partners things like how many of their requests contain spam, misinformation, or violence so they can take mitigating actions."}, {"start": 705.07, "end": 731.89, "text": "We frequently see spikes of abusive content as users discover new ways to circumvent our safety systems, such as creating fictional role-play scenarios. As many of you know, these exploits are often called jailbreaks. When a jailbreak occurs, we take a multilayered approach designed to quickly patch the attack factor and then follow up with a more comprehensive fix in a day or two. Over time, our goal is to improve both the speed and efficacy of these patches and fixes."}, {"start": 732.46, "end": 755.38, "text": "I'm proud to cite a recent Carnegie Mellon paper stating that while they achieved up to 84% attack success on OpenAI models, 66% attack success on Google's Palm II, on the contrary, Claude's robustness resulted in just a 2.1% success rate. This demonstrates the strength of our safety methods, though of course we still have work to do to bring that last 2% down."}, {"start": 755.38, "end": 771.02, "text": "Looking further ahead, we believe the stakes are enormous if AI alignment fails as progress continues. Rapid advances suggest human-level AI may arrive much sooner than expected, raising crucial questions around how to safely steer such powerful systems."}, {"start": 772.59, "end": 796.05, "text": "To proactively address risks from advancing AI, Anthropic has developed a framework called AI Safety Levels modeled after biosafety protocols. It defines escalating safety requirements as AI becomes more capable. Under this framework, Claude is rated AI Safety Level 2, or ASL2, meaning it shows signs of dangerous capabilities but lacks reliability."}, {"start": 796.05, "end": 826.16, "text": "In our responsible scaling policy, we have committed to increasingly strict standards before progressing to ASL 3 and beyond. For example, we will not deploy ASL 3 systems if red team testing reveals meaningful risks of catastrophic misuse. Anthropic's goal is to channel competitive dynamics towards solving alignment challenges in order to safely unlock further progress. We believe empirical research with more advanced models is essential to prepare for human level AI."}, {"start": 826.16, "end": 853.36, "text": "Understanding how to train and test near human systems will provide critical insight. In closing, an empirical iterative approach combining theory and hands-on research offers the most promising path ahead. Progress will require bold ideas, rigor, and deliberation around human values. If we work together proactively, AI can empower humanity while avoiding catastrophic risks."}, {"start": 853.36, "end": 873.81, "text": "Anthropic will continue driving progress toward helpful, honest, and harmless AI benefiting everyone. We urge others to join us in this mission and welcome partnership in charting a course toward a flourishing and just AI-powered future. I'm looking forward to the rest of the conference and to discussing these important issues with all of you. Thank you."}]}