Anthropic’s practical lessons from implementing a Responsible Scaling Policy

On May 20, 2024, Anthropic published a post reflecting on its experience implementing its first Responsible Scaling Policy (RSP), which aims to address catastrophic safety failures and misuse of frontier models. The company framed the RSP as an effort to translate high-level safety concepts into practical guidance for fast-moving technical organisations and to test those ideas as potential standards. Anthropic says it has learned from the operationalisation process and plans to publish an updated RSP.

Why the policy mattered in practice

Anthropic reports that a clearly articulated policy for catastrophic risks provided structure for clarifying organisational priorities and framing discussions about project timelines, headcount, threat models and trade-offs. Implementation surfaced important questions, projects and dependencies that might otherwise have taken longer to identify or remained unaddressed.

At the same time, the company notes the tension between making strong commitments and acknowledging that many answers remain uncertain. Some parts of the original policy required clarification, and where open research questions exist, overly prescriptive requirements are unlikely to be durable. Anthropic expresses the hope that voluntary commitments will move toward industry best practices and, eventually, well-designed regulation as commercial pressures grow.

Five high-level commitments

Anthropic summarises its current framework as five commitments:

• Establishing Red Line Capabilities: identify and publish "Red Line Capabilities" that could appear in future model generations and would pose excessive risk under current safety and security practices (referred to as the ASL-2 Standard).

• Testing for Red Line Capabilities (Frontier Risk Evaluations): demonstrate that Red Line Capabilities are not present in models, or if they cannot prove absence, act as if the capabilities are present. This involves designing empirical Frontier Risk Evaluations with domain experts; failed evaluations would provide strong evidence a model is at or near a red line. Anthropic commits to maintaining a clear evaluation process and public summaries of current evaluations.

• Responding to Red Line Capabilities (ASL-3 Standard): develop and implement the ASL-3 Standard — a set of safety and security measures sufficient to manage models with Red Line Capabilities. Anthropic will define the mitigation measures and the assurance process to validate their effectiveness, and will pause training or deployment if necessary until ASL-3 measures can be applied.

• Iteratively extending the policy: before performing activities that would require ASL-3, publish the upper bound of ASL-3’s suitability by specifying new Red Line Capabilities that would require an even higher ASL-4 standard and build Frontier Risk Evaluations for those capabilities. Maintain a transparent evaluation process.

• Assurance Mechanisms: implement assurance mechanisms to ensure the policy is executed as intended. These mechanisms should stress-test the evaluation process; have mitigations validated publicly or by disinterested experts; give the Board of Directors and the Long-Term Benefit Trust sufficient oversight to identify non-compliance; and ensure the policy is updated appropriately.

Threat modelling and evaluations

Anthropic’s Frontier Red Team and Alignment Science teams have focused on threat modelling and engaging domain experts. Their priorities are (a) improving threat models to determine which capabilities would justify ASL-3, (b) working with teams developing ASL-3 controls so they target the right risks, and (c) mapping capabilities that ASL-3 would not mitigate and that require continued testing.

Key reflections include:

• Emergent capabilities in each new model generation make forecasting future model properties unusually difficult; more threat modelling is needed.
• Experts disagree about which risks to prioritise and how novel capabilities could cause harm, including within established CBRN (Chemical, Biological, Radiological, and Nuclear) domains; therefore, consultations across subdomains are valuable.
• Quantifying threat models has helped prioritise capabilities and scenarios.

Several Anthropic teams (Frontier Red Team, Alignment Science, Finetuning, Alignment Stress Testing) develop evaluations and refine methodology. Currently, they run pre-deployment tests in cybersecurity, CBRN and Model Autonomy for frontier models that reach four times the compute of their most recently tested model; they also test models mid-training if the threshold is reached, and they re-test their most capable model every three months to account for improvements from fine-tuning. Anthropic references detailed results from its evaluations on Claude 3 Opus.

Teams are building evaluations in new domains and making the testing process more robust. Practical lessons include:

• Fast iteration with domain experts is particularly useful to detect poorly calibrated tests or tasks that diverge from the threat model.
• Anthropic should leverage and encourage the growing ecosystem of external researchers and firms; many risks, especially around autonomy and misalignment, are complex and speculative, and internal testing is likely incomplete.
• Different evaluation methodologies have distinct strengths and weaknesses, and the most compelling method depends on the domain and threat model.

Methods discussed:

• Question-and-answer datasets are easy to design and run but may poorly reflect real-world risk due to constrained formats.
• Human trials comparing participants with model access to those using search engines are valuable for misuse-related domains but are time- and resource-intensive; they require robust baselines, sufficient sample sizes, and careful statistical inference.
• Automated task evaluations are informative for autonomous behaviours but building realistic virtual environments and secure infrastructure is engineering-intensive.
• Expert red-teaming and transcript reviews, while less reproducible, have been useful for open-ended exploration of capabilities and gathering expert judgement.

Anthropic lists several open research questions for building a reliable evaluation process and invites broader community engagement.

Elicitation and guarantees

Anthropic seeks evidence of risk and appropriate mitigations before dangerous thresholds are reached, which requires extrapolating from current data to future risk. Ideally, smooth "scaling laws" would allow prediction of when dangerous capabilities arise; in future, Anthropic hopes to predict more precisely how much more capable the next generation will be in particular domains.

However, techniques such as domain-specific reinforcement learning, prompt engineering and supervised fine-tuning can make it impossible to guarantee that testing elicits every relevant capability. A robust testing process therefore involves concerted efforts to pass evaluations and improve elicitation methods, simulating how well-resourced malicious actors might bypass controls and access model weights. Anthropic recognises the lack of a clear boundary between aggressively eliciting a dangerous capability and training a model to have that capability, and aims to make more precise policy claims about sufficient elicitation in future updates.

The company also values making its risk assessment process externally legible, pre-specifying test outcomes that would indicate intolerable unmitigated risk. Clear commitments help resist production pressures that could relax standards, although they can result in crude thresholds; Anthropic plans to explore ways to aggregate diverse evidence sources while keeping verifiable, legible commitments.

ASL-3 standard workstreams

Anthropic’s Security, Alignment Science and Trust and Safety teams are developing the ASL-3 standard. The objective is to design controls to mitigate the risk of model weights being stolen by non-state actors and to reduce misuse across product surfaces. ASL-3 should suffice for many models where low rates of misuse could be catastrophic but would not be sufficient against state-backed actors.

Highlights:

• Plans to scale research on classifier models for automated detection and response, and to strengthen traditional trust & safety operations across product surfaces (e.g., Vertex, Bedrock, Claude.ai).
• For human misuse, a defence-in-depth approach is expected: combining reinforcement learning from human feedback (RLHF) and Constitutional AI, multi-stage classifiers detecting misuse at prompt, completion and conversation levels, plus incident response and jailbreak patching.
• Scaling the security programme required a surge of effort: around 8% of all Anthropic employees now work on security-adjacent areas, and Anthropic expects that share to grow as models become more valuable targets.

Implementing ASL-3 will require changes across day-to-day workflows; the security team has invested in partnerships with researchers and other teams to preserve productivity while applying state-of-the-art cybersecurity controls. Anthropic’s threat model treats insider device compromise as the highest-risk vector, prompting a focus on multi-party authorization and time-bounded, least-privilege access to reduce the risk of model weight exfiltration. The company notes that an existing culture of peer review eased adoption of multi-party controls.

Because the field moves quickly, Anthropic prefers to sketch expected mitigations at a high level and pair them with an attestation process, rather than rigidly pre-specifying all technical controls. Attestation may include detailed control lists, review from disinterested experts and board approval.

Assurance structures and governance

Responsible Scaling, Alignment Stress Testing and Compliance teams are exploring governance, coordination and assurance structures. Anthropic plans to introduce more independent checks over time and is hiring a Risk Manager to develop these structures using best practices from other industries.

Notable points:

• The cross-functional complexity requires strong central coordination; Anthropic will continue building a Responsible Scaling Team to manage dependencies.
• A "second line of defence" that takes an adversarial approach to core workstreams is valuable; the Alignment Stress Testing team has already stress-tested evaluations and policy execution and supplied reflections to the Board of Directors and to the U.S. Department of Commerce Bureau of Industry and Security.
• Anthropic provides regular updates to its Board of Directors and the Long-Term Benefit Trust and shares evaluation reports and quarterly progress updates with employees. It has implemented a non-compliance reporting policy that allows anonymous reporting to the Responsible Scaling Officer.

Conclusion and next steps

Anthropic stresses that responsibly training and deploying future frontier models requires serious investment from both the company and wider industry and governments. The Responsible Scaling Policy has helped focus many teams on shared objectives and driven substantial cross-company engagement, but significant work remains. Anthropic published these reflections ahead of the AI Seoul Summit to continue discussion on empirically grounded frameworks for managing frontier model risks and to encourage other companies to adopt and share their own frameworks and experiences.