On August 12, 2025, Anthropic published details about its Safeguards team and the layered protections it builds around the Claude family of models. The company says it aims to amplify human potential while preventing misuse that could cause real-world harm.
The Safeguards team and its approach
The Safeguards team brings together experts in policy, enforcement, product, data science, threat intelligence, and engineering. Their remit is to identify potential misuse, respond to threats, and construct defenses that keep Claude both helpful and safe. Anthropic describes a lifecycle approach that covers policy development, model training influence, testing for harmful outputs, real-time enforcement, and identification of novel attacks.
Policy development
Safeguards authors the Usage Policy, which defines acceptable and prohibited uses of Claude. The policy addresses critical areas including child safety, election integrity, and cybersecurity, and provides industry-specific guidance for sectors like healthcare and finance.
Two mechanisms guide policy work:
-
Unified Harm Framework: an evolving framework that helps the team assess potential harms across five dimensions — physical, psychological, economic, societal, and individual autonomy. It functions as a structured lens rather than a formal scoring system, taking into account likelihood and scale when shaping policy and enforcement.
-
Policy Vulnerability Testing: Anthropic partners with external domain experts to identify areas of concern and then stress-tests those areas against their policies by prompting the models with challenging inputs. Partners include specialists in terrorism, radicalization, child safety, and mental health. For example, during the 2024 U.S. election cycle, Anthropic worked with the Institute for Strategic Dialogue to surface when Claude might return outdated information; they subsequently added a banner on Claude.ai directing users seeking election information to authoritative sources such as TurboVote.
Training and fine-tuning Claude
Safeguards collaborates closely with fine-tuning teams to reduce the risk of harmful model behaviors. These collaborations involve extended discussions about what behaviors Claude should exhibit and which traits to build into the model during training. Evaluation and detection processes surface problematic outputs; when issues are found, the team can update reward models or adjust system prompts in deployed models.
Anthropic also works with domain specialists to refine how Claude handles sensitive scenarios. For instance, ThroughLine, a leader in online crisis support, provided expertise to inform Claude’s responses to self-harm and mental health situations so the model responds with nuance rather than reflexive refusal or misinterpretation.
As a result of this process, Claude learns to decline assistance with harmful illegal activities, to recognize attempts to generate malicious code or fraudulent content, and to discuss sensitive topics carefully while distinguishing those topics from real attempts to cause harm.
Testing and evaluation before release
Before deploying a new model, Anthropic evaluates its performance and capabilities through several assessments:
-
Safety evaluations: assessing compliance with the Usage Policy on topics such as child exploitation and self-harm. Tests include clear violations, ambiguous contexts, and extended multi-turn dialogues. Anthropic uses models to grade Claude’s responses, with human review as an additional accuracy check.
-
Risk assessments: for high-risk domains (for example cyber harm or CBRNE — chemical, biological, radiological, and nuclear weapons and high-yield explosives), Anthropic conducts AI capability uplift testing with government and industry partners, defines threat models, and tests safeguard performance against those threats.
-
Bias evaluations: checking whether Claude gives reliable and accurate responses across contexts and users. For political bias, prompts representing opposing viewpoints are compared and scored for factuality, comprehensiveness, equivalency, and consistency. The company also tests for biased outputs on topics like jobs and healthcare when identity attributes (gender, race, religion) are included.
Pre-deployment testing informs whether additional guardrails are needed. For example, during pre-launch evaluation of its computer use tool, Anthropic found it could be used to amplify spam generation and distribution; in response, they developed new detection methods and enforcement options before launch, including the ability to disable the tool for accounts showing signs of misuse and added protections against prompt injection. Results from these evaluations are published in system cards released with each model family.
Real-time detection and enforcement
Once models are deployed, Anthropic uses a combination of automated systems and human review to detect harm and enforce the Usage Policy.
Detection and enforcement rely on a set of prompted or specially fine-tuned Claude models called "classifiers". These classifiers detect specific policy violations in real time; multiple classifiers can run simultaneously alongside the main conversation flow. For child sexual abuse material (CSAM) detection on first-party products, uploaded images’ hashes are compared against databases of known CSAM.
Based on classifier outputs, Anthropic can take enforcement actions such as:
-
Response steering: adjusting how Claude interprets and answers prompts in real time to prevent harmful output. For instance, if a classifier detects a user may be attempting to generate spam or malware, the system can automatically add instructions to Claude’s system prompt to steer its response. In narrow cases, Claude’s response can be stopped entirely.
-
Account enforcement actions: investigating patterns of violations and taking account-level measures, from warnings to account termination in severe cases. The company also maintains defenses against fraudulent account creation and misuse of services.
Building these enforcement systems is technically challenging: classifiers must process trillions of input and output tokens while limiting compute overhead and avoiding unnecessary enforcement on benign content.
Ongoing monitoring and investigation
Anthropic monitors harmful Claude traffic beyond single prompts and accounts to understand prevalence and to detect sophisticated attack patterns. Key elements include:
-
Claude insights and observations: an insights tool measures real-world Claude usage and analyzes traffic in a privacy-preserving way by clustering conversations into high-level topics. Research informed by these insights (for example on the emotional impacts of Claude use) can guide additional guardrails.
-
Hierarchical summarization: used to monitor computer-use capabilities or potential harmful cyber use by condensing individual interactions into summaries and analyzing those summaries to identify account-level concerns — behaviors that may only appear violative in aggregate, such as automated influence operations.
-
Threat intelligence: studying the most severe misuses, identifying adversarial patterns that detection systems might miss, and cross-referencing indicators of abuse (for example unusual spikes in account activity) with external threat data such as open-source repositories or industry reporting. Anthropic also watches channels where bad actors may operate — social media, messaging platforms, hacker forums — and shares findings in public threat intelligence reports.
Collaboration, feedback, and hiring
Anthropic emphasizes that safeguarding AI is a challenge that requires external collaboration. The company seeks feedback and partnerships from users, researchers, policymakers, and civil society organizations, and runs a public bug bounty program to test defenses.
To support this work, Anthropic is actively recruiting for roles on the Safeguards team; interested candidates are directed to the company’s Careers page for more information.
