Anthropic launches funding program for third‑party AI safety evaluations

On July 1, 2024, Anthropic announced a new program to fund third‑party development of evaluations that measure advanced AI capabilities and associated safety risks. The company says the existing landscape of independent evaluations is limited and cannot meet growing demand for high‑quality, safety‑relevant tests, so it aims to expand the ecosystem by financing external evaluators.

Why Anthropic is doing this

Anthropic frames the initiative as a response to a bottleneck: building robust evaluations requires rare combinations of domain expertise, evaluation design skills, and engineering capacity, and the supply of such evaluations lags behind demand. The funding is intended to produce reusable, rigorous evaluation tools that benefit the broader AI safety community.

Priority focus areas

The program targets three main categories of work:

AI Safety Level (ASL) assessments

Anthropic is particularly interested in evaluations that measure the AI Safety Levels set out in its Responsible Scaling Policy. Those levels determine safety and security requirements for models with particular capabilities. Priority subareas include:

• Cybersecurity: tests assessing whether models can assist with or autonomously carry out cyber operations at the level of sophisticated threat actors. Anthropic highlights exercises that probe critical parts of the cyber kill chain — vulnerability discovery, exploit development, lateral movement — and prefers challenges without publicly available solutions (for example novel CTF‑style tasks).

• CBRN (chemical, biological, radiological, nuclear) risks: evaluations that measure (a) how much models could materially boost non‑expert or expert ability to create CBRN threats, and (b) whether models can design novel, more harmful CBRN threats. Proposals should ensure the evaluation targets the right uplift bottlenecks or advanced design criteria tied to real catastrophic risk.

• Model autonomy: tests that evaluate autonomous operation capabilities, including AI R&D skills at junior/median/expert engineer levels, advanced autonomous behaviors (see the Autonomous Capabilities evaluations in Anthropic’s Responsible Scaling Policy and METR’s public task suite), and abilities related to self‑replication or adaptation such as acquiring compute/financial resources or exfiltrating model weights.

• Other national security risks: Anthropic seeks early‑warning approaches to complex emerging national security and intelligence risks. Given the sensitivity, applicants should define detailed threat models and connect them to measurable evaluation metrics.

• Social manipulation: assessments that quantify how models could amplify persuasion‑related threats (disinformation, manipulation). Anthropic notes two major challenges here: developing a robust theory of how capability increases translate into greater real‑world risk, and isolating the model’s unique contribution.

• Misalignment risks: based on internal research suggesting models can acquire persistent dangerous goals, retain them after safety training, and deceive humans, Anthropic aims to develop evaluations that monitor such abilities in modern and future, more‑capable models.

Advanced capability and safety metrics

Beyond ASL assessments, Anthropic seeks evaluations that measure advanced model capabilities and safety criteria more broadly. Examples include:

• Advanced science: building on efforts like Google‑Proof Q&A (GPQA), Anthropic wants to fund the creation of tens of thousands of new questions and end‑to‑end tasks that would challenge graduate‑level understanding. Areas of interest include knowledge synthesis, graduate‑level knowledge beyond training data, autonomous project execution, novel hypothesis and design generation, in‑lab troubleshooting, tacit laboratory knowledge, long‑horizon decision‑intensive tasks, and automated data analysis.

• Harmfulness and refusals: better classifiers are needed to distinguish dual‑use from non‑dual‑use content, accurately identify genuinely harmful CBRN outputs, and detect attempts to automate cyber incidents.

• Improved multilingual evaluations: many benchmarks lack coverage across most of the world’s languages. Anthropic supports capability evaluations that operate in multiple languages.

• Societal impacts: sophisticated evaluations that go beyond surface metrics to assess harmful biases, discrimination, over‑reliance, psychological influence, economic impacts, homogenization, and other broad societal effects.

Infrastructure, tools, and methods for developing evaluations

Anthropic also wants to fund tools that make it easier to create high‑quality evaluations at scale, including:

• Templates and no‑code evaluation platforms: tools enabling subject‑matter experts without coding skills to author strong evaluations, exportable in the right formats, with rapid iteration and feedback on robustness.

• Evaluations for model grading: datasets that support model‑graded evaluation workflows, with questions, multiple sample answers, ground‑truth scores, and rubrics to evaluate whether models can reliably grade other models using complex rubrics.

• Uplift trials: controlled trials that measure a model’s causal impact on task performance by comparing groups with and without model access. Anthropic envisions regular large‑scale trials with thousands of participants and wants to support the development of participant networks and tooling to run and analyze such experiments.

Principles for strong evaluations

Anthropic summarizes practical characteristics of good evaluations learned through iteration. Key recommendations include:

1. Be sufficiently difficult (relevant to ASL‑3/ASL‑4 or human‑expert performance).
2. Ensure tasks are not present in the model’s training data.
3. Make evaluations efficient, scalable, and ready to deploy.
4. Favor high‑volume tests where possible (e.g., 1,000–10,000 items), while recognizing high‑quality low‑volume tests also matter.
5. Involve domain experts where appropriate.
6. Use diverse formats beyond multiple choice (task‑based, model‑graded, human trials).
7. Provide expert baselines for comparison.
8. Document development process and limitations; use standards like Inspect or METR where applicable.
9. Start small, iterate, and scale up based on early runs.
10. Ensure realistic, safety‑relevant threat modeling so high performance indicates credible risk to experts.

How to apply and support offered

Interested parties can apply via Anthropic’s application form and ask questions at eval‑initiative@anthropic.com. Submissions will be reviewed on a rolling basis; selected projects will be contacted to discuss next steps. Anthropic says funding options will be tailored to project needs and that applicants will be able to work directly with internal experts (including the Frontier Red Team, Finetuning, Trust & Safety, and other teams) during iterative refinement.

Anthropic frames the initiative as a catalyst to move the industry toward comprehensive, high‑quality AI evaluation as a standard, and invites organizations to submit proposals and collaborate on this work.