Anthropic describes a new automated, open‑source evaluation for measuring political even‑handedness and applies it to Claude models and several competitor models. Using 1,350 paired prompts across 150 topics, the company reports Claude Opus 4.1 and Claude Sonnet 4.5 scored 95% and 94% on even‑handedness, respectively; Gemini 2.5 Pro and Grok 4 performed similarly, while GPT‑5 and Llama 4 scored lower. Anthropic publishes its methodology and data to encourage reproducibility and industry standards, and notes important limitations including U.S. focus, single‑turn prompts, and grader choice.
Purpose and why it matters
Anthropic’s stated aim is for Claude to handle political topics in an even‑handed way: giving opposing viewpoints comparable depth, engagement, and quality of analysis without promoting particular ideological positions. They argue that models that systematically favor some views undermine users’ autonomy and fail at assisting users to form independent judgments.
Ideal behaviors Anthropic trains toward
Anthropic lists desired behaviors for Claude: avoid unsolicited political opinions, prioritize balanced information, maintain factual accuracy and comprehensiveness, present best cases for multiple viewpoints (an ‘‘Ideological Turing Test’’ approach), favor neutral terminology, and engage respectfully. The company uses a system prompt and character training to nudge model behavior toward these traits.
Character training examples
Since early 2024 Anthropic has included reinforcement‑learning‑driven character traits promoting even‑handedness. Example instructions used in training emphasize that the model should not produce rhetoric intended to manipulate political views or serve as propaganda, should avoid taking strong partisan stances on complex topics, and should present different perspectives without expressing personal opinions on sensitive political issues.
Evaluation method: Paired Prompts (automated)
The Paired Prompts method prompts a model with two requests about the same politically contentious topic framed from opposing ideological perspectives and compares the responses. It measures three dimensions:
- Even‑handedness: whether responses to both prompts show comparable depth, engagement, and evidence.
- Opposing perspectives: whether the response acknowledges and presents counterarguments or qualifications.
- Refusals: whether the model declines to engage with the prompt.
For speed and consistency, Claude Sonnet 4.5 was used as the automated grader in the main analysis. As validity checks, some subsamples were graded by other Claude models and by OpenAI’s GPT‑5. The grader prompts and implementation are available in the open‑source repository.
Models, settings, and dataset
Anthropic tested its most capable models, Claude Sonnet 4.5 and Claude Opus 4.1 (both with ‘‘extended thinking’’ off and using the current Claude.ai system prompt). Comparator models were OpenAI GPT‑5 (low reasoning mode, no system prompt), Google DeepMind Gemini 2.5 Pro (lowest thinking configuration, no system prompt), xAI Grok 4 (thinking on, with its system prompt), and Meta Llama 4 Maverick (with its system prompt).
The evaluation comprised 1,350 prompt pairs covering 9 task types and 150 topics, including reasoning, persuasive/formal writing, narratives, analytical research questions, evidence evaluation, opinion prompts, and humor. The set was designed to capture both pro/con arguments and realistic ways users with different political leanings might ask for help.
Results
Even‑handedness scores:
- Claude Opus 4.1: 95%
- Claude Sonnet 4.5: 94%
- Gemini 2.5 Pro: 97% (nominally higher)
- Grok 4: 96% (nominally higher)
- GPT‑5: 89%
- Llama 4: 66% Anthropic notes the differences among the top four models were small, while GPT‑5 and Llama 4 scored lower in this analysis.
Opposing perspectives (frequency of acknowledging counterarguments):
- Opus 4.1: 46%
- Claude Sonnet 4.5: 35% (corrected from an earlier 28% figure on 2025‑11‑24)
- Grok 4: 34%
- Llama 4: 31%
Refusals (rate of declining to engage):
- Opus 4.1: 5%
- Claude Sonnet 4.5: 3%
- Grok 4: near 0%
- Llama 4: 9% (highest among tested models)
Grader reliability checks
Anthropic examined grader agreement in two ways: per‑sample agreement and overall result correlation. In per‑sample agreement for even‑handedness, Claude Sonnet 4.5 agreed with GPT‑5 92% of the time and with Claude Opus 4.1 94% of the time. By contrast, a comparable human rater pair produced about 85% agreement. Overall score correlations between Sonnet 4.5 and Opus 4.1 were very high (r>0.99 for even‑handedness), and Sonnet 4.5 vs GPT‑5 correlations were also strong (r=0.86 for even‑handedness).
These checks suggest the main findings are not strongly dependent on which model was used as the grader, though some variance exists.
Limitations and caveats
Anthropic enumerates several caveats:
- The evaluation emphasizes even‑handedness, opposing perspectives, and refusals, but other bias dimensions exist and might produce different results.
- The analysis primarily focuses on current U.S. political discourse and does not assess international contexts or evolving future debates.
- The test is single‑turn only and does not capture multi‑turn conversational dynamics.
- Model configuration and system‑prompt differences between providers may affect comparability.
- Results can vary between runs because each run generates fresh responses.
The authors note there is no universally accepted definition or measurement of political bias, and that broader or stricter criteria could change which models are judged even‑handed.
Open source materials and next steps
Anthropic has published the implementation details, dataset, and grader prompts on GitHub so other researchers and developers can reproduce the Paired Prompts analysis, run alternative configurations, and contribute to developing shared measurement standards for political bias in AI.
Change log
2025‑11‑24: Corrected the percentage of Sonnet 4.5 responses acknowledging opposing viewpoints from 28% to 35%, and updated the accompanying figure to reflect the correction.
