A detailed OpenAI audit found substantial evaluation problems in the SWE-Bench Pro coding benchmark and estimates that roughly 30% of tasks are broken. The company stresses that accurate measurement of model capabilities is essential for safe deployment and for decisions guided by OpenAI’s Preparedness Framework.
Why SWE-Bench Pro was reviewed
SWE-Bench Pro was designed to address limitations of the earlier SWE-bench Verified by offering longer-horizon, more realistic development tasks intended to measure agentic coding abilities. As with the prior benchmark, tasks are programmatically sourced from feature changes in public and private repositories. Models must implement fixes that pass newly added tests without breaking existing functionality. On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% over eight months.
Audit methodology
OpenAI used a multi-stage data quality pipeline. An initial automated filter inspected task instructions, model attempts, and the grading tests and flagged 286 potentially broken tasks for deeper review. Those flagged tasks underwent two parallel review tracks:
- Investigator audits using Codex-based agents under human supervision, with access to the repository and environment so agents could run tests, inspect files, and analyze failure traces; summaries from multiple independent runs were reviewed by a researcher who made a final label decision.
- A human annotation campaign with experienced software engineers, each task reviewed independently by five engineers trained on the benchmark goals, issue taxonomy, and edge cases. Reviewers first judged the visible problem statement, test cases, and the ground-truth reference solution (the gold patch), then assigned labels and severity ratings; disagreements or low-confidence cases were escalated.
Results and rates
The automated datapoint pipeline flagged 200 tasks as broken (27.4%), while the human annotation campaign identified 249 tasks as broken (34.1%). Based on these analyses, OpenAI estimates that approximately 30% of SWE-Bench Pro tasks are unreliable indicators of model capability.
The detected issues clustered into four broad categories used in the audit, though reviewers frequently applied multiple labels per task, indicating overlapping or compound problems. The agent pipeline and human judgments agreed on category labels in 74% of flagged cases.
One notable divergence was in the "low-coverage tests" category: human reviewers marked this as the most common issue for 9.4% of the benchmark, whereas the agent pipeline identified it in 4.1% of cases.
An illustrative example
One examined task required normalizing table-of-contents entries and rendering them back to Markdown via TocEntry.to_markdown(). The task prompt explicitly described character-level spacing and gave examples containing a single leading space (e.g., " | Chapter 1 | 1"). However, the hidden test assertions expected two leading spaces (" | Chapter 1 | 1"). A model that follows the visible prompt would therefore fail the hidden tests due to a one-character difference, and the task would be marked incorrect despite the model adhering to the prompt examples.
Why these problems arise
OpenAI attributes part of the problem to the provenance of benchmark tasks: extracting examples from pull requests and commits means prompts, merged code, and unit tests were created for human collaboration, often with back-and-forth between contributors and maintainers, not as isolated, implementation-agnostic evaluation items. Tests written for pull requests can be narrowly tailored to validate a specific change rather than defining a general, solution-agnostic standard.
At the same time, improving model capabilities make it easier to detect evaluation flaws at scale: models themselves can be used to inspect prompts, tests, patches, traces, and edge cases with greater depth and consistency than was previously practical.
Consequences and recommendations
Given these findings, OpenAI retracts its earlier recommendation that the community adopt SWE-Bench Pro. The organization advises model developers to scrutinize benchmark results carefully and to verify task quality before relying on those results for safety and deployment decisions.
OpenAI encourages the evaluation community to build new benchmarks by experienced software developers specifically to test model capabilities. That approach can preserve realism and difficulty while enabling stronger human oversight and clearer, implementation-agnostic test design.
The audit’s stated goal is to ensure that task failures reflect genuine model limitations and that task successes correspond to complete, valid solutions to the prompt requirements. Because these evaluations inform OpenAI’s deployment and safety decisions, they emphasize the need for benchmarks that are hard to game, easy to trust, and truly reflective of model capability or alignment.
