For years the SWE-bench family (SWE-bench, SWE-Bench Pro, SWE-bench Multilingual and SWE-bench Verified) served as the primary measurement of large language models' (LLMs) and agentic systems' software-engineering abilities, especially in bug fixing. As models began to routinely perform well on those tasks — partly because benchmark problems may have entered training data — the community has introduced more difficult and less likely-to-be-contaminated benchmarks.
What’s new
Three recently released benchmarks have emerged as strong candidates to succeed the SWE-bench family: DeepSWE, ProgramBench and ITBench-AA. Each raises difficulty in different ways:
- DeepSWE measures agents' ability to implement features in problems that are harder to diagnose and require more code.
- ProgramBench assesses how well an agent can turn ideas into working programs from prompts.
- ITBench-AA evaluates diagnosing incidents in modern hardware and software stacks.
DeepSWE — Datacurve
DeepSWE was developed by Datacurve and is closest in intent to the original SWE-bench, but contains substantially harder examples. Key points:
- It includes 113 problems across 5 programming languages.
- Problems and tests are human-vetted; the benchmark minimizes the risk of contaminating training datasets by drawing inspiration from private codebases rather than reusing solved public code.
- Tasks are based on real repositories but are not direct extracts of existing, solved code. One example task asks to “Extend indexing ranges so arrays and strings support a third slice component: value[start:end:step]” in an ABS language GitHub repository.
Testing is performed with the mini-swe-agent harness. Prompts are relatively brief compared with the detailed prompts used in SWE-Bench Pro; the agent must use an LLM to produce a solution from multiple acceptable possibilities. Solutions require roughly 5.5 times more lines of code than SWE-Bench Pro tasks. DeepSWE uses human-written problems and verification tests rather than automatically generated examples.
Performance: at publication GPT-5.5 (set to xhigh reasoning) leads DeepSWE, solving 70 percent of problems. Claude Opus 4.8 solved 58 percent, while Gemini 3 Flash solved 5 percent — a 65-point spread among the top models.
ProgramBench — Meta, Stanford and Harvard
ProgramBench was developed by researchers at Meta, Stanford University and Harvard. The benchmark tests how well a model controlled by the SWE-agent harness can convert 200 ideas into functional programs without human oversight.
The agent has access to a console that can execute existing programs; its task is to reproduce a program by producing the inputs and expected outputs. To build the benchmark the authors used an agent (either mini-SWE-agent or SWE-agent) with Claude Sonnet 4.5 to (i) identify a candidate repository, (ii) construct an executable program, (iii) generate tests demonstrating program behavior on various inputs, and (iv) assemble a testing environment including a compiled executable, documentation for usage, and test assets the model might not generate itself (for example, images).
Target programs vary from easy to hard. An easy example is entr, which runs a command when a file changes. A harder example is ffmpeg, which encodes, decodes and otherwise processes audio and video.
So far no model has produced programs that pass all tests. Lowering the pass threshold to 95 percent, Claude Opus 4.7 reproduced 3 percent of programs, Claude Opus 4.6 reproduced 2.5 percent, and Claude Sonnet 4.6 reproduced 1.6 percent. At the time of publication no other model had reproduced any program.
ITBench-AA — IBM and Artificial Analysis
ITBench-AA was developed by IBM together with the independent testing lab Artificial Analysis as an update to IBM’s earlier ITBench. It evaluates how well a model controlled by Artificial Analysis’ Stirrup harness can diagnose the technical conditions that cause software systems to fail — for example running out of memory or incorrect configuration changes.
ITBench-AA comprises 59 human-written incidents grounded in real-world events. Each incident includes alerts, events, error traces, system metrics, manifests of involved applications, and a human-provided ground-truth root-cause diagnosis. One example incident included seven different alerts about high error rates; the diagnosis was human error due to servers being taken offline for maintenance.
ITBench-AA measures full recall: the ratio of correct diagnoses to all diagnoses. If a model misses any root cause for an incident, it receives zero for that incident.
Measured results: among tested models Claude Opus 4.7 (set to max reasoning) achieved the highest full recall at 46.7 percent. GPT-5.5 (xhigh reasoning) achieved 45.8 percent. At the low end Llama 3.3 70B achieved 0.6 percent — a spread of more than 40 percentage points between best and worst.
Why this matters
For years SWE-bench variants were the go-to measures of agentic capabilities in software engineering. As models saturated those benchmarks and agents took on longer, more open-ended tasks with fewer and less-consistent human instructions, new benchmarks were needed that (1) add complexity, (2) require more substantive code or reasoning, and (3) are unlikely to appear in models’ training sets.
DeepSWE, ProgramBench and ITBench-AA adopt different strategies to raise the bar: more lines of code and human-vetted problems, end-to-end program synthesis and reproduction, and diagnosis of real incidents in complex stacks. Together they give a more demanding and realistic assessment of what agentic systems can and cannot yet do.
Note
The authors of the source publication observe that progress is encouraging — agents have come a long way — but there remains substantial room for improvement before we have reliable, fully autonomous software-engineering agents.
