Anthropic's Claude Opus 5.5 (xhigh) scored 18.5% "resolved" on ProgramBench, a benchmark published in May that evaluates how well large language models can rebuild software from a compiled program and its documentation. "Resolved" denotes tasks where all tests pass; "almost" denotes tasks where more than 95% of tests pass.
What ProgramBench tests
ProgramBench asks models to reconstruct a variety of real software projects. Tasks include:
- language interpreters (for example PHP, Lua, tinycc),
- databases (DuckDB, SQLite),
- media and compression tools (FFmpeg, zstd, xz),
- developer utilities (ripgrep, fzf, jq).
Scores are computed over 200 tasks: the "resolved" fraction is the share of tasks fully passing their test suites, while "almost" is the share passing over 95% of tests.
How models have improved
When ProgramBench was released in May, every tested model scored 0% "resolved." At that time the best performer was Claude Opus 4.7, with 0% "resolved" and 3% "almost." By comparison, Claude Opus 5.5 (xhigh) now posts 18.5% "resolved" and 70.5% "almost."
These figures indicate a substantial improvement in the months since the benchmark's release in the ability of models to reconstruct non-trivial software components.
Why this matters
Progress on ProgramBench gives a window into how quickly models are improving at tasks related to autonomous or end-to-end software construction. Observers of the benchmark expect scores to continue rising, which could signal broader shifts toward more automated software development workflows if the trend persists.
Limits and caveats
While ProgramBench shows fast progress, benchmarks have noise floors and limitations. The jump to 18.5% "resolved" and 70.5% "almost" is notable but not equivalent to full automation of software engineering. ProgramBench measures a specific reconstruction task; its results should be interpreted cautiously and not overgeneralized.
Sources
Further technical details are available in the ProgramBench paper (ProgramBench: Can Language Models Rebuild Programs From Scratch?, arXiv).
