Epoch and METR have published MirrorCode, a benchmark designed to evaluate how well AI systems can reconstruct software projects that would normally take humans a long time to implement. The benchmark was first announced in April (Import AI #453) and has now been expanded and released with additional tests.
What MirrorCode does
MirrorCode measures whether an AI can reimplement a target program using only command‑line interaction with that program, without access to the original source code or the web. The task requires devising the overall structure and behavior of the program from its input–output behavior, rather than translating code piece‑by‑piece.
Example target programs and sizes
- pkl: a programmable configuration language developed by Apple — 61,000 total lines of code
- gotree: a tool for parsing and manipulating phylogenetic trees — 16,000 lines of code
- qsv_select: a utility for selecting and reordering columns in CSV data — 87,000 lines of code
Key results
MirrorCode is tractable but challenging. Out of 25 target programs, 17 had at least one perfect‑scoring run; four additional targets had near‑perfect runs scoring over 99%. At the same time, 8 of 25 targets were never solved to the 100% threshold, and 4 of 25 were never solved to 99%.
Notable concrete outcomes from the released results:
- Claude Opus 4.7 and GPT‑5.5 successfully reimplemented gotree across several programming languages; these runs incurred inference costs in the range of $100–400.
- Opus 4.7 also reimplemented larger targets such as pkl in at least one run.
- In one case, Opus 4.7 solved a task in 14 hours with an inference cost of $251; Epoch and METR estimate that the same task would take a human between 2 and 17 weeks to complete.
The authors also observe rapid progress: leading models from about a year earlier would have scored roughly 30% and were limited to simpler programs, such as a calendar utility.
Where AI struggled
The program that posed the greatest challenge was ruff, a Python linter and formatter. AI performance was also notably weak on the mathematics package giac_subset and on the email authentication library mailauth.
Release contents
The public release of MirrorCode includes a scaffold and 22 of the 25 MirrorCode target programs, totalling 132 task instances across six programming languages.
Why this matters
The benchmark demonstrates two related facts. First, AI systems have materially improved at coding and can reimplement substantial software with relatively low time and inference cost. Second—and perhaps more consequential—the results show AI models can orient themselves to an unfamiliar environment using only black‑box interaction: they can observe a program’s behavior and produce a ground‑up implementation of it. This indicates that capable agents might be able to bootstrap internal capabilities from external systems they interface with, a dynamic with implications for how AIs could learn from and replicate aspects of the software systems they encounter.
The release gives researchers and industry a standardized way to measure progress on long‑horizon programming tasks and to track improvements in model capabilities over time.
