AI capabilities continue to improve exponentially in some areas, but models still lag behind humans on open-ended tasks requiring judgment, according to a new report from METR, a nonprofit that evaluates artificial intelligence. The distinction could matter especially for the automation of AI research and development.
The report updates the organization’s so-called time horizon plot. In the original assessment, humans and models attempted the same tasks, with difficulty characterized by how long human testers took to complete them. Models were rated according to the length of tasks they could successfully finish. The plot showed that the duration of tasks the most advanced systems could complete doubled every four months. The findings were often misinterpreted, however, because of the complexity of the measures.
Harder skills to measure
The updated assessment adds tasks estimated to take longer, as well as more open-ended work. A central question is whether progress on tasks whose answers can be checked automatically carries over to work where success is harder to establish.
In coding and mathematics, the correctness of a solution can often be verified automatically. That makes reinforcement learning possible: a system tries repeatedly, and training reinforces behavior that leads to success. The same approach is much harder to apply to choosing a new research direction, for example, where the value of an outcome may be unclear or take time to emerge. Such tasks are also harder to evaluate, so they were absent from the original time horizon plot, even though they may be important for automating AI research and development.
Interest in the subject was heightened by another, heavily redacted METR report published earlier in the month. It claimed that AI could be doubling the pace of AI progress at Anthropic. The report prompted concerns about full recursive self-improvement, in which AI systems design their successors without human involvement.
Improvement, but not yet usable results
In one task in the new suite, models had to design AI evaluations that METR could use in its research; employees of the organization scored the results. Although the most advanced models are now better than almost every human at software engineering, they fell short on this task: none of the AI-designed evaluations was judged useful for METR’s research.
Another task asked models to assess philosophical arguments as a test of judgment. Here too they performed below humans, although METR employees disagree about how relevant the task is to automating AI research and development.
The newest models nevertheless show improvement. Opus 5.5, for example, often reaches what METR calls “borderline above the bar” on evaluation design: the result is not yet usable in the organization’s research, but is promising. A year earlier, every model fell into the “clearly below the bar” category.
Debate over what progress means
Sayash Kapoor, a computer scientist and incoming professor at the University of California, Berkeley, welcomed METR’s experimentation with new kinds of evaluation. He said models’ weaker performance on open-ended tasks was unsurprising: in an August preprint, he and his collaborators reported that AI models cannot yet independently reproduce human computer science research. In a Wednesday blog post, OpenAI also reported that its models perform worse on real-world tasks assigned by its employees than on the easier-to-verify tasks in METR’s original suite.
According to Kapoor, there is still disagreement over what the gap between the two types of task means. He believes research judgment—the ability to select promising questions, hypotheses and experiments—will be a major bottleneck to automation. Others contend that models trained on verifiable tasks will eventually succeed at open-ended work too. Progress on verifiable tasks, meanwhile, shows no apparent signs of slowing.
Najoung Kim, a linguist and computer scientist at Boston University, said the improvement observed now reveals something about the present, but it is not yet clear how to extrapolate from it. Matt Putz, a member of METR’s technical staff, said that tasks that are hard to verify or require judgment are inherently harder to study and quantify, making it unlikely that trends can be established as cleanly and robustly as for tasks that are easy to specify.
Limits of the measurements
Practical constraints also affect the updated plot. While humans actually completed the tasks in the original assessment to establish how long they took, most completion times in the new report are estimates. Human testing of tasks that could take hundreds of hours, or that only a few experts are qualified to perform, would be prohibitively expensive and time-consuming.
For other tests, the scoring method is a limitation. Most of the philosophical arguments, for example, were rated by a single person, and models were assessed on how closely they agreed with that individual. Consulting more experts could change the results, but would also take time.
The experts consulted see the inclusion of open-ended, hard-to-verify tasks as a step forward. But the small number of such tests and uncertainties in scoring mean that no definitive conclusion can yet be drawn about whether automated AI research and development will arrive, or how quickly.



