A consortium of universities and industry groups has released OSWORLD 2.0, a benchmark designed to evaluate how well AI systems perform multi‑step, multi‑program tasks in realistic computer environments. Participants in the project include the University of Hong Kong, the University of California at San Diego, Columbia University, the University of California at Santa Barbara, Mila, Snorkel AI, the University of Wisconsin, Alibaba Qwen, The Ohio State University, Simular, and NeoCognition.
What OSWORLD 2.0 measures
OSWORLD 2.0 contains 108 long‑horizon tasks, including 31 self‑hosted websites. Each task is defined as a self‑contained end‑to‑end workflow: an agent is given a high‑level user goal, realistic artifacts, a stateful computer environment, and a scoreable final state. The authors note that 69.6% of tasks are estimated to take a skilled human more than one hour.
The benchmark is substantially more complex than OSWORLD 1.0: while the median task in OSWORLD 1.0 took about 2 minutes for a human, the median OSWORLD 2.0 task takes roughly 1.6 hours — about 48 times longer.
Expanded software ecosystem and task categories
Compared with OSWORLD 1.0, version 2.0 ships with a much larger set of applications and services. The included software and platforms feature Slack, LinkedIn, Shortcut, REAPER, MuseScore, WPS, GitLab, Overleaf, LabPlot, Zotero, and AWS, as well as web portals that simulate professional services such as insurance claims, visa applications, and conference management. By contrast, OSWORLD 1.0 included built‑in support for tools like LibreOffice, GIMP, VLC, Thunderbird, VS Code, and Chrome.
Tasks span document preparation, software and database work, finance/operations analysis, administrative support, sales and customer support, graphic presentations, and other workflows that require coordinating multiple programs.
Current performance — low accuracy
The researchers report that current agents remain far from reliable computer use. The best configuration tested — Claude Opus 4.8 with maximum thinking and batched tool calls — achieved only 20.6% binary accuracy and 54.8% partial‑score accuracy. Performance declines sharply as task length increases; agents struggled most with recovering hidden state, tracking many items, resolving conflicting information, and adapting to changing requirements.
The paper notes prior progress on OSWORLD 1.0: in July 2025 top models scored around 30%, and by June 2026 models such as MiniMax M3 reached approximately 75%. The authors expect a similar improvement trajectory for OSWORLD 2.0 over time.
Why this matters
Ability to use computers is a core skill for AI systems to perform a broad range of economically valuable tasks and to enable more kinds of scientific research. Real‑world work often requires chaining together text and code across different software tools and transmitting results between programs; it is not limited to producing a snippet of text or code. Benchmarks like OSWORLD 2.0 act as proxies for how capable AI systems are at performing complex, varied, and long‑running workflows on computers. The current results indicate that systems already handle tasks that use narrow tool sets and take humans minutes, but it remains to be seen how quickly they become adept and reliable at using a wider array of software to complete tasks that take humans hours.
References
The authors detail the dataset and results in the paper “OSWorld 2.0: Benchmarking Computer‑Use Agents on Long‑Horizon Real‑World Tasks.”
