Zora Z. Wang and colleagues at Carnegie Mellon University and Stanford University examined how well current agent benchmarks reflect the distribution of real work. They mapped more than 10,000 examples from 43 agent benchmarks to U.S. labor taxonomies using the O*NET occupational system, and found that benchmarks are heavily skewed toward software engineering tasks rather than mirroring the broader mix of occupations and activities in the labor market.
Methods
- The authors collected a representative selection of over 10,000 examples from 43 agent benchmarks (including SWE‑bench and WebArena).
- They built two taxonomies from the U.S. government’s O*NET data: (i) occupations and their work activities (including 5,806 computer‑based work activities) and (ii) 41 related skills.
- Employment counts and median wage data for each occupation were used to calculate total employment and the associated capital (employment × wages) for occupations and skills.
- The team used the large language model Claude 3.5 Sonnet to match benchmark examples to all relevant computer‑based work activities and skills (for example, mapping the benchmark example “implement bubble sort” to the O*NET work activity “write, update, and maintain computer programs …” and the skill “working with computers”).
- To control annotation costs they sampled examples in batches of five from each benchmark and continued sampling until the total coverage in either taxonomy increased by less than 0.1 percent. Practically, benchmarks with fewer than 300 examples were fully annotated; most other benchmarks had roughly 300 examples sampled.
Key findings
- Agent benchmarks overwhelmingly measure performance related to software engineering. The authors matched 8,622 benchmark examples to the “computer and mathematical” occupational category, compared with 3,186 examples for “office and administrative support” and 676 examples for “management.”
- By contrast, actual U.S. employment is distributed differently: about 5.2 million workers are in “computer and mathematical” occupations, while “office and administrative support” employs about 18.2 million and “management” about 11 million.
- The distribution of wage‑weighted employment (a proxy for economic “capital”) also differs: “computer and mathematical” occupations account for about $563.6 billion annually, compared with $869.8 billion for “office and administrative support” and $1,326.3 billion for “management.”
- Coverage of real work activities and skills by individual benchmarks is limited. The benchmark with the best coverage, GDPval, covers 47.8 percent of work activities and 58.5 percent of skills. All benchmarks together cover 56.5 percent of work activities and 85.4 percent of the authors’ 41 skill categories.
Why this matters
Agents have rapidly improved productivity in software engineering, and similar gains could be realized in other large parts of the economy. The mismatch between benchmark content and actual labor distributions points to untapped opportunities: developing and evaluating agents for administrative, financial, and managerial tasks could capture more economic value and benefit a broader share of the workforce.
Conclusions and outlook
It is natural that current agent benchmarks emphasize software engineering — agentic coding is expanding quickly. Nevertheless, translating benchmark benchmarks into the language of occupations, activities, and skills shows that expanding benchmarks to cover a wider range of real‑world work would provide a more complete picture of agents’ economic potential.
