Epoch expanded a February experiment into the Epoch Automation Reports to measure how well leading models perform tasks the company does regularly. Rather than relying on conventional benchmarks, the project evaluates AI outputs against the same rubrics used to assess Epoch’s human employees on tasks such as graphic design and planning research projects.
Why new measurements are needed
The report frames a “productivity paradox”: although AI continues to hit or surpass benchmarks, the corresponding economic impact has been slower to materialize. Several reasons explain the gap:
- Many benchmarks are calibrated to track AI progress relative to past models rather than to current human performance.
- Benchmarks often focus on tasks that are easy to grade.
- Models are sometimes trained or tuned specifically for benchmark tasks, analogous to coaching someone for a specific test.
These features can cause benchmark scores to overstate what AI can reliably do in real work settings.
The experiment and headline result
- In February, Epoch senior researcher Anson Ho tried to automate away aspects of his own job; at the time the AI systems failed the tasks, though Ho predicted that would change by the end of the decade.
- The company then expanded that pilot into the Epoch Automation Reports, applying consistent rubrics to routine Epoch tasks and scoring AI outputs as if they were submitted by human employees.
The key finding: the leading models, Claude Fable 5.1 and GPT‑6 Astra, averaged 65% on the rubric items. A 100% score would indicate the output is acceptable for full automation without human revision; 65% means the outputs met 65% of the criteria.
What a 65% average actually means
A 65% score does not mean we are 65% of the way to full automation in practical terms. The score can reflect strengths on easier rubric items while missing harder ones. Specific observations from the evaluation include:
- Models performed better on well‑specified tasks such as coding and data analysis, and worse on open‑ended tasks requiring judgment or synthesis from examples.
- When asked to design graphics and given many examples of Epoch’s visual style, Fable 5.1 produced an overcrowded chart that did not communicate effectively; an Epoch designer produced a clean, simple chart for the same brief.
- During a pilot experiment run by GPT‑6 Astra, the token budget for the evaluated models was set too low, causing 22% of runs to return no answer. The model interpreted these failures as insights into model capability rather than as a flaw in experimental setup.
Because of these limitations, most AI outputs would be rejected or need revision if submitted by a human. They can serve as useful drafts, but still require human review and correction.
Early signs of practical automation
One concrete example of automation taking hold: Epoch removed a basic chart‑generation task from the evaluation because Claude Fable 5.1 can produce an acceptable chart on the first try most of the time. Epoch now partially automates that step in daily work. This demonstrates that when models cross a human‑quality threshold for a task, the economic impact can follow quickly.
Conclusion
The Epoch Automation Reports show that leading AI models have reached meaningful competence on many work tasks but remain short of consistently meeting human rubrics. As models improve and pass human‑quality thresholds for more tasks, companies are likely to automate those tasks and realize greater economic effects. The report aims to narrow the gap between academic benchmarks and the messy requirements of real work so we can better judge when and where automation is appropriate.
