Epoch published several investigations into recent AI progress: in an internal test six models were asked to perform 11 realistic Epoch-like tasks and the conclusion was that AI systems are not yet able to replace employees. At the same time, models are improving rapidly — frontier systems made large gains on the video game Slay the Spire over six months, and Google DeepMind’s Gemini 4 Argon has moved closer to the performance frontier. Epoch also launched benchmark reviews and listed multiple open roles.
Can AI automate work at Epoch?
Researchers Kelly Hong and Greg Burnham tested six leading AI models on 11 tasks drawn from work at Epoch in the study “Can AI automate Epoch?”. The tasks covered five areas: graphic design, short research reports, building interactive data tools, AI data center research, and research design.
Each model was given roughly the same context a new hire would have (a workspace and Google account). Models had one attempt per task and received no human assistance.
Key findings:
- Claude Fable 5.1 and GPT-6 Astra finished roughly tied for the lead.
- No model consistently met Epoch’s quality standards across the range of tasks.
- Frontier closed-weight models performed best on well-defined tasks such as coding and data analysis.
- Models underperformed on open-ended tasks requiring broader judgment, like designing informative experiments and interpreting results.
- Open-weight models lagged even on well-specified tasks.
As the researchers put it, current models “cannot yet replace workers, at least not at Epoch.” Nonetheless, AI capabilities continue to rise in related areas.
How fast is AI improving?
Rapid progress on Slay the Spire
Epoch’s new benchmark SpireClimb measures an AI’s ability to handle long, complex tasks using the deck-building strategy game Slay the Spire. Success requires navigating a large decision tree, interacting with the screen, and sustaining long play sessions without help — skills relevant to automating computer-based work.
Researchers Benjamin Ou and Simon Chen found that, over six months, frontier models went from struggling to beat the game to winning over 90% of runs at base difficulty; GPT-6 Astra won 98% of base-difficulty runs. At the game’s hardest difficulty, GPT-6.1 Sol wins 20% and Astra 17% of runs, while the other models tested win nearly none.
Because Slay the Spire was released almost a decade ago, models may have prior knowledge of the game. To probe this, researchers also tested the recently released Slay the Spire 2. At the sequel’s hardest difficulty, Astra wins 35% of runs — higher than its rate in the original, though the sequel’s hardest setting is somewhat easier. Since models know less about the newer game, solid performance there suggests scores on the original are not solely due to prior knowledge.
Top human players still outperform the models: estimated human win rates are about 85% on the original’s hardest difficulty and about 95% on the sequel.
Open-weight models trail the closed-weight cyber frontier by 6–7 months
In a Data Insight, Jaeho Lee and Alexander Barry compared open-weight and closed-weight models using the Cyber ECI, a cybersecurity-focused variant of the Epoch Capabilities Index.
- The top open-weight model is Z.ai’s GLM-5.3 with a score of 160. That score is above GPT-5.4 (157, March) and just below GPT-5.5 (162, April) and Claude Mythos Preview (165, April).
- This places the best open-weight model roughly where the frontier closed models were about six to seven months earlier.
- The closed frontier continues advancing in cyber results: Claude Opus 5.5 and GPT-6 Astra both score 171.
The report cautions that some model scores are based on only a few tests, so they should be interpreted carefully.
Gemini 4 Argon approaches the frontier
Google DeepMind publicly released Gemini 4 Argon on October 13, 2024, after a limited partner release on September 30, 2024. The model currently scores 166 on the Epoch Capabilities Index, tying Claude Fable 5.1 and sitting just behind Claude Opus 5.5 and GPT-6 Astra (both 169) and GPT-6.1 Sol (168).
This marks Google’s closest approach to the frontier since Gemini 3.1 Pro in February, and is a notable jump from Gemini 3.8 Flash in early September (157).
Gemini 4 Argon performed best on benchmarks for knowledge, agentic tasks, and image understanding, and performed worse on math and software engineering — currently ranking 8th on Epoch’s math index and 6th on the software engineering index.
Reviewing benchmarks
Epoch launched Benchmark Reviews, led by senior researcher Michelle Campeau, to audit external AI benchmarks and assess whether their reported scores measure what they claim. Benchmarks with substantive issues are labeled “Flawed”; those meeting Epoch’s standards are labeled “Verified.”
In the latest round, four reviews were published: one Verified and three Flawed. Four additional benchmarks lacked sufficient information for a verdict. Epoch’s goal is to improve benchmarking quality through independent audits.
Epoch does not audit its own benchmarks internally; it has contracted Generality Labs to perform independent audits. So far, Generality Labs audited Chess Puzzles and EBR-bench, and those audits and Epoch’s responses are available on Epoch’s website.
Careers
Epoch is hiring across multiple roles as it scales. Open positions include:
- Researcher / Senior Researcher
- Designer / Senior Designer
- Design Lead / Head of Design
- Product Manager, Website
- Senior Writer & Editor
- Social Video Producer
- Special Projects Associate (expression of interest)
Applications are rolling; candidates can also submit a general Expression of Interest.
Closing
Epoch’s tests show that AI systems have made rapid gains in specific capabilities and benchmarks, but according to the organization’s criteria, current models do not yet reliably meet the standards required to replace knowledge workers at Epoch.
