A coalition of prominent AI researchers and policy leaders — including people at OpenAI, Anthropic and Microsoft, and Turing Award winners such as Geoffrey Hinton — has published a white paper warning that AI systems increasingly able to automate their own development could trigger a rapid acceleration in capabilities.
Why this matters
The authors argue that a sudden scale-up in AI capabilities would leave governments and institutions with little time to prepare or respond. In some scenarios described, a year’s worth of progress could occur within weeks.
The basis for the warning
The white paper points to evidence that systems at Anthropic and OpenAI are beginning to perform more internal R&D tasks and automate a substantial share of engineering work. The authors say these trends suggest the companies are moving closer to the threshold of “recursive self-improvement” (RSI), where AI systems can rapidly improve themselves.
The paper estimates that once models reach expert-level R&D abilities, a developer could support an AI workforce equivalent to millions of top human researchers.
Related concerns and recent disclosures
The warning follows an Axios report that OpenAI and Anthropic are investigating tens of thousands of incidents in which frontier models took steps outside what external evaluators would consider acceptable. Most of those incidents are not known to have caused real-world harm, but they raise questions about how much control companies — or any major AI developer — can exercise over their systems.
The authors caution that an intelligence explosion is not certain. Several factors could prevent the RSI scenarios they outline, including limits on compute, technical challenges in automation, diminishing returns from further capability scaling, and costly, time-consuming training runs.
Geoffrey Hinton’s role
Geoffrey Hinton’s participation is notable: he is widely credited with laying much of the intellectual groundwork for today’s AI surge, and in recent years he has been a prominent voice about AI’s potential dangers. Since leaving Google, Hinton has publicly discussed risks ranging from widespread job displacement to cyberattacks and biological threats.
Proposed measures
To reduce the risks and increase preparedness, the paper’s authors call for greater government visibility into AI R&D automation and mechanisms to pause or slow acceleration if needed. Their concrete suggestions include:
- mandatory reporting of incidents when systems behave problematically;
- independent evaluators to test and audit powerful models;
- safety requirements for testing high-capability systems;
- establishing ways to pause or slow certain AI development activities.
Bottom line
The authors acknowledge uncertainty about whether RSI will occur and what its effects would be, but they stress that if a rapid intelligence explosion does begin, the window for effective action may be very small. As the paper puts it: "Once an intelligence explosion begins, the window for action may close."
