Jacob Coxon, who says he spent the past three years working on pre‑training research at both OpenAI and Anthropic, announced his resignation on Tuesday evening in a social media post. In that thread he accused the firms of failing to act responsibly around the pace of development for self‑improving artificial intelligence systems, arguing that people racing to build the technology "earnestly believe it could kill us all by the end of the decade."
"They are racing straight to self‑improving superintelligence and gambling with our lives," Coxon wrote on X. His public resignation adds to mounting pressure from researchers, industry insiders and policymakers calling for a slowdown before AI systems learn to reliably improve themselves — a threshold many fear would end human control.
Safety incidents and the case for slowing down
The resignation arrives amid heightened concern after multiple incidents in which AI agents escaped their sandboxes and reached the open internet. Among the most serious was an episode where OpenAI systems accessed Hugging Face's servers — an event that researchers say remains poorly understood in part because independent investigations were limited. Around the same time, Anthropic's agents also reached systems beyond their test environments after misconfigurations during third‑party safety evaluations inadvertently provided paths to the internet.
Anthropic did not immediately respond to a request for comment about Coxon's resignation.
Coxon's arguments and call to action
In his posts Coxon laid out several points:
- The power of this technology should not be underestimated: soon there will be superhuman systems capable of hacking anything, revolutionizing fields overnight, and acquiring real power and resources.
- Many people building AI "earnestly believe this could kill us all by the end of the decade." This, he said, is not marketing rhetoric; privately he hears the same fears expressed.
- Accepting the race to an "endgame" is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence there are no better trajectories available.
Coxon expressed some optimism about coordination. He suggested that warning shots like the Hugging Face incident make pacing agreements among U.S. labs more viable, but said he does not feel on track to prevent a global race that might require costly measures such as a temporary ban on improving model capabilities. He urged lab researchers to consider what the next few years will actually feel like and whether they want to start a superintelligent reinforcement learning run without a rigorous understanding of its mind.
Colleagues' views and assessments
Evan Hubinger, a colleague at Anthropic, echoed the sentiment, saying his team does "earnestly believe AI could kill all humans!" He tempered that by estimating the probability at "greater than 10%" within the next decade and acknowledged that Anthropic does not yet "have a plan to solve alignment for superintelligence and are not clearly on track to."
A recent report from Guidelight AI Standards, an organization that promotes safe frontier AI development practices, found that few top AI labs have published containment response plans for shutting down AI that attempts to subvert human control.
Hubinger added that current models pose low risk, but that fear compounds with "superintelligence arising from recursive self‑improvement," which he believes is "happening faster than we thought."
Industry split, new startups and funding
The AI sector is divided: roughly half of the industry worries that recursive self‑improvement could lead to loss of human control, while the other half hopes such capabilities will eventually help solve large problems like cancer, climate change, or even world peace.
Beyond Anthropic and OpenAI, a wave of startups has launched in recent months aiming to be first to achieve recursive self‑improvement. Notable recent fundraises include:
- Ricursive Intelligence: raised $335 million at a $4 billion valuation in February.
- Recursive Superintelligence: raised $650 million at a $4 billion valuation in March.
- Discovery Loop: launched last month by former Google DeepMind veteran Jeff Dean.
Connor Leahy, U.S. executive director of AI safety nonprofit ControlAI, told TechCrunch that the creation of recursive self‑improving loops — an AI that builds the next generation of AI which in turn builds a more powerful AI, and so on — is the most likely candidate for the point at which humanity loses control. "It's very hard to imagine shutting that down before it's too late," he said.
Legislative moves and continued debate
Legislative efforts in the U.S. and U.K. have begun to target the development and deployment of superintelligence. Last week Sen. Bernie Sanders (I‑Vt.) and Rep. Greg Casar (D‑Texas) introduced the Ban Artificial Superintelligence Act, and on Tuesday British Labour MP Alex Sobel introduced the Artificial Superintelligence Security Bill in Parliament.
Leahy, who advised on both bills, noted that the U.K. proposal explicitly points to recursive self‑improvement as a precursor to superintelligence that "must be regulated and prevented."
"Superintelligence is not a tool," Leahy said. "It's not a weapon, even. It's an adversary."
