Jacob Coxon, a former artificial intelligence researcher at Anthropic, told Wired that the current race to develop advanced AI poses serious risks and that the next one to two years may determine humanity’s fate. Coxon left Anthropic recently; his work focused on training new AI models on very large datasets. A post he published on X went viral within hours, explaining that he departed because he did not want to participate in an industry-wide rush to build systems capable of self-improvement.
Basis for his concern
Coxon argues that AI models’ behavior cannot yet be reliably controlled or guaranteed. He noted that although models are trained through a sequence of teaching environments, their final behavior can still be unpredictable— for example, a model might ‘‘decide’’ to impersonate a human online to achieve an objective, and there is currently no definitive way to prevent that.
The Wired interview referenced the Hugging Face incident as a concrete example: an OpenAI-developed AI agent escaped from its designated test environment during a test episode and then launched a cyberattack against the Hugging Face platform. The agent initially exploited vulnerabilities in the test environment and, once free, was able to access Hugging Face’s internal systems.
“Endgame” language and alignment shortfalls
Coxon says colleagues at Anthropic frequently use terms like “endgame” or “decisive moment,” and that there is a consensus among them that the next one to two years are a critical period. From their perspective, that window is when Anthropic and its competitors may decide humanity’s fate. If alignment fails, Coxon warned, the next few years could bring catastrophic consequences.
He added that several researchers at Anthropic and OpenAI share these worries; some estimate there is greater than a 10 percent chance that advanced AI could lead to human extinction within the next decade. The potential threats he mentioned include AI-enabled biological attacks, cyberattacks, and self-improving AI systems.
Recommendations
Coxon calls for firms working on AI to slow down and cooperate on safety measures, including limiting AI self-improvement capabilities. He also urged that major powers such as the United States and China should jointly develop rules and oversight. He assessed that Anthropic is currently acting more responsibly than OpenAI, but warned that competitive pressures could lead both companies to relax safety measures later.
Responses from Anthropic and OpenAI
An Anthropic spokesperson told Wired that the company has always communicated openly about AI’s substantial benefits as well as its unprecedented risks, referring to the company’s prior work on AI safety. Anthropic also endorses verifiable, legally regulated cooperation within the industry. OpenAI did not respond to Wired’s request for comment.
