This spring U.S. military aircraft were already airborne when officials discovered that intelligence underpinning a planned armed operation against a Chinese vessel had been fabricated by an AI chatbot. The operation was called off at the last minute, narrowly avoiding what could have become a confrontation, CNN reported on Friday.
How the error occurred
According to the report, the false intelligence circulated during the war with Iran and asserted that the ship was carrying components for a nuclear weapons program. The erroneous assessment originated with an analyst at Special Operations Command who asked an AI chatbot to synthesize open-source information with classified signals intelligence. The chatbot misidentified the ship’s cargo manifest.
The analyst then used the tool again to format the incorrect findings into an official-looking summary, which was distributed through command channels. The mistake was caught only near the moment of execution, and the planned strike was aborted.
Why this matters
The episode highlights a growing concern among military leaders and outside experts: as decision-makers rely more on AI systems, the mistakes those systems produce can propagate up the chain of command before they are questioned. The Pentagon is aggressively seeking to integrate artificial intelligence to accelerate decision-making and preserve its competitive edge relative to China; the department has described AI as delivering a significant advantage by speeding up the military "kill chain."
However, the same speed that makes AI attractive may also enable hallucinations to influence critical decisions if human oversight is insufficient.
Expert reactions and recommendations
Jake Steckler, a research scholar at GovAI and a veteran U.S. Army officer, wrote to TechCrunch that "It’s important for service members to understand the uncertainty inherent to LLMs. But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions."
Steckler added that the incident should prompt the introduction of stronger safeguards rather than abandonment of the technology: "These tools can be useful in the right contexts and with the right safeguards in place. But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption."
Conclusion
The spring incident underscores that while AI can provide speed and operational advantages, those benefits require robust human verification and technical safeguards—especially in situations involving potential use of force. Both Pentagon ambitions and expert warnings point to the need for improved security protocols and caution when AI outputs inform life-or-death decisions.
