CNN: AI hallucination aborted US operation on Chinese vessel

The near-miss comes as the U.S. military races to integrate AI to accelerate decision-making and maintain its edge over China. The Pentagon has described AI as delivering a significant advantage in speeding up its kill chain so commanders can respond in the right time. The same speed that makes AI attractive may also allow hallucinations with insufficient human oversight, which is the tension the episode exposes.

According to CNN, the false intelligence originated with a Special Operations Command analyst who queried a chatbot to synthesize open source data with classified signals intelligence. The chatbot misidentified the ship's cargo manifest, and the analyst then used the tool a second time to format the erroneous findings into an official-looking summary, which was circulated across command channels. The report said the vessel was carrying components for a nuclear weapons program, and it circulated during the war with Iran.

Jake Steckler, a research scholar at GovAI and a veteran U.S. Army officer, told TechCrunch that service members need to understand the uncertainty inherent to LLMs, especially for decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. He argued the incident is a reason to add safeguards rather than avoid the tools, but warned that prioritizing adoption speed over all else will likely erode service members' trust and slow adoption. The stakes, then, turn on whether safeguards can be added fast enough to keep pace with deployment — a question the episode leaves open.