A recent security incident between the US and China, detailed in a CNN report, shows that the risks of AI are not about superintelligent machines but about error-prone systems. The report, published on 18 September and not yet verified by other major outlets, describes how the US almost started a war with China based on an intelligence report claiming a Chinese ship was transporting nuclear weapon components. Military personnel made plans to intercept the vessel, with aircraft and soldiers ready to board, a skirmish that could have escalated into world war three.
Near-Miss Incident
It was only moments before executing the plan that American officials investigated further and found the report had been generated with a chatbot, containing erroneous information about the vessel's contents. Unlike the "terminator" scenario of rogue AI, this incident did not galvanize lawmakers into calls for regulation to prevent human extinction, nor did media outlets widely warn the public about the near-trigger of a war due to reliance on error-prone chatbots.
Stochastic Parrots
The authors, Timnit Gebru and Emily M Bender, describe large language models (LLMs) powering such chatbots as "stochastic parrots," designed to regurgitate patterns from training data. They argue that news coverage describing them as "powerful" and "rogue" bolsters perceptions of infallibility, even in high-stakes warfare. CNN's coverage called the chatbot "powerful, new and relatively poorly understood technology," but the authors counter that today's LLMs are not poorly understood nor powerful for such use cases; they are systems for generating plausible-looking text from haphazardly collected datasets, fine-tuned to appeal to users.
Real Harms and Accountability
The authors note that claims of mysterious behavior arise from deliberate mystification by sellers and misapprehension by users. Documented harms, such as hospitals' "AI" scribes misclassifying patients or governments relying on flawed intelligence analysis, stem from misplaced faith in unreliable software, not actual superintelligence. They call for regulating "AI" systems because they are error-prone products, not magical machines, requiring thorough evaluation in life-and-death scenarios and accountability with human decision-makers. As former FTC chair Lina Khan has reminded, there is no exemption from the law for AI, and existing laws can be enforced. The authors urge journalists and lawmakers to hold companies accountable rather than parrot marketing talking points.
