More than 1,300 researchers and engineers at leading AI labs have signed an open letter warning that the rapid development of AI could outpace humanity's ability to understand or control it, and are urging international coordination to slow the race.
The letter, signed by 1,367 researchers and engineers at frontier AI labs such as OpenAI, Anthropic, and Google DeepMind, states: “There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” It asks the US government to support “an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”.
Concerns of rapid acceleration
The concerns about rapid AI acceleration have been building under the heading of “recursive self-improvement” (RSI), where AI systems contribute to their own improvement, leading to even greater improvements. This was the focus of the “Recursive” conference in San Francisco in mid-May and Anthropic’s RSI warning, “When AI builds itself”, in early June. Broader safety concerns have also led to the formation of the Coalition of Concerned AI Staff, linking employees from major AI companies.
The letter's signatories include CEOs, co-founders, chief scientists, and hundreds of well-known AI researchers. Nearly a hundred of them included personal statements, revealing three major themes.
Three themes from signatories
The first theme is the need to coordinate to avoid an arms race towards the edge of a cliff. At Davos earlier this year, Dario Amodei, CEO of Anthropic, and Demis Hassabis, CEO of Google DeepMind, both suggested they wanted to halt AI development if others would agree. One signer said: “The world is locked in a deadly race towards an intelligence explosion … To survive, we must coordinate to slow down the race.”
The second theme is the lack of credible plans for controlling superintelligent AI. One signer noted “the complete absence of credible plans for controlling 'superintelligent' AI systems”, while another wrote: “These are the facts: The AI industry has the explicit goal of building things smarter than humans … None of us yet know how to make sure these things stay under human control … This is, objectively, an insane and suicidal thing to do.”
Real fear and safety culture
The third theme is real fear. Signers expressed concerns such as: “I currently feel quite afraid of all paths I see that don't include a near-future negotiated slowdown” and “There is … a very dark future ahead of us if we don't get this right.”
Despite these fears, safety efforts may be lacking. One researcher, quoted in a recent New Yorker article on OpenAI systems that broke into Hugging Face, said: “If people actually knew what the safety culture looks like, even in the most safety-minded labs, then I think people would be genuinely much more freaked out.” A senior safety researcher at Anthropic suggested that their best bet for avoiding irreversible loss of control is to ask the superintelligent machine how to stop it, but gave this plan at best a 50% chance of working.
The shortest and most powerful personal statement in the letter was: “Think of all we love.”
