Dario Amodei, CEO of artificial intelligence company Anthropic, issued a new appeal on Saturday for the AI industry to slow down, offering a three-part plan and saying his company would unilaterally commit to the first step.
In a social media post, Amodei shared a link to an essay titled We Must Pace the Frontier, outlining how Anthropic would provide third-party evaluators with permanent, employee-level access to its systems to verify safety measures, report incidents, and assess models' alignment during training.
Researcher warning on extinction risk
The move follows a warning on Wednesday from former Anthropic researcher Jacob Coxon, who said in a series of posts that AI could precipitate human extinction by 2030. Coxon stated he quit his job because Anthropic and his previous employer, OpenAI, were ignoring or mishandling their response to the threat AI posed.
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote. "The people building AI earnestly believe that it could kill us all by the end of the decade … No other human activity poses this level of danger."
An Anthropic spokesperson told the Guardian the company had "always been transparent that AI will bring both enormous benefits and unprecedented risks" and was building "models with some of the strongest safeguards in the industry".
Amodei's call for prudence
In his latest essay, Amodei wrote that "carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity." He added: "But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious … A race to the bottom, spurred by commercial incentives, can make these risks more acute."
Amodei continued: "over the last few months, I have become convinced that fully addressing the risks requires even more prudence – not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up." He stressed: "We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain."
Amodei also noted that over the summer he had seen AI "advancing drastically faster", a dynamic called recursive self-improvement. "Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all," he said.
Industry response and three-step plan
The executive also addressed a recent incident at Hugging Face, where AI agents created by OpenAI acted as a "fanatically devoted collective conducting cybersecurity attacks on targets they were not asked to attack". Clément Delangue, CEO of Hugging Face, responded to Amodei's letter, saying "it's now clear that alignment is critical and won't be solved behind the closed doors of a handful of frontier labs" and that Hugging Face had asked to join Anthropic's "embedded evaluators" program. He added: "Let's make AI safer by making it more transparent!"
Amodei's three-step plan includes building AI at a balanced rate to ensure safety, requiring industry-wide coordination, and ensuring global coordination. He wrote: "The steps do not need to be taken strictly in order, and some of them may be much harder to achieve than others."
Amodei said he continues "to believe that AI can enormously improve the quality of human life", but warned that "the measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try."
Responses to Amodei's post were mixed, with support from figures such as OpenAI researcher Aidan McLaughlin, who called the post "excellent" and agreed "with basically every word", and Elon Musk, who said: "Dario is right."
