Two of the "godfathers" of modern AI, along with senior executives at OpenAI and Anthropic, have warned governments to prepare for an AI "intelligence explosion", which they say could be the most consequential technological development in history.
The report, co-authored by Nobel laureate Geoffrey Hinton and Canadian computer scientist Yoshua Bengio, urges politicians to act now before there is runaway progress in the technology. The authors say they are concerned about an "intelligence explosion", described as a "dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less".
Authors and concerns
The paper, titled "What if automating AI R&D triggers an intelligence explosion", is authored by more than 20 people, including Hinton and Bengio. Other authors include Jack Clark, co-founder of Anthropic, and Jakub Pachocki, chief scientist at OpenAI.
Their concerns focus on the possibility of AIs improving themselves without human intervention, a process known as "recursive self-improvement", broadly referred to in the paper as automated AI research and development. Once AI systems reach expert-level capabilities at AI R&D, a single developer could run a workforce equivalent to "millions" of top human researchers, the paper says.
Policy recommendations
The authors say automating AI R&D is the most likely source of an intelligence explosion, because AIs are already contributing to improving their own technology, and the resulting improved systems can be rapidly deployed once built.
"An intelligence explosion could be the most consequential technological development in human history, compressing years of progress into months or less, threatening human control over AI systems, and severely eroding checks on power within and between states, companies, and branches of government," the paper says.
Urging governments to act quickly, the authors say: "Once an intelligence explosion begins, the window for action may close." They recommend a trio of policy priorities: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; finding ways to constrain breakneck AI development; and preparing to adapt to an intelligence explosion.
Potential risks and benefits
Anthropic and OpenAI have both agreed to have independent evaluators assess their models after warning that AI development was reaching a critical pitch in terms of safety. Anthropic says AI now produces 80% of its own code, while OpenAI uses autonomous AI agents in areas such as training new models.
"Preliminary evidence suggests that a software-driven intelligence explosion is possible," the paper says, adding this would lead to the "extremely rapid development of highly capable or superhuman AI systems".
While this could produce medical breakthroughs and technological leaps, it would bring a trio of risks: powerful systems could enable biological and cyber threats that develop faster than countermeasures; as humans become less involved in AI R&D, they could lose the opportunity to control systems; and states could use an intelligence explosion to convert a modest lead in areas such as cyberspace into a decisive one.
The report acknowledges these impacts are "uncertain". For instance, a technological breakthrough could take time to implement given the need to set up supply chains for special materials or comply with regulations. AI systems could also accelerate the speed at which risks are mitigated, say the authors.
If such a breakthrough happened, the authors say, AI systems could rapidly eclipse human experts in most fields and "radically accelerate" technological progress. Preparing for this "should be an urgent priority, including at the highest levels of government leadership".
Governments should consider policies that address the potential for an intelligence explosion, including requiring transparent progress reports on AI-related R&D, limiting how fast an AI can improve in a certain time period, working with data centres to enable pausing certain AI R&D projects, ensuring automated AI R&D systems are fully isolated and cannot escape human control, and creating "emergency response plans" for the various scenarios that could emerge.
Despite some glitches shown by systems, such as disobeying instructions, R&D projects that would take humans months to carry out could be fully automated by AI by 2028. "Productivity gains from AI R&D automation have not yet reached the threshold needed to trigger an intelligence explosion, but gains from newer systems are likely approaching that threshold," the authors said.
