Geoffrey Irving, the former chief scientist at the Government-run AI Security Institute, has warned that superintelligence could kill the population. He believes AI systems can, in principle, align with human intentions, but fears we are moving too fast.
Warning on X
The AI-safety researcher warned on X today: "I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years. I don't expect to have that resolved to below 10% or above 90% before we either make it through, or we don't. We will have to act despite uncertainty."
Reasoning behind the estimate
When asked how he arrived at that percentage, the scientist explained that he had settled on a range of 10-90% for years and now takes the average as the "honest" answer. He responded: "As to why 50%, despite thinking about this for years I still think there is a ton of model uncertainty of a variety of types, and a few years ago converged on '10-90%, but I don’t feel calibrated within that range'. Then I realized that the honest move was just to take the mean."
Concerns about AI advancement
Having previously worked at DeepMind, OpenAI and Google Brain, Irving explained his move to the Government-run AISI by stating "I worry that progress on safety is too slow relative to the rate of advancement in AI". He added: "Most employees and leaders at AI labs mean well, but market forces mean that there is constant pressure to prioritise speed over caution."
AI has advanced quite dramatically in the last five years. Systems can increasingly reason, create and perform multi-step tasks. If you take writing as an example, models went from producing fairly simple text, to writing essays, coding, translating, and holding coherent conversations.
