Martyn Thomas, fellow and emeritus professor of IT at Gresham College, has questioned how any rational person can argue that independent oversight and regulation will be sufficient to stop AI systems killing us all.
His comments follow the withdrawal of a frontier AI model after it failed safety tests in internal testing at OpenAI. The company scrapped the release of the new model over safety concerns.
Calls for independent oversight
Thomas noted the usual call for “independent oversight and regulation” but said there is never any explanation of how this would work or, most importantly, what evidence should persuade a regulator that an AI system is adequately safe.
He pointed to a long engineering tradition of regulating safety-critical software systems, such as those controlling aircraft or nuclear power stations. International standards require “safety cases” – rigorous analyses that show possible hazards have been identified and controlled and that the probability of an accident is extremely low.
Safety cases and probabilities
Typically, the safety case should show, with a probability of at least 99%, that an accident that could cause multiple fatalities will not occur more often than once in 1,000 years. Thomas said it is extremely hard to provide evidence for such low probabilities, even for aircraft that at worst might kill 1,000 people in a single accident.
The developers of frontier AI systems tell us that their systems might kill the whole of humanity, yet they have provided no detailed risk analyses, no safety cases that could be independently assessed, and no evidence that such safety cases could ever be produced for frontier AI systems.
Policy must be based on facts
Thomas concluded that policy must be based on rigorous analysis of the facts, not on fantasy.
