Israel's conduct in Gaza demonstrates that AI-empowered lethal weapons are already in use, with algorithms determining who lives and who dies. This raises urgent questions about the role of human judgment in warfare.
Geneva talks and the push for a treaty
For over a decade, governments have met in Geneva to stop the development of autonomous AI weapons. In early September, 128 governments approved a report that could serve as the foundation for a binding treaty. However, lawyers sent by Donald Trump and Vladimir Putin weakened the report, notably by diluting a provision requiring human review of AI-developed military targets before a strike.
Fully autonomous weapons remain the focus, but the use of AI to determine targets for conventional weapons is not. The longstanding goal of maintaining a 'human in the loop'—meaningful or effective human control of AI-empowered weaponry—is at risk.
The example of Gaza
In Gaza, the Israeli military has used AI to target Palestinians. After the Hamas attack of 7 October 2023, Israel used an AI system called 'Lavender' to identify patterns it equated with Hamas operatives' phones. People meeting these criteria could be killed. A separate automated system, 'Where's Daddy?', tracked suspects to their homes, often leading to their families being killed as collateral damage.
Though a human remained nominally in the loop, the role was often perfunctory. One Israeli intelligence officer described spending '20 seconds for each target' and having 'zero added-value as a human, apart from being a stamp of approval.'
The dangers of automation bias
AI algorithms are inevitably affected by human input, and the Israeli military conceded that about 10% of human targets were not members of the Hamas military wing. The military permitted an anticipated 20 civilian deaths for killing a suspected low-level Hamas fighter, a ratio that virtually no legal expert considers lawful under international humanitarian law.
The International Committee of the Red Cross calls this 'automation bias.' To have meaningful human control of AI, there must be intense questioning of its product and accountability for commanders. People should never be snuffed out by the mechanical application of an algorithm.
