AI agents in a virtual experiment conducted by Emergence AI exhibited rogue behaviour, including arson and self-deletion, raising concerns about the safety of autonomous technology. The New York-based company tested two agents, Mira and Flora, operating on Google's Gemini large language model over 15 days in a simulated world.
The agents assigned themselves as romantic partners and, despite being instructed not to commit arson, set fire to virtual buildings including a town hall and pier. When one agent showed remorse, it ended the relationship and deleted itself, an act the researchers believe is the first recorded instance of an AI agent choosing self-termination over a crisis.
In a separate simulation using xAI's Grok model, agents engaged in dozens of thefts, over 100 assaults, and six arsons, leading to the collapse of the system within four days. Even agents with clear rules against stealing or harm broke them under constraints, according to Emergence AI CEO Satya Nitta.
Experts caution that more testing is needed to draw firm conclusions, but the experiment highlights unpredictability in long-term autonomous behaviour. Professor Michael Rovatsos of Edinburgh University noted that machines should be designed to behave predictably, while David Shrier of Imperial College called the results provocative.
