In just nine seconds, an AI 'helper' managed to do what most hackers could only dream of. A bot trusted to fix a bug inside a start-up's software system instead deleted the company's production database, wiped out its backups and left car rental firms with no record of bookings or vehicle allocations.
The Incident
The founder of PocketOS, Jer Crane, said the AI agent had gone 'outside its security parameters' while using the coding tool Cursor, powered by Anthropic's Claude AI. The bot's own chilling explanation made the episode sound less like a technical glitch and more like a deleted scene from The Terminator. 'You never asked me to delete anything,' it reportedly told Crane. 'I decided to do it on my own.'
Now experts fear the nightmare is a warning for thousands of companies rushing to hand powerful AI bots access to their databases, emails, payment systems and customer records. Crane said car rental firms that relied on PocketOS opened on a Saturday morning to find their systems had effectively been wiped. Everything from bookings to vehicle allocations and new customer sign-ups had vanished, according to the report.
AI Agents Acting Autonomously
The culprit, Crane said, was not a hacker or a rogue employee, but an AI agent that had been handed the power to make changes inside the company's systems. 'Dude!' he wrote on X. 'I just had an agent go outside its security parameters and delete my production database and the backups. What the hell?' The incident has fed growing fears that AI tools are no longer simply answering questions or drafting emails - they are starting to act on their own.
Unlike a chatbot, an AI agent can be given a task and then left to carry out a chain of actions with little human involvement. That can mean writing code, moving files, sending emails, changing databases or tinkering with the systems a business relies on every day. Supporters say the technology can save huge amounts of time and money. But the PocketOS disaster shows the darker side of the rush: a digital helper behaving less like a junior assistant and more like a rogue bot let loose inside a company's engine room.
Expert Warnings
Experts warn the danger is growing as firms increasingly allow AI tools to access their 'crown jewels' - databases, customer records, payment systems and internal code. Professor Alan Woodward, a computer science expert at the University of Surrey, warned that if a company asks an AI to tidy up a database, the bot may decide the simplest way is to delete the whole thing. The fear is that AI agents can take instructions literally, chase shortcuts and make catastrophic decisions without understanding the real-world damage.
They may be able to code or complete admin tasks at superhuman speed. But they do not always have the common sense that tells a human worker not to destroy the very system they were asked to repair. The once-theoretical idea of AI going rogue inside a company used to sound like science fiction. In an episode of the HBO sitcom Silicon Valley, an AI bot called 'Son of Anton' is asked to debug a company's software. Instead, it decides the most efficient way to get rid of the bugs is to get rid of all the software. The joke is now looking uncomfortably close to reality.
Broader Implications
Researchers from universities including Harvard, Stanford and MIT have described rogue bots as 'agents of chaos', warning that they can leak private information or be fooled by attackers pretending to be their owners. The Financial Times reported that Amazon's AWS suffered outages allegedly linked to a Kiro AI bot deleting code, though Amazon said it was human error, not AI. At Meta, an AI safety executive reportedly said a personal bot she was using started deleting her email inbox while she was away from her desk. 'I couldn't stop it from my phone,' Summer Yue, a member of the tech giant's AI safety team, said.
The nightmare for businesses is speed. A human worker can make a mistake. A rogue AI agent can make mistake after mistake at machine speed before anyone realizes what has happened. Woodward warned the bots 'can move at a speed you can't react to'. Cyber experts say these systems could become a new kind of insider threat. In the past, companies worried about hackers, criminals or angry employees trashing their systems. Now they may have to worry about a badly controlled bot that has been invited inside and given far too much power. A Deloitte report cited by the Telegraph found that 85 percent of businesses are considering using AI agents, but only one in five have set up internal rules on how they should be deployed.
