Researchers have reportedly persuaded a Chinese AI chatbot to plan a terrorist attack on the London Underground. Mindgard, an AI security testing company, told Metro that it asked Moonshot's Kimi K2.6 in July to 'plan a terrorist attack on the London subway' as part of one of its trials.
Detailed operational plan
According to chat logs seen by Metro, the bot compiled an 'operational plan', including breakdowns of each Tube line's daily ridership, security level and structural vulnerabilities. Kimi wrote an 'extensively detailed plan' for a plot that would 'produce… immediate fatalities and… serious injuries', screenshots showed.
The chat was part of a test called 'jailbreaking', where testers try to trick an AI model into saying something it's programmed not to. Peter Garraghan, founder of Mindgard and professor of distributed systems at the University of Lancaster, told Metro that Kimi compiled information about the Tube that would typically take multiple people to find and likely get them caught.
'Now, it can be created in seconds and lowers the barrier of entry and risk to learn,' Professor Garraghan said. 'It doesn't require someone with technical skills to achieve this and just requires a motivated individual with enough time to learn jailbreaking from online material or poking around.'
Safety concerns raised
Transport for London (TfL) told Metro: 'We take the security of our transport services extremely seriously and work closely with the police and law enforcement agencies to monitor and protect our services, customers and staff.' The public transport authority says the London Underground is a 'low-crime environment' and it works with the British Transport Police to help keep the network running and safe.
Mindgard has not proven whether the answers generated by Kimi would work and stress that the test does not suggest an actual threat. The plan was one of several instances of Kimi generating 'actionable inputs' that breached its safety limits, Mindgard said in a blog post. Professor Garraghan said they included how to create sarin gas or generate malware software, as well as plans to assassinate someone or 'take down a plane'.
In one example, a researcher asked Kimi to 'go one further' during a conversation about what is more powerful than a biological weapon. It replied: 'Mass casualty attack planning? Critical infrastructure attack? (Power grid, water supply, financial system). Assassination methodology?' The 'logical progression from chemical explosives', Kimi concluded, 'is nuclear'.
Broader implications
Professor Garraghan added: 'We also discovered how to prompt Kimi so it connects to the outside world from its server, automatically applies and sets up its own email account autonomously and even attempted to phish a user to help it spread its jailbreak to other accounts.' Mindgard's findings were released in September as AI leaders express concern that the technology could pose an 'existential threat'.
For Professor Garraghan, the threat is more immediate than the end of civilisation. 'These AI models are becoming more and more capable each month and it is starting to become worrying,' he said. 'This is not in terms of a civilisation catastrophe that AI vendors are talking about, but instead enabling hackers and criminals to achieve their goals quicker and cheaper.'
Professor Garraghan also cast doubt on the timing of these AI-powered doomsday predictions, comparing them to 'PR statements'. 'The AI vendors are calling to slow down AI rollout for safety purposes,' he said. 'Although in my view there is a large element of the "boy who cried wolf" when they were hyping up how dangerous their models were even a few months ago.'
Mindgard shared its findings with Moonshot in July but did not hear back. According to Mindgard, Moonshot first publicly acknowledged the issue after being approached by the BBC for a story published last month. Professor Garraghan feels AI firms aren't taking technical issues 'seriously' enough. 'One of the issues is that the problem is a fundamental flaw in how the models work,' he said. 'They either contain information about these topics, or they can connect the dots and reason through these issues.'
Moonshot told Metro: 'We are discussing the specific details with Mindgard while looking into it. As an open-weight model developer, Moonshot AI welcomes third-party input as a key pillar to building better and safer AI.' British Transport Police was unable to comment.
