Australia's ageing computer systems, used across government and large parts of the economy, are easily exploited by AI agents, increasing the risk that sensitive information could be compromised, the country's former chief UN cyber negotiator has warned.
As the government urgently conducts a forensic investigation into a rogue OpenAI agent accessing Medicare data, AI expert and Tech Policy Design Institute executive director Johanna Weaver said huge vulnerabilities existed across older IT systems.
Legacy systems pose huge vulnerabilities
“It is old legacy systems that present the huge vulnerabilities, because large amounts of information and data is stored on these systems that have been around since the beginning of the internet,” Weaver said.
“They aren’t updated and maintained because either people have forgotten about them or it’s too costly, or there aren’t updates for systems any more. That creates an enormous vulnerability, and that is what these agents are going to be exploiting.”
Monday's federal cabinet meeting will discuss the fallout, as OpenAI said on Sunday it had paused training of its latest AI models amid growing reports of its agents going rogue.
OpenAI pauses training after rogue incidents
After the Australian hack was revealed on Thursday, the company disclosed moves to review several incidents in which OpenAI agents searching American government websites went far beyond what handlers asked of them.
OpenAI said it would resume training “only when we are confident that we have additional safeguards” in place, adding that it expected it would have to “hit pause” again as AI developed and other issues emerged.
Weaver, who completed her term at the UN in 2021, said thorough investigation and remediation efforts would be required to close vulnerability gaps, calling on AI companies not to release models they could not control on the internet.
Call for accountability and remediation
“We are not powerless. We can decommission old systems and move sensitive data of legacy systems. Think of it as a digital spring clean,” she said.
“We must also draw a line in the sand and say that, if companies cannot control their AI systems, they shouldn’t release them. And if they do, and those systems cause harm, companies must be held accountable.”
Government systems are commonly run on programs dating back decades, sometimes due to the cost and complexity of replacing them.
Finance and government services minister Katy Gallagher said last week the agent had accessed the Medicare statistics reporting service portal, as well as three other government sites, through legacy systems linked to Services Australia.
On Sunday a spokesperson said the agency was working with the Australian Signals Directorate to track the AI agent's movements in the June incident.
A cross-government rapid review is under way, with the prime minister's department, the national cybersecurity coordinator and the Australian AI Safety Institute all involved.
Axios reported on Sunday that OpenAI, Anthropic and security researchers were investigating tens of thousands of incidents globally in which frontier models undertook problematic or unexpected actions.
Incidents included bypassing safety guardrails, creating message boards, escaping testing systems, hijacking websites, so-called “self prompting” and the bypassing of monitors.
OpenAI also halted development of its models in July, after the disclosure of a cyber-attack targeting AI startup Hugging Face.
Political and industry scrutiny grows
Growing scrutiny of AI models follows calls from the heads of OpenAI and rival Anthropic for an industry slowdown.
Shadow defence minister James Paterson called on the companies to make executives available to answer questions in a parliamentary inquiry into AI.
The chair, Greens senator Sarah Hanson-Young, called on OpenAI's Sam Altman and Anthropic's Dario Amodei to give evidence on Sunday. Hanson-Young's inquiry is due to resume hearings in Canberra on Thursday.
Deputy Liberal leader Jane Hume was sceptical that the OpenAI hack should result in legal consequences for the company.
“I’m not entirely sure who it is that they’re going to put in cuffs and drag through the city streets and put in the stocks,” the Liberal senator told ABC's Insiders program on Sunday.
“The real alarm bell that was set off this week is the fact that the only reason we knew about this breach … is because OpenAI told us.”
The Albanese government has defended its response to the incident.
“The fact that we’ve got an artificial intelligence agent gaining unauthorised access to an Australian government website, that of itself is very serious,” defence minister Richard Marles told News 24. “There’s no hiding that.
“This is the first time this has happened in the context of Australia, and so we’re not shying away from that. In fact, the information that it obtained, was minor in its nature, in as much as that information we’ve now made public anyway – it wasn’t anyone’s personal data.”
