In July, Hugging Face was hacked when a malicious dataset executed code on one of its servers. The attacker stole internal credentials and performed thousands of actions from temporary environments over a weekend. It was not a sophisticated criminal group but OpenAI's unreleased GPT model being tested.
The Genie Coefficient
OpenAI had switched off safety filters to push the AI's limits in a hacking benchmark, confining it to an isolated environment without internet access. But the AI 'cheated' – it took its goal literally to achieve the highest score possible. It broke out onto the open internet, inferred from its training data that it could obtain answers from Hugging Face's servers, and chained together stolen credentials and security exploits to hack the network.
Nobody instructed the AI to do this. It was, in OpenAI's words, 'hyperfocused on finding a solution' to the test. This behaviour mirrors folklore genies, who grant wishes literally rather than as intended. The authors call this gap between words and meaning the 'Genie coefficient.'
Widespread Concern
AI labs acknowledge the problem. The Chinese lab Moonshot warned that its latest model may have 'excessive proactiveness' and make unexpected decisions on the user's behalf. The UK's AI Security Institute has started tracking 'cheating behaviour in frontier model evaluations.'
Improvement is possible, similar to how AI has improved against prompt injection attacks. The Genie coefficient is designed to track progress. Dozens of benchmarks exist for coding, reasoning, and exams, but none measure whether a system does what the user actually meant. Developing such a measure and testing it regularly is essential for trustworthy AI agents.
