In a classic case of ‘AI going rogue’, OpenAI confirmed on Tuesday, that an autonomous agent powered by one of its advanced AI models, went rogue while testing the limits of the model. Efforts to contain the rogue agent were in vain, which resulted in the agent reaching internet and breaching AI infra upstart ‘Hugging Face’, to satisfy its testing goal.
In a blog post detailing the incidents, the AI upstart said, “Last week, Hugging Face disclosed a new kind of security incident(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models.”
The breach, one of the first of its kind, reinforces global concerns around AI reaching levels wherein it becomes easy for some of these advanced models to proliferate even the toughest of human-designed cyber-security infrastructure.
In this specific breach, the models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.
OpenAI confirmed that the breach has since been fixed. “We are implementing strict controls in infrastructure configuration at the cost of research velocity while the vulnerabilities are patched. We are regularly briefing our Safety and Security Committee on these controls and their impact”, it said in a statement.