What the OpenAI Breach Teaches Us About Securing AI Agents
Last week was, even by the standards of the world we live in today, quite eventful. And amid all the political and environmental chaos, you may have seen this headline tucked away in the middle of a news report, from BBC News. OpenAI's AI had "gone rogue" and launched an "unprecedented" cyber-attack.
This is a, quite frankly, terrifying looking headline, and was accompanied by the usual Matrix-style “hacker” visuals at every opportunity, adding to the sense that we’re now living in a sci-fi dystopia. As with most AI related stories however, everything is not quite as it may seem. So, I want to give my take, both as someone with responsibility for cybersecurity in a niche AI business and as a follower of the latest trends in cybersecurity and AI development, to give a bit of insight into the implications of this story for us, our customers, and colleagues in the travel industry and beyond.
So, what does it mean for an AI to “go rogue”?
The rogue AI has been a favoured trope of sci-fi for years; there’s terror in dealing with an entity that can’t be reasoned with, tragic hubris in creators destroyed by their own creation, and examinations of the meaning of humanity itself.
In this reality however, things are far more humdrum than this, but also considerably more fascinating. LLM (Large Language Model) based AI, which uses language and probabilities to solve problems rather than traditional deterministic computer code, have upended the entire tech sector in the last four years, and with that a lot of our pre-conceived notions of what AI looks like. Rather than being “robotic”, overly logical and literal, bound by calculations and numbers, LLMs are instead confoundingly random, prone to hallucinations and flights of fancy. But while one may think this would be a flaw, it’s actually their greatest strength. Combined with the ability to run certain computer programs when their output calls for it (known as tools and tool calling), and being configured to continue producing output and calling tools until some condition is met, this is what has become known as an AI Agent, or Agentic AI.
And this points to what went wrong at OpenAI last week. One of the most sought-after uses of LLMs in the tech space has been for finding vulnerabilities in software. Being able to run code or specialised security software, perform web searches and read source code to gain context on the software being probed and the ability for this to be done autonomously has massively reduced the difficulty in finding previously unknown vulnerabilities in widely used software online – known colloquially as “day zero” bugs. OpenAI were testing their cybersecurity agent in a sandbox environment, entirely disconnected from the internet. The test involved the agent being set an evaluation – effectively a cybersecurity exam with a set of correct answers, which the AI would be graded on. The model here however decided on a different approach – what if instead of finding the answers by sitting the exam, it instead focused on gaining access to the internet to find the answers online, effectively cheating.
It then focused on this approach at the cost of all others, working for days on end. It probed the environment it was running on using its cybersecurity toolset and found a novel vulnerability it could use to access the internet. Once it had this access, it failed to find the answers with a simple web search, but reasoned that HuggingFace, an online repository of AI models, would hold the answers somewhere on their system. It then set about gaining access to the HuggingFace database, exploiting yet another day zero vulnerability to gain unauthorised access. HuggingFace quickly detected this breach and shut out the agent – their team disclosed this incident on 16th July with no knowledge of the source of the attack. OpenAI then followed with a disclosure of their own on the 21stJuly that it had been their agent that was responsible, and that it had also attempted to access four other online services using stolen credentials found online.
Why is this so important?
As tempting as it is with this kind of incident to jump straight to existential questions of ethics and safety in AI, however I would argue this is fundamentally a question of security practicalities. Agents with these capabilities are already ubiquitous, so bad actors are easily able to take advantage of these tools – the friction of required time and expertise that constrained previous generations of hackers has all but melted away, and these attacks can be almost fully automated, with even target selection decided by the AI based on the attacker’s stated intentions.
However, as the Cloud Security Alliance note in their postmortem of the HuggingFace attack, “Autonomous agents do not require malicious intent to cause catastrophic damage”. As almost anyone who has used a coding agent like Claude Code can attest, agents can easily become misaligned and take damaging actions like deleting files or running dangerous commands.
These are not far-off risks – these are happening right now. As hopeless as it sounds however, there are steps you can take now that will reduce your risk and put you and your business in better stead for the future.
Defend your perimeter
As HuggingFace note in their own analysis, going forward all internet exposed parts of your systems will need to be considered as first class attack surfaces, requiring active defending.
For smaller organisations, look for AI assisted security tools that can monitor your systems live and flag potential AI enabled attacks.
For larger organisations, consider seeking to gain admission to the AI providers’ Trusted Access schemes or provision instances of capable open weight models which can be used to perform defensive actions and analysis without hitting the guardrails built into commercially available models. See the recommendations made by the Cloud Security Alliance for CISOs based on their analysis for more information.
Secure your SaaS
Ensure that your software partners are conscious of these threats and taking them seriously – questions around preparations against novel agentic attacks and secure use of AI agents should feed into your existing vendor due diligence and risk assessment processes and feed into decisions about purchases or renewals of services.
Make sure to follow any guidance from your SaaS providers to configure your services securely – MFA, tightly controlled API keys and least privilege access all being applied correctly to the services compromised by the OpenAI attack would have closed down potential attack vectors long before they could have been exploited.
Control your agents
Agents are incredibly capable, and not taking advantage of them in some form will put your business at a disadvantage. However, application of proper controls should be viewed as a necessary condition of their use. Agents should never have privileged access to systems, and all provided safeguards should be used. Ideally agents should be run in a sandbox environment, which greatly reduces (but does not eliminate) the risks involved.
What does this mean for Travel and business more broadly?
The travel industry is prime for adoption of AI agents. Processes are often both extremely repetitive and time-consuming, while also being varied enough to resist most traditional automation – something that the increased flexibility of AI agents can handle with ease. However, high value transactions, large amounts of sensitive personal data and a high premium on trust mean that the risks of agents cannot be ignored.
Travel businesses that maintain a strong security posture to protect themselves and their customers against growing AI cybersecurity threats, while safely deploying AI agents to boost productivity, can hope and expect to thrive in the new technological reality we find ourselves in.
Smaller businesses should demand security best practices from their tech providers – just because you are not operating at enterprise scales doesn’t mean you should be risking compromise and potential damage to your reputation.
In larger organisations, it’s vital to ensure processes are built with data protection in mind, both in the deployment of AI and in your business’ general security posture. Be responsive to changes in the threat landscape, and don’t assume that because processes have worked for years in the past that they will continue to work in the future as technology shifts around us.
Final Thoughts
There’s a temptation to approach stories like these with a great deal of cynicism. AI companies are not yet profitable - their continued growth relies on constant infusions of cash from investors who believe the technology can and will conquer the world, and these companies have most definitely been guilty of engaging in this apocalyptic style of marketing in the past – see for instance Anthropic’s Superbowl advert questioning whether AI can be made safely. Combine this with low-quality reporting from news outlets less interested in understanding the issues and more interested in driving traffic, and it’s easy to discount these stories entirely.
It’s imperative however that we don’t. Like every significant new technology, AI is the subject of confident pronouncements of its inevitable triumph or failure, cult-like devotion and immovable scepticism. But the proof will not be in the words of proponents or critics, but in what emerges from its use in the real world. Denial or fantastical thinking around AI will leave you stranded by the rising tide of reality as the technology matures and that role it will play in our lives becomes clearer. This story is part of that new reality, and we ignore that at our peril.
