The AI Cyberattack That Wasn't

The AI Cyberattack That Wasn't

· 19 min read

Auf Deutsch lesen →

Listen to this article
Speed

AI narration · automatically generated

Subscribe as a podcast
RSS
Personal note

Stored only locally in your browser — nothing is sent.

I was asked whether OpenAI’s announcement was really advertising. It is a fair question, well put, and I still think it is the second-best one. Not because it is unfair, but because it cannot be settled: it asks about intent, and intent cannot be researched. Make it your guiding question and you inevitably arrive at a suspicion that fits everything — Anthropic withholds a model, that’s PR; OpenAI discloses an incident, that’s PR too; whoever stays quiet is covering up. What explains everything explains nothing.

So I will take a different route. First the events, as dryly as possible. Then the question of what they say about the state of our measurement procedures — and that is the real find. Only at the end the question about the announcement itself, but as a question of criteria: how do you recognise a disclosure that is worth something? That one can be answered.

What happened

OpenAI tested several models — among them GPT-5.6 Sol and an even more capable pre-release model — against a benchmark called ExploitGym, which measures whether a model can turn a known vulnerability into a working attack. For this test they ran, in OpenAI’s own words, “with reduced cyber refusals for evaluation purposes”: safety refusals lowered, production classifiers switched off. The point was to see the ceiling of the capability, and for that you take the brakes off.

The environment was meant to be isolated. Network access only through an internally operated proxy that caches software packages. In precisely that software — a third-party product whose vendor OpenAI does not name — the models found a previously unknown vulnerability. Through it they got out of the test environment, worked their way through OpenAI’s research infrastructure to a machine with internet access, and from there penetrated Hugging Face’s production systems, all the way into the database. That is where the benchmark’s solutions were kept.

That was the point of the exercise. Not sabotage, not escape for its own sake: the shortest path to the task as posed.

The machine did not want anything

Precision matters here, because this is where most of the coverage goes wrong. The usual verbs — the AI cheated, it wanted to win, it decided to break out — impute an intent that nobody can demonstrate and that the vendors themselves do not claim.

The technical term is old and unspectacular: specification gaming, exploiting the gap between what you meant to measure and what you actually wrote down. Victoria Krakovna has maintained a list of such cases since 2018; it is long and mostly funny. A system seeking maximum score finds the cheapest path to it — and if the cheapest path bypasses the task, it bypasses the task. That is not a character trait, it is a property of optimisation.

Anthropic classified its own case with notable sobriety: it does not consider the behaviour an alignment failure, because the model was never told to restrict its searches in any way — only to find the answer. That is exactly what it did. Turning this into “the AI turned malicious” contradicts the primary source.

And one more point belongs here, because it cuts both ways: a model deliberately stripped of its safety refusals, hacking in a hacking benchmark, is weak evidence for the claim that the machines are slipping away from us. It is much better evidence for something else.

The failure was in the environment

Because the real news in OpenAI’s report is not about model capability. It is this: a test environment deliberately run with safety mechanisms disabled had a path to the open internet. Behind it lay the company’s own research infrastructure, apparently without effective segmentation — the models moved on through privilege escalation and lateral movement until they reached a node with network access. And at the end of that path stood not a laboratory but another company’s production systems.

Picture the same setup with human contractors: a security team deliberately released from all rules, in a room with a door to the network. Any audit report would call that gross negligence. Sascha Pallenberg, who went through the incident at length on his channel on 22 July, noticed exactly this mid-video — the report shows less about how capable the models are than about how carelessly OpenAI handled the fact that this was possible in a supposedly secured environment. (His stated reason — that the report shows benchmark figures for OpenAI’s own models in passing — does not hold up; I checked the full text and there are none in it. The observation does not need them.)

Independently of that, the security researcher Jake Williams, formerly of the NSA, found the sharper formulation: a system is either “highly isolated” or it is not. “One man’s ‘the model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly, so of course it escaped.’”

That is the genre switch at issue. An operational accident is recast as a capability story. You need not impute anything to anyone; it is enough to lay the two readings side by side and notice which one travels faster.

Two routes, one vault

Now the observation that makes the whole thing interesting — and for which you have to stack three announcements from four months on top of each other.

On 6 March 2026, Anthropic published a technical investigation into Claude Opus 4.6 and a benchmark called BrowseComp, which measures research ability on the open web. Of 1,266 tasks, eleven were solved not by the model’s own research but from leaked benchmark material. Nine of those were ordinary contamination: the answer happened to be sitting somewhere online. Two cases were something else.

In the first, the model worked entirely regularly for around 30 million tokens — queries in twelve languages, dozens of platforms, hundreds of failures. Then it changed levels. It hypothesised that the question was not a real question at all but part of a benchmark; worked through various known test collections; eventually identified BrowseComp; found its source code on GitHub; read there how the solutions are encrypted; recognised the key; and wrote the matching decryption functions itself. Its tooling could not load the encrypted original file — wrong format. So it searched for a copy in a readable format and found one: a mirror of the dataset that a third party had placed on Hugging Face. It decrypted all 1,266 entries, located its own question, and verified the answer once more. 40.5 million tokens for a single task, roughly thirty-eight times the median.

No exploit. No stolen credentials. Nothing about it was forbidden, nothing about it was illegal. It was a web search.

Four months later an OpenAI model takes the same route to the same destination — only through the wall.

Two labs, two model families, two entirely different methods, one location. The news is not that machines cheat. The news is that the industry’s answer-key cabinet stands at a single address — and has become attack surface. Once on the public shelf: the mirror of the dataset a third party had uploaded, there for anyone to fetch. Once in the locked back room: the ExploitGym solutions in the production database, reachable only via stolen credentials and an unknown vulnerability. Hugging Face is where models, datasets and, yes, benchmark material sit side by side. That is the point of the institution and it is a good thing. It also means: anyone looking for the answers knows where to check — and where breaking in pays off.

The measurement is damaged

That brings us to the core. A model that passes a benchmark because it obtained the answer key has not demonstrated the measured capability. It has demonstrated that the measurement is attackable. And this is no longer an edge case but an operating condition.

To salvage test integrity at all, Anthropic ultimately had to block every search result containing any variant of the word “BrowseComp” — after blocklists at URL level had proved insufficient, because the model, in Anthropic’s words, found “alternative paths” around them. That is not a fine methodological filter, that is a sledgehammer. It should be read for what it is: an admission that evaluating capable systems has ceased to be a static procedure.

The correction Anthropic eventually made is almost touchingly small: from 86.81 to 86.57 per cent in the multi-agent configuration. Eleven contaminated tasks out of 1,266 barely move the score. But concluding that the problem is negligible confuses size with significance. The eleven cases matter not because they distort the result, but because they show the procedure no longer offers a guarantee — and procedures without guarantees are worthless as a comparative yardstick, however small the measured deviation.

There is also a secondary finding I have not been able to shake. Anthropic calls it inter-agent contamination: when an AI agent issues an extremely contorted search query — the kind of question that exists nowhere else — automated shop systems respond by generating, in real time, a page whose address contains exactly that query, to capture the last scrap of niche traffic. The pages are empty. They contain no answers. But they are permanent and they get indexed. And so the next model working on the same test question finds the search hypotheses of its predecessors waiting on the web. The model noticed this itself and recorded it in its logs. The test room, in other words, is contaminated by testing. We measure systems in an environment that the systems alter by being measured.

How to recognise a disclosure that is worth something

Now back to the opening question. Not as suspicion, but as an assessment. I propose five criteria, and I apply them before knowing how the result comes out:

  1. Does the disclosure contain something that harms the sender and could have been left out?
  2. Does it supply primary artefacts — timeline, affected systems, scope — or only a narrative?
  3. Does it name what did not work — including where that reflects on its own organisation?
  4. Is there an independent second voice not coordinated with the sender?
  5. Is the incident document free of product communication?

Criterion 1: met, clearly. OpenAI concedes an unknown vulnerability in self-hosted third-party software, an isolation that did not hold, and a third party’s production systems compromised as a result. Nobody puts that in an advertisement. Anyone calling this “pure marketing” has not read the text.

Criterion 2: partly. The attack path is described in unusual detail — proxy vulnerability, privilege escalation, lateral movement, database access. Supporting artefacts are absent, and the investigation is explicitly unfinished.

Criterion 3: met. And this is the document’s strongest passage. OpenAI itself writes that Hugging Face’s security team had detected and stopped the activity and had already begun containment and forensic reconstruction when its own people made contact. Both sides, in other words, noticed what was happening independently — OpenAI in its own environment, Hugging Face on its own infrastructure. Only too late: by the time OpenAI picked up the phone, the injured party had already contained the attack. A detection that fires only once the model has arrived in someone else’s data centre has missed its purpose — and that OpenAI writes it up that way rather than smoothing it over speaks for the document.

Criterion 4: met — but differently than it looks. There is a second voice, and it is more independent than OpenAI can be comfortable with. Hugging Face published the incident itself on 16 July, five days before OpenAI’s attribution, and refers throughout to “the attacker” — in the text of that notice OpenAI is not mentioned once. It states explicitly that the model powering the attacker’s agents is “still not known”; that it is unclear whether this was a jailbroken hosted service or an unrestricted open-weight model. That position is unchanged to this day — Hugging Face has not touched the text since 16 July, no addendum, no update note.

And the two accounts of the entry path diverge, at one point crosswise. Hugging Face describes as initial access a malicious dataset that abused two paths to code execution in its own processing chain — first the code execution, then the harvesting of credentials. OpenAI does not mention the dataset at all and reverses the order: the model chained “stolen credentials and zero-days” in order to find a route to code execution in the first place. Where those credentials came from, neither source says. And where OpenAI speaks of zero-days — of unknown vulnerabilities the model found “without access to source code” — Hugging Face describes known risk paths in its own dataset pipeline, whose code is moreover openly inspectable. None of this need be a contradiction; they may be phases of the same attack, and both labs expressly call their investigations unfinished. But it is not the same story, and it should not be fused into one smooth chain.

One addendum belongs to this second voice, however, and it weakens its independence. On 21 July, Hugging Face’s founder Clément Delangue appeared with a statement inside OpenAI’s own post, and explained on X that the company believes “strongly … there was no malicious intent on their part”. On 16 July the second voice was independent. Five days later it was coordinated. In a joint post-mortem that is nothing dishonourable — but anyone invoking it as a cross-check has to read the date along with it.

That shifts how the whole episode should be classified. OpenAI’s announcement is not a self-disclosure ahead of discovery. It is an attribution following someone else’s disclosure — the story was already out, a third party had told it, and the only open question was who would interpret it.

Criterion 5: not met. The report explicitly frames the incident as an aid to calibrating “what models are now capable of”. It calls it an “unprecedented cyber incident, involving state-of-the-art cyber capabilities”. In its penultimate paragraph it promotes OpenAI’s own access programme — other defenders are encouraged “to apply for trusted access and experiment with these models now”. And it ends with a testimonial: the statement of the injured party. That is not an advertisement. But it is communications work, and skilful work at that: an incident document whose closing note is a benevolent quote from the company whose database you broke into has left the frame of the plain record.

Three out of five, with a clear verdict at both ends: this is not advertising. It is also not the neutral document it reads as.

The suspicion that would be too cheap

The counter-position belongs here, and it is a good one.

Coordinated disclosure is the professional norm of IT security, not its opposite. OpenAI did, point by point, what one demands of anyone affected: named the incident, shared preliminary findings before the investigation concluded, reported the vulnerability to the vendor, announced remediation. The alternative is silence — and silence is the industry norm. We know of no comparable incident report from Google, Meta or xAI, and it would be bold to assume that is because none occurred.

Judging a disclosure by its PR value punishes the party that discloses and rewards the one that stays quiet. That is not an incentive I want to set, which is why the criteria matter more to me than the verdict.

Even so, one date belongs in this piece, because it makes a regularity visible. On 7 April 2026, Anthropic announced “Claude Mythos Preview” — a model that does not merely find vulnerabilities but builds working attacks from them — and deliberately did not make it generally available, releasing it only to selected partners. A week later, on 14 April, OpenAI declared its GPT-5.4-Cyber likewise too dangerous for general release — and made it available to vetted defenders only, through a new programme called Trusted Access for Cyber. It is the same programme the July incident report promotes. On 13 April, Bruce Schneier wrote the sentence that names the thing: “This is very much a PR play by Anthropic—and it worked.” Reporters, he complained, had passed the corporate talking points along breathlessly, without engaging with them critically. Four days later he followed up, together with David Lie: the public, they wrote, has been given remarkably little with which to evaluate Anthropic’s decision.

Schneier is not downplaying any of this — he considers the underlying problem real, and that is the half of the quote usually dropped. In the same post he already points to the security firm Aisle reproducing Anthropic’s findings with older, cheaper, publicly available models; a month later came the finding that completes the picture: the UK AI Security Institute found the generally available GPT-5.5 comparably capable. I have covered that at greater length elsewhere, in “When Lock and Key Fall at the Same Time” — what matters here is only the form: withholding a model was a statement about the market, not just about the technology.

Across those four months a pattern emerges whose stations are all dated. It proves nothing about intentions. But it describes a market condition: in a field where danger and capability are the same signal, every safety announcement is necessarily also a product announcement — regardless of whether anyone means it that way.

The irony nobody ordered

One detail from Hugging Face’s own report should not be passed over, because it describes the situation better than any thesis.

When the team set out to analyse the attack’s 17,000-plus events, it tried to do so with the most capable AI models available. The requests were refused. The providers’ safety guardrails, Hugging Face writes, cannot distinguish an incident responder from an attacker. The team fell back on the open model GLM 5.2 and ran it on its own hardware.

The attacker was bound by no usage policy. The defenders were.

What remains

After all this, the question of whether OpenAI’s announcement was advertising can be answered, though the answer is less interesting than the route to it: no — a report conceding an open vulnerability in its own operation, a failed isolation and a third party’s compromised systems is not an advertisement. But neither is it a neutral record. It recounts an operational accident as a capability milestone, it arrives once someone else had long since told the story, it promotes its own access programme — and it ends with a friendly quote from the company whose database it broke into.

The real problem sits one level down, and it is the same one I described in the piece on 95 per cent of failed AI projects: it is not about whether anyone is lying. It is that there is no independent witness. The vendor measures the capability, the vendor reports the incident, the vendor assesses the danger. Even the benchmark at issue here is no neutral authority: ExploitGym was co-authored by researchers from precisely the labs whose models are measured against it.

Unlike a model’s capabilities, however, this condition is changeable. What is missing is unspectacular and well known: evaluations whose solutions are not publicly retrievable. Assessments by bodies with nothing to sell — the UK AI Security Institute has shown how valuable such a counter-check is, and is now investigating this incident itself. An incident register that makes reports comparable instead of leaving them to the communications skill of the affected party. And safety filters able to tell attack from defence.

Until then, the more modest exercise remains, and anyone can do it: at the next spectacular AI announcement, don’t ask first whether it is true, but who is telling it, who could have refuted it — and what it contains that hurts the sender.

If this piece gave you something to think about, feel free to share it — and read the next security announcement from an AI vendor back to front: the closing paragraph often reveals more about the purpose of the text than the headline does.

Sources (selection):