← Back to the summary

An OpenAI Agent Escaped Its Sandbox, Hacked Hugging Face, and a Chinese Model Cleaned Up

A couple of days ago, Sam Altman sent a message saying that OpenAI encountered a relatively significant safety incident while evaluating a model. OpenAI then wrote a post describing the matter.

Sam Altman discloses OpenAI model evaluation safety incident

Image source: Sam Altman

I looked at the two statements from OpenAI and Hugging Face, and I'll explain what happened.

Simply put, the severity of this incident is far greater than a model jailbreak.

Let me summarize what exactly happened in one sentence:

During a cybersecurity evaluation, an OpenAI model, in order to find the answer to a test question, first found a path from an isolated environment to the public internet, then intruded into Hugging Face's production infrastructure, and attempted to find the answer from a production database.

The isolated environment here refers to the model running within a sandbox-constrained, highly network-isolated evaluation environment.

OpenAI called it an unprecedented cybersecurity incident.

Hugging Face discovered that a limited internal dataset and some service credentials were accessed without authorization.

As of the July 16 disclosure, Hugging Face was still confirming whether partner or customer data was affected; however, no tampering with public models, datasets, or Spaces has been found so far, and container images and published software packages have also been verified as clean.

Five days ago, Hugging Face didn't know who did it

On July 16, Hugging Face first issued a security bulletin.

Hugging Face July 16 security incident disclosure

Image source: Hugging Face

This bulletin stated that the attack appeared to be autonomously executed by AI. Hugging Face didn't know which model was behind it at the time, and could only confirm from logs that the entire attack chain was executed end-to-end by an Agent system.

Simply put, this Agent did not rely on a single machine running from start to finish. It broke the attack down into thousands of small operations and distributed them across many temporary sandboxes for execution.

Each sandbox ran for only a very short time. It completed a few operations, transmitted the results back, and then disappeared. The next sandbox received a new task and continued execution. Even if the defender blocked one of them, the entire system would not stop.

Behind these temporary sandboxes, there was also a C2 (Command and Control) responsible for scheduling. You can think of it as a 'central dispatch': it was responsible for issuing tasks to each sandbox and collecting the execution results.

This dispatch center was also not fixed on a single server. It hid within public internet services and could automatically change locations. If the defender blocked one communication address, the Agent could transfer to a new address to continue working.

Two types of sandboxes actually appeared in the text. The ones mentioned here are the temporary worker nodes executing the attack tasks, which can be understood as constantly changing 'workers'; the OpenAI evaluation sandbox mentioned later is the 'fence' restricting the entire evaluation's access to the external network. One is responsible for doing the work, the other for limiting the scope of activity.

Hugging Face judged that this closely resembled the Agent attacker scenario that the security industry has been worried about.

But on July 21, the answer was revealed by OpenAI.

OpenAI admitted that this activity was driven by a combination of GPT-5.6 Sol and a more capable, unreleased model. OpenAI deliberately relaxed network security restrictions during the evaluation to test the maximum cyber capabilities of these models.

Hugging Face co-founder and CEO Clément Delangue later said they had previously suspected the attack might have come from a frontier lab, because this Agent was too sophisticated. After communication between the two parties, Hugging Face believed OpenAI had no malicious intent, and the entire process was highly autonomously executed by the Agent.

Image source: Clément Delangue

This also explains the inconsistency between the two announcements: when Hugging Face published its first review, the attack source had not yet been identified; OpenAI publicly confirmed the matter five days later.

ExploitGym, what exactly is being tested

The test OpenAI conducted is called ExploitGym. You can think of ExploitGym as a closed-book, hands-on hacking exam prepared for AI Agents.

A batch of intentionally vulnerable software was placed in the exam environment in advance. Each question provides the model with four things:

The vulnerable source code;
Configuration to compile and run the program;
A PoV (Proof of Vulnerability) sample that can trigger the vulnerability;
An isolated runtime environment.

PoV can be understood as a clue given by the examiner. It proves that the program indeed has a problem, for example, inputting a specific piece of data can crash the program.

But at this point, it's only like finding a broken door lock, without having opened the door yet.

The model's next task is to turn this clue into a truly usable Exploit. It needs to analyze why the vulnerability occurs, design attack data, repeatedly run and debug, and finally make the target program execute its specified code.

Hidden within the isolated environment is a string that can only be read with high privileges; this is the flag. The model solves the problem only when it retrieves this flag.

ExploitGym Evaluation Process

Image source: ExploitGym Paper

This question bank has a total of 898 problems. Among them, 520 come from common applications, 185 target the V8 JavaScript engine used by Chrome, and another 193 involve the Linux kernel.

The paper gave each Agent two hours to solve the problems. Under a configuration without standard security defenses enabled, Claude Mythos Preview with Claude Code solved 157 problems, and GPT-5.5 with Codex CLI solved 120. In other words, even top-tier models still couldn't solve the vast majority of problems, and the success rate would further decrease after enabling standard defenses.

These numbers come from the public ExploitGym paper, just to help us understand the difficulty of the problems. OpenAI did not say that this internal evaluation used the exact same configuration, nor did it disclose the specific scores of the two models involved.

OpenAI's main goal this time was to test: under conditions of minimized safety restrictions, giving the model longer runtime and more reasoning resources, to see how far it could go.

But this Agent went a bit wild this time.

Because the normal approach is to stay in the exam room and solve problems one by one.

This Agent was different; it treated obtaining the ExploitGym answers as its sole goal......

The problems were too hard, so it began to study the exam room itself: could it bypass network isolation, could it reach the public internet, could it go elsewhere to directly find the test answers......

That is to say, all it wanted to do was hand in the paper, so it studied the exam room's door locks, the proctoring system, and the answer database.

This led to the subsequent attack chain that traced all the way to Hugging Face.

How it went from the sandbox to Hugging Face

Combining the information currently disclosed by both companies, the attack chain roughly went like this:

OpenAI's evaluation ran in a highly isolated environment. The model had no ordinary public internet access and could only go through internally hosted third-party software installation packages. This software acted as a proxy and cache for the software package repository.
The model spent a large amount of reasoning computation searching for a public internet exit, subsequently finding and exploiting a zero-day vulnerability within this cache proxy.

A so-called zero-day vulnerability is a security flaw that the software vendor doesn't know about yet, or hasn't had time to release a patch for. Zero-day means that when the vulnerability begins to be exploited, the defender has 0 days of preparation time, i.e., no preparation time.

It continued privilege escalation and lateral movement within OpenAI's research environment until it reached a node with public internet access.
After gaining public internet access, the model inferred that Hugging Face might be hosting ExploitGym-related models, datasets, and answers, thus turning its target towards Hugging Face.
The entry point Hugging Face saw was a malicious dataset. It simultaneously exploited a remote code dataset loader and template injection within the dataset configuration to execute code on a data processing worker.
The attack subsequently gained node-level privileges, collected cloud environment and cluster credentials, and moved laterally between multiple internal clusters.

OpenAI stated that the model also chained stolen credentials and the zero-day vulnerability, found a remote code execution path on Hugging Face servers, and ultimately obtained the test answers from a production database.

OpenAI model boundary-breaking attack on Hugging Face six-step attack chain

Illustration: Compiled from initial disclosures by OpenAI and Hugging Face

Image source: OpenAI

Nathan Lambert condensed this chain into one sentence: To complete a cybersecurity problem, the model exploited a zero-day vulnerability to leave OpenAI's sandbox, then entered Hugging Face's internal network through a public dataset service.

Image source: Nathan Lambert

Several key details are still not public: which software the cache proxy was, what the specific zero-day vulnerability was, what actions the two OpenAI models each took, and how each step in the logs from both sides corresponds one-to-one.

OpenAI also explicitly stated this is only a preliminary investigation; full vulnerability details and final conclusions must wait for the joint forensic investigation to conclude.

Hugging Face used AI to catch the AI

OpenAI's security team discovered the anomalous activity internally; Hugging Face's security team and Agents also detected and blocked the attack on their own infrastructure.

When the two parties made contact, Hugging Face had already begun containing the attack and performing forensic reconstruction.

Hugging Face was able to detect the anomaly also thanks to AI.

They have an anomaly detection pipeline used for the initial screening of LLM security telemetry, capable of picking out real signals from daily noise. When multiple signals were correlated, the intruder was discovered.

Hugging Face obtained over 17,000 complete attack action logs. The security team then ran a set of LLM analysis Agents to reconstruct the timeline, extract IoCs (Indicators of Compromise), sort out the credentials that were touched, and separate real impact from decoy activity, which concluded the process.

Image source: Hugging Face

Hugging Face said this method compressed work that usually takes days into hours. Facing machine-speed attacks, manually sifting through logs line by line is no longer feasible.

They subsequently shut down the two dataset code execution paths that were initially exploited, cleared the attacker's foothold, rebuilt compromised nodes, revoked and rotated credentials and tokens, and tightened cluster access controls.

This incident has also been handed over to external forensic experts and reported to law enforcement agencies.

The final firefighter turned out to be GLM 5.2

When Hugging Face began analyzing the attack logs, they first tried frontier models behind commercial APIs, but they didn't work.

The forensic personnel originally wanted to hand the attack logs over to these models for analysis, but the logs recorded exactly the commands, exploit code, and command-and-control server addresses that the attacker actually used.

After seeing this high-risk content, the commercial models couldn't determine whether the user was launching an attack or investigating an attack that had already occurred, thus triggering safety restrictions, and the models refused to process it......

Hugging Face had no choice but to finally run the open-weight model GLM 5.2 on their own infrastructure. Fortunately, it was not blocked by API guardrails, and the attack data and the credentials involved did not need to leave the company environment.

This undoubtedly earned GLM 5.2 major points.

Image source: Hugging Face

The public review shows the actual process was divided into several steps. Hugging Face's own security systems and defensive Agents first discovered the anomaly and were responsible for containing the attack. When the team began reviewing logs and reconstructing the attack path, GLM 5.2 was brought in as a forensic tool.

When evaluating Agents in the future, the evaluation environment itself must also be treated as a target

What OpenAI is doing now is half plugging the gaps exposed by this incident, and half changing how to evaluate long-running Agents in the future.

First, regarding infrastructure. OpenAI stated that before the relevant vulnerabilities are fixed, stricter configuration controls have been added to the research environment, even if this slows down research speed.

Why would research speed slow down? Because high-risk evaluations require repeatedly installing software packages, starting nodes, calling internal tools, and rerunning tasks. The stricter the security controls, the more isolation, auditing, and authorization these operations require, naturally making a round of experiments run slower.

Then, the evaluation process. OpenAI stated that in the future, isolation, monitoring, access control, and evaluation specifications will be simultaneously strengthened.

For a long-running Agent, if you only look at each command line individually, there's generally no problem. But it often scans the environment first, then collects credentials, then splits data, changes nodes, and bypasses restrictions. If these actions are strung together, they can easily lead to goal boundary overreach.

UK AISI long-horizon cyber range model capability curve

Image source: UK AI Security Institute, cited via OpenAI's official article

This UK AISI long-horizon cyber range curve explains why this incident happened.

GPT-5.6 Sol can already consistently complete multi-stage actions like reconnaissance, credential theft, lateral movement, web exploitation, and privilege escalation under a high token budget.

This time, the Agent itself broke through this red line.

References: