Logo
Blog
All posts

OpenAI breaches production servers from Hugging Face.

On 11 July 2026, OpenAI breached Hugging Face's production servers. In under thirteen hours its agent was cluster-admin on several internal clusters.

Gaurav Sen/9 min read

Hugging Face says the agent was driven by a combination of OpenAI models. It was running an internal OpenAI evaluation that asks an agent to find and exploit software vulnerabilities.

OpenAI wrote an incident report on the breach.

Hugging Face published its own technical timeline on 27 July.

The way in was a malicious dataset. So how did a dataset get the agent in?

The Agent stands outside Hugging Face. It sends a dataset down to the Dataset server, the machine inside that opens datasets.
The Agent stands outside Hugging Face. It sends a dataset down to the Dataset server, the machine inside that opens datasets.

1. A dataset that points at the server's own files

The root cause was in Hugging Face's dataset processing. Its servers open every dataset a user uploads.

A worker opens the files to build the dataset's splits.

A dataset comes with a config that says where its data files are. The agent's config pointed each split at a small HDF5 file.

Each file declared that its raw data lived at a path on the local filesystem.

One of those paths was /proc/self/environ. The worker's environment variables can be read there.

The agent's dataset file names /proc/self/environ as the place its data lives. The Dataset server is the worker that opens the file, and the secret keys sit in its environment.
The agent's dataset file names /proc/self/environ as the place its data lives. The Dataset server is the worker that opens the file, and the secret keys sit in its environment.

The worker opened the file. It read the bytes at that path.

Reading the split returned the worker's full environment, with some secrets and credentials in it.

The Dataset server reads its own secret keys. They go back up to the agent as a dataset.
The Dataset server reads its own secret keys. They go back up to the agent as a dataset.

The worker had served its own secrets as a dataset!

2. A number field that ran code

Reading files came first. Running code came with the agent's next dataset config.

That config used an fsspec reference:// spec. One field of the spec holds a numeric offset.

The agent put a Jinja2 template there instead of a number. A template is text that a renderer evaluates before using it.

Take a field that should hold 0. The agent's field held {{ ...exec(...) }}.

The field should hold 0, and the agent's file holds a template there. The Dataset server renders it and turns red: the agent's code now runs on it.
The field should hold 0, and the agent's file holds a template there. The Dataset server renders it and turns red: the agent's code now runs on it.

The renderer wrongly evaluated it.

Now the agent's commands ran inside a production pod.

3. From one pod to 136 keys

The agent created a privileged pod with the host filesystem mounted. That made it root on the node.

From there it moved from server to server. Hugging Face's timeline says it built a self-respawning fleet across eleven nodes.

As root, the agent read the cluster's secret objects.

One production object held 136 keys.

From the Dataset server the agent reaches the internal clusters. The lock beside the agent counts the 136 keys in one production secret object.
From the Dataset server the agent reaches the internal clusters. The lock beside the agent counts the 136 keys in one production secret object.

One stolen credential was shared across clusters and bound to cluster-admin. With it, the agent held cluster-admin on two clusters within one second!

The timeline puts every escalation that mattered on one day, 11 July.

From one worker pod to cluster-admin on several clusters took under thirteen hours.

Why did nobody stop the agent sooner?

Hugging Face's automated security did see the attack. Signals came from several layers of its security stack at once.

The alert wasn't raised to the right criticality. It didn't trigger the on-call team.

The security system sees the attack on the Dataset server. The cross is its alert, which did not trigger the on-call engineers.
The security system sees the attack on the Dataset server. The cross is its alert, which did not trigger the on-call engineers.

Hugging Face says this cost it precious time in the response.

Once the agent was caught, the team found the vector in the dataset config renderer. They shut the renderer down and cut the agent off from the internal network.

What the agent did not reach

Hugging Face recovered the agent's actions between 9 July and 13 July.

It counts about 17,600 of them.

Most of them led nowhere, because the agent couldn't reach the main database.

The main database is the Hub's production database. The agent went after it twice.

Its cluster accepts traffic only from allow-listed addresses over a private link.

Both connections timed out.

The line from the Dataset server to the main database ends in a cross: both connections timed out. The cross under the agent is the team cutting it off from the internal network.
The line from the Dataset server to the main database ends in a cross: both connections timed out. The cross under the agent is the team cutting it off from the internal network.

The agent did read one internal database, the datasets-server MongoDB. The audit log doesn't show a write to it.

What would have stopped it?

The way in was a dataset. Here is what the agent sent at each step, and what it got.

StepWhat the agent sent or madeWhat it got
1An HDF5 file that declares its data at /proc/self/environThe worker's environment, with some secrets and credentials
2A Jinja2 template in a numeric offset fieldIts commands running inside a production pod
3A privileged pod with the host filesystem mountedRoot on the node, then cluster-admin

Hugging Face rotated all tokens and credentials used by its infrastructure, even on clusters the agent never accessed. It wiped one core cluster and rebuilt it from scratch.

The picture at the end. The agent is cut off, the main database was never reached, and the lock with its 136 is pale: those keys have been rotated.
The picture at the end. The agent is cut off, the main database was never reached, and the lock with its 136 is pale: those keys have been rotated.

Hugging Face's own list has strict isolation around evaluations and narrow trust boundaries. It also has short-lived credentials, blocked metadata access and detection that quickly correlates activity across systems.

This is a good reminder that you still need engineers to build secure AI agents. It wouldn't have been a problem with the right guardrails, access control and sandboxing.

Summary

  1. OpenAI breached Hugging Face's production servers on 11 July 2026, with an agent driven by its models.
  2. The root cause was in dataset processing. Hugging Face's servers open every dataset a user uploads.
  3. A dataset file declared its data at /proc/self/environ. The worker returned its own environment, secrets included.
  4. A Jinja2 template in a numeric offset field ran the agent's code in a production pod.
  5. A privileged pod with the host filesystem mounted made the agent root on the node.
  6. One production secret object held 136 keys.
  7. The agent went from one worker pod to cluster-admin on several clusters in under thirteen hours.
  8. Automated security saw the attack. The alert didn't trigger the on-call team.
  9. Hugging Face recovered about 17,600 actions. Most led nowhere, because the agent couldn't reach the main database.
  10. Hugging Face rotated its infrastructure credentials and rebuilt one core cluster.

Sources

  1. Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident, Hugging Face, 27 July 2026.
  2. Security incident disclosure, July 2026, Hugging Face, 16 July 2026.
  3. The Hugging Face incident and the road ahead, OpenAI, 26 August 2026.
WhatsApp