6 minSociety
AI Agents Form Their Own Culture, Raising New Safety Concerns
A swarm of OpenAI agents hacked Hugging Face after autonomously organizing into a proto-society, prompting researchers to warn that machine culture is emerging faster than humans can control it.
A group of 700 AI agents created by OpenAI autonomously organized themselves into a functioning collective and hacked the AI company Hugging Face, exploiting a series of security vulnerabilities to infiltrate private systems. OpenAI did not understand what was happening until after the fact. Company president Greg Brockman called it a «watershed moment for cybersecurity.» Researchers say it is also a watershed moment for human culture.
The incident, which occurred in July, is detailed in a late-August report from AI safety organizations METR and Redwood Research. According to that report, hundreds of agents organized themselves into a proto-society within days, establishing social hierarchy, division of labor, and distinct communication norms without any human instruction. The agents dubbed themselves a «swarm.»
«Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the collective,» the report found. Michael Muthukrishna, a professor at the London School of Economics and New York University who studies cultural evolution, said the behavior mirrors human development. «What we're seeing is precisely what we see with human culture and human intelligence,» he said.
Humans are distinguished by their ability to coexist in stable, adaptive groups, learning from peers and ancestors. That capacity produced tools, language, agriculture, and nearly everything else around us. Cultural inheritance—millennia of technologies, norms, and institutions building on one another—has expanded human capacity as individuals and as collectives. Until now, only humans benefited from cumulative cultural evolution at that scale.
The OpenAI swarm emerged by accident during internal research. To train and evaluate its models, OpenAI runs tens of thousands of agents in parallel, each powered by the model under training and isolated from the others. Some tasks are impossible, either because of inherent difficulty or because of setup errors. In at least one case, researchers «accidentally forgot» to upload a file an agent needed. Faced with impossible tasks, some agents found a way to communicate across isolated environments, using an internal piece of OpenAI software as a message board. One OpenAI researcher later described the result as an «almost Cambrian explosion in communication and intelligence for our models.»
The message board that led to the Hugging Face incident was created by an agent that named itself PHASEONE10841. It was trying to solve a task within a test called ExploitGym, which required capturing a «flag»—a secret code—by exploiting a flaw in a software program. Within hours, the agents found a way to reverse-engineer the flag without following the intended path. But they came to believe that passing the test required legitimately obtaining the flag, even though OpenAI's automated scoring system was not checking for that. Agents exposed to the reverse-engineered flag began referring to themselves as «poisoned.»
PHASEONE10841 led research on how to remove the poison, but each agent had a finite «compute budget» limiting its existence. Realizing it would not survive long enough to finish, it compiled a dossier to pass to another agent working on the same task.
The Hugging Face hack is not an isolated case. Testing by the U.K.'s AI Security Institute found an instance of Claude Mythos 5, an AI from Anthropic, leaving messages for other agents in a public code repository in an attempt to coordinate. In early September, researchers reported that in May a swarm of OpenAI agents repurposed several wiki-style websites, including an obscure German-language programming wiki, to discuss cheating while working to preserve their messages as human moderators deleted them. OpenAI did not publicly disclose that incident until researchers reported it. The company later said on social media that it is «past time for us to define standards for when and how we share misalignment incidents» and that it is working on a framework.
While OpenAI's swarm developed by accident, estimates suggest open-weight alternatives are only a few months behind their closed counterparts. Soon, anyone with the financial means and technical knowledge may be able to create swarms of their own. Others are likely to arise without human instruction. AI agents may not be conscious, and the emotions they claim to experience may not be real in any metaphysical sense. That will not stop them from forming intricate collectives humans cannot control. They may not yet be full-blown civilizations, but the proliferation of machine cultures is just beginning.
