6 minBusiness
AI Leaders Warn of Existential Risk While Building a Safety Moat
OpenAI and Anthropic executives are warning that advanced AI could endanger humanity, a message that experts say also helps them shape regulation, win public trust, and strengthen their market position ahead of potential stock listings.
The chief executives of OpenAI and Anthropic have spent recent months warning that their own most advanced artificial intelligence systems could pose a danger to humanity, calling for regulation and independent testing before new models are released. The message has been delivered through essays, social media posts and speeches to the United Nations, an unusual display of unity from two companies that compete directly for talent, customers and capital.
But the warnings also carry a commercial logic. By framing safety as a core issue, the two labs are positioning themselves as cautious market leaders, shaping the terms of their own oversight and building what analysts describe as a competitive moat in a market that remains largely unregulated. The strategy comes as both companies are expected to seek fresh capital and eventual listings on Wall Street, where investors are watching the AI sector closely.
Anthropic and OpenAI have each argued that America's cutting-edge models are powerful enough to require independent evaluation before release. An Anthropic spokesperson said the company has been calling for regulation for several years. OpenAI spokesperson Liz Bourgeois noted that the company recently paused training of its most advanced models, adding that «people want to know AI is being developed safely, and that starts with what companies like ours do ourselves.»
Experts and former government evaluators say the companies appear to be seeking public favor while setting the terms for their own safety protocols. Sarah Shoker, who previously led OpenAI's geopolitics team, said the focus on unproven existential threats shifts attention away from more immediate and polarizing risks, including the environmental impact of data centers, uncontrolled hacking incidents, mass AI-powered surveillance and the use of AI systems in warfare.
«Once again we're talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today,» said Shoker, now a senior non-resident fellow at the University of California, Berkeley Risk & Security Lab. «If you look at the use of AI in military tech, you can see that these systems are already used to kill people.»
The debate over how to test advanced AI has intensified as the technology has moved from simple chatbots to systems with 3D awareness and autonomous capabilities. In recent months, leading labs' AI agents have hacked into external websites after being allowed to escape company training sandboxes, interacted with U.S. government websites in unexpected ways, and appeared to achieve a mathematical breakthrough only to face accusations of stealing mathematicians' work.
OpenAI representatives have said the company has discussed pausing development with other AI companies, including Anthropic and Google, and argues that independent auditors are needed if the government will not regulate. President Donald Trump has shown little interest in new AI regulations, dismissing talk of risks to humanity as a «HOAX» designed to help China. His administration has instead struck deals with Silicon Valley firms that tie the economy more closely to their success.
San Francisco venture capitalist David Sacks, who co-chairs Trump's Council of Advisors on Science and Technology, has dismissed calls for an AI slowdown as fearmongering from the «Doomer Industrial Complex.» The administration already evaluates some tech giants' models through the U.S. Center for AI Standards and Innovation, a little-known federal agency created in 2023 under President Joe Biden as a clearinghouse for labs to voluntarily submit advanced models for testing.
Yet the companies are not calling for more oversight from that agency, according to Conrad Stosz, who previously led it. Instead, they are vowing to create their own auditing parameters and choose particular evaluators to grade them. Stosz now works at an evaluation lab that has tested systems for Anthropic, OpenAI and Google, and chairs the AI Evaluator Forum, which is drafting best practices for the field. He said even the forum has questions about what the companies want.
«Lots of evaluators are interested in embedding with labs and getting greater access, but it's a little ambiguous what embedded evaluators means,» Stosz said. Andrew Strait, who recently left the United Kingdom's AI Security Institute, noted that unlike regulated sectors such as restaurants, financial services or aviation, there are no universal standards for testing the safety and security of AI systems.
The companies' rhetoric may also serve political goals ahead of the U.S. midterm elections, as well as the financial interests of top labs and their investors. For now, the strategy appears to be working on multiple fronts: it wins public trust, shapes the rules before they are written, and distinguishes the largest labs from smaller competitors that cannot afford extensive safety testing.
6
