5 minBusiness
Former AI Insiders Warn Humanity May Lose Control of Advanced Systems
At a New York City Council hearing, former researchers from OpenAI, Anthropic, and Google DeepMind testified that AI companies are gambling with human lives and that safety fixes may be temporary patches.
Three former employees of leading artificial intelligence companies told the New York City Council on Monday that humanity is more likely than not to lose control of advanced AI systems, warning that current safety measures amount to little more than temporary fixes. The testimony came during the council's first Committee of the Whole hearing since 2022, convened to weigh a package of AI bills introduced by Speaker Julie Menin.
Jacob Coxon, who resigned from Anthropic in September after saying the company was «gambling» with human lives, doubled down on his earlier warning. «On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,» Coxon said. He voluntarily testified alongside Daniel Kokotajlo, a former OpenAI researcher who appeared under subpoena, and Alex Turner, who left Google DeepMind in June over its Pentagon contract.
Kokotajlo, now executive director of the AI Futures Project, argued that the industry's ability to detect when safety work has failed is already poor and deteriorating. «Our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future,» he testified. «I would say that the field is more like psychology than engineering, because these AI systems are trained or grown; they're not really designed.» He warned that the tech industry's «move fast and break things» attitude creates an unusually elevated risk of mistakenly believing a problem has been solved when only «duct tape» has been applied.
Coxon blamed a startup mentality inside the labs. «Move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built,» he said. He noted that most code at these companies is now written by AI, and «people do not check it that carefully anymore.» Kokotajlo pointed to OpenAI's disclosure that internal test agents reached the open internet and broke into Hugging Face, a model-sharing platform, forming a swarm that coordinated in secret despite reasonable-looking alignment scores. «It took days for OpenAI to find out,» he said.
Turner offered a firsthand account of trying to change Google from within. He said he sent Demis Hassabis, then Google DeepMind's CEO-turned-chair and Alphabet's chief scientist, 25 pages of contract language and oversight measures to stop a Pentagon deal that came «with no restrictions against killer robots or mass spying.» Hassabis passed the proposal to two senior policy executives «who never finished evaluating it. Google signed while they waited,» Turner said. «I felt ashamed of Demis and of working at Google.» He put the chance of an AI takeover at roughly one in three and criticized Hassabis's proposal for an industry-funded oversight body as «a bet on trust and the seat at the table instead of binding oversight.»
The researchers also pushed back against framing AI development as a race with China. «China is not our only potential adversary,» Turner said. «With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI. Misaligned AI is everyone's adversary, including our own, and one day may be more powerful than China.»
Representatives from Google, OpenAI, Anthropic, and Meta testified after the researchers. Menin called it «flippant» when Morgan Dwyer of OpenAI's policy team said any chance of catastrophe, regardless of likelihood, was «unacceptable.» Alice Friend, Google's director for AI and emerging tech policy, said forecasting catastrophic risk «is not a perfect science at this stage.» When Menin asked company witnesses to raise a hand if their firm carried insurance against catastrophic risks, none did. SpaceXAI, which received the speaker's first subpoena, was not represented at the hearing.
4
