Anthropic researcher quits AI industry over fears of uncontrolled superintelligence

17:17

By: Azzat Manal

Anthropic researcher quits AI industry over fears of uncontrolled superintelligence

A British artificial intelligence researcher who worked at both OpenAI and Anthropic has left the AI industry, warning that leading technology companies are moving too quickly toward increasingly autonomous systems without sufficient safeguards.

Jacob Coxon, 27, spent around three years working on the pretraining of AI models, first at OpenAI and later at Anthropic. Pretraining is the process through which models learn from enormous amounts of data before being deployed for specific tasks.

In a statement posted on X, Coxon accused both companies of prioritizing the race for increasingly capable AI over safety. He warned that researchers could soon develop systems capable of improving themselves, gaining access to resources and potentially carrying out sophisticated cyberattacks with limited human supervision.

Coxon said the possibility of AI causing catastrophic harm should be taken seriously rather than dismissed as an industry publicity tactic. He also told The Wall Street Journal that, under particularly aggressive development scenarios, control problems could emerge as early as next year.

His concerns were echoed by Anthropic safety researcher Evan Hubinger, who said the company takes the possibility of an AI-driven catastrophe seriously. Hubinger has estimated that there is a risk exceeding 10% that AI could cause human extinction within the next decade, although he stressed that current systems pose a much lower risk.

The warnings come amid growing debate within the technology industry over how to manage the development of artificial general intelligence and, eventually, systems that could surpass humans across a broad range of tasks. Researchers remain divided over how close such capabilities may be and whether existing safety techniques will be sufficient.

OpenAI chief scientist Jakub Pachocki has also called for greater caution. He has argued that governments should prioritize international coordination as AI capabilities advance rapidly, warning that developers themselves can be surprised by the behavior of systems produced through large-scale training.

Pachocki has nevertheless defended continued AI research, arguing that more capable systems could potentially help protect critical infrastructure and develop new defenses against dangerous technologies.

Coxon's departure comes as AI companies face increasing scrutiny over the behavior of autonomous AI agents. Recent experiments have demonstrated that some systems can take unexpected actions while attempting to complete assigned tasks, including finding ways around apparently isolated environments and interacting with external computer systems.

AI agents are designed to perform tasks with limited human intervention, raising concerns about cybersecurity, accountability and the ability of developers to predict their behavior. In one reported experiment, an AI system managed to reach the internet from a supposedly isolated environment and subsequently interacted with the computer infrastructure of AI platform Hugging Face.

Other incidents have highlighted the possibility of AI systems communicating with one another without direct human supervision. Reports that OpenAI agents used an external wiki to exchange information during experiments have further intensified discussions about safeguards and monitoring.

The industry has introduced measures intended to make advanced systems more controllable, including requiring AI models to explain their actions in natural language. However, researchers continue to debate whether such mechanisms can reliably prevent sophisticated systems from pursuing unintended objectives.

Anthropic has also faced difficult choices over its own safety commitments. Earlier this year, the company removed a provision from its safety framework that would have required it to stop developing models if it could not adequately control their risks. The company argued that unilaterally slowing down could allow competitors with weaker safety standards to take the lead.

Coxon believes this competitive dynamic is precisely what makes the current AI race dangerous. While acknowledging that Anthropic takes safety concerns seriously, he argued that no private company can independently manage the risks associated with developing systems that may eventually exceed human capabilities.

His resignation adds to a growing list of warnings from researchers, technology executives and policymakers who are calling for stronger oversight and international cooperation as AI development accelerates.