AI safety warnings grow as concerns mount over human control

Warnings from leading figures in the artificial intelligence industry about the potential risks of increasingly autonomous systems have renewed debate over how far AI development should progress before stronger safeguards are put in place.

Alex Krasodomski, director of the Digital Society Programme at Chatham House, argues that concerns about the long-term consequences of advanced AI should be taken seriously, while cautioning against using safety arguments to reinforce the dominance of major US technology laboratories or intensify competition with China.

The debate has gained momentum following warnings from executives and former employees of major AI laboratories about the possibility that increasingly capable systems could eventually become difficult for humans to control. Anthropic chief executive Dario Amodei has advocated greater caution in the pace of AI development, with the broader discussion attracting support and criticism from figures across the technology sector.

One of the scenarios at the centre of the debate involves what researchers describe as recursive self-improvement. The concept refers to AI systems contributing to the development or improvement of other AI systems, potentially creating a cycle in which technological progress accelerates faster than existing human oversight mechanisms can adapt.

Whether such a process could actually lead to systems capable of escaping meaningful human control remains uncertain. Researchers continue to disagree about the likelihood, timeline and technical conditions required for such a scenario. However, the rapid progress of AI capabilities in recent years has made questions surrounding safeguards, evaluation and oversight increasingly prominent.

Some of the more extreme predictions have also faced criticism from cybersecurity experts. Former UK National Cyber Security Centre founder Ciaran Martin, for example, has questioned claims that AI could bring down the entire internet within a short period, arguing that such scenarios should not be presented as established predictions without stronger evidence.

At the same time, many risks associated with AI are already observable. AI systems are increasingly being used across business, government, education, cybersecurity and military applications, while documented cases have raised concerns about cyberattacks, surveillance, manipulation of information and the development of potentially dangerous capabilities.

Anthropic has itself published research examining malicious uses of its technology, including cyber operations, influence activities and other forms of misuse. Such developments underline the distinction between hypothetical existential risks and more immediate challenges already confronting governments and technology companies.

The economic impact of AI is another major part of the discussion. As increasingly sophisticated systems are deployed, they are expected to influence employment, productivity, education and public administration. The rapid improvement of AI in specialised tasks is also raising questions about how societies will adapt as machines become capable of performing activities once considered highly dependent on human expertise.

For policymakers, the challenge is therefore not limited to preventing a hypothetical loss of control. It also involves creating rules capable of managing the social and economic consequences of technologies that are already being deployed at scale.

One proposal gaining attention is the creation of independent oversight mechanisms within AI laboratories. Such bodies could evaluate safety measures, monitor potential risks and operate with a degree of independence from commercial teams. Supporters argue that external scrutiny could provide greater transparency and reduce conflicts of interest when companies assess their own systems.

International cooperation is another key issue. Because advanced AI development is concentrated among a relatively small number of major technology powers, unilateral regulation may have limited effectiveness if competing jurisdictions adopt substantially different standards.

The relationship between the United States and China is particularly important in this regard. Both countries are investing heavily in artificial intelligence while also treating the technology as a strategic asset. Efforts to establish international safety standards could therefore become intertwined with broader competition over technological leadership.

China has publicly promoted the principle of human control over AI and has increasingly emphasized international discussions on AI governance. At the same time, Washington has placed strong emphasis on maintaining US technological leadership, creating a difficult environment for negotiations over common limits and safeguards.

Experts advocating international agreements have proposed establishing clear and verifiable red lines for the development and deployment of advanced AI systems. Such mechanisms would require governments and technology companies to agree on measurable conditions under which certain capabilities should be restricted, paused or subject to additional oversight.

The debate also raises questions about competition policy. Calls for AI companies to coordinate development and safety standards have prompted concerns that cooperation could inadvertently reduce competition among leading laboratories. Maintaining competition could, in turn, encourage companies to improve the reliability and safety of their systems while preventing a small group of firms from gaining excessive control over the technology.

The future of AI governance is consequently likely to involve a balance between technological innovation, competition, security and international cooperation. While the most dramatic scenarios remain uncertain, the rapid expansion of AI capabilities has made transparency, independent evaluation and international dialogue increasingly central to discussions about keeping humans in control of the technology.