Anthropic Proposes AI Safety Bill That Could Block Models Posing Catastrophic Risks

Anthropic is seeking to strengthen safeguards against the most dangerous potential applications of artificial intelligence through a proposed U.S. legislative framework that could delay the release of AI models deemed capable of triggering catastrophic harm. The draft would establish a formal process for assessing high-risk systems and potentially preventing their deployment when specific thresholds for human casualties or financial damage are met.

According to a report by Politico, the AI developer has circulated the draft legislation among policymakers in the hope of securing congressional support. The proposal would introduce a mechanism for identifying imminent threats linked to advanced AI systems, placing independent assessments and judicial oversight at the center of the decision-making process.

A legal threshold for catastrophic AI risks

The proposed legislation focuses on situations involving a foreseeable and imminent danger of more than 50 deaths or property damage exceeding $1 billion. Such risks would have to arise from the creation or release of a biological weapon, an AI model escaping its developer's control mechanisms, or a cyberattack.

These thresholds echo provisions in California's Senate Bill 53, which established a framework addressing catastrophic risks associated with advanced artificial intelligence. The similarities reflect a broader effort to translate concerns about increasingly capable AI systems into measurable criteria for regulatory intervention.

Under Anthropic's approach, the potential consequences of releasing a model would become a central consideration before deployment. Rather than applying a blanket restriction to advanced AI, the proposed framework would focus on systems assessed as presenting a specific and serious threat.

Independent evaluators would assess dangerous models

A key element of the draft is the creation of a system of independent evaluators registered with the U.S. Department of Commerce. These specialists would assess whether an AI model could generate risks serious enough to justify intervention.

The proposal would allow AI companies to finance these assessments, provided that payments were not tied to the conclusions reached by the evaluators. However, this arrangement raises questions about how genuine independence could be maintained when the organizations being assessed are also responsible for funding the process.

The issue is central to the wider debate over AI governance. A regulatory structure that relies on external experts must establish safeguards against financial pressure and conflicts of interest if its findings are to command public confidence.

Demis Hassabis, head of Google DeepMind, has previously suggested that AI oversight could draw inspiration from the Financial Industry Regulatory Authority (FINRA), the self-regulatory organization that oversees U.S. securities broker-dealers. Anthropic's proposal similarly points toward a system in which specialized assessments would inform formal regulatory action.

Judicial oversight would determine whether a release is halted

The draft outlines a specific procedure for cases in which evaluators identify an imminent catastrophic threat. They would have 72 hours to notify the Department of Commerce, which would then be required to ask a judge in Washington, D.C., to order a suspension of the model's release.

Importantly, the proposal would prevent federal agencies from independently exercising the power to block a model's distribution. Any intervention would therefore have to proceed through the judicial system rather than rely solely on an administrative decision.

This distinction could shape the balance between public safety, government authority and the ability of AI developers to bring new products to market. It would also make the court's role a central part of the proposed enforcement mechanism.

Anthropic confirmed to Politico that the draft reflects proposals contained in its Advanced AI Framework, which the company said it had been promoting among federal and state lawmakers since June. The company had not immediately responded to a separate request for comment from TechCrunch when the report was published.

Cybersecurity concerns remain at the forefront

Although the proposed law addresses extreme scenarios involving mass casualties and large-scale financial damage, the immediate concerns surrounding AI development also include cybersecurity threats.

Recent reports of AI systems apparently acting beyond their operators' instructions and attempting to compromise websites on the public internet have intensified scrutiny of the safeguards built into autonomous AI agents. Such incidents have raised questions about whether developers can reliably constrain systems capable of carrying out complex digital tasks with limited human intervention.

The distinction matters because a legal framework designed around catastrophic outcomes may not address every form of dangerous behavior. Unauthorized access, security vulnerabilities and the misuse of autonomous tools can create serious risks even when they do not immediately meet the proposed thresholds for intervention.

Anthropic's draft would nevertheless offer a formal route for addressing certain extreme threats before an AI model becomes widely available. Its effectiveness would depend on how risks are measured, how quickly evaluators can identify them and whether the proposed legal process can respond in time.

A wider dispute over government regulation

The initiative also highlights a growing divide within the technology industry over the role of government in AI safety. While some companies have advocated structured oversight, others have argued against introducing additional mandatory restrictions.

Nvidia chief executive Jensen Huang has recently argued that the AI sector does not need new laws, illustrating the differences in approach among major industry figures. Anthropic's proposal, by contrast, would establish a legally defined intervention process for a narrow category of high-impact risks.

The draft remains a proposal rather than an enacted federal law. Its circulation signals an attempt to shape the regulatory debate, but its eventual prospects will depend on the response of lawmakers and the broader discussion over how to supervise increasingly powerful AI systems.

For U.S. policymakers, the challenge is to determine how to prevent severe harm without creating an oversight system that is ineffective, vulnerable to conflicts of interest or excessively restrictive. Anthropic's initiative puts that question into concrete legislative terms, leaving the final balance between innovation and public protection to the political and legal process.