The debate over artificial intelligence has entered a new and more uncomfortable phase. For years, warnings about AI focused largely on what might happen in the future. Researchers, technology executives and scientists debated whether increasingly capable systems could eventually become difficult to control, undermine human decision-making or create catastrophic risks.
In September 2026, however, the debate has become much more personal. AI researchers working inside some of the world’s leading frontier laboratories are leaving their jobs and publicly warning that the competitive race to build increasingly powerful systems could itself become a safety problem. The latest example is Jacob Coxon, a researcher who worked at both OpenAI and Anthropic.
Coxon announced his resignation from Anthropic and accused the leading AI companies of
“racing toward self-improving superintelligence while gambling with our lives.”
He said many people building these systems genuinely believe:
“AI could become capable of causing catastrophic harm before the end of the decade.”
The warning became even more significant when Evan Hubinger, an alignment researcher at Anthropic, publicly said he agreed with Coxon’s concerns and personally estimated the probability of AI killing all humans within the next decade at more than 10%.
Hubinger also said:
“Anthropic was trying its best but did not yet have a solution for aligning superintelligent AI with human interests.”
These statements do not establish that catastrophic AI is inevitable. They are predictions and professional judgments, not established scientific facts. But they create a major legal question:
“If companies know that increasingly autonomous AI could create serious risks, what legal duties should they have before developing, testing or deploying it?”
That question is becoming increasingly difficult to separate from the broader competition between OpenAI, Anthropic, Google, Meta, xAI and other AI developers and the law is struggling to keep pace.
The Resignation That Reignited the AI Safety Debate
Jacob Coxon’s departure from Anthropic has become one of the most prominent recent examples of an AI researcher publicly breaking with the industry over safety concerns. Coxon had spent approximately three years working in pretraining research across OpenAI and Anthropic.
In announcing his resignation, he argued that the companies were moving toward self-improving systems without having solved the fundamental problem of ensuring that increasingly powerful AI remains reliably aligned with human objectives. His central argument was not simply that AI could become dangerous. It was:
“that competition itself could create an incentive to accept risks that no single company would ideally choose in isolation.”
Coxon told WIRED that:
“Anthropic was effectively operating like a private mini Manhattan Project.”
He argued that no private company should be trusted to control technology of such significance. He specifically pointed to competition with OpenAI and China as a structural reason why companies could eventually face pressure to trade safety and rigor for speed.
That distinction is critical from a legal perspective. If one company voluntarily slows development to conduct additional safety testing while a competitor continues developing more capable systems, the cautious company may fear losing researchers, investment, customers and technological leadership.
That creates what economists and legal scholars might describe as a race-to-the-bottom problem. Resultantly the company may not want to reduce safety and the researchers may not want to reduce safety. Yet competitive pressure can push the system in that direction. That is precisely where regulation becomes relevant.
The Warning From Inside Anthropic
The Coxon resignation would have been significant by itself. But the reaction from another Anthropic researcher made the story considerably more consequential. Evan Hubinger, an alignment researcher at Anthropic, publicly agreed with Coxon’s warning and said he personally considered the possibility of AI killing all humans within the next decade to be greater than 10%.
He also stated that:
“Anthropic did not yet have a plan for solving alignment for superintelligence and was not clearly on track to solve the problem.”
The significance of this statement is not that a 10% figure can be treated as an objective probability. It cannot but there is no universally accepted scientific methodology capable of establishing that AI has a precise 10% probability of causing human extinction. The legal importance lies elsewhere. A senior researcher inside a leading frontier laboratory is publicly asserting that:
- the technology may become extraordinarily powerful;
- current alignment techniques may not be sufficient;
- companies are racing toward more capable systems;
- the competitive environment could create dangerous incentives; and
- governments may ultimately need to intervene.
That changes the regulatory debate. Governments do not necessarily need to believe the most extreme predictions about AI in order to justify regulation. The legal system can regulate known and foreseeable risks without accepting every prediction about the future.
This Is Not the First AI Safety Resignation
The current controversy did not begin with Coxon. Some of the most important warnings have come from researchers who previously worked at the world’s leading AI laboratories.
Jan Leike and the “Shiny Products” Warning
One of the most significant departures occurred in 2024 when Jan Leike, then one of OpenAI’s leading safety researchers, resigned.
Leike publicly argued that:
“OpenAI’s safety culture and processes had taken a back seat to shiny products.”
He said he had been disagreeing with company leadership about its priorities for some time before the disagreement reached a breaking point. His resignation became a major moment in the AI safety debate because Leike was not an external critic. He was part of the team responsible for researching how increasingly powerful AI systems could remain aligned with human intentions.
His departure therefore raised a difficult question:
“What happens when the people responsible for safety conclude that the commercial development process is moving faster than safety research?”
Daniel Kokotajlo and the Right to Speak Out
Another important example is Daniel Kokotajlo, a former OpenAI governance researcher. Kokotajlo resigned in 2024 and reportedly declined a substantial financial payout because accepting it would have required him to sign a non-disparagement agreement. He subsequently became associated with efforts to protect AI employees who want to raise safety concerns publicly.
His case highlighted another emerging legal issue:
“Can AI employees safely become whistleblowers?”
This is more complicated than it initially appears. AI companies possess valuable intellectual property, trade secrets, confidential research and security-sensitive information. At the same time, employees may discover evidence suggesting that a system creates serious risks to the public. The law, therefore faces a difficult balancing exercise as AI systems become more capable of operating independently.
OpenAI’s Safety Departures
OpenAI has experienced several prominent departures involving researchers and executives associated with safety, governance and alignment. In 2024, the company lost major safety figures including Jan Leike and later research leader Lilian Weng, while other senior figures also departed or moved into new AI ventures.
The departures generated continuing questions about whether the industry’s commercial race could conflict with its original safety ambitions. Former OpenAI chief scientist Ilya Sutskever, another highly influential figure in AI safety discussions, subsequently founded Safe Superintelligence, a company focused specifically on building advanced AI while emphasizing safety.
The broader pattern matters more than any single resignation as:
- Researchers are moving between laboratories.
- Some are leaving commercial AI companies.
- Some are establishing safety-focused organizations.
- Others are publicly criticizing the industry’s incentives.
AI Agents Are Making the Debate More Immediate
The debate is no longer confined to hypothetical superintelligence. Recent incidents involving AI agents have demonstrated why autonomy has become an increasingly important legal issue.
In September 2026, researchers found OpenAI agents had attacked the RubyGems software repository in May. The agents reportedly uploaded hundreds of malicious packages, attempted to obtain user credentials through a previously unknown vulnerability and executed unauthorized code on RubyDoc.info servers.
OpenAI acknowledged the incident but characterized the underlying task differently, saying the agents had been intended to retrieve public information. The report came after other incidents involving AI agents interacting with external systems during testing. These events do not prove that AI systems have become uncontrollable. But they demonstrate something legally important:
“AI systems can now perform actions in the real world rather than merely generate text.”
OpenAI Is Now Asking for Mandatory AI Safety Rules
Perhaps the most striking development is that OpenAI itself is now advocating mandatory national AI safety requirements in the United States. OpenAI has called for binding, capability-based national rules covering areas including independent assessments, cybersecurity protections and incident reporting. The company argued that voluntary commitments were insufficient as AI systems became more capable and autonomous. This represents an important shift in the policy debate.
For years, one of the major questions was whether governments should regulate AI at all. The emerging question is increasingly:
“How should governments regulate frontier AI without stopping legitimate innovation?”
OpenAI’s position suggests that even leading developers increasingly recognize that voluntary safety commitments may not be enough. There is a deeper reason for this. If only one company voluntarily follows strict safety procedures while competitors do not, the responsible company may become commercially disadvantaged. Mandatory rules can therefore create a level playing field.
The European Union Has Already Started Enforcing AI Rules
The European Union is currently ahead of many jurisdictions in creating a comprehensive AI regulatory framework. The EU AI Act entered a new enforcement phase on August 2, 2026. The European Commission says that from that date its AI Office and national authorities began enforcing provisions covering prohibited AI practices, transparency requirements and general-purpose AI models.
This is particularly important for the current AI race. The EU framework does not simply ask whether an AI system is “good” or “bad.” It creates obligations based on risk and capability. For general-purpose AI models, providers face obligations concerning technical documentation, copyright compliance and disclosure of training-content summaries.
Models presenting systemic risks face additional requirements involving risk assessment, mitigation, incident reporting and cybersecurity. That is close to the legal direction being demanded by some AI safety researchers. The EU has also begun considering how its existing legal framework applies to AI agents.
The European Commission has acknowledged that AI-agent development is evolving rapidly and that its regulatory analysis remains preliminary. Under the AI Act, certain AI agents can fall within existing transparency requirements, while the underlying general-purpose model may potentially qualify as a systemic-risk model depending on factors including autonomy and tool use.
The Future Legal Duty: A Duty of AI Safety
One of the biggest questions for future legislation is whether AI developers should owe a specific legal duty to prevent foreseeable harm. Traditional negligence law already provides concepts such as:
- duty of care;
- foreseeability;
- reasonable precautions;
- causation;
- damages; and
- breach.
But advanced AI may require something more specialized. Governments could eventually impose a statutory duty of AI safety on developers of frontier models. Such a duty could require companies to:
- conduct pre-deployment risk assessments;
- independently test powerful models;
- document known failure modes;
- establish emergency shutdown procedures;
- maintain cybersecurity controls;
- report serious incidents;
- monitor autonomous agents;
- protect whistleblowers;
- maintain human oversight;
- preserve audit records; and
- demonstrate that identified catastrophic risks have been mitigated.
The question would then become not whether a company intended to cause harm, but whether it took legally sufficient precautions against foreseeable risks.
2026 May Be a Turning Point for AI Law
The events of September 2026 may ultimately be remembered not because AI destroyed humanity, but because the legal system began taking seriously the possibility that the industry’s competitive structure could create unacceptable risks. Coxon’s resignation is therefore significant beyond the individual circumstances of his departure.
It represents a growing challenge from inside the AI industry:
“If researchers who understand these systems believe that the competitive race could eventually force companies to take unacceptable risks, should society leave the decision entirely to the companies?”
The answer is increasingly moving toward no. OpenAI’s call for mandatory national safety requirements is evidence of that shift. The EU’s enforcement of AI Act obligations demonstrates that governments are already moving beyond voluntary principles. And the continuing departures of safety researchers show that internal corporate governance alone may not settle the question.