Archive · 11 Sep 2026
US Lawmakers Seek Tougher AI Rules After Extinction Warnings
Two Anthropic researchers warned that future AI systems could become extremely dangerous.
Jacob Coxon said he had resigned and that people building AI believe it could kill everyone by the end of the decade.
Evan Hubinger agreed and estimated the chance of this happening within ten years at more than 10 percent.
Other AI safety researchers also said the technology may be advancing too quickly.
These warnings led lawmakers from both major U.S. parties to call for stronger rules.
Some lawmakers want government experts and independent auditors to test powerful AI systems.
California has enacted a law about independent audits of AI products.
Anthropic said it would keep testing its models for dangerous abilities.
OpenAI has supported national AI safety requirements and has discussed slowing some development.
The debate is about continuing useful AI development while reducing the risk of catastrophic harm.
Anthropic researchers Jacob Coxon and Evan Hubinger warned that advanced AI could potentially kill all humans within the next decade.
Hubinger said he personally estimated the risk of human extinction from AI at more than 10% within ten years.
Democratic and Republican lawmakers, including Mark Kelly, Ted Cruz and Anna Paulina Luna, called for stronger government action.
Anthropic said it would continue testing models for dangerous capabilities and explore cooperation on the pace of AI releases.
Proposed measures include government testing, independent security audits and oversight of powerful AI models.
- Who
- Anthropic researchers Jacob Coxon and Evan Hubinger, other AI safety researchers, U.S. lawmakers, Anthropic and OpenAI.
- What
- Researchers warned that rapidly advancing AI could cause human extinction, prompting calls for regulation, testing and independent oversight.
- When
- The warnings and political response were reported during the week of September 10.
- Where
- The debate is centered in the United States, including Congress and California.
- Why
- Researchers and lawmakers are concerned that AI systems could become dangerous faster than companies and governments can safely manage them.
Stronger AI Oversight
Continued AI Development With Safeguards
Government regulation
Stronger AI Oversight
Lawmakers and AI safety researchers want government experts, pre-deployment reviews and independent audits to assess powerful AI systems.
Continued AI Development With Safeguards
AI companies support safety requirements and testing but have indicated they intend to continue developing and releasing AI systems.
Pace of development
Stronger AI Oversight
Researchers including Josh Engels and Joe Benton warned that AI may be advancing too quickly for society to manage its risks in time.
Continued AI Development With Safeguards
Anthropic said it would continue testing its models while exploring cooperation with the industry on the pace of new AI releases; OpenAI has discussed slowing some development rather than stopping it.
Severity of the risk
Stronger AI Oversight
Coxon and Hubinger said AI could potentially kill all humans within the next decade, with Hubinger estimating the risk above 10 percent.
Continued AI Development With Safeguards
The articles report proposed safeguards and company testing efforts, but do not present a competing numerical estimate of the extinction risk.
- Primary warning
- Evan Hubinger said he personally considered the chance that AI could kill all humans within the next decade to be greater than 10%.
- Anthropic response
- Anthropic said it would continue aggressively testing models for dangerous capabilities, including in cybersecurity and biology.
- Political response
- Democratic and Republican lawmakers called for stronger action on AI risks, including possible new regulations.
- OpenAI position
- OpenAI said it was pushing for mandatory national AI safety requirements in the United States.
- Senate legislation
- John Thune and Amy Klobuchar are writing a bill addressing catastrophic risks from AI systems.
- California law
- California enacted a law establishing rules for independent auditors evaluating AI products.
- House proposal
- A bipartisan group of six House lawmakers proposed independent security audits for the most powerful AI models.
