5 hrs ago
Lawmakers Seek New AI Rules After Extinction Warnings
Some scientists who work on artificial intelligence are worried that future AI systems could become extremely dangerous.
Jacob Coxon said people building AI believe it could kill everyone by the end of this decade.
Evan Hubinger agreed and estimated the risk at more than 10% within the next ten years.
Other former AI safety researchers also said the technology may be moving too quickly.
These warnings led lawmakers from both major U.S. parties to call for stronger rules.
Some lawmakers want government experts and independent auditors to test powerful AI systems.
Anthropic says it will keep testing AI for dangerous abilities.
OpenAI has discussed slowing some development and supports national safety requirements.
The debate is about how to encourage useful AI while reducing the chance of catastrophic harm.
Anthropic researchers Jacob Coxon and Evan Hubinger warned that advanced AI could potentially kill or even extinguish humanity within the next decade.
Democratic and Republican lawmakers, including Mark Kelly, Ted Cruz and Anna Paulina Luna, called for stronger government action and oversight.
Anthropic said it would continue testing models for dangerous capabilities while exploring cooperation with the industry on release timelines.
OpenAI has reportedly discussed slowing development and is advocating mandatory national AI safety requirements.
Proposed federal and state measures include government testing, independent security audits and oversight of powerful AI models.
- Who
- Anthropic researchers, other AI safety specialists, U.S. lawmakers, Anthropic and OpenAI.
- What
- Researchers warned that rapidly advancing AI could cause catastrophic harm, prompting calls for new safety rules and independent oversight.
- Where
- The debate is centered in the United States, including Congress and California.
- When
- The warnings and political response occurred during the week of September 10; several legislative efforts were proposed earlier in the year.
- Why
- Researchers and lawmakers are concerned that AI systems could become dangerous faster than companies and governments can safely manage them.
Stronger AI Oversight
Continued AI Development With Safety Measures
How quickly AI should advance
Stronger AI Oversight
Researchers and lawmakers argue that AI development may be moving too quickly and want government testing, independent audits and possible limits on releases.
Continued AI Development With Safety Measures
Anthropic said it would continue developing and testing AI models, while OpenAI has indicated it may slow parts of development and work toward safety requirements.
Role of government
Stronger AI Oversight
Lawmakers including Mark Kelly, Ted Cruz and Anna Paulina Luna say Washington should take AI risks more seriously and create stronger rules.
Continued AI Development With Safety Measures
The companies’ stated approach emphasizes testing, cooperation with the industry and safety requirements rather than stopping AI development.
Key facts
- Primary warning
- Evan Hubinger said he personally considers the chance that AI could kill all humans within the next decade to be greater than 10%.
- Anthropic response
- Anthropic said it would continue aggressively testing models for dangerous capabilities, including in cybersecurity and biology.
- OpenAI position
- OpenAI said it was pushing for mandatory national AI safety requirements in the United States.
- Senate legislation
- John Thune and Amy Klobuchar are writing a bill addressing catastrophic risks from AI systems.
- California law
- California enacted a law establishing rules for independent auditors evaluating AI products.
- House proposal
- A bipartisan group of six House lawmakers proposed requiring independent security audits for the most powerful AI models.
- Research group
- Former AI safety researchers Josh Engels and Joe Benton joined METR, a nonprofit that measures potential AI threats to society.
Quotes
Amy Klobuchar
Democratic U.S. senator involved in drafting AI regulation legislation
“people building AI earnestly believe that it could kill us all by the end of the decade.”
indianexpress.com
“work with government experts to verify and test models to make sure AI is safe.”
livemint.com
Evan Hubinger
Anthropic scientist who endorsed Jacob Coxon’s warning about AI’s potential risks
“We really do earnestly believe AI could kill all humans. I personally think it is >10% within the next decade.”
livemint.com
indianexpress.com
Sources
Related news
AI Researcher Quits, Lawmakers Demand Tougher US Rules
Former Anthropic Researcher Warns AI Could Kill Humanity This Decade
Anthropic Researcher Resigns, Renewing Debate Over AI Safety Risks
UN Rights Chief Warns Advanced AI Could Threaten Humanity
OpenAI Calls for Binding US Rules as AI Fears Grow
OpenAI Considers Slowing Frontier AI as Safety Concerns Grow
AI Experts Warn Advanced Systems Could Threaten Humanity
Silicon Valley Keeps Moving the Goalposts on AGI Claims
AI Reshapes Patient-Centric Healthcare and Life Sciences Progress
