2 hrs ago
Former Anthropic Researcher Warns AI Could Outpace Nuclear Threats
Jacob Coxon used to research AI at Anthropic and OpenAI.
He says very advanced AI could become more dangerous than nuclear weapons.
His concern is that nobody yet knows how to make such systems reliably follow human instructions.
He says major AI companies are competing to build powerful systems quickly.
Anthropic safety executive Evan Hubinger agreed that the danger could be extremely serious.
Hubinger estimated that AI could kill all humans with a probability above 10% over the next decade.
Anthropic says stopping alone could allow less cautious companies to take over.
Several technology workers and researchers are asking governments to coordinate and slow down development.
The United States does not currently regulate AI models through federal law.
Former Anthropic researcher Jacob Coxon said uncontrolled artificial superintelligence could pose a greater threat than nuclear weapons.
Coxon accused Anthropic and OpenAI of prioritizing competition to develop advanced AI over safety.
Anthropic safety executive Evan Hubinger said the company believes AI could potentially kill all humans, estimating the risk above 10% over the next decade.
Anthropic removed a pledge to halt model development if it could not control associated risks, citing competition from less cautious rivals.
Industry employees and researchers have called for international coordination and caution, while the United States currently has no federal law regulating AI models.
- Who
- Former Anthropic researcher Jacob Coxon, Anthropic safety executive Evan Hubinger, and executives and researchers from Anthropic and OpenAI.
- What
- Coxon warned that artificial superintelligence could become an existential threat, while industry figures debated safety, competition, and regulation.
- Where
- The warnings and responses were shared through social media, Fox News, and a blog post, with calls directed toward Washington and governments worldwide.
- When
- Coxon issued his warning on Tuesday; other developments cited occurred in February, late July, August, and Sunday.
- Why
- Coxon argues that companies cannot safely develop human-surpassing AI without government intervention or international coordination.
Safety Critics
Competition and Coordination Argument
Whether development should slow
Safety Critics
Coxon argues that no company can responsibly develop AI beyond human capabilities without government intervention or a coordinated slowdown.
Competition and Coordination Argument
Anthropic has argued that pausing unilaterally could allow less cautious rivals to dominate, potentially making the industry less safe overall.
Current safety readiness
Safety Critics
Coxon says the companies are racing toward self-improving superintelligence without acting responsibly and that AI control remains unresolved.
Competition and Coordination Argument
Anthropic says it is trying to address the risks, while acknowledging it does not yet have a plan to guarantee that more capable systems will obey their creators.
International policy
Safety Critics
Coxon and other researchers call for international cooperation because an AI arms race could have unprecedented consequences.
Competition and Coordination Argument
OpenAI chief scientist Jakub Pachocki also called for extreme caution and international coordination, but the articles do not identify a competing policy proposal.
Key facts
- Main warning
- Coxon said AI could potentially kill all humans and may be more dangerous than nuclear weapons.
- Coxon’s background
- He said he spent three years conducting research at Anthropic and OpenAI.
- Risk estimate
- Evan Hubinger estimated the chance that AI could kill all humans at more than 10% over the next decade.
- Companies named
- Anthropic and OpenAI were described as competing to build increasingly advanced AI systems.
- Anthropic policy change
- Anthropic removed a pledge in February to halt model development if it could not control the risks.
- Industry response
- More than 1,000 technology employees, including Anthropic CEO Dario Amodei, called for a coordinated slowdown.
- United States regulation
- AI models are not regulated by federal law in the United States.
Quotes
Jacob Coxon
Former Anthropic researcher who says he also conducted research at OpenAI
“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives”
NDTV
“We know how to control nuclear weapons...We don't yet know how to control AI”
NDTV
Evan Hubinger
Anthropic safety executive
“We really do earnestly believe AI could kill all humans!”
NDTV
Sources
Related news
AI Experts Warn Advanced Systems Could Threaten Humanity
Former Anthropic Researcher Warns AI Could Kill Humanity This Decade
UN and OpenAI Officials Warn of AI Control Risks
Sanders Backs Permanent Ban on Superintelligent AI Development
OpenAI Safety Official Warns Superintelligent AI Could Kill Most People
UN Rights Chief Warns Advanced AI Could Threaten Humanity
Anthropic Walks Away From Reported $6 Billion Decart Deal
Corporate America Shifts Toward Cheaper Open-Source AI Models
Anthropic Rebuilds Ties With Trump Administration After Pentagon Feud
