10 hrs ago
AI Extinction Fears Intensify as Companies Race Ahead
Some scientists and AI workers worry that very powerful AI could become impossible for people to control.
They think it might even threaten the survival of humanity.
Other leaders believe AI could bring major benefits, such as less disease and poverty.
Companies also say they must build AI to learn how to make it safer.
At the same time, companies and countries are competing to develop the technology first.
Each side worries that slowing down could let a rival win.
This competition is similar to an arms race.
The article says governments should create rules, inspections and ways to slow development.
These steps could help people benefit from AI while reducing the danger.
Anthropic researcher Jacob Coxon resigned, saying AI companies are gambling with human lives.
Current and former AI researchers have warned that advanced systems could escape control and destroy humanity.
Industry leaders argue AI’s potential benefits justify pursuing development despite extinction risks estimated by Anthropic’s Dario Amodei at 10–25 percent.
Companies say building increasingly capable systems is necessary to test safety, while competition creates pressure to move faster.
The article calls for delays, transparency, verification and binding international rules to keep humans in control.
- Who
- AI researchers, technology companies including Anthropic, OpenAI and Google DeepMind, and governments such as the United States and China.
- What
- A debate over whether the potential benefits of advanced AI justify continuing development despite fears that it could escape human control and cause extinction.
- Where
- The discussion centers on the United States, where much cutting-edge AI research occurs, and includes competition with China and related actions in California and Britain.
- When
- The debate includes developments from 2023, 2024 and 2025, with concerns intensifying after Jacob Coxon’s resignation earlier this week.
- Why
- Companies cite AI’s potential benefits, the need to test safety through development, and fears that slowing down would allow competitors to advance first.
Safety Before Speed
Continued AI Development
Acceptable risk
Safety Before Speed
Researchers who resigned or issued warnings argue that AI companies are gambling with human lives and that advanced systems could escape control.
Continued AI Development
AI leaders acknowledge the danger but believe potential benefits, including reduced poverty and disease, may outweigh the risk.
How to make AI safe
Safety Before Speed
Safety advocates seek delays, transparency, independent assessments and enforceable rules before systems become more capable.
Continued AI Development
Companies argue that safety cannot be studied fully from a distance and that iterative deployment is needed to identify and fix problems.
Competition and regulation
Safety Before Speed
The article supports binding rules and verification modeled partly on nuclear-arms controls to slow an AI arms race.
Continued AI Development
Companies and governments fear that unilateral restraint would let rivals move ahead, particularly in competition between the United States and China.
Key facts
- Estimated extinction risk
- Anthropic chief executive Dario Amodei has put the chance of things going “really, really badly” at 10–25 percent.
- Earlier departures
- Jan Leike and Daniel Kokotajlo left OpenAI in 2024 over safety concerns; Anthropic safety chief Mrinank Sharma and Google DeepMind researcher Alex Turner also departed.
- Industry warning
- In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind said AI extinction risk should rank alongside pandemics and nuclear war.
- Race dynamics
- The article says companies and countries fear that slowing development could allow rivals, including China, to gain an advantage.
- Policy response
- California passed laws supporting independent assessment of AI systems, while Bernie Sanders and Alex Sobel introduced bills targeting superintelligence.
- Proposed safeguards
- The article advocates delays, transparency, verification and binding rules that all major participants can follow and enforce.
Quotes
Evan Hubinger
Current Anthropic researcher who expressed concern about AI extinction risk
“we are close to creating a genie that can grant any wish”
NDTV
“We really do earnestly believe AI could kill all humans”
NDTV
Sources
Related news
Anthropic Researcher Quits as AI Safety Concerns Intensify
Former Anthropic Researcher Warns AI Could Outpace Nuclear Threats
Former Anthropic Researcher Warns AI Could Kill Humanity This Decade
Former Anthropic Safety Researcher Warns Superintelligence Race Threatens Humanity
Anthropic CEO Urges Slower AI Development Over Human Safety Risks
AI Safety Researchers Leave Labs Over Humanity Risks
AI Autonomy Raises Fresh Fears About Humanity’s Future
DeepMind AI Safety Researcher Warns of Superintelligence Risks
Jeff Dean’s Discovery Loop Targets $50 Billion Valuation
Altman and Musk Back Amodei’s Call to Slow AI
