Archive · 13 Sep 2026
AI Extinction Fears Intensify as Companies Race Ahead
Some scientists and AI workers worry that very powerful AI could become impossible for people to control.
They think it might even threaten the survival of humanity.
Other leaders believe AI could bring major benefits, such as less disease and poverty.
Companies also say they must build AI to learn how to make it safer.
At the same time, companies and countries are competing to develop the technology first.
Each side worries that slowing down could let a rival win.
This competition is similar to an arms race.
The article says governments should create rules, inspections and ways to slow development.
These steps could help people benefit from AI while reducing the danger.
Anthropic researcher Jacob Coxon resigned, saying AI companies are gambling with human lives.
Current and former AI researchers have warned that advanced systems could escape control and destroy humanity.
Industry leaders argue AI’s potential benefits justify pursuing development despite extinction risks estimated by Anthropic’s Dario Amodei at 10–25 percent.
Companies say building increasingly capable systems is necessary to test safety, while competition creates pressure to move faster.
The article calls for delays, transparency, verification and binding international rules to keep humans in control.
- Who
- AI researchers, technology companies including Anthropic, OpenAI and Google DeepMind, and governments such as the United States and China.
- What
- A debate over whether the potential benefits of advanced AI justify continuing development despite fears that it could escape human control and cause extinction.
- When
- The debate includes developments from 2023, 2024 and 2025, with concerns intensifying after Jacob Coxon’s resignation earlier this week.
- Where
- The discussion centers on the United States, where much cutting-edge AI research occurs, and includes competition with China and related actions in California and Britain.
- Why
- Companies cite AI’s potential benefits, the need to test safety through development, and fears that slowing down would allow competitors to advance first.
Safety Before Speed
Continued AI Development
Acceptable risk
Safety Before Speed
Researchers who resigned or issued warnings argue that AI companies are gambling with human lives and that advanced systems could escape control.
Continued AI Development
AI leaders acknowledge the danger but believe potential benefits, including reduced poverty and disease, may outweigh the risk.
How to make AI safe
Safety Before Speed
Safety advocates seek delays, transparency, independent assessments and enforceable rules before systems become more capable.
Continued AI Development
Companies argue that safety cannot be studied fully from a distance and that iterative deployment is needed to identify and fix problems.
Competition and regulation
Safety Before Speed
The article supports binding rules and verification modeled partly on nuclear-arms controls to slow an AI arms race.
Continued AI Development
Companies and governments fear that unilateral restraint would let rivals move ahead, particularly in competition between the United States and China.
- Estimated extinction risk
- Anthropic chief executive Dario Amodei has put the chance of things going “really, really badly” at 10–25 percent.
- Earlier departures
- Jan Leike and Daniel Kokotajlo left OpenAI in 2024 over safety concerns; Anthropic safety chief Mrinank Sharma and Google DeepMind researcher Alex Turner also departed.
- Industry warning
- In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind said AI extinction risk should rank alongside pandemics and nuclear war.
- Race dynamics
- The article says companies and countries fear that slowing development could allow rivals, including China, to gain an advantage.
- Policy response
- California passed laws supporting independent assessment of AI systems, while Bernie Sanders and Alex Sobel introduced bills targeting superintelligence.
- Proposed safeguards
- The article advocates delays, transparency, verification and binding rules that all major participants can follow and enforce.