1 day ago
Former Anthropic Safety Researcher Warns Superintelligence Race Threatens Humanity
Joe Benton used to work on AI safety at Anthropic.
He says AI companies are racing to build computers that could become smarter than people.
Benton worries these systems might become difficult to control.
He says companies are not doing enough safety work or sharing enough information.
He wants companies to report accidents, near-misses and progress toward machines that can improve themselves.
He also wants independent groups to check whether companies are following basic safety rules.
Benton plans to work with METR, an organization that evaluates advanced AI systems.
His warning follows similar concerns from Jacob Coxon and comments by India’s Nirmala Sitharaman.
Joe Benton says he left Anthropic’s safety team two weeks ago amid concerns about the race toward superintelligent AI.
He argues AI companies are underinvesting in safety while developing systems potentially more intelligent than humans.
Benton warns that an intelligence explosion or loss of control could occur without the public being informed.
He wants companies to disclose capability gains, recursive self-improvement progress, safety incidents and near-misses.
Benton plans to join METR, also described as METR Evals, while Jacob Coxon and Nirmala Sitharaman raised related concerns.
- Who
- Joe Benton, Anthropic, METR, former researcher Jacob Coxon and India’s Finance Minister Nirmala Sitharaman.
- What
- Benton warned that the race to develop superintelligent and potentially self-improving AI could threaten humanity without stronger safeguards, transparency and independent evaluation.
- Where
- Benton made his comments on X and in a Substack post; Sitharaman discussed the issue at the Global Fintech Fest 2026 in India.
- When
- Benton said he left Anthropic two weeks before his September 12, 2026 statement; Coxon and Sitharaman raised related concerns that same week.
- Why
- Benton said companies are underinvesting in safety and that the public needs more information about capability advances, incidents, near-misses and recursive self-improvement.
AI Safety Advocates
Technology-Assisted Safeguards
How advanced AI risks should be managed
AI Safety Advocates
Joe Benton and Jacob Coxon argue that companies are pursuing self-improving superintelligence without sufficient safety investment, transparency or independent scrutiny.
Technology-Assisted Safeguards
Nirmala Sitharaman said technology itself could help address vulnerabilities created by technological advances, while emphasizing that safeguards must continually evolve.
Public disclosure and oversight
AI Safety Advocates
Benton wants mandatory reporting of capability gains, incidents and near-misses, along with minimum standards and independent evaluations.
Technology-Assisted Safeguards
The articles do not describe a specific opposing disclosure policy, but Sitharaman’s comments emphasize continuously updating safeguards as technology changes.
Key facts
- Former employer
- Anthropic
- New organization
- Benton plans to join METR, also identified in one article as METR Evals.
- Central warning
- Humanity may not survive the transition to superintelligent AI.
- Requested disclosures
- Progress toward recursive self-improvement, safety incidents, near-misses and AI capability gains.
- Requested oversight
- Minimum safety standards and independent assessments of companies’ safety frameworks.
- Related resignation
- Jacob Coxon left Anthropic after saying leading AI companies were not acting responsibly in pursuing self-improving superintelligence.
- Political response
- Nirmala Sitharaman questioned whether AI safeguards are evolving quickly enough and said they require continuous updating.
Quotes
Joe Benton
Former member of Anthropic’s safety team and incoming METR researcher
“Within the next couple of years, we may be sharing the world with AI agents smarter than any human alive today. These AI systems may have drives and desires that diverge from those of any human overseer, with capabilities we can’t effectively constrain. Humanity may not survive this transition. We need a lot more preparation to make this world safe.”
thestatesman.com
“I left Anthropic’s safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this.”
thestatesman.com
Sources
Related news
Anthropic Researcher Quits as AI Safety Concerns Intensify
Anthropic Researchers Warn of AI Risks, MIT Professor Disagrees
Anthropic Researcher Leaves Safety Team, Seeks Greater AI Transparency
Anthropic Researcher Leaves Unvested Equity Over AI Safety Fears
DeepMind AI Safety Researcher Warns of Superintelligence Risks
AI Extinction Fears Intensify as Companies Race Ahead
Anthropic CEO Urges Slower AI Development Over Human Safety Risks
Altman and Musk Back Amodei’s Call to Slow AI
Anthropic CEO Warns AI Swarm Could Seize Internet Within Months
