Archive · 12 Sep 2026
Former Anthropic Safety Researcher Warns Superintelligence Race Threatens Humanity
Joe Benton used to work on AI safety at Anthropic.
He says AI companies are racing to build computers that could become smarter than people.
Benton worries these systems might become difficult to control.
He says companies are not doing enough safety work or sharing enough information.
He wants companies to report accidents, near-misses and progress toward machines that can improve themselves.
He also wants independent groups to check whether companies are following basic safety rules.
Benton plans to work with METR, an organization that evaluates advanced AI systems.
His warning follows similar concerns from Jacob Coxon and comments by India’s Nirmala Sitharaman.
Joe Benton says he left Anthropic’s safety team two weeks ago amid concerns about the race toward superintelligent AI.
He argues AI companies are underinvesting in safety while developing systems potentially more intelligent than humans.
Benton warns that an intelligence explosion or loss of control could occur without the public being informed.
He wants companies to disclose capability gains, recursive self-improvement progress, safety incidents and near-misses.
Benton plans to join METR, also described as METR Evals, while Jacob Coxon and Nirmala Sitharaman raised related concerns.
- Who
- Joe Benton, Anthropic, METR, former researcher Jacob Coxon and India’s Finance Minister Nirmala Sitharaman.
- What
- Benton warned that the race to develop superintelligent and potentially self-improving AI could threaten humanity without stronger safeguards, transparency and independent evaluation.
- When
- Benton said he left Anthropic two weeks before his September 12, 2026 statement; Coxon and Sitharaman raised related concerns that same week.
- Where
- Benton made his comments on X and in a Substack post; Sitharaman discussed the issue at the Global Fintech Fest 2026 in India.
- Why
- Benton said companies are underinvesting in safety and that the public needs more information about capability advances, incidents, near-misses and recursive self-improvement.
AI Safety Advocates
Technology-Assisted Safeguards
How advanced AI risks should be managed
AI Safety Advocates
Joe Benton and Jacob Coxon argue that companies are pursuing self-improving superintelligence without sufficient safety investment, transparency or independent scrutiny.
Technology-Assisted Safeguards
Nirmala Sitharaman said technology itself could help address vulnerabilities created by technological advances, while emphasizing that safeguards must continually evolve.
Public disclosure and oversight
AI Safety Advocates
Benton wants mandatory reporting of capability gains, incidents and near-misses, along with minimum standards and independent evaluations.
Technology-Assisted Safeguards
The articles do not describe a specific opposing disclosure policy, but Sitharaman’s comments emphasize continuously updating safeguards as technology changes.
- Former employer
- Anthropic
- New organization
- Benton plans to join METR, also identified in one article as METR Evals.
- Central warning
- Humanity may not survive the transition to superintelligent AI.
- Requested disclosures
- Progress toward recursive self-improvement, safety incidents, near-misses and AI capability gains.
- Requested oversight
- Minimum safety standards and independent assessments of companies’ safety frameworks.
- Related resignation
- Jacob Coxon left Anthropic after saying leading AI companies were not acting responsibly in pursuing self-improving superintelligence.
- Political response
- Nirmala Sitharaman questioned whether AI safeguards are evolving quickly enough and said they require continuous updating.
