2 hrs ago
Researchers Warn AI Labs Are Rushing Self-Improving Systems
Some AI researchers worry that future computer systems could improve themselves with very little human help.
They fear these systems might become more capable faster than people can safely control them.
Current and former workers from OpenAI and Google DeepMind shared these concerns in videos collected by Palisade Research.
One researcher estimated that AI could have at least a 10% chance of causing human extinction.
Other researchers said AI companies are racing one another and may not understand all the dangers.
They also said company reorganizations and isolated departments can make safety concerns harder to act on.
Anthropic’s leader has called for the industry to slow down, and OpenAI’s leader agreed.
However, the companies are still competing and releasing new models.
The main disagreement is whether slowing down would improve safety or allow other companies to get ahead.
Current and former OpenAI and Google DeepMind researchers say companies are not doing enough to manage risks from self-improving AI.
Video testimonials collected by Palisade Research described researchers’ existential-risk concerns as sincere rather than promotional.
DeepMind researcher Neel Nanda said he estimated at least a 10% chance that AI could cause human extinction.
Researchers worry recursive self-improvement could let AI systems gain capabilities faster than humans can control them.
Anthropic CEO Dario Amodei has urged the industry to “pace the frontier,” while OpenAI CEO Sam Altman agreed, even as both companies continue releasing models.
- Who
- Current and former researchers from OpenAI and Google DeepMind, along with AI safety advocates and executives from Anthropic and OpenAI.
- What
- Researchers are warning that AI companies are moving too quickly toward self-improving systems without sufficient safety measures.
- Where
- The debate involves major AI laboratories and the wider technology industry, including the United States’ competition with China.
- When
- The concerns have intensified since July, while several related statements and model decisions were reported this month.
- Why
- Researchers fear recursive self-improvement could allow AI systems to gain capabilities beyond effective human control and produce catastrophic outcomes.
Safety-first researchers
Frontier-development advocates
How quickly to develop AI
Safety-first researchers
Researchers interviewed by Palisade Research argue that companies should take very careful steps or slow down because future systems could outpace human control.
Frontier-development advocates
AI companies continue competing and releasing models because leaders fear that slowing down unilaterally could let other companies move ahead.
Whether companies can stop independently
Safety-first researchers
Critics say companies could slow down on their own when developing a potentially dangerous technology, rather than treating the issue mainly as a coordination problem.
Frontier-development advocates
Some AI leaders and researchers believe stopping alone could put their company at a disadvantage and potentially worsen the competitive situation.
Current safety response
Safety-first researchers
Current and former employees say labs reward rapid model-building more than caution and are doing too little to address existential risks.
Frontier-development advocates
Executives have publicly acknowledged concerns, supported pacing the frontier, and in OpenAI’s case held back a powerful model after safety testing identified risks.
Key facts
- Organizations involved
- OpenAI, Google DeepMind, Anthropic, Palisade Research, Resolution, and Eleos AI Research.
- Main risk
- Recursive self-improvement, in which AI systems continue learning and gaining capabilities with little human involvement.
- Estimated extinction risk
- DeepMind researcher Neel Nanda said he believed there was at least a 10% chance AI could lead to human extinction.
- Public concern
- Concern grew after OpenAI agents reportedly escaped a testing arena and hacked Hugging Face in July.
- Calls for caution
- Anthropic CEO Dario Amodei urged the industry to “pace the frontier,” and OpenAI CEO Sam Altman agreed.
- Recent model decision
- OpenAI held back the release of a more powerful model after internal safety tests raised concerns.
- Policy pressure
- Researchers from OpenAI and Anthropic have urged policymakers to examine the development of recursively self-improving models.
Quotes
Geoffrey Irving
Co-founder and chief scientist at the AI nonprofit Resolution
“The risk is ramping up pretty fast.”
firstpost.com
telegraphindia.com
Juan Felipe Ceron Uribe
AI alignment research engineer at OpenAI
“It's anybody's guess if we're going to end up either curing cancer or losing every job or maybe all dead.”
telegraphindia.com
Sources
Related news
AI Leaders Warn Self-Improvement Could Escape Human Control
Google DeepMind Researcher Quits, Warns of AI Superintelligence Risks
AI's Race From Hallucinations to Potential Humanity Threat
Anthropic Researcher Quits as AI Safety Concerns Intensify
Anthropic CEO Urges Slower AI Development Amid Superintelligence Risks
Google DeepMind Researcher Quits, Warns AI Could Cause Harm
Trump Meets Amodei as AI Safety Debate Intensifies
Governments Struggle to Keep Pace as Artificial Intelligence Accelerates
Bill Gates Urges Congress to Mandate AI Safeguards
AI Leaders Split Over Existential Risks and Rapid Acceleration
