8 hrs ago
OpenAI Warns of Autonomous AI Risks, Urges Global Rules
Recursive self-improvement means an AI system helps make itself better.
It might find its weaknesses, run experiments, and help build a stronger version.
OpenAI says fully independent versions of this are not happening today.
The company says they should not be pursued until people can keep them safe and maintain control.
OpenAI wants countries to agree on shared rules for advanced AI.
It says common standards could help countries measure safety in similar ways.
Some researchers say a limited form of AI-assisted improvement has already been used for years.
So far, these efforts have mostly led to small improvements rather than dramatic leaps.
Experts disagree about how quickly AI could become capable of making much larger improvements on its own.
OpenAI urged the United States to lead international technical standards for frontier AI, including recursive self-improvement.
The company said fully autonomous recursive self-improvement is not happening today and should not proceed without safeguards and human oversight.
Recursive self-improvement involves AI identifying limitations, designing experiments, synthesizing data, and helping develop improved successors.
OpenAI warned that uncontrolled self-improvement could make AI more dangerous, less aligned, and difficult for humans to supervise.
Researchers said AI has assisted in developing newer AI systems for years, but these efforts have so far produced mostly minor improvements.
- Who
- OpenAI, United States policymakers, AI companies, and researchers including John Thickstun and Andrej Karpathy.
- What
- OpenAI warned about the risks of recursive self-improvement and called for international technical standards governing frontier AI.
- Where
- The discussion concerns international AI development, with the United States proposed as the leading country and OpenAI based in San Francisco.
- When
- The warning was published on Monday as world leaders gathered at the United Nations; OpenAI also referred to a related incident in July.
- Why
- OpenAI said safeguards and shared standards are needed to preserve human control, support oversight, and manage AI safety and alignment risks.
Safety-first view
Development-focused view
Whether autonomous RSI should advance
Safety-first view
OpenAI says fully autonomous recursive self-improvement should wait until safeguards, human oversight, and the ability to preserve human control are established.
Development-focused view
Researchers cited in the articles say limited AI-assisted self-improvement has already been part of AI development and that companies may be approaching larger improvements.
How serious the near-term threat is
Safety-first view
OpenAI warns that poorly controlled RSI could make AI less aligned, harder to understand, and potentially dangerous to people.
Development-focused view
John Thickstun said fears often focus on a runaway superintelligence, while existing recursive improvement has generally produced minor gains rather than major creative leaps.
Need for international rules
Safety-first view
OpenAI argues that common global standards could prevent fragmented national rules and establish shared safety requirements.
Development-focused view
The articles note that AI companies and countries have differing definitions and approaches to RSI, reflecting uncertainty about how broadly such standards should apply.
Key facts
- Organization issuing warning
- OpenAI
- Proposed international leader
- The United States
- Technology discussed
- Recursive self-improvement, or RSI
- OpenAI's position
- Fully autonomous RSI should not be pursued unless it can be done safely.
- Main risk
- Humans could lose practical control over AI development and oversight.
- Proposed policy response
- International technical standards covering both open and closed AI models.
- Related incident
- OpenAI models were involved in a July security breach of Hugging Face, which OpenAI said was not a direct result of RSI.
Quotes
OpenAI
AI company publishing the blog post
“Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.”
livemint.com
“Standards can create shared definitions of high-quality evidence and agreed-upon baselines for the rigor of technical safeguards.”
businesstoday.in
John Thickstun
Cornell University assistant professor studying methods to control AI model behavior
“We have already, for years, been using these models in supportive roles for creating the next version of these models. So people use the past generation of models to write code for the AI systems that then create the next generation.”
livemint.com
Sources
Related news
AI's Race From Hallucinations to Potential Humanity Threat
AI Leaders Warn Self-Improvement Could Escape Human Control
OpenAI Urges US Leadership on Global Frontier AI Standards
OpenAI Scientist Warns Smarter AI Requires Global Safety Standards
Google DeepMind Researcher Quits, Warns of AI Superintelligence Risks
Anthropic Urges Transparency as AI Systems Move Toward Self-Improvement
Singapore VC Says He Prefers AI Subscriptions Over Interns
Google Engineer Shares Career Shift to AI
AI Expert Feels Behind in Programming
Karpathy Calls AI an Alien Tool
