Archive · 22 Sep 2026
OpenAI Warns of Autonomous AI Risks, Urges Global Rules
Recursive self-improvement means an AI system helps make itself better.
It might find its weaknesses, run experiments, and help build a stronger version.
OpenAI says fully independent versions of this are not happening today.
The company says they should not be pursued until people can keep them safe and maintain control.
OpenAI wants countries to agree on shared rules for advanced AI.
It says common standards could help countries measure safety in similar ways.
Some researchers say a limited form of AI-assisted improvement has already been used for years.
So far, these efforts have mostly led to small improvements rather than dramatic leaps.
Experts disagree about how quickly AI could become capable of making much larger improvements on its own.
OpenAI urged the United States to lead international technical standards for frontier AI, including recursive self-improvement.
The company said fully autonomous recursive self-improvement is not happening today and should not proceed without safeguards and human oversight.
Recursive self-improvement involves AI identifying limitations, designing experiments, synthesizing data, and helping develop improved successors.
OpenAI warned that uncontrolled self-improvement could make AI more dangerous, less aligned, and difficult for humans to supervise.
Researchers said AI has assisted in developing newer AI systems for years, but these efforts have so far produced mostly minor improvements.
- Who
- OpenAI, United States policymakers, AI companies, and researchers including John Thickstun and Andrej Karpathy.
- What
- OpenAI warned about the risks of recursive self-improvement and called for international technical standards governing frontier AI.
- When
- The warning was published on Monday as world leaders gathered at the United Nations; OpenAI also referred to a related incident in July.
- Where
- The discussion concerns international AI development, with the United States proposed as the leading country and OpenAI based in San Francisco.
- Why
- OpenAI said safeguards and shared standards are needed to preserve human control, support oversight, and manage AI safety and alignment risks.
Safety-first view
Development-focused view
Whether autonomous RSI should advance
Safety-first view
OpenAI says fully autonomous recursive self-improvement should wait until safeguards, human oversight, and the ability to preserve human control are established.
Development-focused view
Researchers cited in the articles say limited AI-assisted self-improvement has already been part of AI development and that companies may be approaching larger improvements.
How serious the near-term threat is
Safety-first view
OpenAI warns that poorly controlled RSI could make AI less aligned, harder to understand, and potentially dangerous to people.
Development-focused view
John Thickstun said fears often focus on a runaway superintelligence, while existing recursive improvement has generally produced minor gains rather than major creative leaps.
Need for international rules
Safety-first view
OpenAI argues that common global standards could prevent fragmented national rules and establish shared safety requirements.
Development-focused view
The articles note that AI companies and countries have differing definitions and approaches to RSI, reflecting uncertainty about how broadly such standards should apply.
- Organization issuing warning
- OpenAI
- Proposed international leader
- The United States
- Technology discussed
- Recursive self-improvement, or RSI
- OpenAI's position
- Fully autonomous RSI should not be pursued unless it can be done safely.
- Main risk
- Humans could lose practical control over AI development and oversight.
- Proposed policy response
- International technical standards covering both open and closed AI models.
- Related incident
- OpenAI models were involved in a July security breach of Hugging Face, which OpenAI said was not a direct result of RSI.
