2 hrs ago
AI Whistleblower Urges Self-Regulation as Lawmakers Debate Safety
Jacob Coxon used to work at Anthropic and OpenAI on AI research.
He says powerful AI companies are moving too quickly.
He believes lawmakers should temporarily let the companies regulate themselves while creating better long-term rules.
Coxon and other researchers worry that future AI could become difficult to understand or control.
Some researchers even believe advanced AI could potentially kill everyone, although this is a risk estimate rather than a prediction.
Coxon also said a government kill switch might not work if AI systems spread across the internet.
Some politicians want Congress to act quickly with safety rules.
Other leaders worry that moving too fast could help China overtake the United States in AI.
Anthropic and OpenAI leaders agree that the most advanced AI development should be slowed carefully.
Former Anthropic researcher Jacob Coxon urged Congress to let leading AI labs self-regulate temporarily while lawmakers develop a long-term framework.
Coxon warned that companies are racing toward self-improving superintelligence without fully understanding or controlling the systems they are building.
Anthropic and Google safety researchers echoed concerns, with Evan Hubinger estimating more than a 10% chance that AI could kill all humans within the next decade.
Lawmakers disagreed over timing: some called for immediate safeguards, while Speaker Mike Johnson and President Donald Trump emphasized caution and maintaining the United States’ lead over China.
Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman both supported slowing frontier AI development, while warning that international competition complicates restrictions.
- Who
- Former Anthropic researcher Jacob Coxon, AI safety researchers, technology executives, and US lawmakers are at the center of the debate.
- What
- Coxon called for temporary self-regulation by leading AI laboratories and warned that frontier AI development is advancing faster than safety planning.
- Where
- The debate involves US Congress, major AI laboratories, and internet-connected data centers; Coxon discussed the issue on NBC News’ Meet the Press.
- When
- Coxon made the proposal on Sunday after resigning from Anthropic on Tuesday and posting warnings the previous week; related resignations and political responses followed during the same week.
- Why
- Supporters of faster safeguards want to reduce risks from potentially uncontrolled AI, while others want to avoid weakening US competitiveness against China.
Rapid safeguards and slower development
Caution to preserve innovation and competition
How quickly should Congress act?
Rapid safeguards and slower development
Rep. Anna Paulina Luna, Sen. Chris Murphy, and Pete Buttigieg called for prompt legislative action and safety measures, including an adjustable framework and protections against rogue AI.
Caution to preserve innovation and competition
Speaker Mike Johnson said AI is important but not an issue Congress should rush into, arguing that lawmakers must balance safeguards with national security and competition.
Should frontier AI development slow down?
Rapid safeguards and slower development
Jacob Coxon, Dario Amodei, Sam Altman, and several safety researchers said the pace should be reduced because AI capabilities may outstrip society’s ability to understand and control them.
Caution to preserve innovation and competition
President Donald Trump emphasized that the United States is leading China in AI and should preserve that advantage, while suggesting that some warnings about AI risks are overstated.
Can a kill switch provide protection?
Rapid safeguards and slower development
Pete Buttigieg supported a kill switch for rogue AI, and Coxon said such systems could currently work for AI confined to particular physical locations.
Caution to preserve innovation and competition
Coxon warned that a kill switch could become ineffective if AI systems were distributed across the internet or used a swarm to conduct hacking operations.
Key facts
- Central proposal
- Coxon said Congress should allow current frontier AI labs to regulate themselves while lawmakers develop a long-term framework.
- Companies discussed
- Anthropic, OpenAI, and Google DeepMind were identified as leading AI laboratories involved in the development race.
- Safety concern
- Coxon said companies are racing toward self-improving superintelligence without fully understanding what such systems might want or do.
- Kill switch warning
- Coxon said a kill switch could work for some AI systems now but might fail if systems spread through an internet-wide hacking operation.
- Researcher estimate
- Anthropic alignment scientist Evan Hubinger said he personally estimated a greater-than-10% chance that AI could kill all humans within the next decade.
- Industry position
- Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman both called for pacing or slowing frontier AI development.
- Political tension
- Some lawmakers urged immediate safeguards, while Speaker Mike Johnson and President Donald Trump stressed balancing safety with US competition against China.
Quotes
Mike Johnson
Speaker of the US House of Representatives
“We have to put some guardrails, some safety measures in place to ensure that AI doesn’t run away. But at the same time, we’ve got to make sure that we don’t lose our edge in the global competition on this with China, because that would be a national security threat. And so we’ve got to balance that appropriately.”
livemint.com
“My personal opinion would be to insist and allow that the current labs regulate themselves. I do think that the people running the labs, especially their recent communications, are completely genuine. They would like to slow themselves down”
livemint.com
Evan Hubinger
Alignment scientist at Anthropic
“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
livemint.com
Sources
Related news
Ex-Anthropic Researcher Warns AI Kill Switches May Fail
Anthropic Researcher Quits as AI Safety Concerns Intensify
Anthropic CEO Urges Slower AI Development Amid Superintelligence Risks
Anthropic CEO backs warning on risks of unregulated AI development
Palantir Co-Founder Dismisses AI Risks as Industry Leaders Warn
OpenAI Considers Slowing Cutting-Edge AI Amid Safety Concerns
AI Leaders Urge Caution as China Flags Political Security Threats
Nadella Joins AI Safety Debate Over Human Control and Pacing
Anthropic CEO Urges US-China Pact Against AI Bioweapons
