Science & Tech · AI · 2 hrs ago
Four AI companies back a shared website scraping opt-out
OpenAI, Google, Microsoft and Anthropic have pledged to recognize a shared instruction that lets websites block their content from AI training.
The proposed signal will go in a site’s existing robots.txt file, which tells web crawlers what they may access.
One standard instruction could replace the separate rules publishers now need for different companies’ bots.
Anthropic has started supporting it; OpenAI says it will do so in the coming months, while Microsoft plans to start on February 1, 2027, and Google sometime in 2027.
The setting applies only to future AI training and does not undo earlier scraping or stop other crawling, including search indexing.
Scraping remains allowed by default until a publisher changes the setting, and opting out could reduce the chance that its content appears in AI apps.
OpenAI, Google, Microsoft and Anthropic have pledged to support a shared website signal for opting out of AI training data scraping.
The proposed signal would let publishers state their preferences once in a website’s robots.txt file instead of setting separate rules for each company.
The signal applies to future AI model training and does not prevent other web crawling or remove content already collected.
Content scraping will remain allowed by default until a publisher changes its settings.
Anthropic has started supporting the signal; the other companies plan to implement it in the coming months or in 2027.
- Who
- OpenAI, Google, Microsoft and Anthropic have pledged to support the signal.
- What
- A shared robots.txt signal will let publishers opt out of having content scraped for AI training.
- When
- The companies announced their commitments on October 30, 2026.
- Where
- The signal will be placed in a website’s robots.txt file.
- Why
- It is intended to give publishers one standard way to express AI training preferences to multiple companies.
The New York Times
OpenAI and Microsoft
Copyright dispute
The New York Times
The Times sued the companies, alleging they used millions of its articles without authorisation or payment to train ChatGPT.
OpenAI and Microsoft
The companies invoke fair use, an exception to copyright law.
The New York Times sued OpenAI and Microsoft over the alleged use of its articles to train ChatGPT.
Cloudflare began blocking AI bots by default on new customer sites.
OpenAI, Google, Microsoft and Anthropic pledged to support the shared signal.
Microsoft plans to begin implementing the signal.
Google plans to support the signal; OpenAI said it would implement it in the coming months.
- Companies
- OpenAI, Google, Microsoft and Anthropic
- Signal location
- Website robots.txt file
- Scope
- Future AI model training
- Microsoft start date
- February 1, 2027
- Google timing
- Sometime in 2027
- Anthropic status
- Has already started supporting the signal
4 sources
What people are saying
We believe creating a common signal that conveys such preferences to multiple providers at once is an important step.
And we look forward to working with other AI companies on the standards and their implementation across the industry.