Science & Tech · Big Tech & Internet · 4 hrs ago
Tech firms back a shared standard for AI scraping of websites
OpenAI, Google, Microsoft and Anthropic have backed a shared standard for websites to control how AI systems collect their content.
The proposed standard would let site owners use one instruction, rather than separate commands for different companies, to set preferences on scraping for AI training.
It is based on robots.txt and is being considered by the Internet Engineering Task Force, which develops open internet standards.
The companies will adopt it at different times: Anthropic says it has started, OpenAI plans to introduce it in the coming months, Google during the next year, and Microsoft from February 1, 2027.
The plan comes amid legal fights over publishers' content, including a 2023 lawsuit by The New York Times accusing OpenAI and Microsoft of using millions of its articles without permission or payment; the companies cite fair use.
The EU has upheld the legitimacy of Italy's rules for fair compensation, while an EU legal adviser has said chatbot reuse of news could in some circumstances violate publishers' rights.
The standard could give publishers more control, but opting out may reduce their content's visibility in AI-based apps and services, Google and Microsoft have warned.
OpenAI, Google, Microsoft and Anthropic have committed to supporting a shared standard for websites to set preferences on AI scraping.
The proposed standard would let publishers use one instruction instead of separate commands for different AI companies.
The standard is based on a proposal under review by the Internet Engineering Task Force.
The companies will adopt it on different schedules, and it will not become operational immediately.
The initiative comes amid legal disputes over publishers’ rights and the use of their content to train AI systems.
- Who
- OpenAI, Google, Microsoft and Anthropic have committed to the standard. It is intended for news organisations and website operators.
- What
- The companies will support a shared robots.txt standard for setting preferences on AI scraping.
- When
- The commitment was reported on November 8, 2026. The companies plan to implement the standard at different times.
- Where
- On websites, through the robots.txt standard. The proposal is being considered by the Internet Engineering Task Force.
- Why
- To give publishers and website operators a common way to regulate automated collection of their content by AI systems.
Publishers and website owners
AI companies
Aspect
Publishers and website owners
Publishers want control over whether their content is used and have challenged its use without permission or compensation.
AI companies
AI companies say publishers opting out of AI training could reduce the chances their content appears in AI applications and experiences.
Aspect
Publishers and website owners
Some publishers and media groups are seeking to prevent AI companies from using journalism without authorisation.
AI companies
OpenAI and Microsoft are relying on a copyright exception known as fair use in response to a lawsuit.
The New York Times sued OpenAI and Microsoft, alleging they used millions of its articles without permission or compensation to train ChatGPT.
Cloudflare began blocking AI-based bots by default on sites belonging to new customers.
The European Union court ruled that Italy’s fair-compensation decision was legitimate.
The Lazio regional administrative court rejected Meta’s appeal against the decision.
ANSA reported that OpenAI, Google, Microsoft and Anthropic had committed to supporting the shared standard.
- Companies
- OpenAI, Google, Microsoft and Anthropic
- Standard
- robots.txt
- Standards body
- Internet Engineering Task Force
- Microsoft implementation
- February 1, 2027
- Google implementation
- During 2027
- Cloudflare policy
- AI bots blocked by default for new customers from July 2025
What people are saying
We are ready to work with other companies in the AI sector to define standards and their large-scale implementation.
The Court should find that the real-time retrieval and reuse of online news publications by chatbots to generate answers to users’ questions may, in certain circumstances, constitute a violation of the rights of news publishers.