Archive · 2 Jul 2025
Cloudflare and Publishers Fight AI Scraping, Seek Revenue
AI is using technology to copy information from news websites, which hurts the websites because people get answers directly from AI.
Cloudflare is helping websites by making it harder for AI to take content without permission and allowing them to charge for its use.
Publishers are worried about this, as AI could take their revenue and the rights to the content that they created.
India is also looking at solutions since content creators' content is being exploited without consent or compensation.
Global momentum is building against unauthorized AI scraping of journalistic content.
Cloudflare will block AI scrapers by default on new domains, allowing publishers to control access.
Publishers, including Condé Nast and TIME, are blocking AI web crawlers.
Indian publishers are calling on the government to address unauthorized content scraping.
Cloudflare offers a "pay-per-crawl" model, enabling publishers to charge AI firms for access.
- Who
- Cloudflare, publishers (e.g., Condé Nast), and AI companies are the main players.
- What
- Cloudflare is implementing measures to restrict and monetize AI access to website content, while publishers seek legal and regulatory protection.
- When
- Cloudflare is now blocking AI scrapers by default.
- Where
- The issue impacts publishers globally, including the US, UK, and India.
- Why
- To protect content creators, ensure fair compensation, and address declining referral traffic due to AI-generated answers.
Publishers
AI Companies
Content Access and Use
Publishers
Need recognition and compensation for their content.
AI Companies
AI should be allowed to take content freely for training AI models
Control and Regulation
Publishers
Require consent for content access, licensing frameworks, and technological tools
AI Companies
Continue current practices without publisher oversight and control.
- Company
- Cloudflare
- Initiative
- Blocking AI crawlers by default
- Product
- Pay-per-crawl
- Key Clients
- Condé Nast, TIME
- Affected Parties
- Publishers and AI companies
- Organizations Involved
- DNPA
Sources
Checkmate for GenAI crawlers: Cloudflare allows publishers to keep tabs on unauthorised AI crawling
‘Urge govt to act against AI’s unauthorised and rampant data scrapping’: News publishers body
Cloudflare will now block AI bots by default on new sites
