Archive · 14 Nov 2025
Websites Fight Back Against AI Data Scrapers Harming Revenue
Imagine the internet is like a big library.
AI companies are like students who want to take notes from all the books really fast to learn for a test.
They used to get permission, and when people took notes, the library got visitors and could earn money from them.
But now, these AI students are just quickly copying the notes without asking, and they don't need to visit the library anymore.
This means the library doesn't get visitors or money to buy new books.
Some libraries and companies are now putting up 'no copying' signs or charging the AI students a fee to take notes.
They want to make sure that people who create the books still get paid and have a reason to write new ones, otherwise, the library might run out of new books.
AI crawlers are extracting data from websites without permission or payment, negatively impacting website revenues.
The traditional online economy model where content creators received traffic and advertising revenue in exchange for search engine access is being broken by generative AI.
Cloudflare has introduced measures to block AI crawlers from accessing content on over 10 million websites without permission or payment.
Startup TollBit offers tools for publishers to monetize AI crawler traffic, charging AI companies a fee for content access.
Content creators fear that without proper compensation, the incentives for creating original online content may disappear.
- Who
- AI companies, traditional content creators (media outlets), internet service providers (Cloudflare), and data management firms (Dataiku, TollBit).
- What
- AI 'crawlers' are harvesting website data without permission or payment, impacting website revenues and disrupting the online economy.
- When
- The issue has escalated with the rise of generative AI and AI chatbots.
- Where
- Across the internet, affecting billions of websites globally.
- Why
- AI companies seek data to train their algorithms and chatbots; this method bypasses traditional traffic-based revenue models for content creators.
Content Creators & Website Owners
AI Companies
Content Access and Monetization
Content Creators & Website Owners
AI crawlers access content without permission or payment, diminishing website traffic and advertising revenue. This disrupts the traditional model where search engines provided visibility in exchange for traffic.
AI Companies
AI companies use web crawlers to harvest data for training algorithms and chatbots, which is essential for their development and services.
Economic Impact of AI Crawlers
Content Creators & Website Owners
Generative AI crawlers are cutting into online operations and advertising revenues of traditional content producers like media outlets, potentially reducing incentives for content creation.
AI Companies
The AI-driven business of the internet doesn't generate traffic for original content creators, as AI summaries reduce the need for users to visit original sites.
Solutions and Scope
Content Creators & Website Owners
Measures like Cloudflare's blocking of AI crawlers and TollBit's monetization tools are being implemented to regain control and revenue. However, some believe these are partial measures for a larger, long-term internet economy evolution.
AI Companies
Currently, AI companies are the primary users of harvested data, with no direct payment mechanism established for accessing this content in the way the internet economy traditionally worked.
- AI Crawler Impact
- Reduced website traffic and advertising revenue for content creators.
- Wikipedia Traffic Decrease
- Fell by 8% between 2024 and 2025 due to AI search engine summaries.
- Cloudflare's Solution
- New measure to block AI crawlers without payment or permission, applied to over 10 million websites.
- TollBit's Service
- Provides tools for publishers to block, monitor, and monetize AI crawler traffic, charging AI companies a transaction fee.
- TollBit Clients
- Includes USA Today, Time magazine, and the Associated Press.