2 days ago
AI Crawlers Threaten Web Quality as Traffic and Trust Collapse
For many years, search engines visited websites and sent people back to read the original information.
AI tools now often read websites and give people answers without making them visit those sites.
This can reduce website traffic and advertising income.
Some website owners are therefore blocking AI crawlers.
However, websites with incorrect information may be less likely to block them.
This could cause AI answers to rely more on poor-quality or AI-written material.
Training AI on too much AI-written material may also make its answers worse, a problem called model collapse.
The article suggests clicking original search results and trying search engines that use less AI.
AI tools increasingly crawl websites to train models and generate answers without consistently sending visitors to source sites.
Website owners lose traffic and revenue while AI crawling creates additional costs and can be more intensive than traditional crawling.
Cloudflare estimates that AI bots now account for more than half of all web traffic.
Many high-quality sites are blocking AI crawlers, potentially leaving AI systems more dependent on low-quality or AI-generated content.
Researchers and commentators warn that AI summaries may reduce information quality, while direct links and non-AI search engines could offer more reliable results.
- Who
- Website owners, search engines, AI companies and people seeking information online.
- What
- AI crawling is changing how online information is collected and delivered, potentially weakening website traffic and answer quality.
- Where
- Across the World Wide Web, including sites managed through Cloudflare and Google AI Overviews.
- When
- The change is occurring recently; Cloudflare sites are scheduled to block AI crawlers by default on advertising-supported pages on September 15, with the year not specified.
- Why
- AI tools can use website content without sending comparable traffic or revenue to publishers, prompting some sites to block crawlers.
Key facts
- Estimated AI-bot traffic
- Cloudflare estimates that more than half of all web traffic is now generated by AI bots.
- Cloudflare site coverage
- Cloudflare manages 30 percent or more of the top 10,000 websites.
- Planned crawler blocking
- From September 15, Cloudflare sites will block AI crawlers by default on pages containing advertising.
- Potential reach
- The article says up to 30 percent of the world's top sites may no longer appear in Google AI Overviews summaries.
- AI-generated sources
- One cited study found that about one in six sources used by AI search tools is itself an AI-generated website.
- Proposed compensation
- Pay-to-crawl models have been suggested to compensate content creators but have not gained traction.
- Recommended alternatives
- The article cites Mojeek, Brave and options listed by Ban the Bots as search alternatives that depend less on AI crawling.
Sources
Related news
AI Summaries Threaten Website Traffic and the Web’s Social Contract
Google's AI Search Cuts Publisher Traffic, Draws Global Regulatory Backlash
Readers Debate AI Content Rights and National Identity
Regulators Scrutinise Google's AI Search Treatment of Publishers
French Media Groups Accuse Google AI Summaries of Cutting Traffic
Tech Giants Move to Clean Up AI Slop Across Platforms
Pug Jinny Lu Wins World's Ugliest Dog 2026 Contest
Wikipedia Founder Jimmy Wales Says AI Can't Be Trusted
Google Search Mistakenly Shows OpenAI CEO Sam Altman Dead
