Archive · 14 Sep 2026
AI Crawling Threatens Web Traffic and Information Quality
For many years, search engines sent people to websites when they used information from them.
AI tools often read websites and give people a summary without sending them to the original page.
This can cause websites to lose visitors and advertising money.
Some website owners are responding by blocking AI programs.
If useful websites block those programs, AI tools may rely more on lower-quality or AI-written websites.
That could make AI answers less accurate.
The article says Wikipedia traffic fell after Google added AI summaries.
It suggests that people click on original search results to support good websites and check information.
It also mentions search engines such as Mojeek and Brave that rely less on AI-based crawling.
AI tools increasingly crawl websites to train models and generate answers rather than consistently sending traffic through source links.
Declining traffic and revenue are prompting many website owners to block AI crawlers, potentially reducing access to high-quality information.
Cloudflare estimates that AI bots now account for more than half of web traffic, while AI crawlers can impose costs through intensive scraping.
Wikipedia traffic declined after Google introduced AI summaries in English and other languages, according to a study cited by the authors.
Cloudflare plans to block AI crawlers by default on advertising-supported pages from September 15, while experts recommend clicking through to original sources.
- Who
- Website owners, AI companies, search engines, and people seeking information online are affected; the article was written by Dana McKay and Damiano Spina of RMIT University.
- What
- AI crawling is weakening the traditional arrangement in which search engines sent website traffic in exchange for access to content, creating concerns about revenue and information quality.
- When
- The shift is described as ongoing; Cloudflare sites are scheduled to block AI crawlers by default on advertising-supported pages from September 15.
- Where
- Across the World Wide Web, including sites managed through Cloudflare and Google search results.
- Why
- AI answers often provide information without requiring users to visit source websites, reducing traffic and revenue and encouraging sites to restrict AI scraping.
- Traditional web arrangement
- Websites allowed search engines to crawl content, while search engines provided links and traffic.
- AI traffic estimate
- Cloudflare estimates that more than half of web traffic is now generated by AI bots.
- Website coverage
- Cloudflare manages 30% or more of the top 10,000 websites.
- Traffic trend
- A cited study found that English-language Wikipedia traffic dropped quickly after Google introduced AI summaries, with similar patterns in other languages.
- Cloudflare policy
- From September 15, Cloudflare sites will block AI crawlers by default on pages containing advertising.
- AI-generated sources
- One study cited in the article found that about one in six sources used by AI search tools is itself an AI-generated website.
- Suggested alternatives
- The article cites Mojeek, Brave, and services listed by Ban the Bots as search options that rely less on AI-based crawling.