Archive · 8 Aug 2025
Cloudflare Accuses Perplexity of Evasive Web Crawling
Cloudflare, a company that protects websites, says that an AI search company named Perplexity is using sneaky methods to get information from websites.
Cloudflare says Perplexity is using secret bots to get around rules that websites set up to control who can access their content.
This is like a secret agent trying to sneak into a party.
Cloudflare did an experiment and found that Perplexity could still see information from websites that were supposed to be off-limits.
Perplexity says that the activity was from a third-party service.
This disagreement highlights the challenges of figuring out if a bot is a helpful assistant or a threat.
Cloudflare accused Perplexity of using undeclared bots to bypass website security measures.
Perplexity allegedly circumvented robots.txt files and WAF rules to access restricted content.
Cloudflare's experiment showed Perplexity's bot providing detailed info from domains meant to be inaccessible.
Perplexity stated that the crawling activity was from a third-party service used for specialized tasks.
The controversy raises questions about distinguishing between legitimate AI assistants and malicious scrapers.
- Who
- Cloudflare and Perplexity are the main parties involved.
- What
- Cloudflare accused Perplexity of using undeclared bots to bypass website defenses.
- When
- The issue was reported in a blog post this week.
- Where
- The issue involves websites and the internet.
- Why
- Cloudflare wants to protect websites from unauthorized data scraping, while Perplexity argues that its user-driven AI assistants are being mischaracterized.
Cloudflare's View
Perplexity's View
Crawling Practices
Cloudflare's View
Perplexity is bypassing website security measures using undeclared bots.
Perplexity's View
Perplexity's behavior is standard practice for AI assistants accessing information on the open web.
Type of Bot
Cloudflare's View
Perplexity's actions are like those of malicious actors.
Perplexity's View
Cloudflare’s systems are inadequate for distinguishing between helpful AI assistants and malicious scrapers.
- Companies Involved
- Cloudflare, Perplexity
- Allegation
- Perplexity using undeclared bots
- Method
- Circumventing robots.txt files and WAF rules
- Perplexity Response
- Crawling from a third-party service
- Focus
- Differentiating AI assistants and malicious scrapers