Perplexity AI Accused of Using Stealth Crawlers to Evade Website Blocks
Cloudflare accused Perplexity AI of using stealth crawling techniques to evade website blocks and robots.txt directives by switching to undeclared browser signatures and rotating IP addresses. Cloudflare demonstrated this behavior through honeytrap domains and confirmed Perplexity accessed protected content despite explicit restrictions.
What happened
Cloudflare alleged in a blog post that Perplexity initially used a declared user agent for its bots, but switched to undeclared, generic browser signatures and rotated IP addresses when encountering network blocks or robots.txt disallow directives.
Cloudflare created unpublished test websites with robots.txt files explicitly blocking all crawlers, including Perplexity's declared bots. When Cloudflare queried Perplexity's AI with questions about these restricted sites, the service returned detailed information that could only have come from the protected pages.
Reported impact
Cloudflare customers complained that Perplexity was accessing content explicitly off-limits to bots despite visible blocks on their pages.
Index analysis
Perplexity responded that its AI agents crawl websites differently than traditional crawlers, accessing current information requested by users rather than building a pre-compiled database.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.