AI web scraping reduces referral traffic to content creators
AI companies deploy web crawlers that scrape content to train generative AI models while sending little referral traffic back to original content sources. This disrupts the traditional web economics that compensated content creators through visitor referrals, reportedly harming media publishers' advertising and subscription revenue.
What happened
AI companies including Anthropic, OpenAI, and Meta deploy web crawlers to scrape content from websites to train AI models. From July 2024 to July 2025, requests from GPTBot rose 147% and requests from Meta-ExternalAgent rose 843%.
Unlike search engine crawlers that historically sent referral traffic back to sites, AI model crawlers send little traffic in return. Cloudflare measures this through a crawl-to-refer ratio that shows how much tech companies take from the web versus what they give back.
Reported impact
Content creators, especially media publishers, receive reduced traffic to their websites because users increasingly trust AI-generated answers and summaries instead of visiting original source websites. This impacts their ability to generate revenue through advertising and subscriptions.
The traditional web bargain, where sites received compensation through referrals in exchange for allowing data scraping, has broken down in the AI era.
Index analysis
AI companies scrape websites without paying for the high-quality human data needed for model training, instead relying on automated collection from public web sources.
AI tools generate derivative content that appears above search results or as direct responses, reducing user motivation to visit original content sources.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.