Skip to content
Submit incident
AII-1002-16667
ConfirmedA court, a regulator or the organization itself has stated the core facts, or the record cites a court or regulator source.

AI crawlers scrape web content without permission, reducing publisher traffic and revenue

AI web crawlers from major tech companies are scraping content from billions of websites without permission or payment to train generative AI systems, reducing traffic and advertising revenues for publishers. Unlike traditional search engines, generative AI systems harvest information for chatbots without directing human visitors back to original sites, fundamentally disrupting the internet's business model. Wikipedia reported an 8 percent decline in human traffic attributable to this practice.

14 November 2025 ยท Google ยท Intellectual property and appropriation
14 November 2025

Drafted by Raidu Incident Detection, an automated system.

AII-1002-166672 sources โ†“
Quick facts
Category
Intellectual property and appropriation
Country
United States
Source
Economic Times
Date
14 November 2025
Drafted by
Raidu Incident Detection

What happened

AI crawlers from major tech companies including Google and OpenAI are scraping content from billions of websites without permission or payment to train generative AI systems.

Unlike traditional search engines which generated traffic for websites in exchange for access, generative AI crawlers harvest information for chatbots without directing human visitors to original sites, fundamentally changing the internet's business model.

Reported impact

Wikipedia reported human internet traffic fell by 8 percent between 2024 and 2025 due to a rise in AI search engine summaries.

Traditional content producers, particularly media outlets, report that AI crawlers have cut into their online operations and advertising revenues.

Index analysis

The scraping occurs without permission or payment, breaking the previous exchange model where sites allowed search engine access in return for increased visibility and traffic.

Cloudflare CEO Matthew Prince stated: 'The fundamental tension is that the new business of the internet that is AI-driven doesn't generate traffic.'

Governance controls

Cloudflare announced a measure in summer 2025 aimed at blocking AI crawlers from accessing content without payment or permission, applying to more than 10 million websites.

TollBit provides tools to online news publishers to block, monitor and monetize AI crawler traffic, with the company working with more than 5,600 sites including USA Today, Time magazine and the Associated Press.

Classification

Category
Intellectual property and appropriation
Industry
Media, social platforms and marketing
Country
United States
Harm status
Actual harm
Intent
Unknown
Timing
Unknown
Harm types
Economic or property
Harmed
Business
Failure modes
Unauthorised scraping
AI systems
AI web crawlers

Sources and evidence

This record was researched and written by the Index, based on the following original reporting.

Economic Times
As AI data scrapers sap websites' revenues, some fight back
14 November 2025
Taipei Times
Companies fighting back against AI 'crawlers' sapping Web sites' revenues - Taipei Times
14 November 2025
Oct 2026Carlsen Verlag sues OpenAI for copyright infringement of "Das NEINhorn"AllegedIntellectual property and appropriationSpiegel OnlineRP OnlineBรถrsenblatt+5Oct 2026Argentine media group Perfil sues OpenAI and Microsoft over unauthorized content useAllegedIntellectual property and appropriationPerfilClarinSep 2026Mathematician Accuses OpenAI of Using Unpublished Research in Model TrainingAllegedIntellectual property and appropriationThe VergeCybernewsQuartz+2Sep 2026OpenAI's GPT-6 Astra shows academic integrity concerns in mathematics researchAllegedIntellectual property and appropriationฤแปc bรกo tin tแปฉc, tin mแป›i Ngร y nay OnlineVietnamPlusAug 2026AI companies buy and destroy rare books for model trainingAllegedIntellectual property and appropriationThe Wall Street JournalOman Observer