AI bot traffic overwhelms web defenses and harms publishers
According to DataDome's 2025 report, AI bot traffic has overwhelmed web defenses, with large language model crawlers increasing from 2.6% to over 10% of verified bot traffic between January and August. Only 2.8% of websites are fully protected despite 88.9% blocking GPTBot in robots.txt, and AI bots increasingly target forms, login pages, and checkout flows beyond scraping.
What happened
According to DataDome's 2025 Global Bot Security Report, LLM crawler traffic skyrocketed from 2.6% of verified bot traffic in January to over 10.1% by August, with DataDome detecting nearly 1.7 billion requests from OpenAI crawlers in a single month.
Unlike older automation, AI-driven traffic goes beyond scraping: in 2025, 64% of AI bot traffic reached forms, 23% reached login pages, and 5% reached checkout flows, creating vectors for fraud, account takeover, and compliance risk.
Analysis of over 16,900 websites across twenty-two industries found that only 2.8% of domains are fully protected against AI bot traffic in 2025, down from 8.4% in 2024, despite 88.9% of domains disallowing GPTBot in their robots.txt files.
Reported impact
Publishers face economic harm as AI-generated traffic overwhelms web defenses. DataDome stated that 'businesses can't tell if the AI traffic they're seeing is good or bad, which leaves them both exposed to fraud and blind to opportunity.'
Even among enterprises with more than 10,000 employees, just 2.2% had full protection and 61% were completely unprotected. Only 2% of domains with over 30 million monthly visits were fully protected.
Index analysis
AI-powered crawlers and browsers can ignore robots.txt directives, rendering static blocking strategies obsolete. Advanced bots such as anti-fingerprinting bots were blocked by just 7% of websites, while fake Chrome and curl bots were detected only 21% of the time.
Governance controls
Cloudflare updated robots.txt with new AI content controls to give creators more power in the AI era, though questions remain over whether major players like Google will respect these controls.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.