AI scrapers strain Wikimedia infrastructure with 50% bandwidth surge
Automated scrapers feeding large language models have caused a 50% surge in bandwidth consumption on Wikimedia Foundation's infrastructure since January 2024. The nonprofit reports that bots now account for 65% of its most resource-intensive traffic, forcing costly infrastructure upgrades and risking service disruptions during peak human usage.
What happened
The Wikimedia Foundation reported that automated scrapers feeding large language models have exponentially increased traffic to Wikipedia and its related projects. Since January 2024, bandwidth used for downloading multimedia content from Wikimedia Commons has grown by 50%, with the bots consuming resources that the foundation's infrastructure was not designed to handle.
Bots account for approximately 65% of the most resource-intensive traffic despite representing only 35% of total page views. The scrapers are accessing both publicly available content and developer infrastructure systems such as code review platforms and bug trackers.
Reported impact
The Wikimedia Foundation states that the surge in bot traffic is straining bandwidth, forcing the organization to add infrastructure capacity and increasing its data center costs. The foundation notes that its infrastructure is built to handle sudden spikes from human users during high-interest events, but the volume from AI scrapers is 'unprecedented and presents growing risks and costs.'
The traffic surge threatens to disrupt access to Wikipedia during periods of high human traffic and interest.
Index analysis
The incidents illustrates the hidden infrastructure costs of large-scale AI training. The scraping activity, while legal under the terms of Wikipedia's Creative Commons licensing, creates an asymmetry: Wikipedia's content is free to users but its infrastructure maintenance is expensive, particularly when non-human traffic dominates bandwidth consumption.
Governance controls
The Wikimedia Foundation has imposed 'case-by-case' rate limiting and bans on offending AI crawlers. The foundation is developing a 'Responsible Use of Infrastructure' plan to address the long-term sustainability of bot traffic.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.