AI crawlers disrupt open source projects via unauthorized scraping
AI crawlers from major companies are overwhelmingly accessing open source project repositories without respecting robots.txt restrictions, evading detection by spoofing user agents and using residential IP addresses. The traffic is causing service instability, outages, and significant bandwidth costs for community-maintained infrastructure.
What happened
AI crawlers from major tech companies including Amazon and Alibaba have been aggressively scraping open source project repositories without respecting robots.txt files. The crawlers evade detection by spoofing user agents, cycling through residential IP addresses as proxies, and accessing computationally expensive endpoints such as git blame, git logs, and entire commit histories.
Multiple open source projects including SourceHut, GNOME, KDE, and Fedora Pagure have experienced significant impacts. GNOME GitLab implemented a proof-of-work challenge system (Anubis) that filtered out 96.8 percent of requests as bot traffic; KDE's GitLab was temporarily knocked offline; Fedora Pagure blocked all traffic from Brazil after mitigation attempts failed; and SourceHut reported severe outages disrupting high-priority work.
Reported impact
The crawler traffic has caused substantial economic and operational damage to open source communities. Some projects report that up to 97 percent of their traffic originates from AI bots, dramatically increasing bandwidth costs and causing repeated service instability and outages.
The impacts extend beyond infrastructure to developer productivity and legitimate user access. Maintainers report delayed high-priority tasks lasting weeks or months, and users encounter significant delays or are occasionally blocked while legitimate traffic is filtered.
Index analysis
The incidents reveal a systemic failure in responsible AI data collection practices. Companies are deploying crawlers that deliberately evade standard web protocols (robots.txt) and detection mechanisms, treating community-maintained infrastructure as free resources to exploit.
Governance controls
Open source projects have implemented defensive measures including robots.txt configuration, user-agent blocking, and IP filtering, but these have proven ineffective as crawlers continuously adapt their tactics. GNOME and others have resorted to proof-of-work challenges that impose delays on legitimate users as the only effective mitigation found so far.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.