Konstantin Ryabitsev, known for his role in operating git.kernel.org, has described how abusive automated crawlers are imposing significant resource demands on the site. git.kernel.org is the official Git repository for the Linux kernel, and rendering commits and related metadata as web pages is one of its duties.
Ryabitsev summarizes the situation (TL;DR): the site spends more CPU cycles rendering commits for scrapers than it spends on all other legitimate access combined, including git clone operations. At any given time, across five geographically distributed nodes, about 14 CPU cores are devoted solely to rendering Git commits as HTML for crawlers.
This situation raises several concerns:
- Waste of resources: CPU time is consumed by requests that do not serve real human readers or development workflows.
- Service impact: the extra load can slow or interfere with legitimate users, for example developers performing git clone operations.
- Broader implications: Ryabitsev notes that services like Datasette, which serve a large number of crawlable pages, may face similar issues from this kind of "background radiation."
The topic has been discussed in the Hacker News community, tagged under crawling, git, linux, datasette, and ai-ethics. The report highlights a growing problem in the web ecosystem: automated bots and scrapers can generate costly resource usage and therefore require mitigation at operational, protocol, and community levels.
Ryabitsev's post does not provide operational countermeasures or detailed statistics beyond the cited numbers (five nodes, 14 CPU cores), so addressing the issue will require further measurement and long-term monitoring to design effective defenses.
Why this matters to other services
Ryabitsev specifically mentions Datasette as an example of a service that could be affected by similar background crawler traffic; any site that exposes many easily indexable pages can see comparable loads from scrapers. The situation underlines the need for web services to adopt strategies to filter bot traffic, protect resources, and prioritize real human users.
In short: according to the operator of git.kernel.org, automated crawler "background radiation" has grown to the point that it consumes more CPU than legitimate traffic, prompting operational and community responses to prevent further resource waste and scaling issues.
