Konstantin Ryabitsev from git.kernel.org discusses the growing issue of 'background radiation' from abusive crawlers. At any given time, across 5 geo-distributed nodes, 14 CPU cores are dedicated to rendering git commits as HTML, far exceeding the resources used for legitimate access like git clones. This highlights the significant strain on resources and the need for better protection against such behavior. Ryabitsev's concerns are particularly relevant for services like Datasette, which serve a large number of crawlable web pages. Source: https://simonwillison.net/2026/Sep/7/creepy-crawlies/
Loading