While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.
Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side
The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts.
Then you've got good old cloudflare which is free to use
While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.
Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
Most of the nonsense comes through botnets/residential proxies, so it's very hard to say.
There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
AI skynet is winning. It is stealing time from humans, thus forcing down their activity, as can be seen here.
A headline would be great if those AI companies would close down. I hold them all responsible for this.
going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side
The hardest-to-mitigate bot traffic tends to come from residential proxies, and most residential connections are still v4-only.
However, it is also easy to get large numbers of v6 addresses cheaply.
This is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.
The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts.
Then you've got good old cloudflare which is free to use