While AI crawlers are a problem I’m also kind of astonished why so many projects don’t use tools like ratelimiters or IP-blocklists.
These are pretty simple to setup, cause no/very little additional load and don’t cause collateral damage for legitimate users that just happend to use a different browser.
IP based blocking is complicated once you are big enough or providing service to users is critical.
For example, if you are providing some critical service such as health care, you cannot have a situation where a user cannot access health care info without hard proof that they are causing an issue and that you did your best to not block the user.
Let’s say you have a household of 5 people with 20 devices in the LAN, one can be infected and running some bot, you do not want to block 5 people and 20 devices.
Another example, double NAT, you could have literally hundreds or even thousands of people behind one IP.
While AI crawlers are a problem I’m also kind of astonished why so many projects don’t use tools like ratelimiters or IP-blocklists. These are pretty simple to setup, cause no/very little additional load and don’t cause collateral damage for legitimate users that just happend to use a different browser.
the article posted yesterday mentioned a lot of these requests are only made once per IP address, the botnet is absolutely huge.
IP based blocking is complicated once you are big enough or providing service to users is critical.
For example, if you are providing some critical service such as health care, you cannot have a situation where a user cannot access health care info without hard proof that they are causing an issue and that you did your best to not block the user.
Let’s say you have a household of 5 people with 20 devices in the LAN, one can be infected and running some bot, you do not want to block 5 people and 20 devices.
Another example, double NAT, you could have literally hundreds or even thousands of people behind one IP.
It’s literally as simple as importing an ipset into iptables and refreshing it from time to time. There is even predefined tools for that.