View Single Post
Old 09-09-2018, 09:15 AM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
If you want to more aggressively block IPs, you can find a number of lists here:

FireHOL IP Lists | IP Blacklists | IP Blocklists | IP Reputation

Be aware that there's going to be some collateral damage - for example, it's going to catch legit surfers using a VPN that also funnels a lot of abuse - but if you block aggressively for a few days it may be enough to make the cloner decide to move on.

Another option is to look at a much lower level to find patterns in the request headers. Most scrapers don't bother to perfectly mimic a browser, so they stand out like a sore thumb when you view the complete HTTP request.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote