|
One other tip regarding HTTP headers. If the scraper is using assorted open proxies, some of those proxies will leak the real IP of the server via headers such as X-Forwarded-For, X-Real-IP, Client-IP, etc. Since the IP of the server is unique the presence of this can be used to automatically ban proxies.
|