Quote:
Originally Posted by Emil
I fucking hate Baidu, their spiders are everywhere!
|
I hear ya. They cause real problems with sites that have a lot of slightly different outbound links, like a TGP using a script for content or trading. They don't seem to respect robots.txt so they just keep hammering away.
I ended up writing a little script that automatically firewalls their IPs.