|
I tried 3 techniques
1. mod_evasive
2. mid_limitipconn
3. Iptables Limits Connections Per IP
None were effective. I never found robot.txt to be effective. I ended up signing up for google webmaster tools just to limit the crawl rate. I feel extorted. I will play around with another solution soon but thanks for all the info. I learned a lot.
|